[Feature] Implement TableQuestion type and improve table answer handling

[JonnyTran]

Description

Implement a dedicated TableQuestion type and improve table answer handling to properly manage table data in questions and responses. This implementation leverages workspace schema configuration for enhanced validation, type safety, and better user experience.

Problem

• Table answers are stored as stringified JSON in TextQuestionAnswer objects
• No dedicated question type for tables in the backend
• Limited validation and processing capabilities for table answers
• Inefficient serialization/deserialization between frontend and backend
• Complex and error-prone table handling in the frontend
• No integration with workspace schema configuration for table validation

Proposed Solution

1. Create Dedicated TableQuestion Type: Add proper table question support to backend models
2. Leverage Workspace Schema Configuration: Use schema definitions for table answer validation
3. Enhance Frontend TableQuestionAnswer: Improve table handling with schema awareness
4. Implement Schema-Aware Validation: Validate table answers against workspace schema definitions
5. Improve User Experience: Create better table editing and review interfaces

Implementation Details

Dependencies

• Workspace schema configuration infrastructure should be available for optimal validation
• TableField implementation ([Feature] Implement TableField type in backend #90) provides foundation for table type support
• SchemaService should provide table schema definitions and validation

Backend Changes

1. Add table question type to enums:

   class QuestionType(str, Enum):
       text = "text"
       multi_label = "multi_label"
       single_label = "single_label"
       rating = "rating"
       ranking = "ranking"
       span = "span"
       table = "table"  # New type

2. Enhance Question model with table support:

   @property
   def is_table(self) -> bool:
       return self.settings.get("type") == QuestionType.table
   
   def validate_table_answer(self, data: dict, workspace_id: UUID) -> List[ValidationError]:
       """Validate table answer against workspace schema configuration"""
       if not self.is_table:
           return []
       
       schema_service = SchemaService(workspace_id)
       return schema_service.validate_table_answer(self.name, data)

3. Update question settings for table-specific configuration:

   # Table question settings structure
   table_settings = {
       "type": "table",
       "schema_name": "result_extraction",  # Reference to workspace schema
       "allow_partial_answers": True,
       "validation_rules": {...},
       "display_options": {...}
   }

Schema Integration

1. Use workspace schema configuration for table answer validation:

   class SchemaService:
       def validate_table_answer(self, question_name: str, data: dict) -> List[ValidationError]:
           """Validate table answer against question schema definition"""
           question_schema = self.get_question_schema(question_name)
           return self._validate_table_data(data, question_schema)
           
       def get_table_question_schema(self, question_name: str) -> Dict:
           """Get schema definition for a table question"""
           pass
           
       def resolve_table_references(self, table_data: dict, workspace_id: UUID) -> dict:
           """Resolve references in table data using workspace schema"""
           pass

2. Enhance answer processing for table questions:

   • Validate table structure against schema
   • Resolve references to other tables
   • Provide suggestion integration for table answers

Frontend Changes

1. Enhance TableQuestionAnswer class:

   export class TableQuestionAnswer extends QuestionAnswer {
       public value: TableAnswer;
       public schemaDefinition: TableSchema;
       public workspaceId: string;
   
       constructor(
           type: QuestionType,
           schemaDefinition: TableSchema,
           workspaceId: string
       ) {
           super(type);
           this.schemaDefinition = schemaDefinition;
           this.workspaceId = workspaceId;
           this.value = new TableData([], schemaDefinition);
       }
   
       protected fill(answer: Answer) {
           this.value = new TableData(
               answer.value.data, 
               this.schemaDefinition,
               answer.value.reference
           );
       }
   
       get isValid(): boolean {
           return this.value?.data?.length > 0 && 
                  this.validateAgainstSchema();
       }
   
       private validateAgainstSchema(): boolean {
           // Validate using workspace schema configuration
           return true;
       }
   }

2. Improve table rendering and editing components:

   • Create schema-aware table editor
   • Add reference resolution UI
   • Implement validation feedback
   • Support partial answer saving

3. Integrate with workspace schema configuration:

   • Load table schemas from workspace config
   • Provide schema-based validation in real-time
   • Enable reference lookup and suggestion

API Changes

1. Update question handlers for table support:

   async def create_question(
       question_create: QuestionCreate,
       workspace_id: UUID = Path(),
       db: AsyncSession = Depends(get_async_db)
   ):
       if question_create.settings.get("type") == "table":
           schema_service = SchemaService(workspace_id, db)
           schema_service.validate_table_question_config(question_create.settings)

2. Enhance answer processing for table questions:

   • Validate table answers against schema definitions
   • Process reference relationships in table data
   • Provide enriched table data in responses

Migration Strategy

1. Create migration scripts for existing table questions:

   def upgrade():
       # Identify existing text questions with table data
       # Convert question type from 'text' to 'table'
       # Update question settings with table configuration
       # Migrate existing table answers to new format

2. Implement backward compatibility:

   • Support existing table answers during transition
   • Gradual migration of question types
   • Fallback handling for legacy table data

Related Files

• extralit/argilla-server/src/argilla_server/models/database.py - Enhanced Question model
• extralit/argilla-server/src/argilla_server/enums.py - Updated QuestionType enum
• extralit/argilla-server/src/argilla_server/services/SchemaService.py - Table validation logic
• extralit/argilla-server/src/argilla_server/api/handlers/v1/datasets/questions.py - Enhanced question handlers
• extralit/argilla-frontend/v1/domain/entities/question/QuestionAnswer.ts - Enhanced answer models
• extralit/argilla-frontend/components/features/table-question/ - New table question components

Acceptance Criteria

• Dedicated TableQuestion type exists in backend models with proper enum support
• Table questions integrate with workspace schema configuration for validation
• Table answers are properly serialized and deserialized with schema awareness
• Schema-based validation is implemented for table answers
• Frontend TableQuestionAnswer provides improved table editing experience
• Reference resolution works correctly in table answers
• API handlers correctly process table questions with workspace context
• Migration scripts successfully convert existing table questions
• Backward compatibility is maintained for existing data during transition
• Performance is optimized for table answer operations
• Integration tests verify table question functionality
• UI provides clear validation feedback and error handling

Related Issues

This is part of the strategic workspace-level schema management enhancement:

• Depends on: Workspace Schema Configuration Infrastructure
• Depends on: [Feature] Implement TableField type in backend #90 (TableField implementation)
• Part of: [Refactor] Phase 2: Introduce dedicated table types #96 (Phase 2: Table types)
• Enables: Enhanced table suggestion handling ([Feature] Enhance table suggestion handling #93)

[JonnyTran]

This issue is part of Phase 2 refactoring for Extralit Document Extraction Data Architecture. See the parent issue #96 for the overall plan and related issues.