cyberhybridhub/question_system_expantion_guide.md

# Question System Expansion Guide

This document explains the architecture, databases, and control flow of the Question System in CyberHybridHub. It serves as a design reference and checklist for developers who wish to introduce new types of questions.

---

## 1. System Overview & Architecture

The question system serves active, unanswered questions to users based on weighted random selection ratios. Currently, it supports:
- **Market History Questions**: Prospective price-movement guessing games where users swipe to guess price directions.
- **Spelling Questions**: Audio-driven quiz items where users listen to a word read aloud via Text-to-Speech (TTS) and type the correct spelling.

### System Diagram

```mermaid
graph TD
    Client[Flutter Client] <-->|SignalR / HTTP| API[Shelf API Router]
    API <--> Dispatcher[QuestionDispatcher]
    Dispatcher <--> |Weighted Mix Roll| Mix[QuestionSourceMix]
    Dispatcher <--> QService[QuestionService]
    Dispatcher <--> Pipeline[QuestionPipeline]
    
    QService <--> |Create & Fetch| DB[(PostgreSQL)]
    Pipeline <--> |Evaluate & Transition| DB
    
    subgraph Client-Side Presentation
        Router[QuestionPresentationRouter] -->|usesSpellingWordTile| SpellingTile[SpellingQuestionTile]
        Router -->|isMarket / Swipe| SwipeTile[SwipeQuestionTile]
    end
    Client --> Router
```

---

## 2. Core Database Schema

The questions and their pipeline states are stored in the following PostgreSQL tables:

### The `questions` Table
Tracks individual questions generated for, queued for, or answered by a user.
- `id` (`UUID PRIMARY KEY`): Unique question identifier.
- `assigned_user_id` (`TEXT`): References the user's Firebase UID.
- `question_text` (`TEXT`): The text prompt displayed to the user.
- `user_response` (`NUMERIC`): Stores numeric answers (e.g., slider inputs or swipe values like `10` / `-10`).
- `user_response_text` (`TEXT`): Stores string-based typed answers (e.g., spelling inputs).
- `correct_answer` (`NUMERIC`): The correct answer/benchmark to grade against.
- `created_at` / `modified_at` (`TIMESTAMPTZ`): Auditing and ordering timestamps.
- `source_tag` (`TEXT`): Category tag (e.g., `market_history:prospective`, `spelling:word_quiz`).
- `pipeline_key` (`TEXT`): The active pipeline context (e.g., `trading`, `spelling`, `weather`).
- `pipeline_step` (`TEXT`): Active step within the pipeline (e.g., `spelling_word_quiz`).
- `metadata` (`JSONB`): Key-value payload for question-specific properties (e.g., symbol tickers, audio URLs, canonical word answers).

### The `user_pipeline_state` Table
Maintains multi-step session progress for branching or sequential question chains.
- `assigned_user_id` (`TEXT PRIMARY KEY`): User's Firebase UID.
- `pipeline_key` (`TEXT`): The active branching pipeline identifier.
- `step` (`TEXT`): The current step index or name (e.g., `temperature_quiz`, `idle`).
- `context` (`JSONB`): Arbitrary session context (e.g., weather facts, selected words, target numbers).
- `updated_at` (`TIMESTAMPTZ`): Last state transition timestamp.

---

## 3. Configuration & Mix Ratios

The ratio of different questions served to a user is governed by **Question Source Preferences**.

### Definition & Mix Allocation
Source definitions are registered statically in [question_source_preferences.dart](file:///home/chh/cyberhybridhub.com/server/lib/question_source_preferences.dart):
```dart
static const List<QuestionSourcePreferenceDefinition> definitions = [
  QuestionSourcePreferenceDefinition(
    id: 'market_history',
    label: 'Market question',
    description: 'Prospective price-move guessing game',
  ),
  QuestionSourcePreferenceDefinition(
    id: 'spelling',
    label: 'Spelling',
    description: 'Spell-check quiz from curated word lists',
  ),
];
```

1. **Preference Selection**: Users can enable/disable individual question sources and configure their relative weights (ratios) from 0 to 100 via the settings panel.
2. **Effective Mix**: The server resolves configuration details and computes the normalized relative percentages using `QuestionSourcePreferences.effectiveMix()`.
3. **Dispatcher Selection**: When dispatching or queueing questions, the system calls `QuestionSourceMix.pickNextSourceId()` to perform a weighted random roll matching the configured percentages.
4. **Zero-Ratio Suppression**: If a source is enabled but configured with a ratio of `0`, it is suppressed and will not yield questions.

---

## 4. Question Lifecycle & Serving Flow

### 1. Queue Maintenance & Seeding
- The server maintains a small cache of unanswered questions for each active user (typically limited to `maxQueuedQuestions = 3`).
- The periodic cron or a login event invokes `QuestionDispatcher.topUpQueue()` in [question_dispatcher.dart](file:///home/chh/cyberhybridhub.com/server/lib/question_dispatcher.dart).
- If the current count of unanswered questions is below the threshold, the dispatcher rolls for a preferred source and attempts to produce one question using `_produceFromSource()`. If it fails or is capped, it rolls over to other enabled sources sorted by descending ratio weight.

### 2. Retrieval & Dispatch
- The client initiates a request via `GET /v1/me/questions`.
- The [questions_handler.dart](file:///home/chh/cyberhybridhub.com/server/lib/handlers/questions_handler.dart) delegates to `QuestionDispatcher.listPanelQuestions()`.
- The dispatcher retrieves unanswered questions and orders them so that the currently rolled source preference resides at the head of the list (using `QuestionSourceMix.orderUnansweredByPreferredSource()`).

### 3. Client Routing & Presentation
- On the Flutter client, the [question_presentation_router.dart](file:///home/chh/cyberhybridhub.com/lib/widgets/question_presentation_router.dart) examines incoming properties:
  - If `IncomingQuestion.usesSpellingWordTile` is true, it renders a [spelling_question_tile.dart](file:///home/chh/cyberhybridhub.com/lib/widgets/spelling_question_tile.dart).
  - Otherwise, it defaults to rendering a [swipe_question_tile.dart](file:///home/chh/cyberhybridhub.com/lib/widgets/swipe_question_tile.dart) (binary choice swipe interface).

### 4. Deferral & Skip Actions
- When a user chooses to skip or defer a question (e.g., swiping left or clicking skip):
  - A `POST /v1/me/questions/<id>/defer` request is sent.
  - The database updates the prioritization parameters (moving the question to the back of the queue by editing `view_order_at` or `created_at`).
  - **Special Spelling Behavior**: In [question_dispatcher.dart](file:///home/chh/cyberhybridhub.com/server/lib/question_dispatcher.dart#L154-L163), if spelling was the rolled preference but the user just deferred the only spelling question in their queue, the system forcefully enqueues a *new* spelling question immediately so the user is not served the exact same word again.

### 5. Deferred Preparation (Tile Load)
- For performance and synchronization, some questions defer part of their configuration until they are actively displayed.
- For example, spelling questions are created with a placeholder title and `needsPrepare = true`.
- When [spelling_question_tile.dart](file:///home/chh/cyberhybridhub.com/lib/widgets/spelling_question_tile.dart) is mounted, it sends a `POST /v1/me/questions/<id>/spelling/prepare` request.
- The pipeline selects a word, saves it in the database and pipeline context, links spelling progress trackers, and replies with the updated client payload (containing the spoken word and audio properties).

### 6. Submission, Grading, & Progression
- When a user submits an answer via `POST /v1/me/questions/<id>/answer`:
  - [questions_db.dart](file:///home/chh/cyberhybridhub.com/server/lib/questions_db.dart) logs and writes the answer to PostgreSQL.
  - The dispatcher runs grading logic:
    - **Spelling**: String comparison matching `userResponseText` against the `canonicalWord` metadata field via [spelling_answer_policy.dart](file:///home/chh/cyberhybridhub.com/server/lib/spelling/spelling_answer_policy.dart).
    - **Market History**: Numeric evaluation using `gradeProspectiveAnswer()`.
  - The pipeline runner (`QuestionPipeline.onAnswerSubmitted()`) processes the answer. Based on the matching/scoring outcome, it transitions the `user_pipeline_state` and optionally enqueues a branched follow-up question.
  - Finally, the server schedules an asynchronous `topUpQueue()` execution to replenish the unanswered queue.

---

## 5. Guide: Adding a New Question Type

To introduce a new type of question (e.g., a "Math Trivia" question), follow this systematic implementation checklist:

### Step 1: Define Database Fields & Custom Tables (Server)
Determine if your question type requires dedicated tracking tables (like spelling word catalogs or trade order details).
- Create a SQL migration file in `server/migrations/` to define any new tables or constraints.
- Ensure any additional metadata needed to present or grade the question can be stored inside the JSONB `metadata` column of the `questions` table.

### Step 2: Register the Question Source
1. Open [question_source_preferences.dart](file:///home/chh/cyberhybridhub.com/server/lib/question_source_preferences.dart).
2. Append a new `QuestionSourcePreferenceDefinition` to the static `definitions` list:
   ```dart
   QuestionSourcePreferenceDefinition(
     id: 'math_trivia',
     label: 'Math Puzzle',
     description: 'Solve arithmetic equations and logic puzzles',
     defaultRatio: 30,
   )
   ```

### Step 3: Implement Database Resolution (Server)
Modify the static resolution helpers in [questions_db.dart](file:///home/chh/cyberhybridhub.com/server/lib/questions_db.dart):
1. Locate `resolveSourceId(Map<String, dynamic> question)`. Add a case to return your new source identifier:
   ```dart
   if (question['pipelineKey'] == 'math_trivia' || (sourceTag != null && sourceTag.startsWith('math:'))) {
     return 'math_trivia';
   }
   ```
2. Locate `resolvePresentation(Map<String, dynamic> question)`. Return the correct frontend layout identifier:
   ```dart
   if (resolveSourceId(question) == 'math_trivia') {
     return 'math_quiz_keyboard'; // Target layout format
   }
   ```

### Step 4: Add Queue Maintenance Production (Server)
Integrate your question generator into the top-up flow:
1. Open [question_dispatcher.dart](file:///home/chh/cyberhybridhub.com/server/lib/question_dispatcher.dart).
2. Inside `_produceFromSource()`, add a case for your source ID that invokes your generator:
   ```dart
   case 'math_trivia':
     await _questionService.tryEnqueueMathQuestions(firebaseUid);
   ```

### Step 5: Build the Generation & Pipeline Engine (Server)
Create a service class (or extend `QuestionPipeline`) to handle generation and state progression:
1. **Generation**: Implement a method that creates a database row using `questionsDb.createQuestion()`. Specify your custom `sourceTag`, `pipelineKey`, `pipelineStep`, and grade variables. Ensure it respects batch parameters to avoid flooding the user's queue.
2. **State Progression**: In `QuestionPipeline.onAnswerSubmitted()`, add a switch case for your new `pipelineKey`. Read the answer, evaluate correctness, update any progress records, and transition the user's pipeline state as required.

### Step 6: Parse the Client Model (Frontend)
1. Open [incoming_question.dart](file:///home/chh/cyberhybridhub.com/lib/models/incoming_question.dart).
2. Add a getter to identify the new question type:
   ```dart
   bool get isMathTrivia => sourceId == 'math_trivia';
   ```
3. Update key mapping properties like `usesSpellingWordTile` or custom tile configuration parameters if you need to bypass default layouts.

### Step 7: Create the UI Component & Router (Frontend)
1. If your question type requires a custom layout (e.g. keyboard input, multi-choice grids), create a new widget in `lib/widgets/` (e.g., `math_question_tile.dart`).
2. Open [question_presentation_router.dart](file:///home/chh/cyberhybridhub.com/lib/widgets/question_presentation_router.dart).
3. Update the `build` method to intercept your presentation format and route to the new widget:
   ```dart
   if (question.presentation == 'math_quiz_keyboard') {
     return MathQuestionTile(
       key: ValueKey<String>(question.id),
       question: question,
       onSubmit: onAnswerText,
       onDefer: onDefer,
     );
   }
   ```