cyberhybridhub/question_system_expantion_guide.md

Question System Expansion Guide

This document explains the architecture, databases, and control flow of the Question System in CyberHybridHub. It serves as a design reference and checklist for developers who wish to introduce new types of questions.

---

1. System Overview & Architecture

The question system serves active, unanswered questions to users based on weighted random selection ratios. Currently, it supports:

• Market History Questions: Prospective price-movement guessing games where users swipe to guess price directions.
• Spelling Questions: Audio-driven quiz items where users listen to a word read aloud via Text-to-Speech (TTS) and type the correct spelling.

System Diagram

graph TD
    Client[Flutter Client] <-->|SignalR / HTTP| API[Shelf API Router]
    API <--> Dispatcher[QuestionDispatcher]
    Dispatcher <--> |Weighted Mix Roll| Mix[QuestionSourceMix]
    Dispatcher <--> QService[QuestionService]
    Dispatcher <--> Pipeline[QuestionPipeline]
    
    QService <--> |Create & Fetch| DB[(PostgreSQL)]
    Pipeline <--> |Evaluate & Transition| DB
    
    subgraph Client-Side Presentation
        Router[QuestionPresentationRouter] -->|usesSpellingWordTile| SpellingTile[SpellingQuestionTile]
        Router -->|isMarket / Swipe| SwipeTile[SwipeQuestionTile]
    end
    Client --> Router

---

2. Core Database Schema

The questions and their pipeline states are stored in the following PostgreSQL tables:

The questions Table

Tracks individual questions generated for, queued for, or answered by a user.

• id (UUID PRIMARY KEY): Unique question identifier.
• assigned_user_id (TEXT): References the user's Firebase UID.
• question_text (TEXT): The text prompt displayed to the user.
• user_response (NUMERIC): Stores numeric answers (e.g., slider inputs or swipe values like 10 / -10).
• user_response_text (TEXT): Stores string-based typed answers (e.g., spelling inputs).
• correct_answer (NUMERIC): The correct answer/benchmark to grade against.
• created_at / modified_at (TIMESTAMPTZ): Auditing and ordering timestamps.
• source_tag (TEXT): Category tag (e.g., market_history:prospective, spelling:word_quiz).
• pipeline_key (TEXT): The active pipeline context (e.g., trading, spelling, weather).
• pipeline_step (TEXT): Active step within the pipeline (e.g., spelling_word_quiz).
• metadata (JSONB): Key-value payload for question-specific properties (e.g., symbol tickers, audio URLs, canonical word answers).

The user_pipeline_state Table

Maintains multi-step session progress for branching or sequential question chains.

• assigned_user_id (TEXT PRIMARY KEY): User's Firebase UID.
• pipeline_key (TEXT): The active branching pipeline identifier.
• step (TEXT): The current step index or name (e.g., temperature_quiz, idle).
• context (JSONB): Arbitrary session context (e.g., weather facts, selected words, target numbers).
• updated_at (TIMESTAMPTZ): Last state transition timestamp.

---

3. Configuration & Mix Ratios

The ratio of different questions served to a user is governed by Question Source Preferences.

Definition & Mix Allocation

Source definitions are registered statically in question_source_preferences.dart:

static const List<QuestionSourcePreferenceDefinition> definitions = [
  QuestionSourcePreferenceDefinition(
    id: 'market_history',
    label: 'Market question',
    description: 'Prospective price-move guessing game',
  ),
  QuestionSourcePreferenceDefinition(
    id: 'spelling',
    label: 'Spelling',
    description: 'Spell-check quiz from curated word lists',
  ),
];

1. Preference Selection: Users can enable/disable individual question sources and configure their relative weights (ratios) from 0 to 100 via the settings panel.
2. Effective Mix: The server resolves configuration details and computes the normalized relative percentages using QuestionSourcePreferences.effectiveMix().
3. Dispatcher Selection: When dispatching or queueing questions, the system calls QuestionSourceMix.pickNextSourceId() to perform a weighted random roll matching the configured percentages.
4. Zero-Ratio Suppression: If a source is enabled but configured with a ratio of 0, it is suppressed and will not yield questions.

---

4. Question Lifecycle & Serving Flow

1. Queue Maintenance & Seeding

• The server maintains a small cache of unanswered questions for each active user (typically limited to maxQueuedQuestions = 3).
• The periodic cron or a login event invokes QuestionDispatcher.topUpQueue() in question_dispatcher.dart.
• If the current count of unanswered questions is below the threshold, the dispatcher rolls for a preferred source and attempts to produce one question using _produceFromSource(). If it fails or is capped, it rolls over to other enabled sources sorted by descending ratio weight.

2. Retrieval & Dispatch

• The client initiates a request via GET /v1/me/questions.
• The questions_handler.dart delegates to QuestionDispatcher.listPanelQuestions().
• The dispatcher retrieves unanswered questions and orders them so that the currently rolled source preference resides at the head of the list (using QuestionSourceMix.orderUnansweredByPreferredSource()).

3. Client Routing & Presentation

• On the Flutter client, the question_presentation_router.dart examines incoming properties:
  • If IncomingQuestion.usesSpellingWordTile is true, it renders a spelling_question_tile.dart.
  • Otherwise, it defaults to rendering a swipe_question_tile.dart (binary choice swipe interface).

4. Deferral & Skip Actions

• When a user chooses to skip or defer a question (e.g., swiping left or clicking skip):
  • A POST /v1/me/questions/<id>/defer request is sent.
  • The database updates the prioritization parameters (moving the question to the back of the queue by editing view_order_at or created_at).
  • Special Spelling Behavior: In question_dispatcher.dart, if spelling was the rolled preference but the user just deferred the only spelling question in their queue, the system forcefully enqueues a new spelling question immediately so the user is not served the exact same word again.

5. Deferred Preparation (Tile Load)

• For performance and synchronization, some questions defer part of their configuration until they are actively displayed.
• For example, spelling questions are created with a placeholder title and needsPrepare = true.
• When spelling_question_tile.dart is mounted, it sends a POST /v1/me/questions/<id>/spelling/prepare request.
• The pipeline selects a word, saves it in the database and pipeline context, links spelling progress trackers, and replies with the updated client payload (containing the spoken word and audio properties).

6. Submission, Grading, & Progression

• When a user submits an answer via POST /v1/me/questions/<id>/answer:
  • questions_db.dart logs and writes the answer to PostgreSQL.
  • The dispatcher runs grading logic:
    • Spelling: String comparison matching userResponseText against the canonicalWord metadata field via spelling_answer_policy.dart.
    • Market History: Numeric evaluation using gradeProspectiveAnswer().
  • The pipeline runner (QuestionPipeline.onAnswerSubmitted()) processes the answer. Based on the matching/scoring outcome, it transitions the user_pipeline_state and optionally enqueues a branched follow-up question.
  • Finally, the server schedules an asynchronous topUpQueue() execution to replenish the unanswered queue.

---

5. Guide: Adding a New Question Type

To introduce a new type of question (e.g., a "Math Trivia" question), follow this systematic implementation checklist:

Step 1: Define Database Fields & Custom Tables (Server)

Determine if your question type requires dedicated tracking tables (like spelling word catalogs or trade order details).

• Create a SQL migration file in server/migrations/ to define any new tables or constraints.
• Ensure any additional metadata needed to present or grade the question can be stored inside the JSONB metadata column of the questions table.

Step 2: Register the Question Source

1. Open question_source_preferences.dart.
2. Append a new QuestionSourcePreferenceDefinition to the static definitions list:

   QuestionSourcePreferenceDefinition(
     id: 'math_trivia',
     label: 'Math Puzzle',
     description: 'Solve arithmetic equations and logic puzzles',
     defaultRatio: 30,
   )
   

Step 3: Implement Database Resolution (Server)

Modify the static resolution helpers in questions_db.dart:

1. Locate resolveSourceId(Map<String, dynamic> question). Add a case to return your new source identifier:

   if (question['pipelineKey'] == 'math_trivia' || (sourceTag != null && sourceTag.startsWith('math:'))) {
     return 'math_trivia';
   }
   

2. Locate resolvePresentation(Map<String, dynamic> question). Return the correct frontend layout identifier:

   if (resolveSourceId(question) == 'math_trivia') {
     return 'math_quiz_keyboard'; // Target layout format
   }
   

Step 4: Add Queue Maintenance Production (Server)

Integrate your question generator into the top-up flow:

1. Open question_dispatcher.dart.
2. Inside _produceFromSource(), add a case for your source ID that invokes your generator:

   case 'math_trivia':
     await _questionService.tryEnqueueMathQuestions(firebaseUid);
   

Step 5: Build the Generation & Pipeline Engine (Server)

Create a service class (or extend QuestionPipeline) to handle generation and state progression:

1. Generation: Implement a method that creates a database row using questionsDb.createQuestion(). Specify your custom sourceTag, pipelineKey, pipelineStep, and grade variables. Ensure it respects batch parameters to avoid flooding the user's queue.
2. State Progression: In QuestionPipeline.onAnswerSubmitted(), add a switch case for your new pipelineKey. Read the answer, evaluate correctness, update any progress records, and transition the user's pipeline state as required.

Step 6: Parse the Client Model (Frontend)

1. Open incoming_question.dart.
2. Add a getter to identify the new question type:

   bool get isMathTrivia => sourceId == 'math_trivia';
   

3. Update key mapping properties like usesSpellingWordTile or custom tile configuration parameters if you need to bypass default layouts.

Step 7: Create the UI Component & Router (Frontend)

1. If your question type requires a custom layout (e.g. keyboard input, multi-choice grids), create a new widget in lib/widgets/ (e.g., math_question_tile.dart).
2. Open question_presentation_router.dart.
3. Update the build method to intercept your presentation format and route to the new widget:

   if (question.presentation == 'math_quiz_keyboard') {
     return MathQuestionTile(
       key: ValueKey<String>(question.id),
       question: question,
       onSubmit: onAnswerText,
       onDefer: onDefer,
     );
   }