ag
This commit is contained in:
parent
a7c370b743
commit
76c73a30b4
File diff suppressed because one or more lines are too long
@ -1,307 +0,0 @@
|
||||
# Flutter Admin Portal TDD Plan
|
||||
|
||||
**Scope:** Validate the functionality defined in `FLUTTER-ADMIN-PORTAL.md` using test-first development across server API + Flutter UI/repository layers.
|
||||
|
||||
**Primary objective:** Ship an admin portal that reliably shows retention/cleanup/backfill history, prioritizes unresolved failures/rate limits, and presents concise expandable details.
|
||||
|
||||
---
|
||||
|
||||
## 1. Test strategy at a glance
|
||||
|
||||
Use a layered pyramid so core behavior is proven at the cheapest layer first:
|
||||
|
||||
| Layer | Purpose | Tools |
|
||||
|---|---|---|
|
||||
| Unit (server) | pin/severity/status logic, query mapping | `dart test` |
|
||||
| Integration (server) | auth, DB query shape/order, endpoint behavior | `dart test` + Postgres test DB |
|
||||
| Unit (Flutter) | DTO mapping, repository merge, parsing, formatting | `flutter test` |
|
||||
| Widget (Flutter) | pinned section, newest-first list, expansion details, pagination interactions | `flutter test` |
|
||||
| Optional e2e/manual | confidence pass with real API + seeded rows | local runbook |
|
||||
|
||||
---
|
||||
|
||||
## 2. Coverage targets (reasonable)
|
||||
|
||||
Aim for practical coverage without over-testing visual polish:
|
||||
|
||||
- **Server admin handler package:** >= **85% line coverage**
|
||||
- **Server pin/severity helper logic:** >= **95% line coverage**
|
||||
- **Flutter admin repository/models:** >= **90% line coverage**
|
||||
- **Flutter admin widgets/screens:** >= **75% line coverage**
|
||||
- **Critical behavior cases:** **100% scenario coverage** for:
|
||||
- unresolved errors pinned
|
||||
- unpin after successful retry by same `kind`
|
||||
- newest-first ordering
|
||||
- expanded full error text display
|
||||
- no raw data fields rendered
|
||||
|
||||
---
|
||||
|
||||
## 3. Test data model fixtures
|
||||
|
||||
Create reusable fixture builders for `market_data_sync_runs` rows:
|
||||
|
||||
```text
|
||||
run(id, kind, startedAt, finishedAt, rowsWritten, rowsRemoved, error)
|
||||
```
|
||||
|
||||
Named fixture sets:
|
||||
|
||||
1. `all_success_recent_first`
|
||||
2. `rate_limit_unresolved`
|
||||
3. `failed_then_success_same_kind`
|
||||
4. `partial_backfill_error`
|
||||
5. `in_progress_stale`
|
||||
6. `mixed_kinds_mixed_outcomes`
|
||||
|
||||
Keep fixtures concise; avoid raw bar payloads by design.
|
||||
|
||||
---
|
||||
|
||||
## 4. Server TDD plan
|
||||
|
||||
Target files (planned):
|
||||
|
||||
- `server/lib/handlers/market_history_admin_handler.dart`
|
||||
- `server/lib/trading/market_history_admin_query.dart` (or equivalent helper)
|
||||
- `server/test/integration/market_history_admin_handler_test.dart`
|
||||
- `server/test/trading/market_history_admin_logic_test.dart`
|
||||
|
||||
### 4.1 RED: pin/severity/status logic unit tests
|
||||
|
||||
Write tests before implementation:
|
||||
|
||||
1. **Pinned unresolved failure**
|
||||
- failed `backfill` with no later success => pinned
|
||||
2. **Unpin after retry success**
|
||||
- failed `backfill` then later successful `backfill` => old failure not pinned
|
||||
3. **Pinned is per-kind**
|
||||
- `cleanup` success does not clear `backfill` failure pin
|
||||
4. **Rate limit classification**
|
||||
- `error` containing `429` or `rate limited` => `severity=rate_limit`
|
||||
5. **Partial success classification**
|
||||
- `rows_written>0 && error!=null` => `status=partial`
|
||||
6. **In-progress classification**
|
||||
- `finished_at==null` recent => `status=in_progress`
|
||||
- stale threshold exceeded => `severity=warning/error` (per chosen policy)
|
||||
7. **Newest-first ordering**
|
||||
- runs sorted by `started_at DESC`
|
||||
|
||||
### 4.2 GREEN: minimal implementation
|
||||
|
||||
Implement pure helper functions first:
|
||||
|
||||
- `deriveSeverity(error, finishedAt, startedAt, now)`
|
||||
- `deriveStatus(error, finishedAt, rowsWritten, rowsRemoved)`
|
||||
- `computePinned(runs)`
|
||||
- `toSummary(run)`
|
||||
|
||||
Keep these deterministic and injectable with `clock` for testability.
|
||||
|
||||
### 4.3 RED: handler integration tests
|
||||
|
||||
`market_history_admin_handler_test.dart`:
|
||||
|
||||
1. **403 for non-admin token**
|
||||
2. **200 + shape for admin**
|
||||
3. **`runs` newest-first**
|
||||
4. **`pinned` contains unresolved rate-limit row**
|
||||
5. **`pinned` clears when later success seeded**
|
||||
6. **`limit` respected**
|
||||
7. **pagination via `before`**
|
||||
8. **`kind` filter**
|
||||
9. **response excludes raw fields** (no `raw`, no snapshot payload)
|
||||
|
||||
### 4.4 GREEN: handler + routing
|
||||
|
||||
Implement route:
|
||||
|
||||
- `GET /v1/admin/market-history/sync-runs`
|
||||
|
||||
Wire auth middleware with admin allowlist/claim gate.
|
||||
|
||||
### 4.5 REFACTOR
|
||||
|
||||
- Move query + transform logic out of handler into service/query class.
|
||||
- Ensure all SQL is parameterized.
|
||||
- Add response DTO class to reduce map-shape drift.
|
||||
|
||||
---
|
||||
|
||||
## 5. Flutter TDD plan
|
||||
|
||||
Planned files:
|
||||
|
||||
- `lib/admin/models/sync_run_event.dart`
|
||||
- `lib/admin/services/market_history_admin_api.dart`
|
||||
- `lib/admin/repositories/sync_run_log_repository.dart`
|
||||
- `lib/admin/screens/market_history_log_screen.dart`
|
||||
- `lib/admin/widgets/sync_run_expansion_tile.dart`
|
||||
|
||||
Tests:
|
||||
|
||||
- `test/admin/models/sync_run_event_test.dart`
|
||||
- `test/admin/repositories/sync_run_log_repository_test.dart`
|
||||
- `test/admin/services/market_history_admin_api_test.dart`
|
||||
- `test/admin/widgets/sync_run_expansion_tile_test.dart`
|
||||
- `test/admin/screens/market_history_log_screen_test.dart`
|
||||
|
||||
### 5.1 RED: model + repository unit tests
|
||||
|
||||
1. Parse API response into domain model
|
||||
2. Severity/status mapping from API and fallback parsing
|
||||
3. Merge pinned + history sections
|
||||
4. Keep pinned visible above chronological list
|
||||
5. Append paged results without reordering newer rows
|
||||
6. Deduplicate by `id` during refresh
|
||||
7. Error state handling (`401/403/500/network`)
|
||||
|
||||
### 5.2 GREEN: implement model/repository
|
||||
|
||||
- Implement immutable model objects
|
||||
- Implement repository APIs:
|
||||
- `loadInitial()`
|
||||
- `refresh()`
|
||||
- `loadMore()`
|
||||
- Include simple in-memory cache for current session
|
||||
|
||||
### 5.3 RED: service HTTP tests
|
||||
|
||||
Using mocked HTTP client:
|
||||
|
||||
1. sends bearer token header
|
||||
2. sends `limit`, `before`, `kind` query params
|
||||
3. handles non-200 responses
|
||||
4. parse malformed payload guardrails
|
||||
|
||||
### 5.4 GREEN: service implementation
|
||||
|
||||
Mirror existing app API style (`apiBaseUrl`, auth token retrieval, robust parse).
|
||||
|
||||
### 5.5 RED: widget/screen tests
|
||||
|
||||
1. **Pinned section shown** when pinned rows exist
|
||||
2. **Pinned section hidden** when none exist
|
||||
3. **Newest row at top** in history list
|
||||
4. **Expansion tile details**
|
||||
- shows full message text
|
||||
- does not show raw data fields
|
||||
5. **Severity styling** (icon/chip text present)
|
||||
6. **Pull-to-refresh triggers repository refresh**
|
||||
7. **Scroll-to-end triggers `loadMore()`**
|
||||
8. **Loading, empty, and error states render correctly**
|
||||
|
||||
### 5.6 GREEN: implement UI
|
||||
|
||||
- Build screen with two sections:
|
||||
- `Needs attention` (pinned)
|
||||
- `History` (paged, newest-first)
|
||||
- Use `ExpansionTile` rows with concise collapsed summary.
|
||||
|
||||
### 5.7 REFACTOR
|
||||
|
||||
- Extract common chips/icons/date formatters
|
||||
- Keep widget tree shallow for testability
|
||||
- Avoid business logic in widget build methods
|
||||
|
||||
---
|
||||
|
||||
## 6. API-trigger actions (optional phase C) TDD
|
||||
|
||||
If adding on-demand actions:
|
||||
|
||||
Server tests:
|
||||
|
||||
- `POST /v1/admin/market-data/resync` returns `202` and creates run rows
|
||||
- `POST /v1/admin/market-data/cleanup` with `archive` flag routes correctly
|
||||
- auth + invalid input tests
|
||||
|
||||
Flutter tests:
|
||||
|
||||
- retry button visible for pinned failures
|
||||
- confirm dialog for cleanup
|
||||
- action triggers refresh and shows new in-progress/success rows
|
||||
|
||||
This phase can be deferred without affecting MVP log validation.
|
||||
|
||||
---
|
||||
|
||||
## 7. Quality gates and CI commands
|
||||
|
||||
### Server
|
||||
|
||||
```bash
|
||||
cd server && dart test test/trading/market_history_admin_logic_test.dart
|
||||
cd server && dart test test/integration/market_history_admin_handler_test.dart
|
||||
```
|
||||
|
||||
### Flutter
|
||||
|
||||
```bash
|
||||
flutter test test/admin/models/
|
||||
flutter test test/admin/repositories/
|
||||
flutter test test/admin/services/
|
||||
flutter test test/admin/widgets/
|
||||
flutter test test/admin/screens/
|
||||
```
|
||||
|
||||
### Coverage checks (recommended)
|
||||
|
||||
Server:
|
||||
|
||||
```bash
|
||||
cd server && dart test --coverage=coverage
|
||||
```
|
||||
|
||||
Flutter:
|
||||
|
||||
```bash
|
||||
flutter test --coverage
|
||||
```
|
||||
|
||||
Add CI thresholds for the targets in Section 2.
|
||||
|
||||
---
|
||||
|
||||
## 8. Risk-driven test checklist
|
||||
|
||||
High-priority cases to lock down early:
|
||||
|
||||
- [x] Pin logic regression when multiple kinds interleave
|
||||
- [x] Rate-limit text variants (`429`, `rate limit`, capitalization differences)
|
||||
- [x] Very long error message expansion rendering
|
||||
- [x] Pagination duplicates/holes on refresh + load-more race
|
||||
- [x] Timezone display mismatch (`started_at` UTC vs local display)
|
||||
- [x] Empty payload safety (no crashes)
|
||||
|
||||
---
|
||||
|
||||
## 9. Definition of done
|
||||
|
||||
The admin portal feature is considered validated when:
|
||||
|
||||
1. All MVP tests listed above pass.
|
||||
2. Coverage targets in Section 2 are met.
|
||||
3. Automated acceptance tests (`test/admin/acceptance/`) cover:
|
||||
- newest-first order,
|
||||
- pinned unresolved issues,
|
||||
- expandable full error detail,
|
||||
- no raw payload details in UI,
|
||||
- cleanup archive toggle when enabled.
|
||||
4. CI includes these test suites and fails on regression.
|
||||
|
||||
Manual sanity pass (live server + seeded rows) remains recommended before production rollout.
|
||||
|
||||
---
|
||||
|
||||
## 10. Suggested implementation order
|
||||
|
||||
1. Server pure logic tests + implementation ✅
|
||||
2. Server handler integration tests + endpoint ✅
|
||||
3. Flutter models/repository/service tests + implementation ✅
|
||||
4. Flutter widget/screen tests + UI implementation ✅
|
||||
5. Optional trigger actions phase ✅
|
||||
6. Risk checklist (§8) + acceptance tests (§9) ✅
|
||||
|
||||
This order keeps risky logic proven first and gives UI teams a stable API contract early.
|
||||
|
||||
121
README.md
121
README.md
@ -1,17 +1,116 @@
|
||||
# cyberhybridhub
|
||||
# Cyber Hybrid Hub
|
||||
|
||||
A new Flutter project.
|
||||
A hybrid, interactive guessing-game and automated trading platform. Cyber Hybrid Hub integrates live Alpaca market data and paper trading with a Flutter frontend and a Postgres-backed Dart backend API.
|
||||
|
||||
## Getting Started
|
||||
---
|
||||
|
||||
This project is a starting point for a Flutter application.
|
||||
## Project Structure
|
||||
|
||||
A few resources to get you started if this is your first Flutter project:
|
||||
The repository consists of two main components:
|
||||
* **Flutter Client (Root)**: The mobile and web application interface where users answer questions, track their scores, and where administrators can audit background synchronization tasks.
|
||||
* **Dart API Server ([server/](file:///home/nathan/cyberhybridhub.com/server/))**: A high-performance Dart API server that manages authentication, stores market data snapshots, serves questions, handles scoring, and executes trades via the Alpaca API.
|
||||
|
||||
- [Learn Flutter](https://docs.flutter.dev/get-started/learn-flutter)
|
||||
- [Write your first Flutter app](https://docs.flutter.dev/get-started/codelab)
|
||||
- [Flutter learning resources](https://docs.flutter.dev/reference/learning-resources)
|
||||
---
|
||||
|
||||
For help getting started with Flutter development, view the
|
||||
[online documentation](https://docs.flutter.dev/), which offers tutorials,
|
||||
samples, guidance on mobile development, and a full API reference.
|
||||
## Key Features
|
||||
|
||||
1. **Market Guessing Game**: Generates obfuscated price-movement questions for active US equities. Users swipe to guess direction (+10 for up, -10 for down) and earn scores based on closeness.
|
||||
2. **Spelling Challenge**: An interactive, catalog-driven spelling game with integrated progress scoring.
|
||||
3. **Alpaca Ingestion & Trading Pipeline**:
|
||||
* Ingests live `1Min` bar data from Alpaca Market Data and aggregates them into two regular-session half-days (Morning: 9:30 AM–12:45 PM ET, Afternoon: 12:45 PM–4:00 PM ET) per trading day.
|
||||
* Executes paper trading orders automatically based on rule-evaluation matches and user confirmation.
|
||||
* Protects against rate-limiting (429) and enforces daily/hourly notional guardrails.
|
||||
4. **Admin Portal**: A dedicated interface for operators to monitor database sync runs (universe sync, backfills, cleanup), pin unresolved failures (rate limits, partial runs), and trigger on-demand sync or retention cleanup tasks.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph flutter [Flutter App]
|
||||
UI[HomeScreen / Swipe UI]
|
||||
AdminUI[Admin Portal Screen]
|
||||
Hub[QuestionsHubService SignalR]
|
||||
end
|
||||
|
||||
subgraph server [Dart API Server]
|
||||
API[Shelf Router Handlers]
|
||||
Worker[QuestionBackgroundWorker]
|
||||
Orchestrator[TradingOrchestrator]
|
||||
QP[QuestionPipeline]
|
||||
AlpacaMD[Alpaca Market Data Client]
|
||||
AlpacaTR[Alpaca Trading Client]
|
||||
end
|
||||
|
||||
subgraph pg [Postgres Database]
|
||||
MD[market_data_snapshots]
|
||||
Runs[market_data_sync_runs]
|
||||
Q[questions]
|
||||
Config[user_trading_config]
|
||||
Orders[trade_orders]
|
||||
end
|
||||
|
||||
AlpacaMD -->|REST Poll / Ingest| MD
|
||||
Worker --> Orchestrator
|
||||
Orchestrator --> MD
|
||||
Orchestrator --> Runs
|
||||
QP --> Q
|
||||
QP --> Hub
|
||||
Hub --> UI
|
||||
API --> UI
|
||||
AdminUI -->|Admin Actions| API
|
||||
API --> AlpacaTR
|
||||
AlpacaTR -->|POST Orders| Orders
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Setup & Running
|
||||
|
||||
### 1. Prerequisites
|
||||
* [Flutter SDK](https://docs.flutter.dev/get-started/install) (latest stable version)
|
||||
* [Dart SDK](https://dart.dev/get-dart)
|
||||
* [PostgreSQL](https://www.postgresql.org/) (running locally)
|
||||
|
||||
### 2. Database Setup
|
||||
Create the development and test databases:
|
||||
```bash
|
||||
createdb cyberhybridhub
|
||||
createdb cyberhybridhub_test
|
||||
```
|
||||
|
||||
### 3. Server Setup
|
||||
Navigate to the `server/` directory and configure environment variables:
|
||||
```bash
|
||||
cd server
|
||||
cp .env.example .env
|
||||
```
|
||||
Update `.env` with your Postgres connection details and Alpaca API credentials.
|
||||
|
||||
Install dependencies and run the server:
|
||||
```bash
|
||||
dart pub get
|
||||
dart run bin/server.dart
|
||||
```
|
||||
|
||||
### 4. Running the Flutter App
|
||||
From the root directory, run the Flutter application pointing to your local server:
|
||||
```bash
|
||||
flutter run --dart-define=API_BASE_URL=http://localhost:3000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Root scripts
|
||||
To run the full validation suites, use the helper scripts located in the `scripts/` directory:
|
||||
|
||||
```bash
|
||||
./scripts/test-server.sh # Runs server unit and integration tests (no live Alpaca)
|
||||
./scripts/test-admin-portal.sh # Runs Flutter + server admin portal test suites
|
||||
./scripts/test-server-alpaca.sh # Runs live Alpaca integration tests (requires keys in server/.env)
|
||||
```
|
||||
|
||||
For more server-specific implementation details, check out the [server README](file:///home/nathan/cyberhybridhub.com/server/README.md).
|
||||
|
||||
@ -1,252 +0,0 @@
|
||||
# TODO — RTH session half bars (morning / afternoon aggregates)
|
||||
|
||||
Companion to [`server/README.md`](./server/README.md) (Market history) and
|
||||
[`FLUTTER-ADMIN-PORTAL.md`](./FLUTTER-ADMIN-PORTAL.md).
|
||||
|
||||
**Goal:** Replace six UTC **4-hour** Alpaca bars per day with **two regular-session
|
||||
aggregates per US trading day**, each built from up to **195 one-minute bars**:
|
||||
|
||||
| Slot | US Eastern (NYSE regular) | Duration |
|
||||
|------|---------------------------|----------|
|
||||
| Morning | 9:30 AM – 12:45 PM | 3h 15m (195 min) |
|
||||
| Afternoon | 12:45 PM – 4:00 PM | 3h 15m (195 min) |
|
||||
|
||||
Persist **one OHLCV row per symbol per slot** (not 195 rows). Use for
|
||||
`guess_weekly_move`, admin week coverage, and question audit.
|
||||
|
||||
**TDD rhythm:** Red → Green → Refactor → Confirm (same as
|
||||
[`TODO.md`](./TODO.md) §0).
|
||||
|
||||
---
|
||||
|
||||
## 0. Design decisions (lock before coding)
|
||||
|
||||
- [ ] **Timezone:** `America/New_York` for slot boundaries (handles DST); store
|
||||
canonical `as_of` / `raw.slot_start` as UTC instants of slot open.
|
||||
- [ ] **Stored `timeframe`:** new value, e.g. `sessionHalf` (do not overload
|
||||
`4Hour`).
|
||||
- [ ] **Alpaca fetch:** `GET /v2/stocks/bars` with `timeframe=1Min` per slot
|
||||
`[start, end]`; aggregate in server (`o`/`h`/`l`/`c`/`v` from minutes).
|
||||
- [ ] **Existing data:** delete or archive all `timeframe = '4Hour'` history rows
|
||||
after migration; full backfill required.
|
||||
- [ ] **`MIN_BARS_FOR_GUESS`:** revisit default (`5` bars ≈ 2.5 trading days at
|
||||
2 slots/day vs ~20h span with 4h bars).
|
||||
- [ ] **Week coverage UI:** 2 dots per **trading day** (not 6 UTC dots).
|
||||
- [ ] **Question audit API (optional):** return `assetCount` only, drop `assets[]`
|
||||
payload to save bandwidth (Flutter already shows count-only).
|
||||
|
||||
---
|
||||
|
||||
## 1. Slot model (replace `MarketHistoryFourHourSlot`)
|
||||
|
||||
**File:** replace or supersede `server/lib/trading/market_history_four_hour_slot.dart`
|
||||
→ e.g. `market_history_session_slot.dart`.
|
||||
|
||||
- [ ] **Red** — `server/test/trading/market_history_session_slot_test.dart`:
|
||||
- [ ] `slotStartContaining` maps instants to morning (9:30 ET) or afternoon
|
||||
(12:45 ET) slot start (UTC).
|
||||
- [ ] `endExclusive` / `endInclusive` for 195-minute windows.
|
||||
- [ ] `hasEnded` / `lastCompletedSlotStart` never returns in-progress slot.
|
||||
- [ ] `completedSlotStartsInWindow` yields 2 × trading days in rolling window;
|
||||
skips weekends + NYSE holidays (`MarketHistoryTradingCalendar`).
|
||||
- [ ] DST: assert 13:30 vs 14:30 UTC morning start across EDT/EST fixtures.
|
||||
- [ ] `wireUtc` / `slotStartWire` include minutes (`…T13:30:00Z`).
|
||||
- [ ] **Green** — implement slot module; `slotsPerDay = 2`,
|
||||
`slotDuration = Duration(hours: 3, minutes: 15)`.
|
||||
- [ ] **Refactor** — update imports project-wide; delete old four-hour module when
|
||||
unused.
|
||||
|
||||
**Confirm:** `cd server && dart test test/trading/market_history_session_slot_test.dart`
|
||||
|
||||
---
|
||||
|
||||
## 2. Config & env
|
||||
|
||||
**Files:** `market_history_config.dart`, `market_history_env.dart`, `env.dart`,
|
||||
`server/README.md`.
|
||||
|
||||
- [ ] `barTimeframe` → `sessionHalf` (or chosen name).
|
||||
- [ ] Remove `slotHours = 4`; document `slotsPerDay = 2`.
|
||||
- [ ] Add `alpacaFetchTimeframe = '1Min'` (fetch only, not stored).
|
||||
- [ ] Document env vars; defaults for `MIN_BARS_FOR_GUESS` if changed.
|
||||
|
||||
---
|
||||
|
||||
## 3. Database migration `010_session_half_bars.sql`
|
||||
|
||||
- [ ] **Red** — extend `market_history_schema_test.dart`:
|
||||
- [ ] `timeframe` CHECK allows `sessionHalf`.
|
||||
- [ ] Partial index on `(symbol, as_of DESC) WHERE metric='bar' AND timeframe='sessionHalf'`.
|
||||
- [ ] **Green** — migration:
|
||||
- [ ] `DELETE` (or archive) `metric='bar' AND timeframe='4Hour'`.
|
||||
- [ ] Update `market_data_snapshots_timeframe_check`.
|
||||
- [ ] `CREATE INDEX market_data_snapshots_bar_session_half_idx …`.
|
||||
- [ ] Apply in integration test harness (`001`–`010`).
|
||||
|
||||
**Confirm:** `cd server && dart test test/integration/market_history_schema_test.dart`
|
||||
|
||||
---
|
||||
|
||||
## 4. Minute fetch + aggregation (backfill)
|
||||
|
||||
**Files:** `market_data_history.dart`, `alpaca_market_data_client.dart` (unchanged
|
||||
API surface; caller passes `1Min`).
|
||||
|
||||
- [ ] **Red** — `market_data_history_sync_test.dart`:
|
||||
- [ ] Mock `1Min` bars spanning 9:30–12:45 ET → one persisted `sessionHalf` row.
|
||||
- [ ] OHLCV aggregation rules: `o`=first, `h`=max, `l`=min, `c`=last, `v`=sum.
|
||||
- [ ] Pagination: merge pages via existing `getBarsRange` + `next_page_token`.
|
||||
- [ ] Wrong-window minutes rejected; empty minutes → placeholder or error per
|
||||
calendar rules.
|
||||
- [ ] Rate-limit / partial run behavior unchanged.
|
||||
- [ ] **Green** — `_fetchBarsWithRateLimitRetry` uses `1Min`; `_persistBars`
|
||||
aggregates then upserts one row per symbol; `raw.slot_start` + optional
|
||||
`raw.minute_bars_count`.
|
||||
- [ ] **Refactor** — extract `aggregateMinuteBars(List<AlpacaBar>)` helper.
|
||||
|
||||
**Confirm:** `cd server && dart test test/integration/market_data_history_sync_test.dart`
|
||||
|
||||
---
|
||||
|
||||
## 5. DB slot matching
|
||||
|
||||
**File:** `market_data_db.dart`
|
||||
|
||||
- [ ] Replace `_slotStartBucketSql` (4-hour UTC `div(hour,4)`) with session-slot
|
||||
equality on `raw.slot_start` wire or shared Dart/SQL slot function.
|
||||
- [ ] **Red** — `market_data_db_test.dart` for `symbolsWithBarForSlot` at 9:30 / 12:45
|
||||
ET boundaries.
|
||||
|
||||
---
|
||||
|
||||
## 6. Read paths
|
||||
|
||||
| File | Work |
|
||||
|------|------|
|
||||
| `market_history_query.dart` | Filter `timeframe = sessionHalf`; update comments. |
|
||||
| `market_history_question_audit.dart` | Step by **one session slot** (not ±4h); slot pair query. |
|
||||
| `market_history_week_coverage.dart` | 2 slots per trading day; `slotsPerDay: 2`. |
|
||||
| `market_history_trading_calendar.dart` | Trading-day helpers keyed on ET date of slot. |
|
||||
| `market_history_admin_logic.dart` | Error strings / slot labels. |
|
||||
| `backfill_sync_item.dart` | Wire format with minute-precision `slotStart`. |
|
||||
|
||||
- [ ] **Red** — unit + integration tests for each area (see existing `*_test.dart`
|
||||
files under `server/test/`).
|
||||
- [ ] **Green** — implement.
|
||||
|
||||
**Confirm:** `./scripts/test-server.sh` (no live Alpaca).
|
||||
|
||||
---
|
||||
|
||||
## 7. Flutter admin
|
||||
|
||||
| File | Work |
|
||||
|------|------|
|
||||
| `lib/admin/utils/sync_run_formatters.dart` | `formatMarketHistorySlotWire` — no `hour ~/ 4`. |
|
||||
| `lib/admin/models/market_history_week_coverage.dart` | Default `slotsPerDay: 2`. |
|
||||
| `lib/admin/widgets/market_history_week_coverage_sheet.dart` | Copy: 2 slots/day. |
|
||||
| `lib/admin/widgets/market_history_question_audit_sheet.dart` | ET slot labels; count-only UI (done). |
|
||||
| `lib/admin/widgets/sync_run_expansion_tile.dart` | Backfill row: count only (done). |
|
||||
|
||||
- [ ] **Red** — widget tests under `test/admin/`.
|
||||
- [ ] **Green** — implement remaining slot-label / coverage changes after server
|
||||
ships new slot times.
|
||||
|
||||
**Confirm:** `./scripts/test-admin-portal.sh`
|
||||
|
||||
---
|
||||
|
||||
## 8. Fixtures & live tests
|
||||
|
||||
- [ ] Add `server/test/fixtures/alpaca_bars_1min_session.json` (195 minutes × 1 symbol).
|
||||
- [ ] Update `alpaca_bars_4h_window.json` usages or remove.
|
||||
- [ ] `@Tags(['alpaca'])` live test: `1Min` range for one slot, aggregate locally.
|
||||
|
||||
---
|
||||
|
||||
## 9. Documentation
|
||||
|
||||
- [ ] `server/README.md` — Market history section (2 session slots, `1Min` fetch).
|
||||
- [ ] `FLUTTER-ADMIN-PORTAL.md` — week coverage + question audit behavior.
|
||||
- [ ] Link from [`TODO.md`](./TODO.md) (legacy 4h work is complete; this doc
|
||||
supersedes granularity for Phase 3).
|
||||
|
||||
---
|
||||
|
||||
## 10. Deploy / ops
|
||||
|
||||
- [ ] Run migration `010` on prod/staging (deletes legacy `4Hour` bar rows automatically).
|
||||
- [ ] **Or** manually clear history before backfill (if migration already applied without `010`):
|
||||
|
||||
```sql
|
||||
-- Required: remove old 4-hour bar rows (wrong shape for session-half logic)
|
||||
DELETE FROM market_data_snapshots
|
||||
WHERE metric = 'bar' AND timeframe = '4Hour';
|
||||
|
||||
-- Optional: archived 4Hour copies (if any)
|
||||
DELETE FROM market_data_archive
|
||||
WHERE metric = 'bar' AND timeframe = '4Hour';
|
||||
|
||||
-- Optional: force a clean sync audit trail (not required for backfill to run)
|
||||
TRUNCATE market_data_sync_runs;
|
||||
```
|
||||
|
||||
**Do not truncate** `tradable_assets` — universe sync is independent.
|
||||
|
||||
After clearing, run admin **Resync** or wait for the worker; `hasPendingSlots` will
|
||||
enqueue backfill for every missing `sessionHalf` slot in the rolling window.
|
||||
|
||||
- [ ] Verify week-coverage calendar green for 2 slots × trading days.
|
||||
- [ ] Verify `guess_weekly_move` eligibility with new `MIN_BARS` threshold.
|
||||
|
||||
---
|
||||
|
||||
## 11. Progress log
|
||||
|
||||
| Date | Step | Notes |
|
||||
|------|------|-------|
|
||||
| | | |
|
||||
|
||||
---
|
||||
|
||||
## Appendix — affected files (checklist)
|
||||
|
||||
**Server (implement):**
|
||||
|
||||
- `server/lib/trading/market_history_four_hour_slot.dart` → session slot module
|
||||
- `server/lib/trading/market_history_config.dart`
|
||||
- `server/lib/trading/market_data_history.dart`
|
||||
- `server/lib/trading/market_data_db.dart`
|
||||
- `server/lib/trading/market_history_query.dart`
|
||||
- `server/lib/trading/market_history_question_audit.dart`
|
||||
- `server/lib/trading/market_history_week_coverage.dart`
|
||||
- `server/lib/trading/market_history_trading_calendar.dart`
|
||||
- `server/lib/trading/backfill_sync_item.dart`
|
||||
- `server/lib/alpaca/alpaca_market_data_client.dart` (wire helper import only)
|
||||
- `server/migrations/010_session_half_bars.sql` (new)
|
||||
|
||||
**Server tests:**
|
||||
|
||||
- `server/test/trading/market_history_four_hour_slot_test.dart` → session tests
|
||||
- `server/test/trading/market_history_week_coverage_test.dart`
|
||||
- `server/test/trading/market_history_question_audit_test.dart`
|
||||
- `server/test/integration/market_data_history_sync_test.dart`
|
||||
- `server/test/integration/market_data_db_test.dart`
|
||||
- `server/test/integration/market_history_admin_handler_test.dart`
|
||||
- `server/test/integration/market_history_week_coverage_test.dart`
|
||||
- (+ admin logic, schema, scheduler tests as needed)
|
||||
|
||||
**Flutter:**
|
||||
|
||||
- `lib/admin/widgets/market_history_question_audit_sheet.dart`
|
||||
- `lib/admin/widgets/sync_run_expansion_tile.dart`
|
||||
- `lib/admin/widgets/market_history_week_coverage_sheet.dart`
|
||||
- `lib/admin/utils/sync_run_formatters.dart`
|
||||
- `lib/admin/models/market_history_week_coverage.dart`
|
||||
- `test/admin/widgets/market_history_question_audit_sheet_test.dart`
|
||||
- `test/admin/widgets/sync_run_expansion_tile_test.dart`
|
||||
|
||||
**Unrelated (do not change for slot work):**
|
||||
|
||||
- `server/lib/trading/guardrails.dart` (4h **notional** window for orders)
|
||||
- `server/lib/trading/market_data_ingest.dart` (daily bars for live ingest)
|
||||
748
TODO.md
748
TODO.md
@ -1,748 +0,0 @@
|
||||
# TODO — Rolling 7-Day Market Data Window + Cleanup
|
||||
|
||||
> **Next milestone:** ET morning/afternoon session half bars (195-minute aggregates)
|
||||
> — see [`TODO-SESSION-HALF-BARS.md`](./TODO-SESSION-HALF-BARS.md).
|
||||
|
||||
Companion to [`TRADING_DEVELOPMENT_PLAN.md`](./TRADING_DEVELOPMENT_PLAN.md) and
|
||||
[`TRADING_TDD_PLAN.md`](./TRADING_TDD_PLAN.md).
|
||||
|
||||
**Goal:** maintain a rolling **7-day history** of market data for **all active
|
||||
tradable assets** so the question pipeline can generate obfuscated
|
||||
*guessing-game* questions about market movement, while pruning (or archiving)
|
||||
anything older than the window.
|
||||
|
||||
**TDD rhythm (mandatory for every step):**
|
||||
|
||||
1. **Red** — write the failing test(s) first; commit if you like.
|
||||
2. **Green** — minimum implementation that turns every test in this step green.
|
||||
3. **Refactor** — tidy without changing behavior; rerun tests.
|
||||
4. **Confirm** — run the full step-level confirm command listed in the step.
|
||||
5. **Log** — check the box, add a row to [§12 Progress log](#12-progress-log).
|
||||
|
||||
> Do not skip the Red phase. Do not start the next step while any test in the
|
||||
> current step is failing or pending. No live Alpaca calls in default
|
||||
> `dart test` jobs — guard with `@Tags(['alpaca'])`.
|
||||
|
||||
---
|
||||
|
||||
## 0. Scope & design constraints
|
||||
|
||||
- **Window:** rolling 7 calendar days, UTC. Configurable via
|
||||
`MARKET_HISTORY_WINDOW_DAYS` (default `7`).
|
||||
- **Granularity (Phase 1):** `1Day` bars for every active tradable, plus the
|
||||
existing `last_trade` / `prev_close` snapshots for watchlist symbols.
|
||||
- **Granularity (Phase 2):** `1Hour` bars for the union of all enabled users'
|
||||
watchlist symbols (≤30 on Alpaca Basic).
|
||||
- **Universe source of truth:** Alpaca `/v2/assets?status=active&tradable=true`,
|
||||
refreshed daily, cached in Postgres (`tradable_assets`).
|
||||
- **Idempotency:** repeated backfill of the same
|
||||
`(symbol, metric, timeframe, as_of)` MUST NOT create duplicate rows.
|
||||
- **Cleanup vs. archive:** rows with `as_of < now() - window` are either
|
||||
hard-deleted (Phase 1) or moved to `market_data_archive` (Phase 2).
|
||||
- **Worker isolation:** historical sync + cleanup run on their own cadence
|
||||
(default once per day), not on every 60s per-user tick.
|
||||
- **Rate-limit safety:** batch symbols (Alpaca `bars` accepts multi-symbol);
|
||||
cap concurrent symbols; never call Alpaca in tests
|
||||
(`QUESTION_PIPELINE_TEST_MODE=true`).
|
||||
- **No Flutter changes** required for this milestone.
|
||||
|
||||
---
|
||||
|
||||
## 1. Schema additions (migration `005_market_history.sql`)
|
||||
|
||||
### 1.1 Red — failing tests first
|
||||
|
||||
- [x] Create `server/test/integration/market_history_schema_test.dart`:
|
||||
- [x] Test: `INSERT` two snapshots with the same
|
||||
`(symbol, metric, timeframe, as_of)` → second one **upserts**,
|
||||
does not duplicate (current schema lacks the unique constraint, so
|
||||
this MUST fail Red).
|
||||
- [x] Test: `timeframe` defaults to `'tick'` for existing rows; new rows
|
||||
accept `'1Min' | '1Hour' | '1Day'`.
|
||||
- [x] Test: `tradable_assets` PK rejects duplicate symbol; query by
|
||||
`(status='active', tradable=true)` uses the new index (verify via
|
||||
`EXPLAIN` returning `Index Scan`).
|
||||
- [x] Test: `market_data_sync_runs` records `kind`, `started_at`,
|
||||
`finished_at`, `rows_written`, `rows_removed`, `error` shape.
|
||||
|
||||
### 1.2 Green — minimum migration
|
||||
|
||||
- [x] Write `server/migrations/005_market_history.sql`:
|
||||
- [x] `ALTER TABLE market_data_snapshots ADD COLUMN timeframe TEXT NOT NULL
|
||||
DEFAULT 'tick'`.
|
||||
- [x] `ALTER TABLE market_data_snapshots ADD CONSTRAINT
|
||||
market_data_snapshots_unique_obs UNIQUE
|
||||
(symbol, metric, timeframe, as_of)`.
|
||||
- [x] `CREATE INDEX market_data_snapshots_asof_idx
|
||||
ON market_data_snapshots (as_of DESC)`.
|
||||
- [x] `CREATE TABLE tradable_assets (…)` with columns
|
||||
`symbol PK, asset_class, exchange, name, tradable, fractionable,
|
||||
status, raw JSONB, refreshed_at`.
|
||||
- [x] `CREATE INDEX tradable_assets_status_idx
|
||||
ON tradable_assets (status, tradable)`.
|
||||
- [x] `CREATE TABLE market_data_sync_runs (…)` (see §0 plan).
|
||||
- [ ] (Phase 2 stub, commented) `CREATE TABLE market_data_archive (…)` —
|
||||
deferred to §4.2 (the migration runner splits on `;`, which would
|
||||
slice a commented stub mid-block; the archive table will be added
|
||||
in §4.2.2 when it is actually wired up).
|
||||
|
||||
### 1.3 Refactor
|
||||
|
||||
- [x] Confirm `MarketDataDb._rowToSnapshot` still reads correctly with the
|
||||
new column (read-side back-compat — no test changes needed, just verify
|
||||
existing `market_data_db_test.dart` still passes).
|
||||
- [x] Move shared SQL fragments into the migration runner if duplication
|
||||
appeared. _(none observed in 005; nothing to extract yet.)_
|
||||
|
||||
### 1.4 Confirm
|
||||
|
||||
- [x] `cd server && dart test test/integration/migration_test.dart
|
||||
test/integration/market_history_schema_test.dart` — green.
|
||||
- [x] `psql cyberhybridhub_test -c '\d market_data_snapshots'` shows the
|
||||
unique constraint and new column.
|
||||
|
||||
---
|
||||
|
||||
## 2. Tradable-asset universe sync
|
||||
|
||||
**Files (new):** `server/lib/alpaca/alpaca_assets_client.dart`,
|
||||
`server/lib/trading/tradable_assets_db.dart`,
|
||||
`server/lib/trading/tradable_assets_sync.dart`.
|
||||
|
||||
### 2.1 Alpaca assets client
|
||||
|
||||
#### 2.1.1 Red
|
||||
|
||||
- [x] Add fixture `server/test/fixtures/alpaca_assets_active.json` (≥5
|
||||
representative assets, mix of `tradable=true/false` and
|
||||
`fractionable=true/false`).
|
||||
- [x] Add `server/test/alpaca/alpaca_assets_client_test.dart`:
|
||||
- [x] Test: `listActiveTradable()` issues `GET` to
|
||||
`${tradingBaseUrl}/v2/assets?status=active&asset_class=us_equity`
|
||||
with `APCA-API-KEY-ID` + `APCA-API-SECRET-KEY` headers.
|
||||
- [x] Test: parses fixture into `List<AlpacaAsset>` — verifies symbol,
|
||||
exchange, fractionable, tradable, status fields.
|
||||
- [x] Test: 401 / 500 → throws `AlpacaAssetsException` with status code
|
||||
and body in the message.
|
||||
- [x] Test: empty response array → returns `[]`, does not throw.
|
||||
|
||||
#### 2.1.2 Green
|
||||
|
||||
- [x] Add `AlpacaAsset` model in `server/lib/alpaca/alpaca_models.dart`.
|
||||
- [x] Implement `AlpacaAssetsClient` with injectable `http.Client`
|
||||
(mirror `AlpacaMarketDataClient` shape).
|
||||
- [x] Add `AlpacaAssetsException`.
|
||||
|
||||
#### 2.1.3 Refactor
|
||||
|
||||
- [x] Extract a private `_authHeaders` helper if duplicated across
|
||||
Alpaca clients (DRY — but only if you actually duplicate). _(Lifted to
|
||||
`AlpacaEnv.authHeaders`; now reused by all three Alpaca clients.)_
|
||||
|
||||
#### 2.1.4 Confirm
|
||||
|
||||
- [x] `dart test test/alpaca/alpaca_assets_client_test.dart` — green.
|
||||
- [x] Tagged live test
|
||||
`server/test/alpaca/alpaca_assets_live_test.dart`
|
||||
(`@Tags(['alpaca'])`) — returns >100 symbols when keys present;
|
||||
skipped otherwise. Run manually:
|
||||
`dart test --tags=alpaca test/alpaca/alpaca_assets_live_test.dart`.
|
||||
|
||||
### 2.2 Universe persistence + diff
|
||||
|
||||
#### 2.2.1 Red
|
||||
|
||||
- [x] Create `server/test/integration/tradable_assets_db_test.dart`:
|
||||
- [x] Test: `upsertAll([A, B, C])` inserts 3 rows.
|
||||
- [x] Test: re-running `upsertAll([B*, C, D])` updates `B`, leaves `C`
|
||||
unchanged-by-content but `refreshed_at` bumped, inserts `D`, and
|
||||
marks `A` as `tradable=false, status='inactive'` (we never delete
|
||||
history).
|
||||
- [x] Test: `listActiveTradableSymbols()` returns only
|
||||
`tradable=true AND status='active'`.
|
||||
|
||||
#### 2.2.2 Red — sync orchestration
|
||||
|
||||
- [x] Create `server/test/integration/tradable_assets_sync_test.dart`:
|
||||
- [x] Test: `TradableAssetsSync.runOnce()` with mocked client returning
|
||||
the fixture → DB rows match; one row in `market_data_sync_runs`
|
||||
with `kind='universe'` and non-null `finished_at`.
|
||||
- [x] Test: client throws → sync run row recorded with `error` populated,
|
||||
`finished_at` non-null, and `rows_written = 0`.
|
||||
- [x] Test: two consecutive runs are safe (idempotent counts).
|
||||
|
||||
#### 2.2.3 Green
|
||||
|
||||
- [x] Implement `TradableAssetsDb.upsertAll`,
|
||||
`TradableAssetsDb.listActiveTradableSymbols`.
|
||||
- [x] Implement `TradableAssetsSync.runOnce()` that writes a
|
||||
`market_data_sync_runs` row around the upsert.
|
||||
|
||||
#### 2.2.4 Refactor
|
||||
|
||||
- [x] Pull "wrap a closure with a `sync_runs` audit row" into a small
|
||||
helper (`SyncRunRecorder.record(kind, body)`); reuse in §3 and §4.
|
||||
_(Landed at `server/lib/trading/sync_run_recorder.dart`;
|
||||
`TradableAssetsSync` already consumes it.)_
|
||||
|
||||
#### 2.2.5 Confirm
|
||||
|
||||
- [x] `dart test test/integration/tradable_assets_db_test.dart
|
||||
test/integration/tradable_assets_sync_test.dart` — green.
|
||||
|
||||
---
|
||||
|
||||
## 3. Historical backfill (1Day bars × 7 days)
|
||||
|
||||
**Files:** extend `server/lib/alpaca/alpaca_market_data_client.dart`,
|
||||
new `server/lib/trading/market_data_history.dart`,
|
||||
extend `server/lib/trading/market_data_db.dart`.
|
||||
|
||||
### 3.1 Alpaca client — time-range bars with pagination
|
||||
|
||||
#### 3.1.1 Red
|
||||
|
||||
- [x] Add fixtures:
|
||||
- [x] `server/test/fixtures/alpaca_bars_7d_multi_page1.json` — includes
|
||||
`next_page_token`.
|
||||
- [x] `server/test/fixtures/alpaca_bars_7d_multi_page2.json` — final
|
||||
page, `next_page_token: null`.
|
||||
- [x] Extend `server/test/alpaca/alpaca_market_data_client_test.dart`:
|
||||
- [x] Test: `getBarsRange(['SPY','AAPL'], timeframe: '1Day',
|
||||
start, end)` builds correct query string (`start`, `end`,
|
||||
`timeframe`, `feed`, `symbols`, `limit`).
|
||||
- [x] Test: follows pagination — when page1 returns
|
||||
`next_page_token='abc'`, client issues second request with
|
||||
`page_token=abc`; merges both pages' bars per symbol.
|
||||
- [x] Test: stops after a configurable `maxPages` (default 20) to
|
||||
prevent runaway loops.
|
||||
- [x] Test: 429 → throws `AlpacaMarketDataException` containing the
|
||||
word `rate` (so caller can detect & back off).
|
||||
|
||||
#### 3.1.2 Green
|
||||
|
||||
- [x] Implement `Future<AlpacaBarsResponse> getBarsRange({
|
||||
List<String> symbols, String timeframe, DateTime start, DateTime end,
|
||||
int maxPages = 20})` on `AlpacaMarketDataClient`.
|
||||
- [x] Extend `AlpacaBarsResponse` with a `merge(AlpacaBarsResponse other)`
|
||||
method so paginated chunks combine cleanly.
|
||||
|
||||
#### 3.1.3 Refactor
|
||||
|
||||
- [x] If the pagination loop is non-trivial, extract a private
|
||||
`_paginate<T>(initialUri, parsePage)` generic to reuse later for
|
||||
orders/positions endpoints. _(Loop kept inline in `getBarsRange` —
|
||||
~25 lines, clear enough; extract when a second consumer appears.)_
|
||||
|
||||
#### 3.1.4 Confirm
|
||||
|
||||
- [x] `dart test test/alpaca/alpaca_market_data_client_test.dart` — green.
|
||||
- [x] Tagged live test
|
||||
`server/test/alpaca/alpaca_market_data_history_live_test.dart`
|
||||
fetches 7-day bars for `SPY` and asserts ≥3 bars.
|
||||
|
||||
### 3.2 `MarketDataDb` — idempotent upsert + range query
|
||||
|
||||
#### 3.2.1 Red
|
||||
|
||||
- [x] Extend `server/test/integration/market_data_db_test.dart`:
|
||||
- [x] Test: `upsertSnapshot(symbol:'SPY', metric:'bar',
|
||||
timeframe:'1Day', as_of:T, price:500)` then re-upsert with
|
||||
`price:505` → exactly **one** row remains; price is `505`; `raw`
|
||||
is overwritten (volume also overwritten).
|
||||
- [x] Test: `barsForSymbol(symbol, timeframe, since, until)` returns
|
||||
rows ordered by `as_of ASC`; range is inclusive of `since`,
|
||||
exclusive of `until`.
|
||||
- [x] Test: `barsForSymbol` returns `[]` when no rows match; does not
|
||||
throw.
|
||||
- [x] Test: `latestSyncedAsOf(symbol, timeframe)` returns the newest
|
||||
`as_of` or `null`.
|
||||
|
||||
#### 3.2.2 Green
|
||||
|
||||
- [x] Implement `MarketDataDb.upsertSnapshot(...)` using
|
||||
`ON CONFLICT (symbol, metric, timeframe, as_of) DO UPDATE
|
||||
SET price = EXCLUDED.price, volume = EXCLUDED.volume,
|
||||
raw = EXCLUDED.raw`.
|
||||
- [x] Implement `MarketDataDb.barsForSymbol(...)` and
|
||||
`MarketDataDb.latestSyncedAsOf(...)`.
|
||||
|
||||
#### 3.2.3 Refactor
|
||||
|
||||
- [x] Replace existing `insertSnapshot` call sites in
|
||||
`market_data_ingest.dart` with `upsertSnapshot` (tick data has
|
||||
`timeframe='tick'`; same call shape). Re-run
|
||||
`test/integration/market_data_ingest_test.dart` — still green.
|
||||
|
||||
#### 3.2.4 Confirm
|
||||
|
||||
- [x] `dart test test/integration/market_data_db_test.dart
|
||||
test/integration/market_data_ingest_test.dart` — green.
|
||||
|
||||
### 3.3 `MarketDataHistorySync`
|
||||
|
||||
#### 3.3.1 Red
|
||||
|
||||
- [x] Add fixture
|
||||
`server/test/fixtures/alpaca_bars_7d_3symbols.json` — 7 bars × 3
|
||||
symbols (SPY/AAPL/MSFT), realistic timestamps.
|
||||
- [x] Add `server/test/integration/market_data_history_sync_test.dart`:
|
||||
- [x] Test: with mocked Alpaca returning the fixture → 21 rows upserted
|
||||
with `metric='bar'`, `timeframe='1Day'`; sync run row written.
|
||||
- [x] Test: re-running with the same fixture → still 21 rows; zero
|
||||
duplicates; `rows_written` reflects rows touched (not inserted).
|
||||
- [x] Test: partial outage — Alpaca returns 200 for batch 1
|
||||
(AAPL/MSFT), 500 for batch 2 (SPY) → AAPL/MSFT rows persisted;
|
||||
sync run row has `error` mentioning SPY; method does NOT throw.
|
||||
- [x] Test: respects `HISTORY_SYNC_MAX_SYMBOLS` cap (set to 2 → only
|
||||
first 2 symbols fetched).
|
||||
- [x] Test: batching — with `HISTORY_SYNC_BATCH_SIZE=2` and 5 symbols,
|
||||
Alpaca is called 3 times (mock call counter).
|
||||
|
||||
#### 3.3.2 Green
|
||||
|
||||
- [x] Implement `MarketDataHistorySync.runOnce({int windowDays = 7})`:
|
||||
- [x] Reads symbols from
|
||||
`TradableAssetsDb.listActiveTradableSymbols()`.
|
||||
- [x] Batches into `HISTORY_SYNC_BATCH_SIZE` groups; calls
|
||||
`getBarsRange` per batch.
|
||||
- [x] Upserts via `MarketDataDb.upsertSnapshot`.
|
||||
- [x] Captures per-batch errors without aborting; aggregates them into
|
||||
the sync run row (`SyncRunCounts.error`).
|
||||
|
||||
#### 3.3.3 Refactor
|
||||
|
||||
- [x] Extract batching helper if used by §3.4 incremental path too.
|
||||
_(Landed as `chunkList` in `market_data_history.dart`.)_
|
||||
|
||||
#### 3.3.4 Confirm
|
||||
|
||||
- [x] `dart test test/integration/market_data_history_sync_test.dart`
|
||||
— green.
|
||||
|
||||
### 3.4 Incremental daily catch-up
|
||||
|
||||
#### 3.4.1 Red
|
||||
|
||||
- [x] Extend `market_data_history_sync_test.dart`:
|
||||
- [x] Test: with prior `latestSyncedAsOf(symbol)` = `T-2d`, sync issues
|
||||
bars with `start = T-2d` (not `T-7d`); mock HTTP call records
|
||||
the requested start.
|
||||
- [x] Test: with prior sync `T-10d` (outside window), `start` is
|
||||
clamped to `T-windowDays`.
|
||||
- [x] Test: cold start (no prior sync) → `start = T-windowDays`.
|
||||
|
||||
#### 3.4.2 Green
|
||||
|
||||
- [x] Compute per-symbol `start` using `latestSyncedAsOf`; pass to
|
||||
`getBarsRange`.
|
||||
|
||||
#### 3.4.3 Refactor
|
||||
|
||||
- [x] If per-symbol starts vary inside a batch, fall back to
|
||||
`min(starts)` for the batched call and let `upsertSnapshot`
|
||||
dedupe the overlap — document the tradeoff in a code comment.
|
||||
|
||||
#### 3.4.4 Confirm
|
||||
|
||||
- [x] `dart test test/integration/market_data_history_sync_test.dart`
|
||||
— green.
|
||||
|
||||
---
|
||||
|
||||
## 4. Retention & cleanup (older than 7 days)
|
||||
|
||||
**Files (new):** `server/lib/trading/market_data_retention.dart`.
|
||||
|
||||
### 4.1 Hard-delete (Phase 1)
|
||||
|
||||
#### 4.1.1 Red
|
||||
|
||||
- [x] Create `server/test/integration/market_data_retention_test.dart`:
|
||||
- [x] Test: seed 10 snapshots spanning 14 days →
|
||||
`runCleanup({windowDays: 7})` deletes rows with
|
||||
`as_of < now() - 7d`, keeps the rest; returns `rowsRemoved`
|
||||
matching deleted count.
|
||||
- [x] Test: empty table → returns `rowsRemoved = 0`, does not throw.
|
||||
- [x] Test: `batchSize` honored — with 5000 rows older than window and
|
||||
`batchSize=1000`, the underlying `DELETE` is issued ≥5 times
|
||||
(use a counting wrapper around `_connection.execute`).
|
||||
- [x] Test: each invocation appends a `market_data_sync_runs` row
|
||||
with `kind='cleanup'`, `rows_removed` populated.
|
||||
- [x] Test: rows within window are NEVER touched (assert specific IDs
|
||||
survive).
|
||||
|
||||
#### 4.1.2 Green
|
||||
|
||||
- [x] Implement
|
||||
`MarketDataRetention.runCleanup({int windowDays = 7,
|
||||
int batchSize = 5000})`:
|
||||
- [x] Loop: `DELETE FROM market_data_snapshots WHERE as_of < $cutoff
|
||||
LIMIT $batchSize` (use CTE if Postgres version requires it),
|
||||
return rows removed; repeat until 0.
|
||||
- [x] Write a `market_data_sync_runs` row around the operation.
|
||||
|
||||
#### 4.1.3 Refactor
|
||||
|
||||
- [x] Reuse `SyncRunRecorder` from §2.2.4.
|
||||
|
||||
#### 4.1.4 Confirm
|
||||
|
||||
- [x] `dart test test/integration/market_data_retention_test.dart`
|
||||
— green.
|
||||
|
||||
### 4.2 Archive (Phase 2 — opt-in)
|
||||
|
||||
#### 4.2.1 Red
|
||||
|
||||
- [x] Extend `market_data_retention_test.dart`:
|
||||
- [x] Test: with `archiveEnabled: true`, expired rows are copied into
|
||||
`market_data_archive` with `archived_at = now()` BEFORE being
|
||||
deleted; archive count grows by exactly `rowsRemoved`.
|
||||
- [x] Test: archive run is transactional — if archive `INSERT` fails,
|
||||
no `DELETE` happens; sync run row records the error.
|
||||
- [x] Test: `archiveEnabled: false` (default) → archive table
|
||||
untouched.
|
||||
|
||||
#### 4.2.2 Green
|
||||
|
||||
- [x] Uncomment `market_data_archive` table in migration 005 (or add it
|
||||
now if you deferred it). _(Added `006_market_data_archive.sql`.)_
|
||||
- [x] Implement
|
||||
`MarketDataRetention.runArchiveAndCleanup({int windowDays})`
|
||||
with explicit `BEGIN; INSERT … SELECT …; DELETE …; COMMIT`.
|
||||
|
||||
#### 4.2.3 Refactor
|
||||
|
||||
- [x] Consider a single unified entry point
|
||||
`MarketDataRetention.run({int windowDays, bool archive})` that
|
||||
dispatches; only do this if it doesn't muddy the failure-isolation
|
||||
story.
|
||||
|
||||
#### 4.2.4 Confirm
|
||||
|
||||
- [x] `dart test test/integration/market_data_retention_test.dart`
|
||||
— green.
|
||||
|
||||
---
|
||||
|
||||
## 5. Scheduler — daily cadence inside the worker
|
||||
|
||||
**Files:** new `server/lib/workers/market_history_scheduler.dart`,
|
||||
extend `server/lib/workers/question_background_worker.dart`,
|
||||
extend `server/bin/server.dart`.
|
||||
|
||||
### 5.1 Red
|
||||
|
||||
- [x] Add `server/test/integration/market_history_scheduler_test.dart`:
|
||||
- [x] Test: cold start → `runIfDue(now=T0)` runs all 3 stages
|
||||
(`universe`, `backfill`, `cleanup`) in that order;
|
||||
`market_data_sync_runs` has 3 rows.
|
||||
- [x] Test: same-day re-run (`now=T0 + 1h`) → no stages run; zero new
|
||||
sync rows.
|
||||
- [x] Test: next day (`now=T0 + 24h`) → all 3 stages run again.
|
||||
- [x] Test: per-stage cadence — set
|
||||
`MARKET_UNIVERSE_REFRESH_HOURS=48`, `MARKET_HISTORY_SYNC_HOURS=24`,
|
||||
`MARKET_HISTORY_CLEANUP_HOURS=24`; at T0+24h only backfill +
|
||||
cleanup run.
|
||||
- [x] Test: failure isolation — backfill throws → cleanup still runs;
|
||||
both stages logged in `market_data_sync_runs` (one with `error`,
|
||||
one without).
|
||||
- [x] Test: `MARKET_HISTORY_SYNC_HOUR_UTC=10` (optional alignment) →
|
||||
scheduler runs only when local UTC hour ≥ 10 AND last run was on
|
||||
a prior UTC day.
|
||||
- [x] Add `server/test/integration/market_history_worker_wireup_test.dart`:
|
||||
- [x] Test: `QuestionBackgroundWorker._tick` invokes
|
||||
`MarketHistoryScheduler.runIfDue` **before** the
|
||||
`TradingOrchestrator` per-user loop. Use a spy scheduler that
|
||||
records the call order.
|
||||
- [x] Test: scheduler exception is caught — worker tick continues into
|
||||
the orchestrator loop; stderr contains the error.
|
||||
|
||||
### 5.2 Green
|
||||
|
||||
- [x] Implement `MarketHistoryScheduler` with `runIfDue(DateTime now)`,
|
||||
reading the last `finished_at` per `kind` from
|
||||
`market_data_sync_runs`.
|
||||
- [x] Wire `QuestionBackgroundWorker` to accept an optional
|
||||
`MarketHistoryScheduler` and call it at the top of `_tick`.
|
||||
- [x] Wire `bin/server.dart` to construct the scheduler only when
|
||||
`MARKET_HISTORY_SYNC_ENABLED=true && TRADING_ENABLED=true`.
|
||||
|
||||
### 5.3 Refactor
|
||||
|
||||
- [x] If the three stages each need similar before/after logic, abstract
|
||||
a small `_runStage(kind, body)` inside the scheduler.
|
||||
(`_maybeRunStage` — no further refactor needed.)
|
||||
|
||||
### 5.4 Confirm
|
||||
|
||||
- [x] `dart test test/integration/market_history_scheduler_test.dart
|
||||
test/integration/market_history_worker_wireup_test.dart` — green.
|
||||
|
||||
---
|
||||
|
||||
## 6. Question pipeline — "guess the move" rule
|
||||
|
||||
**Files:** extend `server/lib/trading/rule_engine.dart`,
|
||||
extend `server/lib/trading/trading_pipeline.dart`,
|
||||
new `server/lib/trading/market_history_query.dart`.
|
||||
|
||||
The guessing game uses the rolling 7-day window — questions must reveal
|
||||
just enough for the user to guess (obfuscated symbol/price/direction).
|
||||
**No trade is placed for this rule** — answers feed scoring only.
|
||||
|
||||
### 6.1 Red — `MarketHistoryQuery`
|
||||
|
||||
- [x] Add `server/test/integration/market_history_query_test.dart`:
|
||||
- [x] Test: `weeklyMovers({minBars: 5, asOf})` returns only symbols
|
||||
with ≥5 daily bars in the window; each entry exposes
|
||||
`(symbol, openClose, currentClose, days)`.
|
||||
- [x] Test: deterministic — supply a `random: Random(42)` and assert a
|
||||
stable selection order across runs.
|
||||
- [x] Test: symbols with stale data (newest bar > 2d old) are
|
||||
excluded.
|
||||
|
||||
### 6.2 Red — rule engine extension
|
||||
|
||||
- [x] Add `server/test/trading/rule_engine_guess_weekly_move_test.dart`:
|
||||
- [x] Test: rule kind `guess_weekly_move` with mocked
|
||||
`MarketHistoryQuery` returning SPY {ref=500, current=510, days=5}
|
||||
→ produces a `RuleEvaluation` with:
|
||||
- obfuscated `symbol_token='ASSET_A'`,
|
||||
- `correct_answer = 10` (up direction),
|
||||
- `question_text` substituting `{{token}}`, `{{ref_price}}`,
|
||||
`{{ref_days_ago}}`, NEVER `{{symbol}}`.
|
||||
- [x] Test: down move (ref=510, current=500) → `correct_answer = -10`.
|
||||
- [x] Test: insufficient bars → no fire.
|
||||
- [x] Test: `questions.metadata.guess_symbol` is set to real symbol
|
||||
(server-side only) when the question is created in §6.3.
|
||||
|
||||
### 6.3 Red — pipeline wiring
|
||||
|
||||
- [x] Add
|
||||
`server/test/integration/trading_pipeline_guess_weekly_move_test.dart`:
|
||||
- [x] Test: end-to-end with seeded 7 daily bars for SPY → pipeline
|
||||
creates a question with obfuscated text; `metadata.guess_symbol
|
||||
= 'SPY'`; `pipeline_key='trading'`,
|
||||
`pipeline_step='guess_weekly_move:await_answer'`.
|
||||
- [x] Test: `onAnswerSubmitted` with matching direction (e.g., +10 on
|
||||
an up move) records `score_delta = +1` in
|
||||
`user_trading_state.context.guess_score`; non-matching records
|
||||
`score_delta = -1`.
|
||||
- [x] Test: `TradeActuator.processPendingOrders` is **NEVER called**
|
||||
for `guess_weekly_move` answers (assert via spy).
|
||||
- [x] Test: cooldown — after a fire, the same symbol is not re-picked
|
||||
for `GUESS_COOLDOWN_HOURS` (default 24).
|
||||
|
||||
### 6.4 Green
|
||||
|
||||
- [x] Implement `MarketHistoryQuery.weeklyMovers({...})`.
|
||||
- [x] Add rule kind to `RuleEngine` with the new template tokens.
|
||||
- [x] Extend `TradingPipeline.evaluate` + `onAnswerSubmitted` for the
|
||||
new rule kind, including the cooldown bookkeeping.
|
||||
|
||||
### 6.5 Refactor
|
||||
|
||||
- [x] If the token mapping (real symbol ↔ `ASSET_A`/`ASSET_B`/…) is used
|
||||
in multiple places, lift it into a `SymbolObfuscator` helper with
|
||||
its own focused unit test.
|
||||
|
||||
### 6.6 Confirm
|
||||
|
||||
- [x] `dart test test/integration/market_history_query_test.dart
|
||||
test/trading/rule_engine_guess_weekly_move_test.dart
|
||||
test/integration/trading_pipeline_guess_weekly_move_test.dart`
|
||||
— green.
|
||||
|
||||
---
|
||||
|
||||
## 7. Env additions (`server/.env.example`)
|
||||
|
||||
```bash
|
||||
# Rolling history feature gate
|
||||
MARKET_HISTORY_SYNC_ENABLED=false
|
||||
MARKET_HISTORY_WINDOW_DAYS=7
|
||||
MARKET_HISTORY_RETENTION_DAYS=7
|
||||
MARKET_HISTORY_ARCHIVE_ENABLED=false
|
||||
|
||||
# Cadence (hours)
|
||||
MARKET_UNIVERSE_REFRESH_HOURS=24
|
||||
MARKET_HISTORY_SYNC_HOURS=24
|
||||
MARKET_HISTORY_CLEANUP_HOURS=24
|
||||
MARKET_HISTORY_SYNC_HOUR_UTC=10 # optional alignment hour
|
||||
|
||||
# Batching / safety
|
||||
HISTORY_SYNC_BATCH_SIZE=50
|
||||
HISTORY_SYNC_MAX_SYMBOLS=2000 # hard cap; Alpaca Basic-friendly
|
||||
MIN_BARS_FOR_GUESS=5
|
||||
GUESS_COOLDOWN_HOURS=24
|
||||
```
|
||||
|
||||
### 7.1 Red
|
||||
|
||||
- [x] Add `server/test/env/market_history_env_test.dart`:
|
||||
- [x] Test: defaults parsed when env empty (`enabled=false`,
|
||||
`windowDays=7`, etc.).
|
||||
- [x] Test: `MARKET_HISTORY_SYNC_ENABLED=true` while
|
||||
`TRADING_ENABLED=false` → `Env.assertConsistent()` throws.
|
||||
- [x] Test: `MARKET_HISTORY_WINDOW_DAYS=0` or negative → throws.
|
||||
- [x] Test: `MARKET_HISTORY_SYNC_HOUR_UTC=24` → throws (valid range
|
||||
`0..23`).
|
||||
|
||||
### 7.2 Green
|
||||
|
||||
- [x] Extend `server/lib/env.dart` to load and validate these vars.
|
||||
- [x] Append the block above to `server/.env.example`.
|
||||
|
||||
### 7.3 Refactor
|
||||
|
||||
- [x] If `Env` has grown unwieldy, split market-history vars into
|
||||
`MarketHistoryEnv` (typed value object) and have `Env` expose it.
|
||||
|
||||
### 7.4 Confirm
|
||||
|
||||
- [x] `dart test test/env/market_history_env_test.dart` — green.
|
||||
- [x] Document each var in `server/README.md` under a new
|
||||
**"Market history window"** subsection.
|
||||
|
||||
---
|
||||
|
||||
## 8. Operational tooling
|
||||
|
||||
### 8.1 Red
|
||||
|
||||
- [ ] Add `server/test/bin/sync_market_history_smoke_test.dart`:
|
||||
- [ ] Test: imports `bin/sync_market_history.dart` `main` function and
|
||||
runs it with `QUESTION_PIPELINE_TEST_MODE=true` + a fake DB →
|
||||
exits 0 and emits the expected one-line log.
|
||||
- [ ] Add equivalent smoke test for `bin/cleanup_market_history.dart`.
|
||||
|
||||
### 8.2 Green
|
||||
|
||||
- [ ] Add CLI `server/bin/sync_market_history.dart` with
|
||||
`--window=<days>` flag (default 7); honors test mode.
|
||||
- [ ] Add CLI `server/bin/cleanup_market_history.dart` with
|
||||
`--window=<days>` and `--archive` flags.
|
||||
- [ ] Add structured one-line log:
|
||||
`kind=… symbols=… rows_written=… rows_removed=… duration_ms=… error=…`.
|
||||
|
||||
### 8.3 Refactor
|
||||
|
||||
- [ ] Share argument parsing between the two CLIs if duplicated.
|
||||
|
||||
### 8.4 Confirm
|
||||
|
||||
- [ ] `dart test test/bin/` — green.
|
||||
- [ ] Manual: `dart run server:bin/sync_market_history.dart --window=7`
|
||||
against `cyberhybridhub_test` works end-to-end.
|
||||
|
||||
### 8.5 Optional admin endpoint (defer until needed)
|
||||
|
||||
- [ ] Behind Firebase admin auth, `POST /v1/admin/market-data/resync?window=7`
|
||||
enqueues a sync run; **not exposed to Flutter**.
|
||||
|
||||
---
|
||||
|
||||
## 9. Test pyramid for this milestone
|
||||
|
||||
| Layer | Test files |
|
||||
|------|------------|
|
||||
| Unit | `test/alpaca/alpaca_assets_client_test.dart` |
|
||||
| Unit | `test/alpaca/alpaca_market_data_client_test.dart` (extended) |
|
||||
| Unit | `test/trading/rule_engine_guess_weekly_move_test.dart` |
|
||||
| Unit | `test/env/market_history_env_test.dart` |
|
||||
| DB integration | `test/integration/market_history_schema_test.dart` |
|
||||
| DB integration | `test/integration/tradable_assets_db_test.dart` |
|
||||
| DB integration | `test/integration/tradable_assets_sync_test.dart` |
|
||||
| DB integration | `test/integration/market_data_db_test.dart` (extended) |
|
||||
| DB integration | `test/integration/market_data_history_sync_test.dart` |
|
||||
| DB integration | `test/integration/market_data_retention_test.dart` |
|
||||
| DB integration | `test/integration/market_history_query_test.dart` |
|
||||
| DB integration | `test/integration/trading_pipeline_guess_weekly_move_test.dart` |
|
||||
| Worker integration | `test/integration/market_history_scheduler_test.dart` |
|
||||
| Worker integration | `test/integration/market_history_worker_wireup_test.dart` |
|
||||
| Bin smoke | `test/bin/sync_market_history_smoke_test.dart` |
|
||||
| Bin smoke | `test/bin/cleanup_market_history_smoke_test.dart` |
|
||||
| Tagged (`alpaca`) | `test/alpaca/alpaca_assets_live_test.dart` |
|
||||
| Tagged (`alpaca`) | `test/alpaca/alpaca_market_data_history_live_test.dart` |
|
||||
|
||||
**CI gating:**
|
||||
|
||||
```bash
|
||||
# default job — no Alpaca keys, must pass on every PR
|
||||
cd server && dart test
|
||||
|
||||
# nightly / manual — requires ALPACA_API_KEY_ID / ALPACA_API_SECRET_KEY
|
||||
cd server && dart test --tags=alpaca
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Acceptance criteria (Gate H — Rolling history)
|
||||
|
||||
- [ ] `market_data_snapshots` contains rows for every active tradable with
|
||||
`as_of` within the last 7 days, and no rows older.
|
||||
- [ ] Re-running backfill is a no-op (zero duplicate rows; deterministic
|
||||
`rows_written` count when nothing changed upstream).
|
||||
- [ ] Cleanup removes only rows older than the window and never touches
|
||||
newer rows.
|
||||
- [ ] Worker performs one full cycle (universe → backfill → cleanup) per
|
||||
day with stage isolation; failure in one stage does not block the
|
||||
others.
|
||||
- [ ] A `guess_weekly_move` question can be generated end-to-end from
|
||||
pure DB data — no live Alpaca call at evaluation time.
|
||||
- [ ] `dart test` is green; `dart test --tags=alpaca` is green when keys
|
||||
are present.
|
||||
- [ ] `MARKET_HISTORY_SYNC_ENABLED=false` is the default; nothing runs
|
||||
unless explicitly enabled.
|
||||
- [ ] Safety: `MARKET_HISTORY_SYNC_ENABLED=true` without
|
||||
`TRADING_ENABLED=true` fails fast at server boot.
|
||||
|
||||
---
|
||||
|
||||
## 11. Risks & mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Alpaca rate limits on full-universe pull | Batched `bars` calls (`HISTORY_SYNC_BATCH_SIZE`); per-batch error isolation; 429 → exception logged in sync run, retry next day. |
|
||||
| Migration deadlocks on large `market_data_snapshots` | Cleanup batches via `LIMIT` + loop; unique constraint added with `NOT VALID` then `VALIDATE CONSTRAINT` if existing dataset is huge (document in migration). |
|
||||
| Duplicate Alpaca asset entries between runs | `upsertAll` PK-on-symbol; we mark missing symbols inactive instead of deleting. |
|
||||
| Guessing game leaks the real symbol | Question text uses tokens only; real symbol lives in `questions.metadata` (server side); add a regex test that scans `question_text` for any known ticker. |
|
||||
| Backfill blowing past disk budget | Hard caps via `HISTORY_SYNC_MAX_SYMBOLS` and `MARKET_HISTORY_WINDOW_DAYS`; retention deletes daily so steady-state size is bounded. |
|
||||
|
||||
---
|
||||
|
||||
## 12. Progress log
|
||||
|
||||
<!-- Newest entries on top. One line per completed step. -->
|
||||
|
||||
| Date | Step | Result |
|
||||
|------|------|--------|
|
||||
| 2026-05-26 | §7 Env additions | Green: 6/6 env tests; `dart test` 133/133. `MarketHistoryEnv.fromMap` + `assertConsistent`; `ServerEnv.marketHistory`; wired scheduler/sync/retention/guess; `server/.env.example`; README **Market history window**. |
|
||||
| 2026-05-26 | §6 Guess-the-move rule | Green: 12 new tests; `dart test` 127/127. `MarketHistoryQuery.weeklyMovers`; `RuleEngine.evaluateGuessWeeklyMove`; `SymbolObfuscator`; `TradingPipeline` scoring + per-symbol cooldown; `questions.metadata` migration `007`; no pending orders on guess answers. |
|
||||
| 2026-05-26 | §5 Scheduler (worker cadence) | Green: 8/8 scheduler + wireup tests; `dart test` 115/115. `MarketHistoryScheduler.runIfDue` (per-kind cadence + optional `syncHourUtc`); worker calls scheduler before pipeline/trading; `server.dart` wires universe→backfill→cleanup when `MARKET_HISTORY_SYNC_ENABLED` + real Alpaca; `ServerEnv.marketHistorySyncEnabled`; `SyncRunRecorder` uses injected `now` for `finished_at`. |
|
||||
| 2026-05-26 | §4 Retention & cleanup | Green: 8/8 retention tests; `dart test` 107/107. `MarketDataRetention.runCleanup` (batched hard-delete via CTE+RETURNING); `runArchiveAndCleanup` (transactional archive-then-delete); unified `run(archive:)`; migration `006_market_data_archive.sql`; reuses `SyncRunRecorder` kind=`cleanup`. |
|
||||
| 2026-05-26 | §3 Historical backfill (1Day × 7d) | Green: 17 new tests (6 client + 4 db + 8 sync); `dart test` 99/99; live `alpaca_market_data_history_live_test` ≥3 SPY bars. `getBarsRange` + pagination; `upsertSnapshot`/`barsForSymbol`/`latestSyncedAsOf`; `MarketDataHistorySync` with incremental catch-up + partial batch errors via `SyncRunCounts.error`. Defaults in `MarketHistoryConfig` (batch=100). |
|
||||
| 2026-05-26 | §2 Tradable-asset universe sync | Green: 11/11 §2 tests pass (5 client + 3 db + 3 sync); `dart test` 82/82 green; tagged live `alpaca_assets_live_test` returned >100 active us_equity assets. Refactor 2.1.3 lifted auth headers to `AlpacaEnv.authHeaders`; 2.2.4 lifted `SyncRunRecorder` for §3/§4 reuse. |
|
||||
| 2026-05-26 | §1 Schema additions (migration `005_market_history.sql`) | Green: 5/5 schema tests pass; `dart test` 70/70 green; `\d market_data_snapshots` shows `timeframe` col + `market_data_snapshots_unique_obs` unique constraint. Archive stub deferred to §4.2 to keep `;`-split migration runner happy. |
|
||||
|
||||
---
|
||||
|
||||
## 13. References
|
||||
|
||||
- Existing snapshot writer: `server/lib/trading/market_data_ingest.dart`
|
||||
- Existing snapshot DB: `server/lib/trading/market_data_db.dart`
|
||||
- Existing migration to extend: `server/migrations/004_trading.sql`
|
||||
- Orchestrator hook point: `server/lib/trading/trading_orchestrator.dart`
|
||||
- Worker hook point: `server/lib/workers/question_background_worker.dart`
|
||||
- Plans: [`TRADING_DEVELOPMENT_PLAN.md`](./TRADING_DEVELOPMENT_PLAN.md),
|
||||
[`TRADING_TDD_PLAN.md`](./TRADING_TDD_PLAN.md)
|
||||
- Alpaca docs: [Market Data](https://docs.alpaca.markets/docs/market-data-api),
|
||||
[Trading / Assets](https://docs.alpaca.markets/docs/trading-api),
|
||||
[Bars](https://docs.alpaca.markets/reference/stockbars)
|
||||
|
||||
---
|
||||
|
||||
*Document version: 1.0 — Rolling 7-day market data window, cleanup, and
|
||||
guessing-game question integration.*
|
||||
@ -1,453 +0,0 @@
|
||||
# Trading TDD Plan — Progress Tracker
|
||||
|
||||
Companion to [TRADING_DEVELOPMENT_PLAN.md](./TRADING_DEVELOPMENT_PLAN.md). Each step follows **Red → Green → Confirm** before moving on.
|
||||
|
||||
**How agents should use this file:**
|
||||
|
||||
1. Pick the first unchecked step in order.
|
||||
2. Write the failing test (Red), implement minimal code (Green), run the Confirm gate.
|
||||
3. Check off boxes and add a one-line note under **Progress log** with date and result.
|
||||
4. Do not skip Confirm gates or start Phase 2 until Gate B passes.
|
||||
|
||||
---
|
||||
|
||||
## Overall status
|
||||
|
||||
| Milestone | Status | Notes |
|
||||
|-----------|--------|-------|
|
||||
| Step 0 — Test harness | ✅ Done | 2026-05-23 |
|
||||
| Steps 1–11 — Phase 1 MVP | 🔄 In progress | Day 7: Steps 9–10 done; Gate A scripted test green |
|
||||
| Gate A — Local loop (test mode) | ✅ Done | 2026-05-26 — `trading_orchestrator_gate_a_test.dart` |
|
||||
| Gate B — Paper Alpaca E2E | ⬜ Not started | |
|
||||
| Steps 12–15 — Phase 2 | ⬜ Not started | |
|
||||
| Steps 16–19 — Phase 3 | ⬜ Deferred | |
|
||||
|
||||
Legend: ⬜ Not started · 🔄 In progress · ✅ Done · ⏸ Blocked
|
||||
|
||||
---
|
||||
|
||||
## Progress log
|
||||
|
||||
<!-- Agents: append newest entries at the top -->
|
||||
|
||||
| Date | Step | Result |
|
||||
|------|------|--------|
|
||||
| 2026-05-26 | 10 + Gate A | `dart test` green (57 tests); `TradingOrchestrator` (ingest → evaluate → actuate per user) + `QuestionBackgroundWorker` integration; Gate A scripted test (seed → tick → question → +10 → tick → trade_orders row → tick → cooldown) passing in test mode |
|
||||
| 2026-05-26 | 9 | `dart test` green (56 tests); AlpacaTradingClient (POST /v2/orders, GET by_client_order_id, dup detection) + TradeActuator (test mode + mocked Alpaca, guardrails, idempotency); tagged `alpaca` live roundtrip script |
|
||||
| 2026-05-25 | 7–8 | `dart test` green (47 tests); TradingPipeline evaluate/handleAnswer + QuestionPipeline branch |
|
||||
| 2026-05-25 | 5–6 | `dart test` green (39 tests); rule engine + guardrails (pure logic) |
|
||||
| 2026-05-23 | 3–4 | `dart test` green (21 tests); Alpaca MD client, ingest, poll interval; live Alpaca OK |
|
||||
| 2026-05-23 | 1.3–1.4, 2 | `dart test` green (17 tests); config merge, trade orders, Alpaca env/models |
|
||||
| 2026-05-23 | 0, 1.1–1.2 | `dart test` green (4 tests); migration 004 + MarketDataDb |
|
||||
|
||||
---
|
||||
|
||||
## Principles
|
||||
|
||||
| Principle | How it applies |
|
||||
|-----------|----------------|
|
||||
| **One behavior per cycle** | One failing test → minimal code → gate check |
|
||||
| **No Alpaca in unit tests** | Mock HTTP clients; inject fixture JSON |
|
||||
| **DB tests are integration** | Real Postgres test DB (`cyberhybridhub_test`) |
|
||||
| **E2E gates are manual/scripted** | Flutter swipe + Alpaca paper dashboard at defined checkpoints |
|
||||
| **Feature flags off until wired** | `TRADING_ENABLED=false` until Step 10 complete |
|
||||
|
||||
---
|
||||
|
||||
## Step 0 — Test harness
|
||||
|
||||
### 0.1 Server test setup
|
||||
|
||||
- [x] Add `dev_dependencies: test: ^1.25.0` to `server/pubspec.yaml`
|
||||
- [x] Create `server/test/` smoke test
|
||||
- [x] Create `server/test/helpers/fixture_loader.dart`
|
||||
- [x] Create `server/test/helpers/mock_http_client.dart`
|
||||
- [x] Create `server/test/helpers/test_env.dart`
|
||||
- [x] Create fixture files:
|
||||
- [x] `server/test/fixtures/alpaca_latest_trade.json`
|
||||
- [x] `server/test/fixtures/alpaca_daily_bars.json`
|
||||
- [x] `server/test/fixtures/trading_config_default.json`
|
||||
- [x] `server/test/fixtures/market_snapshots_spy_dip.json`
|
||||
|
||||
**Red:** Smoke test fails until layout exists.
|
||||
|
||||
**Green:** Minimal test directory and helpers.
|
||||
|
||||
**Confirm:** `cd server && dart test` passes with 1 smoke test.
|
||||
|
||||
---
|
||||
|
||||
### 0.2 Integration test DB
|
||||
|
||||
- [x] Test helper applies migrations 001–004 on `cyberhybridhub_test`
|
||||
- [x] `server/test/integration/migration_test.dart` — migration applies cleanly
|
||||
|
||||
**Red:** Migration test fails (no `004_trading.sql` yet).
|
||||
|
||||
**Green:** Migration runner + empty-schema apply.
|
||||
|
||||
**Confirm:** `dart test test/integration/migration_test.dart` passes locally and in CI.
|
||||
|
||||
---
|
||||
|
||||
### 0.3 Flutter (minimal)
|
||||
|
||||
- [ ] No new Flutter tests until Step 9 — existing `SwipeQuestionTile` + SignalR contract unchanged
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Foundation (MVP)
|
||||
|
||||
Maps to TRADING_DEVELOPMENT_PLAN §5, §6, §7, §8, §9, §10, §14 Phase 1.
|
||||
|
||||
---
|
||||
|
||||
### Step 1 — Schema & DB accessors
|
||||
|
||||
**Refs:** §5 Postgres schema · `market_data_db.dart` · `trading_config_db.dart` · `trade_orders_db.dart`
|
||||
|
||||
#### 1.1 Migration
|
||||
|
||||
- [x] **Red:** Integration test — INSERT into `market_data_snapshots`, `user_trading_config`, `trade_orders`; FK to `users` enforced
|
||||
- [x] **Green:** `server/migrations/004_trading.sql`
|
||||
- [x] **Green:** Seed `trading_config_templates` row `default_paper_watchlist`
|
||||
- [x] **Confirm:** `\d market_data_snapshots` in psql; seed template queryable
|
||||
|
||||
#### 1.2 MarketDataDb
|
||||
|
||||
- [x] **Red:** `insertSnapshot()` then `latestForSymbol(symbol, metric)` returns newest row by `as_of`
|
||||
- [x] **Green:** `server/lib/trading/market_data_db.dart`
|
||||
- [x] **Confirm:** Two SPY/`last_trade` inserts → latest returns newer price
|
||||
|
||||
#### 1.3 TradingConfigDb
|
||||
|
||||
- [x] **Red:** `resolveEffectiveConfig(uid)` merges template + user JSONB; user override wins
|
||||
- [x] **Green:** `server/lib/trading/trading_config_db.dart`
|
||||
- [x] **Green:** `server/lib/trading/trading_config.dart` parser
|
||||
- [x] **Confirm:** Partial user override yields merged `enabled`, `data_inputs`, `rules`
|
||||
|
||||
#### 1.4 TradeOrdersDb
|
||||
|
||||
- [x] **Red:** Duplicate `client_order_id` raises unique violation; `findByClientOrderId` returns existing row
|
||||
- [x] **Green:** `server/lib/trading/trade_orders_db.dart`
|
||||
- [x] **Confirm:** Idempotency check prevents double-insert
|
||||
|
||||
---
|
||||
|
||||
### Step 2 — Alpaca env & models
|
||||
|
||||
**Refs:** §3 · §11 · `alpaca_env.dart` · `alpaca_models.dart`
|
||||
|
||||
- [x] **Red:** `AlpacaEnv.fromMap()` reads keys; `assertPaperOnly()` throws when live URL + `ALPACA_ALLOW_LIVE=false`
|
||||
- [x] **Green:** `server/lib/alpaca/alpaca_env.dart`
|
||||
- [x] **Green:** `server/lib/alpaca/alpaca_models.dart` (Trade, Bar, OrderRequest)
|
||||
- [x] **Green:** Extend `server/lib/env.dart` with trading flags (after env tests pass)
|
||||
- [x] **Confirm:** Unit tests — paper default, live blocked, missing keys
|
||||
|
||||
---
|
||||
|
||||
### Step 3 — Alpaca Market Data client (REST)
|
||||
|
||||
**Refs:** §9.1 · `alpaca_market_data_client.dart`
|
||||
|
||||
- [x] **Red:** Mock HTTP — `getLatestTrade('SPY')` parses fixture; sends `APCA-API-KEY-ID` headers
|
||||
- [x] **Green:** `server/lib/alpaca/alpaca_market_data_client.dart` with injectable `http.Client`
|
||||
- [x] **Confirm:** Optional tagged test `@Tags(['alpaca'])` — one real call (skipped in CI)
|
||||
|
||||
---
|
||||
|
||||
### Step 4 — Snapshot normalization & ingest
|
||||
|
||||
**Refs:** §9.3 · `market_data_ingest.dart`
|
||||
|
||||
- [x] **Red:** Config `data_inputs` with `[last_trade, daily_bar, prev_close]` → 3 snapshot rows with correct `metric`, `price`, `as_of`
|
||||
- [x] **Green:** `server/lib/trading/market_data_ingest.dart` — `runIfDue()` writes via `MarketDataDb`
|
||||
- [x] **Confirm:** Integration — ingest for seeded user; `SELECT * FROM market_data_snapshots WHERE symbol='SPY'`
|
||||
|
||||
#### 4.1 Poll interval
|
||||
|
||||
- [x] **Red:** Second call within `poll_interval_seconds` does not fetch (uses `user_trading_state.context`)
|
||||
- [x] **Green:** Update `user_trading_state` on ingest
|
||||
- [x] **Confirm:** Two rapid `runIfDue` calls → mock HTTP call count = 1
|
||||
|
||||
---
|
||||
|
||||
### Step 5 — Rule engine (pure logic)
|
||||
|
||||
**Refs:** §4.2 · §8 · `rule_engine.dart`
|
||||
|
||||
- [x] **Red:** `price_below_pct_of_ref` — SPY `last_trade=492`, `prev_close=500`, threshold `-1.5` → fires with `pct ≈ -1.6`
|
||||
- [x] **Green:** `server/lib/trading/rule_engine.dart` — `RuleEngine.evaluate(rule, snapshots) → RuleEvaluation?`
|
||||
- [x] **Confirm:** Table-driven unit tests:
|
||||
|
||||
| Case | Expected |
|
||||
|------|----------|
|
||||
| Above threshold (-0.5%) | No fire |
|
||||
| Missing metric | No fire |
|
||||
| Stale `as_of` (> max_staleness) | No fire |
|
||||
| Cooldown: rule fired today | No fire |
|
||||
| Template `{{pct}}`, `{{symbol}}`, `{{price}}` | Substituted |
|
||||
|
||||
---
|
||||
|
||||
### Step 6 — Guardrails
|
||||
|
||||
**Refs:** §8.3
|
||||
|
||||
- [x] **Red:** `Guardrails.check()` rejects when `max_orders_per_day` exceeded
|
||||
- [x] **Red:** Rejects blocklisted symbol
|
||||
- [x] **Red:** Rejects when `require_question_before_order` violated
|
||||
- [x] **Green:** `server/lib/trading/guardrails.dart`
|
||||
- [x] **Confirm:** Server-side max notional ceiling enforced even if config allows more
|
||||
|
||||
---
|
||||
|
||||
### Step 7 — Trading pipeline — question creation
|
||||
|
||||
**Refs:** §8.1 · `trading_pipeline.dart` · `QuestionService`
|
||||
|
||||
- [x] **Red:** Rule fires + guardrails pass + queue room → `createAndDeliverQuestion` with `pipeline_key=trading`, `pipeline_step=dip_confirm:await_confirm`, `correct_answer=10`, substituted text
|
||||
- [x] **Green:** `server/lib/trading/trading_pipeline.dart` — `evaluate()` with mocked deps
|
||||
- [x] **Confirm:** Integration — seeded snapshots + config → row in `questions` with correct tags
|
||||
- [x] **Green:** `user_trading_state.context` records pending rule id and phase
|
||||
|
||||
---
|
||||
|
||||
### Step 8 — Answer handling → order proposal
|
||||
|
||||
**Refs:** §8.2 · extend `QuestionPipeline.onAnswerSubmitted`
|
||||
|
||||
- [x] **Red:** `pipeline_key=trading`, user `+10`, `BranchDecision.yesNo` → match → stages order (`submit_order`), no Alpaca yet
|
||||
- [x] **Red:** User `-10` → skip logged, no order
|
||||
- [x] **Green:** `PipelineKeys.trading` in `question_pipeline.dart`
|
||||
- [x] **Green:** `_handleTradingAnswer` in `question_pipeline.dart` (delegates to `TradingPipeline.handleAnswer`)
|
||||
- [x] **Confirm:** Integration test exercises the `QuestionPipeline.onAnswerSubmitted` switch for `pipeline_key=trading`
|
||||
|
||||
---
|
||||
|
||||
### Step 9 — Trade actuator
|
||||
|
||||
**Refs:** §10 · `alpaca_trading_client.dart` · `TradeActuator`
|
||||
|
||||
- [x] **Red:** Mock HTTP — `submitOrder(notional: 10, side: buy)` POSTs paper URL with `client_order_id`; `trade_orders` row `status=accepted`
|
||||
- [x] **Red:** Duplicate `client_order_id` → 422 → resolved via `getOrderByClientOrderId`
|
||||
- [x] **Green:** `server/lib/alpaca/alpaca_trading_client.dart`
|
||||
- [x] **Green:** `server/lib/trading/trade_actuator.dart` (test mode short-circuit + Alpaca path)
|
||||
- [x] **Confirm:** Tagged integration test (`server/test/alpaca/alpaca_trading_live_test.dart`) — paper roundtrip, skipped without credentials
|
||||
- [x] **Confirm:** `client_order_id` format `{firebase_uid}-{rule_id}-{question_id}` unique (set by `TradingPipeline.handleAnswer`, enforced by `trade_orders.client_order_id UNIQUE`)
|
||||
|
||||
---
|
||||
|
||||
### Step 10 — Worker integration
|
||||
|
||||
**Refs:** §7 · `question_background_worker.dart` · `trading_orchestrator.dart`
|
||||
|
||||
- [x] **Red:** `TRADING_ENABLED=true`, test mode (no Alpaca client) — one orchestrator tick runs evaluate + actuate against fixture snapshots
|
||||
- [x] **Green:** `server/lib/trading/trading_orchestrator.dart` (ingest → evaluate → actuate, per-stage failure isolation)
|
||||
- [x] **Green:** `QuestionBackgroundWorker` accepts optional `TradingOrchestrator`, runs it after `QuestionPipeline.runMaintenanceCycle`
|
||||
- [x] **Green:** `bin/server.dart` builds the trading stack only when `TRADING_ENABLED=true`; uses real Alpaca clients only when `QUESTION_PIPELINE_TEST_MODE=false` and credentials present
|
||||
- [x] **Confirm:** **Gate A** below — `test/integration/trading_orchestrator_gate_a_test.dart` green
|
||||
|
||||
---
|
||||
|
||||
### Step 11 — End-to-end with real Alpaca paper
|
||||
|
||||
**Refs:** §17 · §15 integration
|
||||
|
||||
- [ ] **Red:** Manual/scripted E2E checklist (or tagged automated test)
|
||||
- [ ] **Green:** Real keys in `.env`, `QUESTION_PIPELINE_TEST_MODE=false`
|
||||
- [ ] **Confirm:** **Gate B** (see below)
|
||||
|
||||
---
|
||||
|
||||
## Gate A — Local loop (test mode)
|
||||
|
||||
**Prerequisite:** Steps 0–10 complete.
|
||||
|
||||
Automated as `server/test/integration/trading_orchestrator_gate_a_test.dart`.
|
||||
|
||||
- [x] Seed user with `user_trading_config.enabled=true` and dip rule
|
||||
- [x] Insert fixture snapshots (or test-mode ingest)
|
||||
- [x] Run one worker tick → question queued
|
||||
- [x] Submit answer +10 via API
|
||||
- [x] Run next tick → `trade_orders` row (fake Alpaca id in test mode)
|
||||
- [x] Third tick → rule does not re-fire (cooldown)
|
||||
- [x] Set `TRADING_ENABLED=true` in dev `.env` only after Gate A passes (manual — flip the flag when ready to start using the live stack against paper Alpaca)
|
||||
|
||||
---
|
||||
|
||||
## Gate B — Paper Alpaca E2E
|
||||
|
||||
**Prerequisite:** Gate A complete.
|
||||
|
||||
- [ ] Ingest real SPY prices
|
||||
- [ ] Rule fires (threshold or market conditions)
|
||||
- [ ] Flutter: question via SignalR
|
||||
- [ ] User swipes +10
|
||||
- [ ] Order visible in Alpaca paper UI
|
||||
- [ ] `trade_orders.question_id` and `rule_id` populated
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Configuration API & hardening
|
||||
|
||||
**Start only after Gate B passes.** Maps to TRADING_DEVELOPMENT_PLAN §13, §14 Phase 2.
|
||||
|
||||
---
|
||||
|
||||
### Step 12 — Config validation
|
||||
|
||||
- [ ] **Red:** Validator rejects >30 symbols, missing `rules[].id`, invalid `mode`
|
||||
- [ ] **Green:** `TradingConfigValidator`
|
||||
- [ ] **Confirm:** Invalid JSON → 400 on PUT
|
||||
|
||||
---
|
||||
|
||||
### Step 13 — HTTP endpoints
|
||||
|
||||
- [ ] **Red:** Handler tests — `GET/PUT /v1/me/trading/config`
|
||||
- [ ] **Red:** `GET /v1/me/trading/orders`
|
||||
- [ ] **Red:** `GET /v1/me/trading/snapshots?symbol=SPY`
|
||||
- [ ] **Green:** `server/lib/handlers/trading_handler.dart`
|
||||
- [ ] **Confirm:** curl with Firebase token; sanitized config (no secrets)
|
||||
|
||||
---
|
||||
|
||||
### Step 14 — Daily guardrail counters
|
||||
|
||||
- [ ] **Red:** After order submit, `context.daily_order_count` increments; resets UTC midnight
|
||||
- [ ] **Green:** Counter in `user_trading_state.context`
|
||||
- [ ] **Confirm:** Third order same day blocked when `max_orders_per_day=3`
|
||||
|
||||
---
|
||||
|
||||
### Step 15 — Order status polling
|
||||
|
||||
- [ ] **Red:** Mock Alpaca status `filled` → `trade_orders.status` and `filled_at` updated
|
||||
- [ ] **Green:** Poll on worker tick or post-submit
|
||||
- [ ] **Confirm:** Failed order → follow-up question (optional)
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Streaming & observability
|
||||
|
||||
**Deferred until Phase 2 stable.** Maps to TRADING_DEVELOPMENT_PLAN §9.2, §14 Phase 3.
|
||||
|
||||
| Step | Red test | Green impl | Done |
|
||||
|------|----------|------------|------|
|
||||
| 16 WS ingest | Mock WS message → snapshot upsert | `alpaca_ws_ingest.dart` | ⬜ |
|
||||
| 17 Symbol cap | 31st symbol rejected | Config validator + WS manager | ⬜ |
|
||||
| 18 SignalR `ReceiveTradeUpdate` | Hub mock receives on fill | Optional Flutter listener | ⬜ |
|
||||
| 19 Metrics | Log ingest lag, rule fires, order latency | Structured logging | ⬜ |
|
||||
|
||||
---
|
||||
|
||||
## Test pyramid (target)
|
||||
|
||||
```text
|
||||
┌─────────────────┐
|
||||
│ Gate B (manual) │ 1 scenario / release
|
||||
├─────────────────┤
|
||||
│ Gate A (worker) │ 1 scripted integration
|
||||
├─────────────────┤
|
||||
│ HTTP handlers │ ~8 tests
|
||||
├─────────────────┤
|
||||
│ DB integration │ ~12 tests
|
||||
├─────────────────┤
|
||||
│ Unit (engine, │ ~40 tests
|
||||
│ clients, config)│
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Suggested schedule (first 2 weeks)
|
||||
|
||||
| Day | Steps | Deliverable |
|
||||
|-----|-------|-------------|
|
||||
| 1 | 0, 1.1–1.2 | Test harness + migration + snapshot DB |
|
||||
| 2 | 1.3–1.4, 2 | Config merge + env |
|
||||
| 3 | 3–4 | Alpaca MD client + ingest |
|
||||
| 4 | 5–6 | Rule engine + guardrails |
|
||||
| 5 | 7–8 | Questions from rules + answer branch |
|
||||
| 6 | 9 | Trade actuator |
|
||||
| 7 | 10 | Worker wire-up → Gate A |
|
||||
| 8 | 11 | Gate B with paper account |
|
||||
|
||||
---
|
||||
|
||||
## CI configuration
|
||||
|
||||
```yaml
|
||||
# Suggested CI jobs
|
||||
- dart test # unit + DB integration (no Alpaca)
|
||||
- dart test --tags=alpaca # optional; secrets required; nightly/manual
|
||||
```
|
||||
|
||||
**CI env:**
|
||||
|
||||
```bash
|
||||
TRADING_ENABLED=true
|
||||
QUESTION_PIPELINE_TEST_MODE=true
|
||||
DATABASE_URL=postgres://.../cyberhybridhub_test
|
||||
# No ALPACA_* keys in default CI job
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Safety tests (must pass before live)
|
||||
|
||||
These should **fail the build** if removed:
|
||||
|
||||
- [ ] `AlpacaEnv` refuses live URL when `ALPACA_ALLOW_LIVE=false`
|
||||
- [ ] Server-side max notional ceiling enforced regardless of config
|
||||
- [ ] Every order has non-null `question_id` when `require_question_before_order=true`
|
||||
- [ ] `client_order_id` UNIQUE prevents duplicate Alpaca POST
|
||||
|
||||
---
|
||||
|
||||
## Mapping to existing code
|
||||
|
||||
| New piece | Extends |
|
||||
|-----------|---------|
|
||||
| `TradingPipeline.evaluate` | `QuestionPipeline._canEnqueue` queue limits |
|
||||
| `onAnswerSubmitted` trading branch | geography/weather switch in `question_pipeline.dart` |
|
||||
| `BranchDecision.yesNo` | +10/-10 swipe (already used for weather) |
|
||||
| `ExternalDataFetcher` pattern | Alpaca clients with injectable `http.Client` |
|
||||
| `QUESTION_PIPELINE_TEST_MODE` | Skip Alpaca; fixture snapshots |
|
||||
|
||||
**Existing touchpoints:**
|
||||
|
||||
| Component | Path |
|
||||
|-----------|------|
|
||||
| Interval worker | `server/lib/workers/question_background_worker.dart` |
|
||||
| Question pipeline | `server/lib/pipeline/question_pipeline.dart` |
|
||||
| Branch decisions | `server/lib/pipeline/branch_decision.dart` |
|
||||
| External fetcher | `server/lib/pipeline/external_data_fetcher.dart` |
|
||||
| Env | `server/lib/env.dart` |
|
||||
| Flutter hub | `lib/services/questions_hub_service.dart` |
|
||||
| Flutter swipe UI | `lib/widgets/swipe_question_tile.dart` |
|
||||
|
||||
---
|
||||
|
||||
## MVP definition of done
|
||||
|
||||
- [ ] `dart test` in `server/` green (~40+ tests, no network in default job)
|
||||
- [ ] Gate A passes with test mode
|
||||
- [ ] Gate B passes on Alpaca paper
|
||||
- [ ] TRADING_DEVELOPMENT_PLAN §17 scenario reproducible from seed SQL
|
||||
- [ ] `TRADING_ENABLED=false` default; no secrets in Flutter
|
||||
- [ ] Every order traceable via `trade_orders.question_id` + `rule_id`
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [TRADING_DEVELOPMENT_PLAN.md](./TRADING_DEVELOPMENT_PLAN.md)
|
||||
- [server/README.md](./server/README.md)
|
||||
- [Alpaca Market Data API](https://docs.alpaca.markets/docs/market-data-api)
|
||||
- [Alpaca Trading API](https://docs.alpaca.markets/docs/trading-api)
|
||||
|
||||
---
|
||||
|
||||
*Document version: 1.0 — TDD progress tracker for Alpaca trading MVP.*
|
||||
@ -250,6 +250,8 @@ class QuestionsDb {
|
||||
SELECT source_tag, pipeline_key
|
||||
FROM questions
|
||||
WHERE assigned_user_id = @uid AND user_response IS NOT NULL
|
||||
ORDER BY modified_at DESC
|
||||
LIMIT 20
|
||||
''',
|
||||
),
|
||||
parameters: <String, dynamic>{'uid': assignedUserId},
|
||||
|
||||
11
server/query_prefs.dart
Normal file
11
server/query_prefs.dart
Normal file
@ -0,0 +1,11 @@
|
||||
import 'dart:io';
|
||||
import 'package:postgres/postgres.dart';
|
||||
|
||||
void main() async {
|
||||
final conn = await Connection.open(Endpoint(host: 'localhost', database: 'cyberhybridhub', username: 'postgres'));
|
||||
final result = await conn.execute('SELECT * FROM question_source_preferences');
|
||||
for (final row in result) {
|
||||
print(row);
|
||||
}
|
||||
await conn.close();
|
||||
}
|
||||
Loading…
x
Reference in New Issue
Block a user