cyberhybridhub/TODO.md

TODO — Rolling 7-Day Market Data Window + Cleanup

Next milestone: ET morning/afternoon session half bars (195-minute aggregates) — see TODO-SESSION-HALF-BARS.md.

Companion to TRADING_DEVELOPMENT_PLAN.md and TRADING_TDD_PLAN.md.

Goal: maintain a rolling 7-day history of market data for all active tradable assets so the question pipeline can generate obfuscated guessing-game questions about market movement, while pruning (or archiving) anything older than the window.

TDD rhythm (mandatory for every step):

1. Red — write the failing test(s) first; commit if you like.
2. Green — minimum implementation that turns every test in this step green.
3. Refactor — tidy without changing behavior; rerun tests.
4. Confirm — run the full step-level confirm command listed in the step.
5. Log — check the box, add a row to §12 Progress log.

Do not skip the Red phase. Do not start the next step while any test in the current step is failing or pending. No live Alpaca calls in default dart test jobs — guard with @Tags(['alpaca']).

---

0. Scope & design constraints

• Window: rolling 7 calendar days, UTC. Configurable via MARKET_HISTORY_WINDOW_DAYS (default 7).
• Granularity (Phase 1): 1Day bars for every active tradable, plus the existing last_trade / prev_close snapshots for watchlist symbols.
• Granularity (Phase 2): 1Hour bars for the union of all enabled users' watchlist symbols (≤30 on Alpaca Basic).
• Universe source of truth: Alpaca /v2/assets?status=active&tradable=true, refreshed daily, cached in Postgres (tradable_assets).
• Idempotency: repeated backfill of the same (symbol, metric, timeframe, as_of) MUST NOT create duplicate rows.
• Cleanup vs. archive: rows with as_of < now() - window are either hard-deleted (Phase 1) or moved to market_data_archive (Phase 2).
• Worker isolation: historical sync + cleanup run on their own cadence (default once per day), not on every 60s per-user tick.
• Rate-limit safety: batch symbols (Alpaca bars accepts multi-symbol); cap concurrent symbols; never call Alpaca in tests (QUESTION_PIPELINE_TEST_MODE=true).
• No Flutter changes required for this milestone.

---

1. Schema additions (migration 005_market_history.sql)

1.1 Red — failing tests first

• Create server/test/integration/market_history_schema_test.dart:
  • Test: INSERT two snapshots with the same (symbol, metric, timeframe, as_of) → second one upserts, does not duplicate (current schema lacks the unique constraint, so this MUST fail Red).
  • Test: timeframe defaults to 'tick' for existing rows; new rows accept '1Min' | '1Hour' | '1Day'.
  • Test: tradable_assets PK rejects duplicate symbol; query by (status='active', tradable=true) uses the new index (verify via EXPLAIN returning Index Scan).
  • Test: market_data_sync_runs records kind, started_at, finished_at, rows_written, rows_removed, error shape.

1.2 Green — minimum migration

• Write server/migrations/005_market_history.sql:
  • ALTER TABLE market_data_snapshots ADD COLUMN timeframe TEXT NOT NULL DEFAULT 'tick'.
  • ALTER TABLE market_data_snapshots ADD CONSTRAINT market_data_snapshots_unique_obs UNIQUE (symbol, metric, timeframe, as_of).
  • CREATE INDEX market_data_snapshots_asof_idx ON market_data_snapshots (as_of DESC).
  • CREATE TABLE tradable_assets (…) with columns symbol PK, asset_class, exchange, name, tradable, fractionable, status, raw JSONB, refreshed_at.
  • CREATE INDEX tradable_assets_status_idx ON tradable_assets (status, tradable).
  • CREATE TABLE market_data_sync_runs (…) (see §0 plan).
  • (Phase 2 stub, commented) CREATE TABLE market_data_archive (…) — deferred to §4.2 (the migration runner splits on ;, which would slice a commented stub mid-block; the archive table will be added in §4.2.2 when it is actually wired up).

1.3 Refactor

• Confirm MarketDataDb._rowToSnapshot still reads correctly with the new column (read-side back-compat — no test changes needed, just verify existing market_data_db_test.dart still passes).
• Move shared SQL fragments into the migration runner if duplication appeared. (none observed in 005; nothing to extract yet.)

1.4 Confirm

• cd server && dart test test/integration/migration_test.dart test/integration/market_history_schema_test.dart — green.
• psql cyberhybridhub_test -c '\d market_data_snapshots' shows the unique constraint and new column.

---

2. Tradable-asset universe sync

Files (new): server/lib/alpaca/alpaca_assets_client.dart, server/lib/trading/tradable_assets_db.dart, server/lib/trading/tradable_assets_sync.dart.

2.1 Alpaca assets client

2.1.1 Red

• Add fixture server/test/fixtures/alpaca_assets_active.json (≥5 representative assets, mix of tradable=true/false and fractionable=true/false).
• Add server/test/alpaca/alpaca_assets_client_test.dart:
  • Test: listActiveTradable() issues GET to ${tradingBaseUrl}/v2/assets?status=active&asset_class=us_equity with APCA-API-KEY-ID + APCA-API-SECRET-KEY headers.
  • Test: parses fixture into List<AlpacaAsset> — verifies symbol, exchange, fractionable, tradable, status fields.
  • Test: 401 / 500 → throws AlpacaAssetsException with status code and body in the message.
  • Test: empty response array → returns [], does not throw.

2.1.2 Green

• Add AlpacaAsset model in server/lib/alpaca/alpaca_models.dart.
• Implement AlpacaAssetsClient with injectable http.Client (mirror AlpacaMarketDataClient shape).
• Add AlpacaAssetsException.

2.1.3 Refactor

• Extract a private _authHeaders helper if duplicated across Alpaca clients (DRY — but only if you actually duplicate). (Lifted to AlpacaEnv.authHeaders; now reused by all three Alpaca clients.)

2.1.4 Confirm

• dart test test/alpaca/alpaca_assets_client_test.dart — green.
• Tagged live test server/test/alpaca/alpaca_assets_live_test.dart (@Tags(['alpaca'])) — returns >100 symbols when keys present; skipped otherwise. Run manually: dart test --tags=alpaca test/alpaca/alpaca_assets_live_test.dart.

2.2 Universe persistence + diff

2.2.1 Red

• Create server/test/integration/tradable_assets_db_test.dart:
  • Test: upsertAll([A, B, C]) inserts 3 rows.
  • Test: re-running upsertAll([B*, C, D]) updates B, leaves C unchanged-by-content but refreshed_at bumped, inserts D, and marks A as tradable=false, status='inactive' (we never delete history).
  • Test: listActiveTradableSymbols() returns only tradable=true AND status='active'.

2.2.2 Red — sync orchestration

• Create server/test/integration/tradable_assets_sync_test.dart:
  • Test: TradableAssetsSync.runOnce() with mocked client returning the fixture → DB rows match; one row in market_data_sync_runs with kind='universe' and non-null finished_at.
  • Test: client throws → sync run row recorded with error populated, finished_at non-null, and rows_written = 0.
  • Test: two consecutive runs are safe (idempotent counts).

2.2.3 Green

• Implement TradableAssetsDb.upsertAll, TradableAssetsDb.listActiveTradableSymbols.
• Implement TradableAssetsSync.runOnce() that writes a market_data_sync_runs row around the upsert.

2.2.4 Refactor

• Pull "wrap a closure with a sync_runs audit row" into a small helper (SyncRunRecorder.record(kind, body)); reuse in §3 and §4. (Landed at server/lib/trading/sync_run_recorder.dart; TradableAssetsSync already consumes it.)

2.2.5 Confirm

• dart test test/integration/tradable_assets_db_test.dart test/integration/tradable_assets_sync_test.dart — green.

---

3. Historical backfill (1Day bars × 7 days)

Files: extend server/lib/alpaca/alpaca_market_data_client.dart, new server/lib/trading/market_data_history.dart, extend server/lib/trading/market_data_db.dart.

3.1 Alpaca client — time-range bars with pagination

3.1.1 Red

• Add fixtures:
  • server/test/fixtures/alpaca_bars_7d_multi_page1.json — includes next_page_token.
  • server/test/fixtures/alpaca_bars_7d_multi_page2.json — final page, next_page_token: null.
• Extend server/test/alpaca/alpaca_market_data_client_test.dart:
  • Test: getBarsRange(['SPY','AAPL'], timeframe: '1Day', start, end) builds correct query string (start, end, timeframe, feed, symbols, limit).
  • Test: follows pagination — when page1 returns next_page_token='abc', client issues second request with page_token=abc; merges both pages' bars per symbol.
  • Test: stops after a configurable maxPages (default 20) to prevent runaway loops.
  • Test: 429 → throws AlpacaMarketDataException containing the word rate (so caller can detect & back off).

3.1.2 Green

• Implement Future<AlpacaBarsResponse> getBarsRange({ List<String> symbols, String timeframe, DateTime start, DateTime end, int maxPages = 20}) on AlpacaMarketDataClient.
• Extend AlpacaBarsResponse with a merge(AlpacaBarsResponse other) method so paginated chunks combine cleanly.

3.1.3 Refactor

• If the pagination loop is non-trivial, extract a private _paginate<T>(initialUri, parsePage) generic to reuse later for orders/positions endpoints. (Loop kept inline in getBarsRange — ~25 lines, clear enough; extract when a second consumer appears.)

3.1.4 Confirm

• dart test test/alpaca/alpaca_market_data_client_test.dart — green.
• Tagged live test server/test/alpaca/alpaca_market_data_history_live_test.dart fetches 7-day bars for SPY and asserts ≥3 bars.

3.2 MarketDataDb — idempotent upsert + range query

3.2.1 Red

• Extend server/test/integration/market_data_db_test.dart:
  • Test: upsertSnapshot(symbol:'SPY', metric:'bar', timeframe:'1Day', as_of:T, price:500) then re-upsert with price:505 → exactly one row remains; price is 505; raw is overwritten (volume also overwritten).
  • Test: barsForSymbol(symbol, timeframe, since, until) returns rows ordered by as_of ASC; range is inclusive of since, exclusive of until.
  • Test: barsForSymbol returns [] when no rows match; does not throw.
  • Test: latestSyncedAsOf(symbol, timeframe) returns the newest as_of or null.

3.2.2 Green

• Implement MarketDataDb.upsertSnapshot(...) using ON CONFLICT (symbol, metric, timeframe, as_of) DO UPDATE SET price = EXCLUDED.price, volume = EXCLUDED.volume, raw = EXCLUDED.raw.
• Implement MarketDataDb.barsForSymbol(...) and MarketDataDb.latestSyncedAsOf(...).

3.2.3 Refactor

• Replace existing insertSnapshot call sites in market_data_ingest.dart with upsertSnapshot (tick data has timeframe='tick'; same call shape). Re-run test/integration/market_data_ingest_test.dart — still green.

3.2.4 Confirm

• dart test test/integration/market_data_db_test.dart test/integration/market_data_ingest_test.dart — green.

3.3 MarketDataHistorySync

3.3.1 Red

• Add fixture server/test/fixtures/alpaca_bars_7d_3symbols.json — 7 bars × 3 symbols (SPY/AAPL/MSFT), realistic timestamps.
• Add server/test/integration/market_data_history_sync_test.dart:
  • Test: with mocked Alpaca returning the fixture → 21 rows upserted with metric='bar', timeframe='1Day'; sync run row written.
  • Test: re-running with the same fixture → still 21 rows; zero duplicates; rows_written reflects rows touched (not inserted).
  • Test: partial outage — Alpaca returns 200 for batch 1 (AAPL/MSFT), 500 for batch 2 (SPY) → AAPL/MSFT rows persisted; sync run row has error mentioning SPY; method does NOT throw.
  • Test: respects HISTORY_SYNC_MAX_SYMBOLS cap (set to 2 → only first 2 symbols fetched).
  • Test: batching — with HISTORY_SYNC_BATCH_SIZE=2 and 5 symbols, Alpaca is called 3 times (mock call counter).

3.3.2 Green

• Implement MarketDataHistorySync.runOnce({int windowDays = 7}):
  • Reads symbols from TradableAssetsDb.listActiveTradableSymbols().
  • Batches into HISTORY_SYNC_BATCH_SIZE groups; calls getBarsRange per batch.
  • Upserts via MarketDataDb.upsertSnapshot.
  • Captures per-batch errors without aborting; aggregates them into the sync run row (SyncRunCounts.error).

3.3.3 Refactor

• Extract batching helper if used by §3.4 incremental path too. (Landed as chunkList in market_data_history.dart.)

3.3.4 Confirm

• dart test test/integration/market_data_history_sync_test.dart — green.

3.4 Incremental daily catch-up

3.4.1 Red

• Extend market_data_history_sync_test.dart:
  • Test: with prior latestSyncedAsOf(symbol) = T-2d, sync issues bars with start = T-2d (not T-7d); mock HTTP call records the requested start.
  • Test: with prior sync T-10d (outside window), start is clamped to T-windowDays.
  • Test: cold start (no prior sync) → start = T-windowDays.

3.4.2 Green

• Compute per-symbol start using latestSyncedAsOf; pass to getBarsRange.

3.4.3 Refactor

• If per-symbol starts vary inside a batch, fall back to min(starts) for the batched call and let upsertSnapshot dedupe the overlap — document the tradeoff in a code comment.

3.4.4 Confirm

• dart test test/integration/market_data_history_sync_test.dart — green.

---

4. Retention & cleanup (older than 7 days)

Files (new): server/lib/trading/market_data_retention.dart.

4.1 Hard-delete (Phase 1)

4.1.1 Red

• Create server/test/integration/market_data_retention_test.dart:
  • Test: seed 10 snapshots spanning 14 days → runCleanup({windowDays: 7}) deletes rows with as_of < now() - 7d, keeps the rest; returns rowsRemoved matching deleted count.
  • Test: empty table → returns rowsRemoved = 0, does not throw.
  • Test: batchSize honored — with 5000 rows older than window and batchSize=1000, the underlying DELETE is issued ≥5 times (use a counting wrapper around _connection.execute).
  • Test: each invocation appends a market_data_sync_runs row with kind='cleanup', rows_removed populated.
  • Test: rows within window are NEVER touched (assert specific IDs survive).

4.1.2 Green

• Implement MarketDataRetention.runCleanup({int windowDays = 7, int batchSize = 5000}):
  • Loop: DELETE FROM market_data_snapshots WHERE as_of < $cutoff LIMIT $batchSize (use CTE if Postgres version requires it), return rows removed; repeat until 0.
  • Write a market_data_sync_runs row around the operation.

4.1.3 Refactor

• Reuse SyncRunRecorder from §2.2.4.

4.1.4 Confirm

• dart test test/integration/market_data_retention_test.dart — green.

4.2 Archive (Phase 2 — opt-in)

4.2.1 Red

• Extend market_data_retention_test.dart:
  • Test: with archiveEnabled: true, expired rows are copied into market_data_archive with archived_at = now() BEFORE being deleted; archive count grows by exactly rowsRemoved.
  • Test: archive run is transactional — if archive INSERT fails, no DELETE happens; sync run row records the error.
  • Test: archiveEnabled: false (default) → archive table untouched.

4.2.2 Green

• Uncomment market_data_archive table in migration 005 (or add it now if you deferred it). (Added 006_market_data_archive.sql.)
• Implement MarketDataRetention.runArchiveAndCleanup({int windowDays}) with explicit BEGIN; INSERT … SELECT …; DELETE …; COMMIT.

4.2.3 Refactor

• Consider a single unified entry point MarketDataRetention.run({int windowDays, bool archive}) that dispatches; only do this if it doesn't muddy the failure-isolation story.

4.2.4 Confirm

• dart test test/integration/market_data_retention_test.dart — green.

---

5. Scheduler — daily cadence inside the worker

Files: new server/lib/workers/market_history_scheduler.dart, extend server/lib/workers/question_background_worker.dart, extend server/bin/server.dart.

5.1 Red

• Add server/test/integration/market_history_scheduler_test.dart:
  • Test: cold start → runIfDue(now=T0) runs all 3 stages (universe, backfill, cleanup) in that order; market_data_sync_runs has 3 rows.
  • Test: same-day re-run (now=T0 + 1h) → no stages run; zero new sync rows.
  • Test: next day (now=T0 + 24h) → all 3 stages run again.
  • Test: per-stage cadence — set MARKET_UNIVERSE_REFRESH_HOURS=48, MARKET_HISTORY_SYNC_HOURS=24, MARKET_HISTORY_CLEANUP_HOURS=24; at T0+24h only backfill + cleanup run.
  • Test: failure isolation — backfill throws → cleanup still runs; both stages logged in market_data_sync_runs (one with error, one without).
  • Test: MARKET_HISTORY_SYNC_HOUR_UTC=10 (optional alignment) → scheduler runs only when local UTC hour ≥ 10 AND last run was on a prior UTC day.
• Add server/test/integration/market_history_worker_wireup_test.dart:
  • Test: QuestionBackgroundWorker._tick invokes MarketHistoryScheduler.runIfDue before the TradingOrchestrator per-user loop. Use a spy scheduler that records the call order.
  • Test: scheduler exception is caught — worker tick continues into the orchestrator loop; stderr contains the error.

5.2 Green

• Implement MarketHistoryScheduler with runIfDue(DateTime now), reading the last finished_at per kind from market_data_sync_runs.
• Wire QuestionBackgroundWorker to accept an optional MarketHistoryScheduler and call it at the top of _tick.
• Wire bin/server.dart to construct the scheduler only when MARKET_HISTORY_SYNC_ENABLED=true && TRADING_ENABLED=true.

5.3 Refactor

• If the three stages each need similar before/after logic, abstract a small _runStage(kind, body) inside the scheduler. (_maybeRunStage — no further refactor needed.)

5.4 Confirm

• dart test test/integration/market_history_scheduler_test.dart test/integration/market_history_worker_wireup_test.dart — green.

---

6. Question pipeline — "guess the move" rule

Files: extend server/lib/trading/rule_engine.dart, extend server/lib/trading/trading_pipeline.dart, new server/lib/trading/market_history_query.dart.

The guessing game uses the rolling 7-day window — questions must reveal just enough for the user to guess (obfuscated symbol/price/direction). No trade is placed for this rule — answers feed scoring only.

6.1 Red — MarketHistoryQuery

• Add server/test/integration/market_history_query_test.dart:
  • Test: weeklyMovers({minBars: 5, asOf}) returns only symbols with ≥5 daily bars in the window; each entry exposes (symbol, openClose, currentClose, days).
  • Test: deterministic — supply a random: Random(42) and assert a stable selection order across runs.
  • Test: symbols with stale data (newest bar > 2d old) are excluded.

6.2 Red — rule engine extension

• Add server/test/trading/rule_engine_guess_weekly_move_test.dart:
  • Test: rule kind guess_weekly_move with mocked MarketHistoryQuery returning SPY {ref=500, current=510, days=5} → produces a RuleEvaluation with: - obfuscated symbol_token='ASSET_A', - correct_answer = 10 (up direction), - question_text substituting {{token}}, {{ref_price}}, {{ref_days_ago}}, NEVER {{symbol}}.
  • Test: down move (ref=510, current=500) → correct_answer = -10.
  • Test: insufficient bars → no fire.
  • Test: questions.metadata.guess_symbol is set to real symbol (server-side only) when the question is created in §6.3.

6.3 Red — pipeline wiring

• Add server/test/integration/trading_pipeline_guess_weekly_move_test.dart:
  • Test: end-to-end with seeded 7 daily bars for SPY → pipeline creates a question with obfuscated text; metadata.guess_symbol = 'SPY'; pipeline_key='trading', pipeline_step='guess_weekly_move:await_answer'.
  • Test: onAnswerSubmitted with matching direction (e.g., +10 on an up move) records score_delta = +1 in user_trading_state.context.guess_score; non-matching records score_delta = -1.
  • Test: TradeActuator.processPendingOrders is NEVER called for guess_weekly_move answers (assert via spy).
  • Test: cooldown — after a fire, the same symbol is not re-picked for GUESS_COOLDOWN_HOURS (default 24).

6.4 Green

• Implement MarketHistoryQuery.weeklyMovers({...}).
• Add rule kind to RuleEngine with the new template tokens.
• Extend TradingPipeline.evaluate + onAnswerSubmitted for the new rule kind, including the cooldown bookkeeping.

6.5 Refactor

• If the token mapping (real symbol ↔ ASSET_A/ASSET_B/…) is used in multiple places, lift it into a SymbolObfuscator helper with its own focused unit test.

6.6 Confirm

• dart test test/integration/market_history_query_test.dart test/trading/rule_engine_guess_weekly_move_test.dart test/integration/trading_pipeline_guess_weekly_move_test.dart — green.

---

7. Env additions (server/.env.example)

# Rolling history feature gate
MARKET_HISTORY_SYNC_ENABLED=false
MARKET_HISTORY_WINDOW_DAYS=7
MARKET_HISTORY_RETENTION_DAYS=7
MARKET_HISTORY_ARCHIVE_ENABLED=false

# Cadence (hours)
MARKET_UNIVERSE_REFRESH_HOURS=24
MARKET_HISTORY_SYNC_HOURS=24
MARKET_HISTORY_CLEANUP_HOURS=24
MARKET_HISTORY_SYNC_HOUR_UTC=10   # optional alignment hour

# Batching / safety
HISTORY_SYNC_BATCH_SIZE=50
HISTORY_SYNC_MAX_SYMBOLS=2000     # hard cap; Alpaca Basic-friendly
MIN_BARS_FOR_GUESS=5
GUESS_COOLDOWN_HOURS=24

7.1 Red

• Add server/test/env/market_history_env_test.dart:
  • Test: defaults parsed when env empty (enabled=false, windowDays=7, etc.).
  • Test: MARKET_HISTORY_SYNC_ENABLED=true while TRADING_ENABLED=false → Env.assertConsistent() throws.
  • Test: MARKET_HISTORY_WINDOW_DAYS=0 or negative → throws.
  • Test: MARKET_HISTORY_SYNC_HOUR_UTC=24 → throws (valid range 0..23).

7.2 Green

• Extend server/lib/env.dart to load and validate these vars.
• Append the block above to server/.env.example.

7.3 Refactor

• If Env has grown unwieldy, split market-history vars into MarketHistoryEnv (typed value object) and have Env expose it.

7.4 Confirm

• dart test test/env/market_history_env_test.dart — green.
• Document each var in server/README.md under a new "Market history window" subsection.

---

8. Operational tooling

8.1 Red

• Add server/test/bin/sync_market_history_smoke_test.dart:
  • Test: imports bin/sync_market_history.dart main function and runs it with QUESTION_PIPELINE_TEST_MODE=true + a fake DB → exits 0 and emits the expected one-line log.
• Add equivalent smoke test for bin/cleanup_market_history.dart.

8.2 Green

• Add CLI server/bin/sync_market_history.dart with --window=<days> flag (default 7); honors test mode.
• Add CLI server/bin/cleanup_market_history.dart with --window=<days> and --archive flags.
• Add structured one-line log: kind=… symbols=… rows_written=… rows_removed=… duration_ms=… error=….

8.3 Refactor

• Share argument parsing between the two CLIs if duplicated.

8.4 Confirm

• dart test test/bin/ — green.
• Manual: dart run server:bin/sync_market_history.dart --window=7 against cyberhybridhub_test works end-to-end.

8.5 Optional admin endpoint (defer until needed)

• Behind Firebase admin auth, POST /v1/admin/market-data/resync?window=7 enqueues a sync run; not exposed to Flutter.

---

9. Test pyramid for this milestone

Layer	Test files
Unit	test/alpaca/alpaca_assets_client_test.dart
Unit	test/alpaca/alpaca_market_data_client_test.dart (extended)
Unit	test/trading/rule_engine_guess_weekly_move_test.dart
Unit	test/env/market_history_env_test.dart
DB integration	test/integration/market_history_schema_test.dart
DB integration	test/integration/tradable_assets_db_test.dart
DB integration	test/integration/tradable_assets_sync_test.dart
DB integration	test/integration/market_data_db_test.dart (extended)
DB integration	test/integration/market_data_history_sync_test.dart
DB integration	test/integration/market_data_retention_test.dart
DB integration	test/integration/market_history_query_test.dart
DB integration	test/integration/trading_pipeline_guess_weekly_move_test.dart
Worker integration	test/integration/market_history_scheduler_test.dart
Worker integration	test/integration/market_history_worker_wireup_test.dart
Bin smoke	test/bin/sync_market_history_smoke_test.dart
Bin smoke	test/bin/cleanup_market_history_smoke_test.dart
Tagged (alpaca)	test/alpaca/alpaca_assets_live_test.dart
Tagged (alpaca)	test/alpaca/alpaca_market_data_history_live_test.dart

CI gating:

# default job — no Alpaca keys, must pass on every PR
cd server && dart test

# nightly / manual — requires ALPACA_API_KEY_ID / ALPACA_API_SECRET_KEY
cd server && dart test --tags=alpaca

---

10. Acceptance criteria (Gate H — Rolling history)

• market_data_snapshots contains rows for every active tradable with as_of within the last 7 days, and no rows older.
• Re-running backfill is a no-op (zero duplicate rows; deterministic rows_written count when nothing changed upstream).
• Cleanup removes only rows older than the window and never touches newer rows.
• Worker performs one full cycle (universe → backfill → cleanup) per day with stage isolation; failure in one stage does not block the others.
• A guess_weekly_move question can be generated end-to-end from pure DB data — no live Alpaca call at evaluation time.
• dart test is green; dart test --tags=alpaca is green when keys are present.
• MARKET_HISTORY_SYNC_ENABLED=false is the default; nothing runs unless explicitly enabled.
• Safety: MARKET_HISTORY_SYNC_ENABLED=true without TRADING_ENABLED=true fails fast at server boot.

---

11. Risks & mitigations

Risk	Mitigation
Alpaca rate limits on full-universe pull	Batched bars calls (HISTORY_SYNC_BATCH_SIZE); per-batch error isolation; 429 → exception logged in sync run, retry next day.
Migration deadlocks on large market_data_snapshots	Cleanup batches via LIMIT + loop; unique constraint added with NOT VALID then VALIDATE CONSTRAINT if existing dataset is huge (document in migration).
Duplicate Alpaca asset entries between runs	upsertAll PK-on-symbol; we mark missing symbols inactive instead of deleting.
Guessing game leaks the real symbol	Question text uses tokens only; real symbol lives in questions.metadata (server side); add a regex test that scans question_text for any known ticker.
Backfill blowing past disk budget	Hard caps via HISTORY_SYNC_MAX_SYMBOLS and MARKET_HISTORY_WINDOW_DAYS; retention deletes daily so steady-state size is bounded.

---

12. Progress log

Date	Step	Result
2026-05-26	§7 Env additions	Green: 6/6 env tests; dart test 133/133. MarketHistoryEnv.fromMap + assertConsistent; ServerEnv.marketHistory; wired scheduler/sync/retention/guess; server/.env.example; README Market history window.
2026-05-26	§6 Guess-the-move rule	Green: 12 new tests; dart test 127/127. MarketHistoryQuery.weeklyMovers; RuleEngine.evaluateGuessWeeklyMove; SymbolObfuscator; TradingPipeline scoring + per-symbol cooldown; questions.metadata migration 007; no pending orders on guess answers.
2026-05-26	§5 Scheduler (worker cadence)	Green: 8/8 scheduler + wireup tests; dart test 115/115. MarketHistoryScheduler.runIfDue (per-kind cadence + optional syncHourUtc); worker calls scheduler before pipeline/trading; server.dart wires universe→backfill→cleanup when MARKET_HISTORY_SYNC_ENABLED + real Alpaca; ServerEnv.marketHistorySyncEnabled; SyncRunRecorder uses injected now for finished_at.
2026-05-26	§4 Retention & cleanup	Green: 8/8 retention tests; dart test 107/107. MarketDataRetention.runCleanup (batched hard-delete via CTE+RETURNING); runArchiveAndCleanup (transactional archive-then-delete); unified run(archive:); migration 006_market_data_archive.sql; reuses SyncRunRecorder kind=cleanup.
2026-05-26	§3 Historical backfill (1Day × 7d)	Green: 17 new tests (6 client + 4 db + 8 sync); dart test 99/99; live alpaca_market_data_history_live_test ≥3 SPY bars. getBarsRange + pagination; upsertSnapshot/barsForSymbol/latestSyncedAsOf; MarketDataHistorySync with incremental catch-up + partial batch errors via SyncRunCounts.error. Defaults in MarketHistoryConfig (batch=100).
2026-05-26	§2 Tradable-asset universe sync	Green: 11/11 §2 tests pass (5 client + 3 db + 3 sync); dart test 82/82 green; tagged live alpaca_assets_live_test returned >100 active us_equity assets. Refactor 2.1.3 lifted auth headers to AlpacaEnv.authHeaders; 2.2.4 lifted SyncRunRecorder for §3/§4 reuse.
2026-05-26	§1 Schema additions (migration 005_market_history.sql)	Green: 5/5 schema tests pass; dart test 70/70 green; \d market_data_snapshots shows timeframe col + market_data_snapshots_unique_obs unique constraint. Archive stub deferred to §4.2 to keep ;-split migration runner happy.

---

13. References

• Existing snapshot writer: server/lib/trading/market_data_ingest.dart
• Existing snapshot DB: server/lib/trading/market_data_db.dart
• Existing migration to extend: server/migrations/004_trading.sql
• Orchestrator hook point: server/lib/trading/trading_orchestrator.dart
• Worker hook point: server/lib/workers/question_background_worker.dart
• Plans: TRADING_DEVELOPMENT_PLAN.md, TRADING_TDD_PLAN.md
• Alpaca docs: Market Data, Trading / Assets, Bars

---

Document version: 1.0 — Rolling 7-day market data window, cleanup, and guessing-game question integration.