# TODO — RTH session half bars (morning / afternoon aggregates) Companion to [`server/README.md`](./server/README.md) (Market history) and [`FLUTTER-ADMIN-PORTAL.md`](./FLUTTER-ADMIN-PORTAL.md). **Goal:** Replace six UTC **4-hour** Alpaca bars per day with **two regular-session aggregates per US trading day**, each built from up to **195 one-minute bars**: | Slot | US Eastern (NYSE regular) | Duration | |------|---------------------------|----------| | Morning | 9:30 AM – 12:45 PM | 3h 15m (195 min) | | Afternoon | 12:45 PM – 4:00 PM | 3h 15m (195 min) | Persist **one OHLCV row per symbol per slot** (not 195 rows). Use for `guess_weekly_move`, admin week coverage, and question audit. **TDD rhythm:** Red → Green → Refactor → Confirm (same as [`TODO.md`](./TODO.md) §0). --- ## 0. Design decisions (lock before coding) - [ ] **Timezone:** `America/New_York` for slot boundaries (handles DST); store canonical `as_of` / `raw.slot_start` as UTC instants of slot open. - [ ] **Stored `timeframe`:** new value, e.g. `sessionHalf` (do not overload `4Hour`). - [ ] **Alpaca fetch:** `GET /v2/stocks/bars` with `timeframe=1Min` per slot `[start, end]`; aggregate in server (`o`/`h`/`l`/`c`/`v` from minutes). - [ ] **Existing data:** delete or archive all `timeframe = '4Hour'` history rows after migration; full backfill required. - [ ] **`MIN_BARS_FOR_GUESS`:** revisit default (`5` bars ≈ 2.5 trading days at 2 slots/day vs ~20h span with 4h bars). - [ ] **Week coverage UI:** 2 dots per **trading day** (not 6 UTC dots). - [ ] **Question audit API (optional):** return `assetCount` only, drop `assets[]` payload to save bandwidth (Flutter already shows count-only). --- ## 1. Slot model (replace `MarketHistoryFourHourSlot`) **File:** replace or supersede `server/lib/trading/market_history_four_hour_slot.dart` → e.g. `market_history_session_slot.dart`. - [ ] **Red** — `server/test/trading/market_history_session_slot_test.dart`: - [ ] `slotStartContaining` maps instants to morning (9:30 ET) or afternoon (12:45 ET) slot start (UTC). - [ ] `endExclusive` / `endInclusive` for 195-minute windows. - [ ] `hasEnded` / `lastCompletedSlotStart` never returns in-progress slot. - [ ] `completedSlotStartsInWindow` yields 2 × trading days in rolling window; skips weekends + NYSE holidays (`MarketHistoryTradingCalendar`). - [ ] DST: assert 13:30 vs 14:30 UTC morning start across EDT/EST fixtures. - [ ] `wireUtc` / `slotStartWire` include minutes (`…T13:30:00Z`). - [ ] **Green** — implement slot module; `slotsPerDay = 2`, `slotDuration = Duration(hours: 3, minutes: 15)`. - [ ] **Refactor** — update imports project-wide; delete old four-hour module when unused. **Confirm:** `cd server && dart test test/trading/market_history_session_slot_test.dart` --- ## 2. Config & env **Files:** `market_history_config.dart`, `market_history_env.dart`, `env.dart`, `server/README.md`. - [ ] `barTimeframe` → `sessionHalf` (or chosen name). - [ ] Remove `slotHours = 4`; document `slotsPerDay = 2`. - [ ] Add `alpacaFetchTimeframe = '1Min'` (fetch only, not stored). - [ ] Document env vars; defaults for `MIN_BARS_FOR_GUESS` if changed. --- ## 3. Database migration `010_session_half_bars.sql` - [ ] **Red** — extend `market_history_schema_test.dart`: - [ ] `timeframe` CHECK allows `sessionHalf`. - [ ] Partial index on `(symbol, as_of DESC) WHERE metric='bar' AND timeframe='sessionHalf'`. - [ ] **Green** — migration: - [ ] `DELETE` (or archive) `metric='bar' AND timeframe='4Hour'`. - [ ] Update `market_data_snapshots_timeframe_check`. - [ ] `CREATE INDEX market_data_snapshots_bar_session_half_idx …`. - [ ] Apply in integration test harness (`001`–`010`). **Confirm:** `cd server && dart test test/integration/market_history_schema_test.dart` --- ## 4. Minute fetch + aggregation (backfill) **Files:** `market_data_history.dart`, `alpaca_market_data_client.dart` (unchanged API surface; caller passes `1Min`). - [ ] **Red** — `market_data_history_sync_test.dart`: - [ ] Mock `1Min` bars spanning 9:30–12:45 ET → one persisted `sessionHalf` row. - [ ] OHLCV aggregation rules: `o`=first, `h`=max, `l`=min, `c`=last, `v`=sum. - [ ] Pagination: merge pages via existing `getBarsRange` + `next_page_token`. - [ ] Wrong-window minutes rejected; empty minutes → placeholder or error per calendar rules. - [ ] Rate-limit / partial run behavior unchanged. - [ ] **Green** — `_fetchBarsWithRateLimitRetry` uses `1Min`; `_persistBars` aggregates then upserts one row per symbol; `raw.slot_start` + optional `raw.minute_bars_count`. - [ ] **Refactor** — extract `aggregateMinuteBars(List)` helper. **Confirm:** `cd server && dart test test/integration/market_data_history_sync_test.dart` --- ## 5. DB slot matching **File:** `market_data_db.dart` - [ ] Replace `_slotStartBucketSql` (4-hour UTC `div(hour,4)`) with session-slot equality on `raw.slot_start` wire or shared Dart/SQL slot function. - [ ] **Red** — `market_data_db_test.dart` for `symbolsWithBarForSlot` at 9:30 / 12:45 ET boundaries. --- ## 6. Read paths | File | Work | |------|------| | `market_history_query.dart` | Filter `timeframe = sessionHalf`; update comments. | | `market_history_question_audit.dart` | Step by **one session slot** (not ±4h); slot pair query. | | `market_history_week_coverage.dart` | 2 slots per trading day; `slotsPerDay: 2`. | | `market_history_trading_calendar.dart` | Trading-day helpers keyed on ET date of slot. | | `market_history_admin_logic.dart` | Error strings / slot labels. | | `backfill_sync_item.dart` | Wire format with minute-precision `slotStart`. | - [ ] **Red** — unit + integration tests for each area (see existing `*_test.dart` files under `server/test/`). - [ ] **Green** — implement. **Confirm:** `./scripts/test-server.sh` (no live Alpaca). --- ## 7. Flutter admin | File | Work | |------|------| | `lib/admin/utils/sync_run_formatters.dart` | `formatMarketHistorySlotWire` — no `hour ~/ 4`. | | `lib/admin/models/market_history_week_coverage.dart` | Default `slotsPerDay: 2`. | | `lib/admin/widgets/market_history_week_coverage_sheet.dart` | Copy: 2 slots/day. | | `lib/admin/widgets/market_history_question_audit_sheet.dart` | ET slot labels; count-only UI (done). | | `lib/admin/widgets/sync_run_expansion_tile.dart` | Backfill row: count only (done). | - [ ] **Red** — widget tests under `test/admin/`. - [ ] **Green** — implement remaining slot-label / coverage changes after server ships new slot times. **Confirm:** `./scripts/test-admin-portal.sh` --- ## 8. Fixtures & live tests - [ ] Add `server/test/fixtures/alpaca_bars_1min_session.json` (195 minutes × 1 symbol). - [ ] Update `alpaca_bars_4h_window.json` usages or remove. - [ ] `@Tags(['alpaca'])` live test: `1Min` range for one slot, aggregate locally. --- ## 9. Documentation - [ ] `server/README.md` — Market history section (2 session slots, `1Min` fetch). - [ ] `FLUTTER-ADMIN-PORTAL.md` — week coverage + question audit behavior. - [ ] Link from [`TODO.md`](./TODO.md) (legacy 4h work is complete; this doc supersedes granularity for Phase 3). --- ## 10. Deploy / ops - [ ] Run migration `010` on prod/staging (deletes legacy `4Hour` bar rows automatically). - [ ] **Or** manually clear history before backfill (if migration already applied without `010`): ```sql -- Required: remove old 4-hour bar rows (wrong shape for session-half logic) DELETE FROM market_data_snapshots WHERE metric = 'bar' AND timeframe = '4Hour'; -- Optional: archived 4Hour copies (if any) DELETE FROM market_data_archive WHERE metric = 'bar' AND timeframe = '4Hour'; -- Optional: force a clean sync audit trail (not required for backfill to run) TRUNCATE market_data_sync_runs; ``` **Do not truncate** `tradable_assets` — universe sync is independent. After clearing, run admin **Resync** or wait for the worker; `hasPendingSlots` will enqueue backfill for every missing `sessionHalf` slot in the rolling window. - [ ] Verify week-coverage calendar green for 2 slots × trading days. - [ ] Verify `guess_weekly_move` eligibility with new `MIN_BARS` threshold. --- ## 11. Progress log | Date | Step | Notes | |------|------|-------| | | | | --- ## Appendix — affected files (checklist) **Server (implement):** - `server/lib/trading/market_history_four_hour_slot.dart` → session slot module - `server/lib/trading/market_history_config.dart` - `server/lib/trading/market_data_history.dart` - `server/lib/trading/market_data_db.dart` - `server/lib/trading/market_history_query.dart` - `server/lib/trading/market_history_question_audit.dart` - `server/lib/trading/market_history_week_coverage.dart` - `server/lib/trading/market_history_trading_calendar.dart` - `server/lib/trading/backfill_sync_item.dart` - `server/lib/alpaca/alpaca_market_data_client.dart` (wire helper import only) - `server/migrations/010_session_half_bars.sql` (new) **Server tests:** - `server/test/trading/market_history_four_hour_slot_test.dart` → session tests - `server/test/trading/market_history_week_coverage_test.dart` - `server/test/trading/market_history_question_audit_test.dart` - `server/test/integration/market_data_history_sync_test.dart` - `server/test/integration/market_data_db_test.dart` - `server/test/integration/market_history_admin_handler_test.dart` - `server/test/integration/market_history_week_coverage_test.dart` - (+ admin logic, schema, scheduler tests as needed) **Flutter:** - `lib/admin/widgets/market_history_question_audit_sheet.dart` - `lib/admin/widgets/sync_run_expansion_tile.dart` - `lib/admin/widgets/market_history_week_coverage_sheet.dart` - `lib/admin/utils/sync_run_formatters.dart` - `lib/admin/models/market_history_week_coverage.dart` - `test/admin/widgets/market_history_question_audit_sheet_test.dart` - `test/admin/widgets/sync_run_expansion_tile_test.dart` **Unrelated (do not change for slot work):** - `server/lib/trading/guardrails.dart` (4h **notional** window for orders) - `server/lib/trading/market_data_ingest.dart` (daily bars for live ingest)