253 lines
10 KiB
Markdown
253 lines
10 KiB
Markdown
# TODO — RTH session half bars (morning / afternoon aggregates)
|
||
|
||
Companion to [`server/README.md`](./server/README.md) (Market history) and
|
||
[`FLUTTER-ADMIN-PORTAL.md`](./FLUTTER-ADMIN-PORTAL.md).
|
||
|
||
**Goal:** Replace six UTC **4-hour** Alpaca bars per day with **two regular-session
|
||
aggregates per US trading day**, each built from up to **195 one-minute bars**:
|
||
|
||
| Slot | US Eastern (NYSE regular) | Duration |
|
||
|------|---------------------------|----------|
|
||
| Morning | 9:30 AM – 12:45 PM | 3h 15m (195 min) |
|
||
| Afternoon | 12:45 PM – 4:00 PM | 3h 15m (195 min) |
|
||
|
||
Persist **one OHLCV row per symbol per slot** (not 195 rows). Use for
|
||
`guess_weekly_move`, admin week coverage, and question audit.
|
||
|
||
**TDD rhythm:** Red → Green → Refactor → Confirm (same as
|
||
[`TODO.md`](./TODO.md) §0).
|
||
|
||
---
|
||
|
||
## 0. Design decisions (lock before coding)
|
||
|
||
- [ ] **Timezone:** `America/New_York` for slot boundaries (handles DST); store
|
||
canonical `as_of` / `raw.slot_start` as UTC instants of slot open.
|
||
- [ ] **Stored `timeframe`:** new value, e.g. `sessionHalf` (do not overload
|
||
`4Hour`).
|
||
- [ ] **Alpaca fetch:** `GET /v2/stocks/bars` with `timeframe=1Min` per slot
|
||
`[start, end]`; aggregate in server (`o`/`h`/`l`/`c`/`v` from minutes).
|
||
- [ ] **Existing data:** delete or archive all `timeframe = '4Hour'` history rows
|
||
after migration; full backfill required.
|
||
- [ ] **`MIN_BARS_FOR_GUESS`:** revisit default (`5` bars ≈ 2.5 trading days at
|
||
2 slots/day vs ~20h span with 4h bars).
|
||
- [ ] **Week coverage UI:** 2 dots per **trading day** (not 6 UTC dots).
|
||
- [ ] **Question audit API (optional):** return `assetCount` only, drop `assets[]`
|
||
payload to save bandwidth (Flutter already shows count-only).
|
||
|
||
---
|
||
|
||
## 1. Slot model (replace `MarketHistoryFourHourSlot`)
|
||
|
||
**File:** replace or supersede `server/lib/trading/market_history_four_hour_slot.dart`
|
||
→ e.g. `market_history_session_slot.dart`.
|
||
|
||
- [ ] **Red** — `server/test/trading/market_history_session_slot_test.dart`:
|
||
- [ ] `slotStartContaining` maps instants to morning (9:30 ET) or afternoon
|
||
(12:45 ET) slot start (UTC).
|
||
- [ ] `endExclusive` / `endInclusive` for 195-minute windows.
|
||
- [ ] `hasEnded` / `lastCompletedSlotStart` never returns in-progress slot.
|
||
- [ ] `completedSlotStartsInWindow` yields 2 × trading days in rolling window;
|
||
skips weekends + NYSE holidays (`MarketHistoryTradingCalendar`).
|
||
- [ ] DST: assert 13:30 vs 14:30 UTC morning start across EDT/EST fixtures.
|
||
- [ ] `wireUtc` / `slotStartWire` include minutes (`…T13:30:00Z`).
|
||
- [ ] **Green** — implement slot module; `slotsPerDay = 2`,
|
||
`slotDuration = Duration(hours: 3, minutes: 15)`.
|
||
- [ ] **Refactor** — update imports project-wide; delete old four-hour module when
|
||
unused.
|
||
|
||
**Confirm:** `cd server && dart test test/trading/market_history_session_slot_test.dart`
|
||
|
||
---
|
||
|
||
## 2. Config & env
|
||
|
||
**Files:** `market_history_config.dart`, `market_history_env.dart`, `env.dart`,
|
||
`server/README.md`.
|
||
|
||
- [ ] `barTimeframe` → `sessionHalf` (or chosen name).
|
||
- [ ] Remove `slotHours = 4`; document `slotsPerDay = 2`.
|
||
- [ ] Add `alpacaFetchTimeframe = '1Min'` (fetch only, not stored).
|
||
- [ ] Document env vars; defaults for `MIN_BARS_FOR_GUESS` if changed.
|
||
|
||
---
|
||
|
||
## 3. Database migration `010_session_half_bars.sql`
|
||
|
||
- [ ] **Red** — extend `market_history_schema_test.dart`:
|
||
- [ ] `timeframe` CHECK allows `sessionHalf`.
|
||
- [ ] Partial index on `(symbol, as_of DESC) WHERE metric='bar' AND timeframe='sessionHalf'`.
|
||
- [ ] **Green** — migration:
|
||
- [ ] `DELETE` (or archive) `metric='bar' AND timeframe='4Hour'`.
|
||
- [ ] Update `market_data_snapshots_timeframe_check`.
|
||
- [ ] `CREATE INDEX market_data_snapshots_bar_session_half_idx …`.
|
||
- [ ] Apply in integration test harness (`001`–`010`).
|
||
|
||
**Confirm:** `cd server && dart test test/integration/market_history_schema_test.dart`
|
||
|
||
---
|
||
|
||
## 4. Minute fetch + aggregation (backfill)
|
||
|
||
**Files:** `market_data_history.dart`, `alpaca_market_data_client.dart` (unchanged
|
||
API surface; caller passes `1Min`).
|
||
|
||
- [ ] **Red** — `market_data_history_sync_test.dart`:
|
||
- [ ] Mock `1Min` bars spanning 9:30–12:45 ET → one persisted `sessionHalf` row.
|
||
- [ ] OHLCV aggregation rules: `o`=first, `h`=max, `l`=min, `c`=last, `v`=sum.
|
||
- [ ] Pagination: merge pages via existing `getBarsRange` + `next_page_token`.
|
||
- [ ] Wrong-window minutes rejected; empty minutes → placeholder or error per
|
||
calendar rules.
|
||
- [ ] Rate-limit / partial run behavior unchanged.
|
||
- [ ] **Green** — `_fetchBarsWithRateLimitRetry` uses `1Min`; `_persistBars`
|
||
aggregates then upserts one row per symbol; `raw.slot_start` + optional
|
||
`raw.minute_bars_count`.
|
||
- [ ] **Refactor** — extract `aggregateMinuteBars(List<AlpacaBar>)` helper.
|
||
|
||
**Confirm:** `cd server && dart test test/integration/market_data_history_sync_test.dart`
|
||
|
||
---
|
||
|
||
## 5. DB slot matching
|
||
|
||
**File:** `market_data_db.dart`
|
||
|
||
- [ ] Replace `_slotStartBucketSql` (4-hour UTC `div(hour,4)`) with session-slot
|
||
equality on `raw.slot_start` wire or shared Dart/SQL slot function.
|
||
- [ ] **Red** — `market_data_db_test.dart` for `symbolsWithBarForSlot` at 9:30 / 12:45
|
||
ET boundaries.
|
||
|
||
---
|
||
|
||
## 6. Read paths
|
||
|
||
| File | Work |
|
||
|------|------|
|
||
| `market_history_query.dart` | Filter `timeframe = sessionHalf`; update comments. |
|
||
| `market_history_question_audit.dart` | Step by **one session slot** (not ±4h); slot pair query. |
|
||
| `market_history_week_coverage.dart` | 2 slots per trading day; `slotsPerDay: 2`. |
|
||
| `market_history_trading_calendar.dart` | Trading-day helpers keyed on ET date of slot. |
|
||
| `market_history_admin_logic.dart` | Error strings / slot labels. |
|
||
| `backfill_sync_item.dart` | Wire format with minute-precision `slotStart`. |
|
||
|
||
- [ ] **Red** — unit + integration tests for each area (see existing `*_test.dart`
|
||
files under `server/test/`).
|
||
- [ ] **Green** — implement.
|
||
|
||
**Confirm:** `./scripts/test-server.sh` (no live Alpaca).
|
||
|
||
---
|
||
|
||
## 7. Flutter admin
|
||
|
||
| File | Work |
|
||
|------|------|
|
||
| `lib/admin/utils/sync_run_formatters.dart` | `formatMarketHistorySlotWire` — no `hour ~/ 4`. |
|
||
| `lib/admin/models/market_history_week_coverage.dart` | Default `slotsPerDay: 2`. |
|
||
| `lib/admin/widgets/market_history_week_coverage_sheet.dart` | Copy: 2 slots/day. |
|
||
| `lib/admin/widgets/market_history_question_audit_sheet.dart` | ET slot labels; count-only UI (done). |
|
||
| `lib/admin/widgets/sync_run_expansion_tile.dart` | Backfill row: count only (done). |
|
||
|
||
- [ ] **Red** — widget tests under `test/admin/`.
|
||
- [ ] **Green** — implement remaining slot-label / coverage changes after server
|
||
ships new slot times.
|
||
|
||
**Confirm:** `./scripts/test-admin-portal.sh`
|
||
|
||
---
|
||
|
||
## 8. Fixtures & live tests
|
||
|
||
- [ ] Add `server/test/fixtures/alpaca_bars_1min_session.json` (195 minutes × 1 symbol).
|
||
- [ ] Update `alpaca_bars_4h_window.json` usages or remove.
|
||
- [ ] `@Tags(['alpaca'])` live test: `1Min` range for one slot, aggregate locally.
|
||
|
||
---
|
||
|
||
## 9. Documentation
|
||
|
||
- [ ] `server/README.md` — Market history section (2 session slots, `1Min` fetch).
|
||
- [ ] `FLUTTER-ADMIN-PORTAL.md` — week coverage + question audit behavior.
|
||
- [ ] Link from [`TODO.md`](./TODO.md) (legacy 4h work is complete; this doc
|
||
supersedes granularity for Phase 3).
|
||
|
||
---
|
||
|
||
## 10. Deploy / ops
|
||
|
||
- [ ] Run migration `010` on prod/staging (deletes legacy `4Hour` bar rows automatically).
|
||
- [ ] **Or** manually clear history before backfill (if migration already applied without `010`):
|
||
|
||
```sql
|
||
-- Required: remove old 4-hour bar rows (wrong shape for session-half logic)
|
||
DELETE FROM market_data_snapshots
|
||
WHERE metric = 'bar' AND timeframe = '4Hour';
|
||
|
||
-- Optional: archived 4Hour copies (if any)
|
||
DELETE FROM market_data_archive
|
||
WHERE metric = 'bar' AND timeframe = '4Hour';
|
||
|
||
-- Optional: force a clean sync audit trail (not required for backfill to run)
|
||
TRUNCATE market_data_sync_runs;
|
||
```
|
||
|
||
**Do not truncate** `tradable_assets` — universe sync is independent.
|
||
|
||
After clearing, run admin **Resync** or wait for the worker; `hasPendingSlots` will
|
||
enqueue backfill for every missing `sessionHalf` slot in the rolling window.
|
||
|
||
- [ ] Verify week-coverage calendar green for 2 slots × trading days.
|
||
- [ ] Verify `guess_weekly_move` eligibility with new `MIN_BARS` threshold.
|
||
|
||
---
|
||
|
||
## 11. Progress log
|
||
|
||
| Date | Step | Notes |
|
||
|------|------|-------|
|
||
| | | |
|
||
|
||
---
|
||
|
||
## Appendix — affected files (checklist)
|
||
|
||
**Server (implement):**
|
||
|
||
- `server/lib/trading/market_history_four_hour_slot.dart` → session slot module
|
||
- `server/lib/trading/market_history_config.dart`
|
||
- `server/lib/trading/market_data_history.dart`
|
||
- `server/lib/trading/market_data_db.dart`
|
||
- `server/lib/trading/market_history_query.dart`
|
||
- `server/lib/trading/market_history_question_audit.dart`
|
||
- `server/lib/trading/market_history_week_coverage.dart`
|
||
- `server/lib/trading/market_history_trading_calendar.dart`
|
||
- `server/lib/trading/backfill_sync_item.dart`
|
||
- `server/lib/alpaca/alpaca_market_data_client.dart` (wire helper import only)
|
||
- `server/migrations/010_session_half_bars.sql` (new)
|
||
|
||
**Server tests:**
|
||
|
||
- `server/test/trading/market_history_four_hour_slot_test.dart` → session tests
|
||
- `server/test/trading/market_history_week_coverage_test.dart`
|
||
- `server/test/trading/market_history_question_audit_test.dart`
|
||
- `server/test/integration/market_data_history_sync_test.dart`
|
||
- `server/test/integration/market_data_db_test.dart`
|
||
- `server/test/integration/market_history_admin_handler_test.dart`
|
||
- `server/test/integration/market_history_week_coverage_test.dart`
|
||
- (+ admin logic, schema, scheduler tests as needed)
|
||
|
||
**Flutter:**
|
||
|
||
- `lib/admin/widgets/market_history_question_audit_sheet.dart`
|
||
- `lib/admin/widgets/sync_run_expansion_tile.dart`
|
||
- `lib/admin/widgets/market_history_week_coverage_sheet.dart`
|
||
- `lib/admin/utils/sync_run_formatters.dart`
|
||
- `lib/admin/models/market_history_week_coverage.dart`
|
||
- `test/admin/widgets/market_history_question_audit_sheet_test.dart`
|
||
- `test/admin/widgets/sync_run_expansion_tile_test.dart`
|
||
|
||
**Unrelated (do not change for slot work):**
|
||
|
||
- `server/lib/trading/guardrails.dart` (4h **notional** window for orders)
|
||
- `server/lib/trading/market_data_ingest.dart` (daily bars for live ingest)
|