mygit/cyberhybridhub
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
cyberhybridhub/TODO-SESSION-HALF-BARS.md
Nathan Anderson eb5f57361c morning evening
2026-05-31 12:40:54 -05:00

10 KiB
Raw Blame History

TODO — RTH session half bars (morning / afternoon aggregates)

Companion to server/README.md (Market history) and FLUTTER-ADMIN-PORTAL.md.

Goal: Replace six UTC 4-hour Alpaca bars per day with two regular-session aggregates per US trading day, each built from up to 195 one-minute bars:

Slot US Eastern (NYSE regular) Duration
Morning 9:30 AM – 12:45 PM 3h 15m (195 min)
Afternoon 12:45 PM – 4:00 PM 3h 15m (195 min)

Persist one OHLCV row per symbol per slot (not 195 rows). Use for guess_weekly_move, admin week coverage, and question audit.

TDD rhythm: Red → Green → Refactor → Confirm (same as TODO.md §0).

0. Design decisions (lock before coding)

  • Timezone: America/New_York for slot boundaries (handles DST); store canonical as_of / raw.slot_start as UTC instants of slot open.
  • Stored timeframe: new value, e.g. sessionHalf (do not overload 4Hour).
  • Alpaca fetch: GET /v2/stocks/bars with timeframe=1Min per slot [start, end]; aggregate in server (o/h/l/c/v from minutes).
  • Existing data: delete or archive all timeframe = '4Hour' history rows after migration; full backfill required.
  • MIN_BARS_FOR_GUESS: revisit default (5 bars ≈ 2.5 trading days at 2 slots/day vs ~20h span with 4h bars).
  • Week coverage UI: 2 dots per trading day (not 6 UTC dots).
  • Question audit API (optional): return assetCount only, drop assets[] payload to save bandwidth (Flutter already shows count-only).

1. Slot model (replace MarketHistoryFourHourSlot)

File: replace or supersede server/lib/trading/market_history_four_hour_slot.dart → e.g. market_history_session_slot.dart.

  • Red — server/test/trading/market_history_session_slot_test.dart:
    • slotStartContaining maps instants to morning (9:30 ET) or afternoon (12:45 ET) slot start (UTC).
    • endExclusive / endInclusive for 195-minute windows.
    • hasEnded / lastCompletedSlotStart never returns in-progress slot.
    • completedSlotStartsInWindow yields 2 × trading days in rolling window; skips weekends + NYSE holidays (MarketHistoryTradingCalendar).
    • DST: assert 13:30 vs 14:30 UTC morning start across EDT/EST fixtures.
    • wireUtc / slotStartWire include minutes (…T13:30:00Z).
  • Green — implement slot module; slotsPerDay = 2, slotDuration = Duration(hours: 3, minutes: 15).
  • Refactor — update imports project-wide; delete old four-hour module when unused.

Confirm: cd server && dart test test/trading/market_history_session_slot_test.dart

2. Config & env

Files: market_history_config.dart, market_history_env.dart, env.dart, server/README.md.

  • barTimeframe → sessionHalf (or chosen name).
  • Remove slotHours = 4; document slotsPerDay = 2.
  • Add alpacaFetchTimeframe = '1Min' (fetch only, not stored).
  • Document env vars; defaults for MIN_BARS_FOR_GUESS if changed.

3. Database migration 010_session_half_bars.sql

  • Red — extend market_history_schema_test.dart:
    • timeframe CHECK allows sessionHalf.
    • Partial index on (symbol, as_of DESC) WHERE metric='bar' AND timeframe='sessionHalf'.
  • Green — migration:
    • DELETE (or archive) metric='bar' AND timeframe='4Hour'.
    • Update market_data_snapshots_timeframe_check.
    • CREATE INDEX market_data_snapshots_bar_session_half_idx ….
  • Apply in integration test harness (001–010).

Confirm: cd server && dart test test/integration/market_history_schema_test.dart

4. Minute fetch + aggregation (backfill)

Files: market_data_history.dart, alpaca_market_data_client.dart (unchanged API surface; caller passes 1Min).

  • Red — market_data_history_sync_test.dart:
    • Mock 1Min bars spanning 9:30–12:45 ET → one persisted sessionHalf row.
    • OHLCV aggregation rules: o=first, h=max, l=min, c=last, v=sum.
    • Pagination: merge pages via existing getBarsRange + next_page_token.
    • Wrong-window minutes rejected; empty minutes → placeholder or error per calendar rules.
    • Rate-limit / partial run behavior unchanged.
  • Green — _fetchBarsWithRateLimitRetry uses 1Min; _persistBars aggregates then upserts one row per symbol; raw.slot_start + optional raw.minute_bars_count.
  • Refactor — extract aggregateMinuteBars(List<AlpacaBar>) helper.

Confirm: cd server && dart test test/integration/market_data_history_sync_test.dart

5. DB slot matching

File: market_data_db.dart

  • Replace _slotStartBucketSql (4-hour UTC div(hour,4)) with session-slot equality on raw.slot_start wire or shared Dart/SQL slot function.
  • Red — market_data_db_test.dart for symbolsWithBarForSlot at 9:30 / 12:45 ET boundaries.

6. Read paths

File Work
market_history_query.dart Filter timeframe = sessionHalf; update comments.
market_history_question_audit.dart Step by one session slot (not ±4h); slot pair query.
market_history_week_coverage.dart 2 slots per trading day; slotsPerDay: 2.
market_history_trading_calendar.dart Trading-day helpers keyed on ET date of slot.
market_history_admin_logic.dart Error strings / slot labels.
backfill_sync_item.dart Wire format with minute-precision slotStart.
  • Red — unit + integration tests for each area (see existing *_test.dart files under server/test/).
  • Green — implement.

Confirm: ./scripts/test-server.sh (no live Alpaca).

7. Flutter admin

File Work
lib/admin/utils/sync_run_formatters.dart formatMarketHistorySlotWire — no hour ~/ 4.
lib/admin/models/market_history_week_coverage.dart Default slotsPerDay: 2.
lib/admin/widgets/market_history_week_coverage_sheet.dart Copy: 2 slots/day.
lib/admin/widgets/market_history_question_audit_sheet.dart ET slot labels; count-only UI (done).
lib/admin/widgets/sync_run_expansion_tile.dart Backfill row: count only (done).
  • Red — widget tests under test/admin/.
  • Green — implement remaining slot-label / coverage changes after server ships new slot times.

Confirm: ./scripts/test-admin-portal.sh

8. Fixtures & live tests

  • Add server/test/fixtures/alpaca_bars_1min_session.json (195 minutes × 1 symbol).
  • Update alpaca_bars_4h_window.json usages or remove.
  • @Tags(['alpaca']) live test: 1Min range for one slot, aggregate locally.

9. Documentation

  • server/README.md — Market history section (2 session slots, 1Min fetch).
  • FLUTTER-ADMIN-PORTAL.md — week coverage + question audit behavior.
  • Link from TODO.md (legacy 4h work is complete; this doc supersedes granularity for Phase 3).

10. Deploy / ops

  • Run migration 010 on prod/staging (deletes legacy 4Hour bar rows automatically).

  • Or manually clear history before backfill (if migration already applied without 010):

    -- Required: remove old 4-hour bar rows (wrong shape for session-half logic)
DELETE FROM market_data_snapshots
WHERE metric = 'bar' AND timeframe = '4Hour';

-- Optional: archived 4Hour copies (if any)
DELETE FROM market_data_archive
WHERE metric = 'bar' AND timeframe = '4Hour';

-- Optional: force a clean sync audit trail (not required for backfill to run)
TRUNCATE market_data_sync_runs;

    Do not truncate tradable_assets — universe sync is independent.

    After clearing, run admin Resync or wait for the worker; hasPendingSlots will enqueue backfill for every missing sessionHalf slot in the rolling window.

  • Verify week-coverage calendar green for 2 slots × trading days.

  • Verify guess_weekly_move eligibility with new MIN_BARS threshold.

11. Progress log

Date Step Notes

Appendix — affected files (checklist)

Server (implement):

  • server/lib/trading/market_history_four_hour_slot.dart → session slot module
  • server/lib/trading/market_history_config.dart
  • server/lib/trading/market_data_history.dart
  • server/lib/trading/market_data_db.dart
  • server/lib/trading/market_history_query.dart
  • server/lib/trading/market_history_question_audit.dart
  • server/lib/trading/market_history_week_coverage.dart
  • server/lib/trading/market_history_trading_calendar.dart
  • server/lib/trading/backfill_sync_item.dart
  • server/lib/alpaca/alpaca_market_data_client.dart (wire helper import only)
  • server/migrations/010_session_half_bars.sql (new)

Server tests:

  • server/test/trading/market_history_four_hour_slot_test.dart → session tests
  • server/test/trading/market_history_week_coverage_test.dart
  • server/test/trading/market_history_question_audit_test.dart
  • server/test/integration/market_data_history_sync_test.dart
  • server/test/integration/market_data_db_test.dart
  • server/test/integration/market_history_admin_handler_test.dart
  • server/test/integration/market_history_week_coverage_test.dart
  • (+ admin logic, schema, scheduler tests as needed)

Flutter:

  • lib/admin/widgets/market_history_question_audit_sheet.dart
  • lib/admin/widgets/sync_run_expansion_tile.dart
  • lib/admin/widgets/market_history_week_coverage_sheet.dart
  • lib/admin/utils/sync_run_formatters.dart
  • lib/admin/models/market_history_week_coverage.dart
  • test/admin/widgets/market_history_question_audit_sheet_test.dart
  • test/admin/widgets/sync_run_expansion_tile_test.dart

Unrelated (do not change for slot work):

  • server/lib/trading/guardrails.dart (4h notional window for orders)
  • server/lib/trading/market_data_ingest.dart (daily bars for live ingest)