morning evening
This commit is contained in:
parent
3af1e31fac
commit
eb5f57361c
252
TODO-SESSION-HALF-BARS.md
Normal file
252
TODO-SESSION-HALF-BARS.md
Normal file
@ -0,0 +1,252 @@
|
|||||||
|
# TODO — RTH session half bars (morning / afternoon aggregates)
|
||||||
|
|
||||||
|
Companion to [`server/README.md`](./server/README.md) (Market history) and
|
||||||
|
[`FLUTTER-ADMIN-PORTAL.md`](./FLUTTER-ADMIN-PORTAL.md).
|
||||||
|
|
||||||
|
**Goal:** Replace six UTC **4-hour** Alpaca bars per day with **two regular-session
|
||||||
|
aggregates per US trading day**, each built from up to **195 one-minute bars**:
|
||||||
|
|
||||||
|
| Slot | US Eastern (NYSE regular) | Duration |
|
||||||
|
|------|---------------------------|----------|
|
||||||
|
| Morning | 9:30 AM – 12:45 PM | 3h 15m (195 min) |
|
||||||
|
| Afternoon | 12:45 PM – 4:00 PM | 3h 15m (195 min) |
|
||||||
|
|
||||||
|
Persist **one OHLCV row per symbol per slot** (not 195 rows). Use for
|
||||||
|
`guess_weekly_move`, admin week coverage, and question audit.
|
||||||
|
|
||||||
|
**TDD rhythm:** Red → Green → Refactor → Confirm (same as
|
||||||
|
[`TODO.md`](./TODO.md) §0).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0. Design decisions (lock before coding)
|
||||||
|
|
||||||
|
- [ ] **Timezone:** `America/New_York` for slot boundaries (handles DST); store
|
||||||
|
canonical `as_of` / `raw.slot_start` as UTC instants of slot open.
|
||||||
|
- [ ] **Stored `timeframe`:** new value, e.g. `sessionHalf` (do not overload
|
||||||
|
`4Hour`).
|
||||||
|
- [ ] **Alpaca fetch:** `GET /v2/stocks/bars` with `timeframe=1Min` per slot
|
||||||
|
`[start, end]`; aggregate in server (`o`/`h`/`l`/`c`/`v` from minutes).
|
||||||
|
- [ ] **Existing data:** delete or archive all `timeframe = '4Hour'` history rows
|
||||||
|
after migration; full backfill required.
|
||||||
|
- [ ] **`MIN_BARS_FOR_GUESS`:** revisit default (`5` bars ≈ 2.5 trading days at
|
||||||
|
2 slots/day vs ~20h span with 4h bars).
|
||||||
|
- [ ] **Week coverage UI:** 2 dots per **trading day** (not 6 UTC dots).
|
||||||
|
- [ ] **Question audit API (optional):** return `assetCount` only, drop `assets[]`
|
||||||
|
payload to save bandwidth (Flutter already shows count-only).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Slot model (replace `MarketHistoryFourHourSlot`)
|
||||||
|
|
||||||
|
**File:** replace or supersede `server/lib/trading/market_history_four_hour_slot.dart`
|
||||||
|
→ e.g. `market_history_session_slot.dart`.
|
||||||
|
|
||||||
|
- [ ] **Red** — `server/test/trading/market_history_session_slot_test.dart`:
|
||||||
|
- [ ] `slotStartContaining` maps instants to morning (9:30 ET) or afternoon
|
||||||
|
(12:45 ET) slot start (UTC).
|
||||||
|
- [ ] `endExclusive` / `endInclusive` for 195-minute windows.
|
||||||
|
- [ ] `hasEnded` / `lastCompletedSlotStart` never returns in-progress slot.
|
||||||
|
- [ ] `completedSlotStartsInWindow` yields 2 × trading days in rolling window;
|
||||||
|
skips weekends + NYSE holidays (`MarketHistoryTradingCalendar`).
|
||||||
|
- [ ] DST: assert 13:30 vs 14:30 UTC morning start across EDT/EST fixtures.
|
||||||
|
- [ ] `wireUtc` / `slotStartWire` include minutes (`…T13:30:00Z`).
|
||||||
|
- [ ] **Green** — implement slot module; `slotsPerDay = 2`,
|
||||||
|
`slotDuration = Duration(hours: 3, minutes: 15)`.
|
||||||
|
- [ ] **Refactor** — update imports project-wide; delete old four-hour module when
|
||||||
|
unused.
|
||||||
|
|
||||||
|
**Confirm:** `cd server && dart test test/trading/market_history_session_slot_test.dart`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Config & env
|
||||||
|
|
||||||
|
**Files:** `market_history_config.dart`, `market_history_env.dart`, `env.dart`,
|
||||||
|
`server/README.md`.
|
||||||
|
|
||||||
|
- [ ] `barTimeframe` → `sessionHalf` (or chosen name).
|
||||||
|
- [ ] Remove `slotHours = 4`; document `slotsPerDay = 2`.
|
||||||
|
- [ ] Add `alpacaFetchTimeframe = '1Min'` (fetch only, not stored).
|
||||||
|
- [ ] Document env vars; defaults for `MIN_BARS_FOR_GUESS` if changed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Database migration `010_session_half_bars.sql`
|
||||||
|
|
||||||
|
- [ ] **Red** — extend `market_history_schema_test.dart`:
|
||||||
|
- [ ] `timeframe` CHECK allows `sessionHalf`.
|
||||||
|
- [ ] Partial index on `(symbol, as_of DESC) WHERE metric='bar' AND timeframe='sessionHalf'`.
|
||||||
|
- [ ] **Green** — migration:
|
||||||
|
- [ ] `DELETE` (or archive) `metric='bar' AND timeframe='4Hour'`.
|
||||||
|
- [ ] Update `market_data_snapshots_timeframe_check`.
|
||||||
|
- [ ] `CREATE INDEX market_data_snapshots_bar_session_half_idx …`.
|
||||||
|
- [ ] Apply in integration test harness (`001`–`010`).
|
||||||
|
|
||||||
|
**Confirm:** `cd server && dart test test/integration/market_history_schema_test.dart`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Minute fetch + aggregation (backfill)
|
||||||
|
|
||||||
|
**Files:** `market_data_history.dart`, `alpaca_market_data_client.dart` (unchanged
|
||||||
|
API surface; caller passes `1Min`).
|
||||||
|
|
||||||
|
- [ ] **Red** — `market_data_history_sync_test.dart`:
|
||||||
|
- [ ] Mock `1Min` bars spanning 9:30–12:45 ET → one persisted `sessionHalf` row.
|
||||||
|
- [ ] OHLCV aggregation rules: `o`=first, `h`=max, `l`=min, `c`=last, `v`=sum.
|
||||||
|
- [ ] Pagination: merge pages via existing `getBarsRange` + `next_page_token`.
|
||||||
|
- [ ] Wrong-window minutes rejected; empty minutes → placeholder or error per
|
||||||
|
calendar rules.
|
||||||
|
- [ ] Rate-limit / partial run behavior unchanged.
|
||||||
|
- [ ] **Green** — `_fetchBarsWithRateLimitRetry` uses `1Min`; `_persistBars`
|
||||||
|
aggregates then upserts one row per symbol; `raw.slot_start` + optional
|
||||||
|
`raw.minute_bars_count`.
|
||||||
|
- [ ] **Refactor** — extract `aggregateMinuteBars(List<AlpacaBar>)` helper.
|
||||||
|
|
||||||
|
**Confirm:** `cd server && dart test test/integration/market_data_history_sync_test.dart`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. DB slot matching
|
||||||
|
|
||||||
|
**File:** `market_data_db.dart`
|
||||||
|
|
||||||
|
- [ ] Replace `_slotStartBucketSql` (4-hour UTC `div(hour,4)`) with session-slot
|
||||||
|
equality on `raw.slot_start` wire or shared Dart/SQL slot function.
|
||||||
|
- [ ] **Red** — `market_data_db_test.dart` for `symbolsWithBarForSlot` at 9:30 / 12:45
|
||||||
|
ET boundaries.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Read paths
|
||||||
|
|
||||||
|
| File | Work |
|
||||||
|
|------|------|
|
||||||
|
| `market_history_query.dart` | Filter `timeframe = sessionHalf`; update comments. |
|
||||||
|
| `market_history_question_audit.dart` | Step by **one session slot** (not ±4h); slot pair query. |
|
||||||
|
| `market_history_week_coverage.dart` | 2 slots per trading day; `slotsPerDay: 2`. |
|
||||||
|
| `market_history_trading_calendar.dart` | Trading-day helpers keyed on ET date of slot. |
|
||||||
|
| `market_history_admin_logic.dart` | Error strings / slot labels. |
|
||||||
|
| `backfill_sync_item.dart` | Wire format with minute-precision `slotStart`. |
|
||||||
|
|
||||||
|
- [ ] **Red** — unit + integration tests for each area (see existing `*_test.dart`
|
||||||
|
files under `server/test/`).
|
||||||
|
- [ ] **Green** — implement.
|
||||||
|
|
||||||
|
**Confirm:** `./scripts/test-server.sh` (no live Alpaca).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Flutter admin
|
||||||
|
|
||||||
|
| File | Work |
|
||||||
|
|------|------|
|
||||||
|
| `lib/admin/utils/sync_run_formatters.dart` | `formatMarketHistorySlotWire` — no `hour ~/ 4`. |
|
||||||
|
| `lib/admin/models/market_history_week_coverage.dart` | Default `slotsPerDay: 2`. |
|
||||||
|
| `lib/admin/widgets/market_history_week_coverage_sheet.dart` | Copy: 2 slots/day. |
|
||||||
|
| `lib/admin/widgets/market_history_question_audit_sheet.dart` | ET slot labels; count-only UI (done). |
|
||||||
|
| `lib/admin/widgets/sync_run_expansion_tile.dart` | Backfill row: count only (done). |
|
||||||
|
|
||||||
|
- [ ] **Red** — widget tests under `test/admin/`.
|
||||||
|
- [ ] **Green** — implement remaining slot-label / coverage changes after server
|
||||||
|
ships new slot times.
|
||||||
|
|
||||||
|
**Confirm:** `./scripts/test-admin-portal.sh`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Fixtures & live tests
|
||||||
|
|
||||||
|
- [ ] Add `server/test/fixtures/alpaca_bars_1min_session.json` (195 minutes × 1 symbol).
|
||||||
|
- [ ] Update `alpaca_bars_4h_window.json` usages or remove.
|
||||||
|
- [ ] `@Tags(['alpaca'])` live test: `1Min` range for one slot, aggregate locally.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Documentation
|
||||||
|
|
||||||
|
- [ ] `server/README.md` — Market history section (2 session slots, `1Min` fetch).
|
||||||
|
- [ ] `FLUTTER-ADMIN-PORTAL.md` — week coverage + question audit behavior.
|
||||||
|
- [ ] Link from [`TODO.md`](./TODO.md) (legacy 4h work is complete; this doc
|
||||||
|
supersedes granularity for Phase 3).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Deploy / ops
|
||||||
|
|
||||||
|
- [ ] Run migration `010` on prod/staging (deletes legacy `4Hour` bar rows automatically).
|
||||||
|
- [ ] **Or** manually clear history before backfill (if migration already applied without `010`):
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Required: remove old 4-hour bar rows (wrong shape for session-half logic)
|
||||||
|
DELETE FROM market_data_snapshots
|
||||||
|
WHERE metric = 'bar' AND timeframe = '4Hour';
|
||||||
|
|
||||||
|
-- Optional: archived 4Hour copies (if any)
|
||||||
|
DELETE FROM market_data_archive
|
||||||
|
WHERE metric = 'bar' AND timeframe = '4Hour';
|
||||||
|
|
||||||
|
-- Optional: force a clean sync audit trail (not required for backfill to run)
|
||||||
|
TRUNCATE market_data_sync_runs;
|
||||||
|
```
|
||||||
|
|
||||||
|
**Do not truncate** `tradable_assets` — universe sync is independent.
|
||||||
|
|
||||||
|
After clearing, run admin **Resync** or wait for the worker; `hasPendingSlots` will
|
||||||
|
enqueue backfill for every missing `sessionHalf` slot in the rolling window.
|
||||||
|
|
||||||
|
- [ ] Verify week-coverage calendar green for 2 slots × trading days.
|
||||||
|
- [ ] Verify `guess_weekly_move` eligibility with new `MIN_BARS` threshold.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Progress log
|
||||||
|
|
||||||
|
| Date | Step | Notes |
|
||||||
|
|------|------|-------|
|
||||||
|
| | | |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix — affected files (checklist)
|
||||||
|
|
||||||
|
**Server (implement):**
|
||||||
|
|
||||||
|
- `server/lib/trading/market_history_four_hour_slot.dart` → session slot module
|
||||||
|
- `server/lib/trading/market_history_config.dart`
|
||||||
|
- `server/lib/trading/market_data_history.dart`
|
||||||
|
- `server/lib/trading/market_data_db.dart`
|
||||||
|
- `server/lib/trading/market_history_query.dart`
|
||||||
|
- `server/lib/trading/market_history_question_audit.dart`
|
||||||
|
- `server/lib/trading/market_history_week_coverage.dart`
|
||||||
|
- `server/lib/trading/market_history_trading_calendar.dart`
|
||||||
|
- `server/lib/trading/backfill_sync_item.dart`
|
||||||
|
- `server/lib/alpaca/alpaca_market_data_client.dart` (wire helper import only)
|
||||||
|
- `server/migrations/010_session_half_bars.sql` (new)
|
||||||
|
|
||||||
|
**Server tests:**
|
||||||
|
|
||||||
|
- `server/test/trading/market_history_four_hour_slot_test.dart` → session tests
|
||||||
|
- `server/test/trading/market_history_week_coverage_test.dart`
|
||||||
|
- `server/test/trading/market_history_question_audit_test.dart`
|
||||||
|
- `server/test/integration/market_data_history_sync_test.dart`
|
||||||
|
- `server/test/integration/market_data_db_test.dart`
|
||||||
|
- `server/test/integration/market_history_admin_handler_test.dart`
|
||||||
|
- `server/test/integration/market_history_week_coverage_test.dart`
|
||||||
|
- (+ admin logic, schema, scheduler tests as needed)
|
||||||
|
|
||||||
|
**Flutter:**
|
||||||
|
|
||||||
|
- `lib/admin/widgets/market_history_question_audit_sheet.dart`
|
||||||
|
- `lib/admin/widgets/sync_run_expansion_tile.dart`
|
||||||
|
- `lib/admin/widgets/market_history_week_coverage_sheet.dart`
|
||||||
|
- `lib/admin/utils/sync_run_formatters.dart`
|
||||||
|
- `lib/admin/models/market_history_week_coverage.dart`
|
||||||
|
- `test/admin/widgets/market_history_question_audit_sheet_test.dart`
|
||||||
|
- `test/admin/widgets/sync_run_expansion_tile_test.dart`
|
||||||
|
|
||||||
|
**Unrelated (do not change for slot work):**
|
||||||
|
|
||||||
|
- `server/lib/trading/guardrails.dart` (4h **notional** window for orders)
|
||||||
|
- `server/lib/trading/market_data_ingest.dart` (daily bars for live ingest)
|
||||||
748
TODO.md
Normal file
748
TODO.md
Normal file
@ -0,0 +1,748 @@
|
|||||||
|
# TODO — Rolling 7-Day Market Data Window + Cleanup
|
||||||
|
|
||||||
|
> **Next milestone:** ET morning/afternoon session half bars (195-minute aggregates)
|
||||||
|
> — see [`TODO-SESSION-HALF-BARS.md`](./TODO-SESSION-HALF-BARS.md).
|
||||||
|
|
||||||
|
Companion to [`TRADING_DEVELOPMENT_PLAN.md`](./TRADING_DEVELOPMENT_PLAN.md) and
|
||||||
|
[`TRADING_TDD_PLAN.md`](./TRADING_TDD_PLAN.md).
|
||||||
|
|
||||||
|
**Goal:** maintain a rolling **7-day history** of market data for **all active
|
||||||
|
tradable assets** so the question pipeline can generate obfuscated
|
||||||
|
*guessing-game* questions about market movement, while pruning (or archiving)
|
||||||
|
anything older than the window.
|
||||||
|
|
||||||
|
**TDD rhythm (mandatory for every step):**
|
||||||
|
|
||||||
|
1. **Red** — write the failing test(s) first; commit if you like.
|
||||||
|
2. **Green** — minimum implementation that turns every test in this step green.
|
||||||
|
3. **Refactor** — tidy without changing behavior; rerun tests.
|
||||||
|
4. **Confirm** — run the full step-level confirm command listed in the step.
|
||||||
|
5. **Log** — check the box, add a row to [§12 Progress log](#12-progress-log).
|
||||||
|
|
||||||
|
> Do not skip the Red phase. Do not start the next step while any test in the
|
||||||
|
> current step is failing or pending. No live Alpaca calls in default
|
||||||
|
> `dart test` jobs — guard with `@Tags(['alpaca'])`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0. Scope & design constraints
|
||||||
|
|
||||||
|
- **Window:** rolling 7 calendar days, UTC. Configurable via
|
||||||
|
`MARKET_HISTORY_WINDOW_DAYS` (default `7`).
|
||||||
|
- **Granularity (Phase 1):** `1Day` bars for every active tradable, plus the
|
||||||
|
existing `last_trade` / `prev_close` snapshots for watchlist symbols.
|
||||||
|
- **Granularity (Phase 2):** `1Hour` bars for the union of all enabled users'
|
||||||
|
watchlist symbols (≤30 on Alpaca Basic).
|
||||||
|
- **Universe source of truth:** Alpaca `/v2/assets?status=active&tradable=true`,
|
||||||
|
refreshed daily, cached in Postgres (`tradable_assets`).
|
||||||
|
- **Idempotency:** repeated backfill of the same
|
||||||
|
`(symbol, metric, timeframe, as_of)` MUST NOT create duplicate rows.
|
||||||
|
- **Cleanup vs. archive:** rows with `as_of < now() - window` are either
|
||||||
|
hard-deleted (Phase 1) or moved to `market_data_archive` (Phase 2).
|
||||||
|
- **Worker isolation:** historical sync + cleanup run on their own cadence
|
||||||
|
(default once per day), not on every 60s per-user tick.
|
||||||
|
- **Rate-limit safety:** batch symbols (Alpaca `bars` accepts multi-symbol);
|
||||||
|
cap concurrent symbols; never call Alpaca in tests
|
||||||
|
(`QUESTION_PIPELINE_TEST_MODE=true`).
|
||||||
|
- **No Flutter changes** required for this milestone.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Schema additions (migration `005_market_history.sql`)
|
||||||
|
|
||||||
|
### 1.1 Red — failing tests first
|
||||||
|
|
||||||
|
- [x] Create `server/test/integration/market_history_schema_test.dart`:
|
||||||
|
- [x] Test: `INSERT` two snapshots with the same
|
||||||
|
`(symbol, metric, timeframe, as_of)` → second one **upserts**,
|
||||||
|
does not duplicate (current schema lacks the unique constraint, so
|
||||||
|
this MUST fail Red).
|
||||||
|
- [x] Test: `timeframe` defaults to `'tick'` for existing rows; new rows
|
||||||
|
accept `'1Min' | '1Hour' | '1Day'`.
|
||||||
|
- [x] Test: `tradable_assets` PK rejects duplicate symbol; query by
|
||||||
|
`(status='active', tradable=true)` uses the new index (verify via
|
||||||
|
`EXPLAIN` returning `Index Scan`).
|
||||||
|
- [x] Test: `market_data_sync_runs` records `kind`, `started_at`,
|
||||||
|
`finished_at`, `rows_written`, `rows_removed`, `error` shape.
|
||||||
|
|
||||||
|
### 1.2 Green — minimum migration
|
||||||
|
|
||||||
|
- [x] Write `server/migrations/005_market_history.sql`:
|
||||||
|
- [x] `ALTER TABLE market_data_snapshots ADD COLUMN timeframe TEXT NOT NULL
|
||||||
|
DEFAULT 'tick'`.
|
||||||
|
- [x] `ALTER TABLE market_data_snapshots ADD CONSTRAINT
|
||||||
|
market_data_snapshots_unique_obs UNIQUE
|
||||||
|
(symbol, metric, timeframe, as_of)`.
|
||||||
|
- [x] `CREATE INDEX market_data_snapshots_asof_idx
|
||||||
|
ON market_data_snapshots (as_of DESC)`.
|
||||||
|
- [x] `CREATE TABLE tradable_assets (…)` with columns
|
||||||
|
`symbol PK, asset_class, exchange, name, tradable, fractionable,
|
||||||
|
status, raw JSONB, refreshed_at`.
|
||||||
|
- [x] `CREATE INDEX tradable_assets_status_idx
|
||||||
|
ON tradable_assets (status, tradable)`.
|
||||||
|
- [x] `CREATE TABLE market_data_sync_runs (…)` (see §0 plan).
|
||||||
|
- [ ] (Phase 2 stub, commented) `CREATE TABLE market_data_archive (…)` —
|
||||||
|
deferred to §4.2 (the migration runner splits on `;`, which would
|
||||||
|
slice a commented stub mid-block; the archive table will be added
|
||||||
|
in §4.2.2 when it is actually wired up).
|
||||||
|
|
||||||
|
### 1.3 Refactor
|
||||||
|
|
||||||
|
- [x] Confirm `MarketDataDb._rowToSnapshot` still reads correctly with the
|
||||||
|
new column (read-side back-compat — no test changes needed, just verify
|
||||||
|
existing `market_data_db_test.dart` still passes).
|
||||||
|
- [x] Move shared SQL fragments into the migration runner if duplication
|
||||||
|
appeared. _(none observed in 005; nothing to extract yet.)_
|
||||||
|
|
||||||
|
### 1.4 Confirm
|
||||||
|
|
||||||
|
- [x] `cd server && dart test test/integration/migration_test.dart
|
||||||
|
test/integration/market_history_schema_test.dart` — green.
|
||||||
|
- [x] `psql cyberhybridhub_test -c '\d market_data_snapshots'` shows the
|
||||||
|
unique constraint and new column.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Tradable-asset universe sync
|
||||||
|
|
||||||
|
**Files (new):** `server/lib/alpaca/alpaca_assets_client.dart`,
|
||||||
|
`server/lib/trading/tradable_assets_db.dart`,
|
||||||
|
`server/lib/trading/tradable_assets_sync.dart`.
|
||||||
|
|
||||||
|
### 2.1 Alpaca assets client
|
||||||
|
|
||||||
|
#### 2.1.1 Red
|
||||||
|
|
||||||
|
- [x] Add fixture `server/test/fixtures/alpaca_assets_active.json` (≥5
|
||||||
|
representative assets, mix of `tradable=true/false` and
|
||||||
|
`fractionable=true/false`).
|
||||||
|
- [x] Add `server/test/alpaca/alpaca_assets_client_test.dart`:
|
||||||
|
- [x] Test: `listActiveTradable()` issues `GET` to
|
||||||
|
`${tradingBaseUrl}/v2/assets?status=active&asset_class=us_equity`
|
||||||
|
with `APCA-API-KEY-ID` + `APCA-API-SECRET-KEY` headers.
|
||||||
|
- [x] Test: parses fixture into `List<AlpacaAsset>` — verifies symbol,
|
||||||
|
exchange, fractionable, tradable, status fields.
|
||||||
|
- [x] Test: 401 / 500 → throws `AlpacaAssetsException` with status code
|
||||||
|
and body in the message.
|
||||||
|
- [x] Test: empty response array → returns `[]`, does not throw.
|
||||||
|
|
||||||
|
#### 2.1.2 Green
|
||||||
|
|
||||||
|
- [x] Add `AlpacaAsset` model in `server/lib/alpaca/alpaca_models.dart`.
|
||||||
|
- [x] Implement `AlpacaAssetsClient` with injectable `http.Client`
|
||||||
|
(mirror `AlpacaMarketDataClient` shape).
|
||||||
|
- [x] Add `AlpacaAssetsException`.
|
||||||
|
|
||||||
|
#### 2.1.3 Refactor
|
||||||
|
|
||||||
|
- [x] Extract a private `_authHeaders` helper if duplicated across
|
||||||
|
Alpaca clients (DRY — but only if you actually duplicate). _(Lifted to
|
||||||
|
`AlpacaEnv.authHeaders`; now reused by all three Alpaca clients.)_
|
||||||
|
|
||||||
|
#### 2.1.4 Confirm
|
||||||
|
|
||||||
|
- [x] `dart test test/alpaca/alpaca_assets_client_test.dart` — green.
|
||||||
|
- [x] Tagged live test
|
||||||
|
`server/test/alpaca/alpaca_assets_live_test.dart`
|
||||||
|
(`@Tags(['alpaca'])`) — returns >100 symbols when keys present;
|
||||||
|
skipped otherwise. Run manually:
|
||||||
|
`dart test --tags=alpaca test/alpaca/alpaca_assets_live_test.dart`.
|
||||||
|
|
||||||
|
### 2.2 Universe persistence + diff
|
||||||
|
|
||||||
|
#### 2.2.1 Red
|
||||||
|
|
||||||
|
- [x] Create `server/test/integration/tradable_assets_db_test.dart`:
|
||||||
|
- [x] Test: `upsertAll([A, B, C])` inserts 3 rows.
|
||||||
|
- [x] Test: re-running `upsertAll([B*, C, D])` updates `B`, leaves `C`
|
||||||
|
unchanged-by-content but `refreshed_at` bumped, inserts `D`, and
|
||||||
|
marks `A` as `tradable=false, status='inactive'` (we never delete
|
||||||
|
history).
|
||||||
|
- [x] Test: `listActiveTradableSymbols()` returns only
|
||||||
|
`tradable=true AND status='active'`.
|
||||||
|
|
||||||
|
#### 2.2.2 Red — sync orchestration
|
||||||
|
|
||||||
|
- [x] Create `server/test/integration/tradable_assets_sync_test.dart`:
|
||||||
|
- [x] Test: `TradableAssetsSync.runOnce()` with mocked client returning
|
||||||
|
the fixture → DB rows match; one row in `market_data_sync_runs`
|
||||||
|
with `kind='universe'` and non-null `finished_at`.
|
||||||
|
- [x] Test: client throws → sync run row recorded with `error` populated,
|
||||||
|
`finished_at` non-null, and `rows_written = 0`.
|
||||||
|
- [x] Test: two consecutive runs are safe (idempotent counts).
|
||||||
|
|
||||||
|
#### 2.2.3 Green
|
||||||
|
|
||||||
|
- [x] Implement `TradableAssetsDb.upsertAll`,
|
||||||
|
`TradableAssetsDb.listActiveTradableSymbols`.
|
||||||
|
- [x] Implement `TradableAssetsSync.runOnce()` that writes a
|
||||||
|
`market_data_sync_runs` row around the upsert.
|
||||||
|
|
||||||
|
#### 2.2.4 Refactor
|
||||||
|
|
||||||
|
- [x] Pull "wrap a closure with a `sync_runs` audit row" into a small
|
||||||
|
helper (`SyncRunRecorder.record(kind, body)`); reuse in §3 and §4.
|
||||||
|
_(Landed at `server/lib/trading/sync_run_recorder.dart`;
|
||||||
|
`TradableAssetsSync` already consumes it.)_
|
||||||
|
|
||||||
|
#### 2.2.5 Confirm
|
||||||
|
|
||||||
|
- [x] `dart test test/integration/tradable_assets_db_test.dart
|
||||||
|
test/integration/tradable_assets_sync_test.dart` — green.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Historical backfill (1Day bars × 7 days)
|
||||||
|
|
||||||
|
**Files:** extend `server/lib/alpaca/alpaca_market_data_client.dart`,
|
||||||
|
new `server/lib/trading/market_data_history.dart`,
|
||||||
|
extend `server/lib/trading/market_data_db.dart`.
|
||||||
|
|
||||||
|
### 3.1 Alpaca client — time-range bars with pagination
|
||||||
|
|
||||||
|
#### 3.1.1 Red
|
||||||
|
|
||||||
|
- [x] Add fixtures:
|
||||||
|
- [x] `server/test/fixtures/alpaca_bars_7d_multi_page1.json` — includes
|
||||||
|
`next_page_token`.
|
||||||
|
- [x] `server/test/fixtures/alpaca_bars_7d_multi_page2.json` — final
|
||||||
|
page, `next_page_token: null`.
|
||||||
|
- [x] Extend `server/test/alpaca/alpaca_market_data_client_test.dart`:
|
||||||
|
- [x] Test: `getBarsRange(['SPY','AAPL'], timeframe: '1Day',
|
||||||
|
start, end)` builds correct query string (`start`, `end`,
|
||||||
|
`timeframe`, `feed`, `symbols`, `limit`).
|
||||||
|
- [x] Test: follows pagination — when page1 returns
|
||||||
|
`next_page_token='abc'`, client issues second request with
|
||||||
|
`page_token=abc`; merges both pages' bars per symbol.
|
||||||
|
- [x] Test: stops after a configurable `maxPages` (default 20) to
|
||||||
|
prevent runaway loops.
|
||||||
|
- [x] Test: 429 → throws `AlpacaMarketDataException` containing the
|
||||||
|
word `rate` (so caller can detect & back off).
|
||||||
|
|
||||||
|
#### 3.1.2 Green
|
||||||
|
|
||||||
|
- [x] Implement `Future<AlpacaBarsResponse> getBarsRange({
|
||||||
|
List<String> symbols, String timeframe, DateTime start, DateTime end,
|
||||||
|
int maxPages = 20})` on `AlpacaMarketDataClient`.
|
||||||
|
- [x] Extend `AlpacaBarsResponse` with a `merge(AlpacaBarsResponse other)`
|
||||||
|
method so paginated chunks combine cleanly.
|
||||||
|
|
||||||
|
#### 3.1.3 Refactor
|
||||||
|
|
||||||
|
- [x] If the pagination loop is non-trivial, extract a private
|
||||||
|
`_paginate<T>(initialUri, parsePage)` generic to reuse later for
|
||||||
|
orders/positions endpoints. _(Loop kept inline in `getBarsRange` —
|
||||||
|
~25 lines, clear enough; extract when a second consumer appears.)_
|
||||||
|
|
||||||
|
#### 3.1.4 Confirm
|
||||||
|
|
||||||
|
- [x] `dart test test/alpaca/alpaca_market_data_client_test.dart` — green.
|
||||||
|
- [x] Tagged live test
|
||||||
|
`server/test/alpaca/alpaca_market_data_history_live_test.dart`
|
||||||
|
fetches 7-day bars for `SPY` and asserts ≥3 bars.
|
||||||
|
|
||||||
|
### 3.2 `MarketDataDb` — idempotent upsert + range query
|
||||||
|
|
||||||
|
#### 3.2.1 Red
|
||||||
|
|
||||||
|
- [x] Extend `server/test/integration/market_data_db_test.dart`:
|
||||||
|
- [x] Test: `upsertSnapshot(symbol:'SPY', metric:'bar',
|
||||||
|
timeframe:'1Day', as_of:T, price:500)` then re-upsert with
|
||||||
|
`price:505` → exactly **one** row remains; price is `505`; `raw`
|
||||||
|
is overwritten (volume also overwritten).
|
||||||
|
- [x] Test: `barsForSymbol(symbol, timeframe, since, until)` returns
|
||||||
|
rows ordered by `as_of ASC`; range is inclusive of `since`,
|
||||||
|
exclusive of `until`.
|
||||||
|
- [x] Test: `barsForSymbol` returns `[]` when no rows match; does not
|
||||||
|
throw.
|
||||||
|
- [x] Test: `latestSyncedAsOf(symbol, timeframe)` returns the newest
|
||||||
|
`as_of` or `null`.
|
||||||
|
|
||||||
|
#### 3.2.2 Green
|
||||||
|
|
||||||
|
- [x] Implement `MarketDataDb.upsertSnapshot(...)` using
|
||||||
|
`ON CONFLICT (symbol, metric, timeframe, as_of) DO UPDATE
|
||||||
|
SET price = EXCLUDED.price, volume = EXCLUDED.volume,
|
||||||
|
raw = EXCLUDED.raw`.
|
||||||
|
- [x] Implement `MarketDataDb.barsForSymbol(...)` and
|
||||||
|
`MarketDataDb.latestSyncedAsOf(...)`.
|
||||||
|
|
||||||
|
#### 3.2.3 Refactor
|
||||||
|
|
||||||
|
- [x] Replace existing `insertSnapshot` call sites in
|
||||||
|
`market_data_ingest.dart` with `upsertSnapshot` (tick data has
|
||||||
|
`timeframe='tick'`; same call shape). Re-run
|
||||||
|
`test/integration/market_data_ingest_test.dart` — still green.
|
||||||
|
|
||||||
|
#### 3.2.4 Confirm
|
||||||
|
|
||||||
|
- [x] `dart test test/integration/market_data_db_test.dart
|
||||||
|
test/integration/market_data_ingest_test.dart` — green.
|
||||||
|
|
||||||
|
### 3.3 `MarketDataHistorySync`
|
||||||
|
|
||||||
|
#### 3.3.1 Red
|
||||||
|
|
||||||
|
- [x] Add fixture
|
||||||
|
`server/test/fixtures/alpaca_bars_7d_3symbols.json` — 7 bars × 3
|
||||||
|
symbols (SPY/AAPL/MSFT), realistic timestamps.
|
||||||
|
- [x] Add `server/test/integration/market_data_history_sync_test.dart`:
|
||||||
|
- [x] Test: with mocked Alpaca returning the fixture → 21 rows upserted
|
||||||
|
with `metric='bar'`, `timeframe='1Day'`; sync run row written.
|
||||||
|
- [x] Test: re-running with the same fixture → still 21 rows; zero
|
||||||
|
duplicates; `rows_written` reflects rows touched (not inserted).
|
||||||
|
- [x] Test: partial outage — Alpaca returns 200 for batch 1
|
||||||
|
(AAPL/MSFT), 500 for batch 2 (SPY) → AAPL/MSFT rows persisted;
|
||||||
|
sync run row has `error` mentioning SPY; method does NOT throw.
|
||||||
|
- [x] Test: respects `HISTORY_SYNC_MAX_SYMBOLS` cap (set to 2 → only
|
||||||
|
first 2 symbols fetched).
|
||||||
|
- [x] Test: batching — with `HISTORY_SYNC_BATCH_SIZE=2` and 5 symbols,
|
||||||
|
Alpaca is called 3 times (mock call counter).
|
||||||
|
|
||||||
|
#### 3.3.2 Green
|
||||||
|
|
||||||
|
- [x] Implement `MarketDataHistorySync.runOnce({int windowDays = 7})`:
|
||||||
|
- [x] Reads symbols from
|
||||||
|
`TradableAssetsDb.listActiveTradableSymbols()`.
|
||||||
|
- [x] Batches into `HISTORY_SYNC_BATCH_SIZE` groups; calls
|
||||||
|
`getBarsRange` per batch.
|
||||||
|
- [x] Upserts via `MarketDataDb.upsertSnapshot`.
|
||||||
|
- [x] Captures per-batch errors without aborting; aggregates them into
|
||||||
|
the sync run row (`SyncRunCounts.error`).
|
||||||
|
|
||||||
|
#### 3.3.3 Refactor
|
||||||
|
|
||||||
|
- [x] Extract batching helper if used by §3.4 incremental path too.
|
||||||
|
_(Landed as `chunkList` in `market_data_history.dart`.)_
|
||||||
|
|
||||||
|
#### 3.3.4 Confirm
|
||||||
|
|
||||||
|
- [x] `dart test test/integration/market_data_history_sync_test.dart`
|
||||||
|
— green.
|
||||||
|
|
||||||
|
### 3.4 Incremental daily catch-up
|
||||||
|
|
||||||
|
#### 3.4.1 Red
|
||||||
|
|
||||||
|
- [x] Extend `market_data_history_sync_test.dart`:
|
||||||
|
- [x] Test: with prior `latestSyncedAsOf(symbol)` = `T-2d`, sync issues
|
||||||
|
bars with `start = T-2d` (not `T-7d`); mock HTTP call records
|
||||||
|
the requested start.
|
||||||
|
- [x] Test: with prior sync `T-10d` (outside window), `start` is
|
||||||
|
clamped to `T-windowDays`.
|
||||||
|
- [x] Test: cold start (no prior sync) → `start = T-windowDays`.
|
||||||
|
|
||||||
|
#### 3.4.2 Green
|
||||||
|
|
||||||
|
- [x] Compute per-symbol `start` using `latestSyncedAsOf`; pass to
|
||||||
|
`getBarsRange`.
|
||||||
|
|
||||||
|
#### 3.4.3 Refactor
|
||||||
|
|
||||||
|
- [x] If per-symbol starts vary inside a batch, fall back to
|
||||||
|
`min(starts)` for the batched call and let `upsertSnapshot`
|
||||||
|
dedupe the overlap — document the tradeoff in a code comment.
|
||||||
|
|
||||||
|
#### 3.4.4 Confirm
|
||||||
|
|
||||||
|
- [x] `dart test test/integration/market_data_history_sync_test.dart`
|
||||||
|
— green.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Retention & cleanup (older than 7 days)
|
||||||
|
|
||||||
|
**Files (new):** `server/lib/trading/market_data_retention.dart`.
|
||||||
|
|
||||||
|
### 4.1 Hard-delete (Phase 1)
|
||||||
|
|
||||||
|
#### 4.1.1 Red
|
||||||
|
|
||||||
|
- [x] Create `server/test/integration/market_data_retention_test.dart`:
|
||||||
|
- [x] Test: seed 10 snapshots spanning 14 days →
|
||||||
|
`runCleanup({windowDays: 7})` deletes rows with
|
||||||
|
`as_of < now() - 7d`, keeps the rest; returns `rowsRemoved`
|
||||||
|
matching deleted count.
|
||||||
|
- [x] Test: empty table → returns `rowsRemoved = 0`, does not throw.
|
||||||
|
- [x] Test: `batchSize` honored — with 5000 rows older than window and
|
||||||
|
`batchSize=1000`, the underlying `DELETE` is issued ≥5 times
|
||||||
|
(use a counting wrapper around `_connection.execute`).
|
||||||
|
- [x] Test: each invocation appends a `market_data_sync_runs` row
|
||||||
|
with `kind='cleanup'`, `rows_removed` populated.
|
||||||
|
- [x] Test: rows within window are NEVER touched (assert specific IDs
|
||||||
|
survive).
|
||||||
|
|
||||||
|
#### 4.1.2 Green
|
||||||
|
|
||||||
|
- [x] Implement
|
||||||
|
`MarketDataRetention.runCleanup({int windowDays = 7,
|
||||||
|
int batchSize = 5000})`:
|
||||||
|
- [x] Loop: `DELETE FROM market_data_snapshots WHERE as_of < $cutoff
|
||||||
|
LIMIT $batchSize` (use CTE if Postgres version requires it),
|
||||||
|
return rows removed; repeat until 0.
|
||||||
|
- [x] Write a `market_data_sync_runs` row around the operation.
|
||||||
|
|
||||||
|
#### 4.1.3 Refactor
|
||||||
|
|
||||||
|
- [x] Reuse `SyncRunRecorder` from §2.2.4.
|
||||||
|
|
||||||
|
#### 4.1.4 Confirm
|
||||||
|
|
||||||
|
- [x] `dart test test/integration/market_data_retention_test.dart`
|
||||||
|
— green.
|
||||||
|
|
||||||
|
### 4.2 Archive (Phase 2 — opt-in)
|
||||||
|
|
||||||
|
#### 4.2.1 Red
|
||||||
|
|
||||||
|
- [x] Extend `market_data_retention_test.dart`:
|
||||||
|
- [x] Test: with `archiveEnabled: true`, expired rows are copied into
|
||||||
|
`market_data_archive` with `archived_at = now()` BEFORE being
|
||||||
|
deleted; archive count grows by exactly `rowsRemoved`.
|
||||||
|
- [x] Test: archive run is transactional — if archive `INSERT` fails,
|
||||||
|
no `DELETE` happens; sync run row records the error.
|
||||||
|
- [x] Test: `archiveEnabled: false` (default) → archive table
|
||||||
|
untouched.
|
||||||
|
|
||||||
|
#### 4.2.2 Green
|
||||||
|
|
||||||
|
- [x] Uncomment `market_data_archive` table in migration 005 (or add it
|
||||||
|
now if you deferred it). _(Added `006_market_data_archive.sql`.)_
|
||||||
|
- [x] Implement
|
||||||
|
`MarketDataRetention.runArchiveAndCleanup({int windowDays})`
|
||||||
|
with explicit `BEGIN; INSERT … SELECT …; DELETE …; COMMIT`.
|
||||||
|
|
||||||
|
#### 4.2.3 Refactor
|
||||||
|
|
||||||
|
- [x] Consider a single unified entry point
|
||||||
|
`MarketDataRetention.run({int windowDays, bool archive})` that
|
||||||
|
dispatches; only do this if it doesn't muddy the failure-isolation
|
||||||
|
story.
|
||||||
|
|
||||||
|
#### 4.2.4 Confirm
|
||||||
|
|
||||||
|
- [x] `dart test test/integration/market_data_retention_test.dart`
|
||||||
|
— green.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Scheduler — daily cadence inside the worker
|
||||||
|
|
||||||
|
**Files:** new `server/lib/workers/market_history_scheduler.dart`,
|
||||||
|
extend `server/lib/workers/question_background_worker.dart`,
|
||||||
|
extend `server/bin/server.dart`.
|
||||||
|
|
||||||
|
### 5.1 Red
|
||||||
|
|
||||||
|
- [x] Add `server/test/integration/market_history_scheduler_test.dart`:
|
||||||
|
- [x] Test: cold start → `runIfDue(now=T0)` runs all 3 stages
|
||||||
|
(`universe`, `backfill`, `cleanup`) in that order;
|
||||||
|
`market_data_sync_runs` has 3 rows.
|
||||||
|
- [x] Test: same-day re-run (`now=T0 + 1h`) → no stages run; zero new
|
||||||
|
sync rows.
|
||||||
|
- [x] Test: next day (`now=T0 + 24h`) → all 3 stages run again.
|
||||||
|
- [x] Test: per-stage cadence — set
|
||||||
|
`MARKET_UNIVERSE_REFRESH_HOURS=48`, `MARKET_HISTORY_SYNC_HOURS=24`,
|
||||||
|
`MARKET_HISTORY_CLEANUP_HOURS=24`; at T0+24h only backfill +
|
||||||
|
cleanup run.
|
||||||
|
- [x] Test: failure isolation — backfill throws → cleanup still runs;
|
||||||
|
both stages logged in `market_data_sync_runs` (one with `error`,
|
||||||
|
one without).
|
||||||
|
- [x] Test: `MARKET_HISTORY_SYNC_HOUR_UTC=10` (optional alignment) →
|
||||||
|
scheduler runs only when local UTC hour ≥ 10 AND last run was on
|
||||||
|
a prior UTC day.
|
||||||
|
- [x] Add `server/test/integration/market_history_worker_wireup_test.dart`:
|
||||||
|
- [x] Test: `QuestionBackgroundWorker._tick` invokes
|
||||||
|
`MarketHistoryScheduler.runIfDue` **before** the
|
||||||
|
`TradingOrchestrator` per-user loop. Use a spy scheduler that
|
||||||
|
records the call order.
|
||||||
|
- [x] Test: scheduler exception is caught — worker tick continues into
|
||||||
|
the orchestrator loop; stderr contains the error.
|
||||||
|
|
||||||
|
### 5.2 Green
|
||||||
|
|
||||||
|
- [x] Implement `MarketHistoryScheduler` with `runIfDue(DateTime now)`,
|
||||||
|
reading the last `finished_at` per `kind` from
|
||||||
|
`market_data_sync_runs`.
|
||||||
|
- [x] Wire `QuestionBackgroundWorker` to accept an optional
|
||||||
|
`MarketHistoryScheduler` and call it at the top of `_tick`.
|
||||||
|
- [x] Wire `bin/server.dart` to construct the scheduler only when
|
||||||
|
`MARKET_HISTORY_SYNC_ENABLED=true && TRADING_ENABLED=true`.
|
||||||
|
|
||||||
|
### 5.3 Refactor
|
||||||
|
|
||||||
|
- [x] If the three stages each need similar before/after logic, abstract
|
||||||
|
a small `_runStage(kind, body)` inside the scheduler.
|
||||||
|
(`_maybeRunStage` — no further refactor needed.)
|
||||||
|
|
||||||
|
### 5.4 Confirm
|
||||||
|
|
||||||
|
- [x] `dart test test/integration/market_history_scheduler_test.dart
|
||||||
|
test/integration/market_history_worker_wireup_test.dart` — green.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Question pipeline — "guess the move" rule
|
||||||
|
|
||||||
|
**Files:** extend `server/lib/trading/rule_engine.dart`,
|
||||||
|
extend `server/lib/trading/trading_pipeline.dart`,
|
||||||
|
new `server/lib/trading/market_history_query.dart`.
|
||||||
|
|
||||||
|
The guessing game uses the rolling 7-day window — questions must reveal
|
||||||
|
just enough for the user to guess (obfuscated symbol/price/direction).
|
||||||
|
**No trade is placed for this rule** — answers feed scoring only.
|
||||||
|
|
||||||
|
### 6.1 Red — `MarketHistoryQuery`
|
||||||
|
|
||||||
|
- [x] Add `server/test/integration/market_history_query_test.dart`:
|
||||||
|
- [x] Test: `weeklyMovers({minBars: 5, asOf})` returns only symbols
|
||||||
|
with ≥5 daily bars in the window; each entry exposes
|
||||||
|
`(symbol, openClose, currentClose, days)`.
|
||||||
|
- [x] Test: deterministic — supply a `random: Random(42)` and assert a
|
||||||
|
stable selection order across runs.
|
||||||
|
- [x] Test: symbols with stale data (newest bar > 2d old) are
|
||||||
|
excluded.
|
||||||
|
|
||||||
|
### 6.2 Red — rule engine extension
|
||||||
|
|
||||||
|
- [x] Add `server/test/trading/rule_engine_guess_weekly_move_test.dart`:
|
||||||
|
- [x] Test: rule kind `guess_weekly_move` with mocked
|
||||||
|
`MarketHistoryQuery` returning SPY {ref=500, current=510, days=5}
|
||||||
|
→ produces a `RuleEvaluation` with:
|
||||||
|
- obfuscated `symbol_token='ASSET_A'`,
|
||||||
|
- `correct_answer = 10` (up direction),
|
||||||
|
- `question_text` substituting `{{token}}`, `{{ref_price}}`,
|
||||||
|
`{{ref_days_ago}}`, NEVER `{{symbol}}`.
|
||||||
|
- [x] Test: down move (ref=510, current=500) → `correct_answer = -10`.
|
||||||
|
- [x] Test: insufficient bars → no fire.
|
||||||
|
- [x] Test: `questions.metadata.guess_symbol` is set to real symbol
|
||||||
|
(server-side only) when the question is created in §6.3.
|
||||||
|
|
||||||
|
### 6.3 Red — pipeline wiring
|
||||||
|
|
||||||
|
- [x] Add
|
||||||
|
`server/test/integration/trading_pipeline_guess_weekly_move_test.dart`:
|
||||||
|
- [x] Test: end-to-end with seeded 7 daily bars for SPY → pipeline
|
||||||
|
creates a question with obfuscated text; `metadata.guess_symbol
|
||||||
|
= 'SPY'`; `pipeline_key='trading'`,
|
||||||
|
`pipeline_step='guess_weekly_move:await_answer'`.
|
||||||
|
- [x] Test: `onAnswerSubmitted` with matching direction (e.g., +10 on
|
||||||
|
an up move) records `score_delta = +1` in
|
||||||
|
`user_trading_state.context.guess_score`; non-matching records
|
||||||
|
`score_delta = -1`.
|
||||||
|
- [x] Test: `TradeActuator.processPendingOrders` is **NEVER called**
|
||||||
|
for `guess_weekly_move` answers (assert via spy).
|
||||||
|
- [x] Test: cooldown — after a fire, the same symbol is not re-picked
|
||||||
|
for `GUESS_COOLDOWN_HOURS` (default 24).
|
||||||
|
|
||||||
|
### 6.4 Green
|
||||||
|
|
||||||
|
- [x] Implement `MarketHistoryQuery.weeklyMovers({...})`.
|
||||||
|
- [x] Add rule kind to `RuleEngine` with the new template tokens.
|
||||||
|
- [x] Extend `TradingPipeline.evaluate` + `onAnswerSubmitted` for the
|
||||||
|
new rule kind, including the cooldown bookkeeping.
|
||||||
|
|
||||||
|
### 6.5 Refactor
|
||||||
|
|
||||||
|
- [x] If the token mapping (real symbol ↔ `ASSET_A`/`ASSET_B`/…) is used
|
||||||
|
in multiple places, lift it into a `SymbolObfuscator` helper with
|
||||||
|
its own focused unit test.
|
||||||
|
|
||||||
|
### 6.6 Confirm
|
||||||
|
|
||||||
|
- [x] `dart test test/integration/market_history_query_test.dart
|
||||||
|
test/trading/rule_engine_guess_weekly_move_test.dart
|
||||||
|
test/integration/trading_pipeline_guess_weekly_move_test.dart`
|
||||||
|
— green.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Env additions (`server/.env.example`)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Rolling history feature gate
|
||||||
|
MARKET_HISTORY_SYNC_ENABLED=false
|
||||||
|
MARKET_HISTORY_WINDOW_DAYS=7
|
||||||
|
MARKET_HISTORY_RETENTION_DAYS=7
|
||||||
|
MARKET_HISTORY_ARCHIVE_ENABLED=false
|
||||||
|
|
||||||
|
# Cadence (hours)
|
||||||
|
MARKET_UNIVERSE_REFRESH_HOURS=24
|
||||||
|
MARKET_HISTORY_SYNC_HOURS=24
|
||||||
|
MARKET_HISTORY_CLEANUP_HOURS=24
|
||||||
|
MARKET_HISTORY_SYNC_HOUR_UTC=10 # optional alignment hour
|
||||||
|
|
||||||
|
# Batching / safety
|
||||||
|
HISTORY_SYNC_BATCH_SIZE=50
|
||||||
|
HISTORY_SYNC_MAX_SYMBOLS=2000 # hard cap; Alpaca Basic-friendly
|
||||||
|
MIN_BARS_FOR_GUESS=5
|
||||||
|
GUESS_COOLDOWN_HOURS=24
|
||||||
|
```
|
||||||
|
|
||||||
|
### 7.1 Red
|
||||||
|
|
||||||
|
- [x] Add `server/test/env/market_history_env_test.dart`:
|
||||||
|
- [x] Test: defaults parsed when env empty (`enabled=false`,
|
||||||
|
`windowDays=7`, etc.).
|
||||||
|
- [x] Test: `MARKET_HISTORY_SYNC_ENABLED=true` while
|
||||||
|
`TRADING_ENABLED=false` → `Env.assertConsistent()` throws.
|
||||||
|
- [x] Test: `MARKET_HISTORY_WINDOW_DAYS=0` or negative → throws.
|
||||||
|
- [x] Test: `MARKET_HISTORY_SYNC_HOUR_UTC=24` → throws (valid range
|
||||||
|
`0..23`).
|
||||||
|
|
||||||
|
### 7.2 Green
|
||||||
|
|
||||||
|
- [x] Extend `server/lib/env.dart` to load and validate these vars.
|
||||||
|
- [x] Append the block above to `server/.env.example`.
|
||||||
|
|
||||||
|
### 7.3 Refactor
|
||||||
|
|
||||||
|
- [x] If `Env` has grown unwieldy, split market-history vars into
|
||||||
|
`MarketHistoryEnv` (typed value object) and have `Env` expose it.
|
||||||
|
|
||||||
|
### 7.4 Confirm
|
||||||
|
|
||||||
|
- [x] `dart test test/env/market_history_env_test.dart` — green.
|
||||||
|
- [x] Document each var in `server/README.md` under a new
|
||||||
|
**"Market history window"** subsection.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Operational tooling
|
||||||
|
|
||||||
|
### 8.1 Red
|
||||||
|
|
||||||
|
- [ ] Add `server/test/bin/sync_market_history_smoke_test.dart`:
|
||||||
|
- [ ] Test: imports `bin/sync_market_history.dart` `main` function and
|
||||||
|
runs it with `QUESTION_PIPELINE_TEST_MODE=true` + a fake DB →
|
||||||
|
exits 0 and emits the expected one-line log.
|
||||||
|
- [ ] Add equivalent smoke test for `bin/cleanup_market_history.dart`.
|
||||||
|
|
||||||
|
### 8.2 Green
|
||||||
|
|
||||||
|
- [ ] Add CLI `server/bin/sync_market_history.dart` with
|
||||||
|
`--window=<days>` flag (default 7); honors test mode.
|
||||||
|
- [ ] Add CLI `server/bin/cleanup_market_history.dart` with
|
||||||
|
`--window=<days>` and `--archive` flags.
|
||||||
|
- [ ] Add structured one-line log:
|
||||||
|
`kind=… symbols=… rows_written=… rows_removed=… duration_ms=… error=…`.
|
||||||
|
|
||||||
|
### 8.3 Refactor
|
||||||
|
|
||||||
|
- [ ] Share argument parsing between the two CLIs if duplicated.
|
||||||
|
|
||||||
|
### 8.4 Confirm
|
||||||
|
|
||||||
|
- [ ] `dart test test/bin/` — green.
|
||||||
|
- [ ] Manual: `dart run server:bin/sync_market_history.dart --window=7`
|
||||||
|
against `cyberhybridhub_test` works end-to-end.
|
||||||
|
|
||||||
|
### 8.5 Optional admin endpoint (defer until needed)
|
||||||
|
|
||||||
|
- [ ] Behind Firebase admin auth, `POST /v1/admin/market-data/resync?window=7`
|
||||||
|
enqueues a sync run; **not exposed to Flutter**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Test pyramid for this milestone
|
||||||
|
|
||||||
|
| Layer | Test files |
|
||||||
|
|------|------------|
|
||||||
|
| Unit | `test/alpaca/alpaca_assets_client_test.dart` |
|
||||||
|
| Unit | `test/alpaca/alpaca_market_data_client_test.dart` (extended) |
|
||||||
|
| Unit | `test/trading/rule_engine_guess_weekly_move_test.dart` |
|
||||||
|
| Unit | `test/env/market_history_env_test.dart` |
|
||||||
|
| DB integration | `test/integration/market_history_schema_test.dart` |
|
||||||
|
| DB integration | `test/integration/tradable_assets_db_test.dart` |
|
||||||
|
| DB integration | `test/integration/tradable_assets_sync_test.dart` |
|
||||||
|
| DB integration | `test/integration/market_data_db_test.dart` (extended) |
|
||||||
|
| DB integration | `test/integration/market_data_history_sync_test.dart` |
|
||||||
|
| DB integration | `test/integration/market_data_retention_test.dart` |
|
||||||
|
| DB integration | `test/integration/market_history_query_test.dart` |
|
||||||
|
| DB integration | `test/integration/trading_pipeline_guess_weekly_move_test.dart` |
|
||||||
|
| Worker integration | `test/integration/market_history_scheduler_test.dart` |
|
||||||
|
| Worker integration | `test/integration/market_history_worker_wireup_test.dart` |
|
||||||
|
| Bin smoke | `test/bin/sync_market_history_smoke_test.dart` |
|
||||||
|
| Bin smoke | `test/bin/cleanup_market_history_smoke_test.dart` |
|
||||||
|
| Tagged (`alpaca`) | `test/alpaca/alpaca_assets_live_test.dart` |
|
||||||
|
| Tagged (`alpaca`) | `test/alpaca/alpaca_market_data_history_live_test.dart` |
|
||||||
|
|
||||||
|
**CI gating:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# default job — no Alpaca keys, must pass on every PR
|
||||||
|
cd server && dart test
|
||||||
|
|
||||||
|
# nightly / manual — requires ALPACA_API_KEY_ID / ALPACA_API_SECRET_KEY
|
||||||
|
cd server && dart test --tags=alpaca
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Acceptance criteria (Gate H — Rolling history)
|
||||||
|
|
||||||
|
- [ ] `market_data_snapshots` contains rows for every active tradable with
|
||||||
|
`as_of` within the last 7 days, and no rows older.
|
||||||
|
- [ ] Re-running backfill is a no-op (zero duplicate rows; deterministic
|
||||||
|
`rows_written` count when nothing changed upstream).
|
||||||
|
- [ ] Cleanup removes only rows older than the window and never touches
|
||||||
|
newer rows.
|
||||||
|
- [ ] Worker performs one full cycle (universe → backfill → cleanup) per
|
||||||
|
day with stage isolation; failure in one stage does not block the
|
||||||
|
others.
|
||||||
|
- [ ] A `guess_weekly_move` question can be generated end-to-end from
|
||||||
|
pure DB data — no live Alpaca call at evaluation time.
|
||||||
|
- [ ] `dart test` is green; `dart test --tags=alpaca` is green when keys
|
||||||
|
are present.
|
||||||
|
- [ ] `MARKET_HISTORY_SYNC_ENABLED=false` is the default; nothing runs
|
||||||
|
unless explicitly enabled.
|
||||||
|
- [ ] Safety: `MARKET_HISTORY_SYNC_ENABLED=true` without
|
||||||
|
`TRADING_ENABLED=true` fails fast at server boot.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Risks & mitigations
|
||||||
|
|
||||||
|
| Risk | Mitigation |
|
||||||
|
|------|------------|
|
||||||
|
| Alpaca rate limits on full-universe pull | Batched `bars` calls (`HISTORY_SYNC_BATCH_SIZE`); per-batch error isolation; 429 → exception logged in sync run, retry next day. |
|
||||||
|
| Migration deadlocks on large `market_data_snapshots` | Cleanup batches via `LIMIT` + loop; unique constraint added with `NOT VALID` then `VALIDATE CONSTRAINT` if existing dataset is huge (document in migration). |
|
||||||
|
| Duplicate Alpaca asset entries between runs | `upsertAll` PK-on-symbol; we mark missing symbols inactive instead of deleting. |
|
||||||
|
| Guessing game leaks the real symbol | Question text uses tokens only; real symbol lives in `questions.metadata` (server side); add a regex test that scans `question_text` for any known ticker. |
|
||||||
|
| Backfill blowing past disk budget | Hard caps via `HISTORY_SYNC_MAX_SYMBOLS` and `MARKET_HISTORY_WINDOW_DAYS`; retention deletes daily so steady-state size is bounded. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. Progress log
|
||||||
|
|
||||||
|
<!-- Newest entries on top. One line per completed step. -->
|
||||||
|
|
||||||
|
| Date | Step | Result |
|
||||||
|
|------|------|--------|
|
||||||
|
| 2026-05-26 | §7 Env additions | Green: 6/6 env tests; `dart test` 133/133. `MarketHistoryEnv.fromMap` + `assertConsistent`; `ServerEnv.marketHistory`; wired scheduler/sync/retention/guess; `server/.env.example`; README **Market history window**. |
|
||||||
|
| 2026-05-26 | §6 Guess-the-move rule | Green: 12 new tests; `dart test` 127/127. `MarketHistoryQuery.weeklyMovers`; `RuleEngine.evaluateGuessWeeklyMove`; `SymbolObfuscator`; `TradingPipeline` scoring + per-symbol cooldown; `questions.metadata` migration `007`; no pending orders on guess answers. |
|
||||||
|
| 2026-05-26 | §5 Scheduler (worker cadence) | Green: 8/8 scheduler + wireup tests; `dart test` 115/115. `MarketHistoryScheduler.runIfDue` (per-kind cadence + optional `syncHourUtc`); worker calls scheduler before pipeline/trading; `server.dart` wires universe→backfill→cleanup when `MARKET_HISTORY_SYNC_ENABLED` + real Alpaca; `ServerEnv.marketHistorySyncEnabled`; `SyncRunRecorder` uses injected `now` for `finished_at`. |
|
||||||
|
| 2026-05-26 | §4 Retention & cleanup | Green: 8/8 retention tests; `dart test` 107/107. `MarketDataRetention.runCleanup` (batched hard-delete via CTE+RETURNING); `runArchiveAndCleanup` (transactional archive-then-delete); unified `run(archive:)`; migration `006_market_data_archive.sql`; reuses `SyncRunRecorder` kind=`cleanup`. |
|
||||||
|
| 2026-05-26 | §3 Historical backfill (1Day × 7d) | Green: 17 new tests (6 client + 4 db + 8 sync); `dart test` 99/99; live `alpaca_market_data_history_live_test` ≥3 SPY bars. `getBarsRange` + pagination; `upsertSnapshot`/`barsForSymbol`/`latestSyncedAsOf`; `MarketDataHistorySync` with incremental catch-up + partial batch errors via `SyncRunCounts.error`. Defaults in `MarketHistoryConfig` (batch=100). |
|
||||||
|
| 2026-05-26 | §2 Tradable-asset universe sync | Green: 11/11 §2 tests pass (5 client + 3 db + 3 sync); `dart test` 82/82 green; tagged live `alpaca_assets_live_test` returned >100 active us_equity assets. Refactor 2.1.3 lifted auth headers to `AlpacaEnv.authHeaders`; 2.2.4 lifted `SyncRunRecorder` for §3/§4 reuse. |
|
||||||
|
| 2026-05-26 | §1 Schema additions (migration `005_market_history.sql`) | Green: 5/5 schema tests pass; `dart test` 70/70 green; `\d market_data_snapshots` shows `timeframe` col + `market_data_snapshots_unique_obs` unique constraint. Archive stub deferred to §4.2 to keep `;`-split migration runner happy. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13. References
|
||||||
|
|
||||||
|
- Existing snapshot writer: `server/lib/trading/market_data_ingest.dart`
|
||||||
|
- Existing snapshot DB: `server/lib/trading/market_data_db.dart`
|
||||||
|
- Existing migration to extend: `server/migrations/004_trading.sql`
|
||||||
|
- Orchestrator hook point: `server/lib/trading/trading_orchestrator.dart`
|
||||||
|
- Worker hook point: `server/lib/workers/question_background_worker.dart`
|
||||||
|
- Plans: [`TRADING_DEVELOPMENT_PLAN.md`](./TRADING_DEVELOPMENT_PLAN.md),
|
||||||
|
[`TRADING_TDD_PLAN.md`](./TRADING_TDD_PLAN.md)
|
||||||
|
- Alpaca docs: [Market Data](https://docs.alpaca.markets/docs/market-data-api),
|
||||||
|
[Trading / Assets](https://docs.alpaca.markets/docs/trading-api),
|
||||||
|
[Bars](https://docs.alpaca.markets/reference/stockbars)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*Document version: 1.0 — Rolling 7-day market data window, cleanup, and
|
||||||
|
guessing-game question integration.*
|
||||||
@ -44,7 +44,7 @@ class MarketHistoryDayCoverage {
|
|||||||
json['slots'] as List<dynamic>? ?? <dynamic>[];
|
json['slots'] as List<dynamic>? ?? <dynamic>[];
|
||||||
return MarketHistoryDayCoverage(
|
return MarketHistoryDayCoverage(
|
||||||
date: DateTime.parse('${json['date']}T00:00:00Z').toUtc(),
|
date: DateTime.parse('${json['date']}T00:00:00Z').toUtc(),
|
||||||
slotsPerDay: (json['slotsPerDay'] as num?)?.toInt() ?? 6,
|
slotsPerDay: (json['slotsPerDay'] as num?)?.toInt() ?? 2,
|
||||||
completedSlots: (json['completedSlots'] as num?)?.toInt() ?? 0,
|
completedSlots: (json['completedSlots'] as num?)?.toInt() ?? 0,
|
||||||
fullySyncedSlots: (json['fullySyncedSlots'] as num?)?.toInt() ?? 0,
|
fullySyncedSlots: (json['fullySyncedSlots'] as num?)?.toInt() ?? 0,
|
||||||
slots: rawSlots
|
slots: rawSlots
|
||||||
@ -81,7 +81,7 @@ class MarketHistoryWeekCoverageReport {
|
|||||||
return MarketHistoryWeekCoverageReport(
|
return MarketHistoryWeekCoverageReport(
|
||||||
asOf: DateTime.parse(json['asOf'] as String).toUtc(),
|
asOf: DateTime.parse(json['asOf'] as String).toUtc(),
|
||||||
windowDays: (json['windowDays'] as num?)?.toInt() ?? 7,
|
windowDays: (json['windowDays'] as num?)?.toInt() ?? 7,
|
||||||
slotsPerDay: (json['slotsPerDay'] as num?)?.toInt() ?? 6,
|
slotsPerDay: (json['slotsPerDay'] as num?)?.toInt() ?? 2,
|
||||||
symbolCount: (json['symbolCount'] as num?)?.toInt() ?? 0,
|
symbolCount: (json['symbolCount'] as num?)?.toInt() ?? 0,
|
||||||
isConsistent: json['isConsistent'] as bool? ?? false,
|
isConsistent: json['isConsistent'] as bool? ?? false,
|
||||||
days: rawDays
|
days: rawDays
|
||||||
|
|||||||
@ -60,6 +60,9 @@ class QuestionAuditAsset {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// US RTH session-half slot length (9:30–12:45 and 12:45–16:00 ET).
|
||||||
|
const Duration _sessionHalfSlotDuration = Duration(hours: 3, minutes: 15);
|
||||||
|
|
||||||
class QuestionAuditReport {
|
class QuestionAuditReport {
|
||||||
const QuestionAuditReport({
|
const QuestionAuditReport({
|
||||||
required this.compareUntil,
|
required this.compareUntil,
|
||||||
@ -91,10 +94,10 @@ class QuestionAuditReport {
|
|||||||
return QuestionAuditReport(
|
return QuestionAuditReport(
|
||||||
compareUntil: compareUntil,
|
compareUntil: compareUntil,
|
||||||
newerSlotStart: json['newerSlotStart'] == null
|
newerSlotStart: json['newerSlotStart'] == null
|
||||||
? compareUntil.subtract(const Duration(hours: 4))
|
? compareUntil.subtract(_sessionHalfSlotDuration)
|
||||||
: DateTime.parse(json['newerSlotStart']! as String).toUtc(),
|
: DateTime.parse(json['newerSlotStart']! as String).toUtc(),
|
||||||
olderSlotStart: json['olderSlotStart'] == null
|
olderSlotStart: json['olderSlotStart'] == null
|
||||||
? compareUntil.subtract(const Duration(hours: 8))
|
? compareUntil.subtract(_sessionHalfSlotDuration * 2)
|
||||||
: DateTime.parse(json['olderSlotStart']! as String).toUtc(),
|
: DateTime.parse(json['olderSlotStart']! as String).toUtc(),
|
||||||
windowDays: (json['windowDays'] as num).toInt(),
|
windowDays: (json['windowDays'] as num).toInt(),
|
||||||
canStepOlder: json['canStepOlder'] as bool? ?? false,
|
canStepOlder: json['canStepOlder'] as bool? ?? false,
|
||||||
|
|||||||
@ -20,17 +20,10 @@ String formatRelativeTime(DateTime startedAt, {DateTime? now}) {
|
|||||||
|
|
||||||
String formatMarketHistorySlotWire(DateTime value) {
|
String formatMarketHistorySlotWire(DateTime value) {
|
||||||
final DateTime utc = value.toUtc();
|
final DateTime utc = value.toUtc();
|
||||||
final int slotHour = (utc.hour ~/ 4) * 4;
|
|
||||||
final DateTime slotStart = DateTime.utc(
|
|
||||||
utc.year,
|
|
||||||
utc.month,
|
|
||||||
utc.day,
|
|
||||||
slotHour,
|
|
||||||
);
|
|
||||||
String two(int n) => n.toString().padLeft(2, '0');
|
String two(int n) => n.toString().padLeft(2, '0');
|
||||||
return '${slotStart.year.toString().padLeft(4, '0')}-'
|
return '${utc.year.toString().padLeft(4, '0')}-'
|
||||||
'${two(slotStart.month)}-${two(slotStart.day)}T'
|
'${two(utc.month)}-${two(utc.day)}T'
|
||||||
'${two(slotStart.hour)}:${two(slotStart.minute)}:${two(slotStart.second)}Z';
|
'${two(utc.hour)}:${two(utc.minute)}:${two(utc.second)}Z';
|
||||||
}
|
}
|
||||||
|
|
||||||
String formatUtcTimestamp(DateTime? value) {
|
String formatUtcTimestamp(DateTime? value) {
|
||||||
|
|||||||
@ -4,7 +4,7 @@ import '../../theme/app_theme.dart';
|
|||||||
import '../models/question_audit_asset.dart';
|
import '../models/question_audit_asset.dart';
|
||||||
import '../services/market_history_admin_api.dart';
|
import '../services/market_history_admin_api.dart';
|
||||||
|
|
||||||
/// Scrollable audit of last-two 4-hour bar price and volume deltas per symbol.
|
/// Scrollable audit of last-two session-half bar price and volume deltas per symbol.
|
||||||
class MarketHistoryQuestionAuditSheet extends StatefulWidget {
|
class MarketHistoryQuestionAuditSheet extends StatefulWidget {
|
||||||
const MarketHistoryQuestionAuditSheet({
|
const MarketHistoryQuestionAuditSheet({
|
||||||
super.key,
|
super.key,
|
||||||
@ -517,7 +517,7 @@ class _SlotRow extends StatelessWidget {
|
|||||||
}
|
}
|
||||||
|
|
||||||
abstract final class _AuditFormat {
|
abstract final class _AuditFormat {
|
||||||
/// Older → newer UTC 4-hour slot starts being compared.
|
/// Older → newer session-half slot opens (UTC instants of 9:30 / 12:45 ET).
|
||||||
static String compareSlotRange({
|
static String compareSlotRange({
|
||||||
required DateTime older,
|
required DateTime older,
|
||||||
required DateTime newer,
|
required DateTime newer,
|
||||||
@ -525,20 +525,17 @@ abstract final class _AuditFormat {
|
|||||||
return '${_slotLabel(older)} – ${_slotLabel(newer)} UTC';
|
return '${_slotLabel(older)} – ${_slotLabel(newer)} UTC';
|
||||||
}
|
}
|
||||||
|
|
||||||
static String _slotLabel(DateTime asOf) {
|
static String _slotLabel(DateTime slotStart) {
|
||||||
final DateTime utc = asOf.toUtc();
|
final DateTime utc = slotStart.toUtc();
|
||||||
final String month = utc.month.toString().padLeft(2, '0');
|
final String month = utc.month.toString().padLeft(2, '0');
|
||||||
final String day = utc.day.toString().padLeft(2, '0');
|
final String day = utc.day.toString().padLeft(2, '0');
|
||||||
final String hour = utc.hour.toString().padLeft(2, '0');
|
final String hour = utc.hour.toString().padLeft(2, '0');
|
||||||
return '$month/$day $hour:00';
|
final String minute = utc.minute.toString().padLeft(2, '0');
|
||||||
|
return '$month/$day $hour:$minute';
|
||||||
}
|
}
|
||||||
|
|
||||||
static String slotTime(DateTime asOf) {
|
static String slotTime(DateTime slotStart) {
|
||||||
final DateTime utc = asOf.toUtc();
|
return '${_slotLabel(slotStart)} UTC';
|
||||||
final String month = utc.month.toString().padLeft(2, '0');
|
|
||||||
final String day = utc.day.toString().padLeft(2, '0');
|
|
||||||
final String hour = utc.hour.toString().padLeft(2, '0');
|
|
||||||
return '$month/$day ${hour}:00 UTC';
|
|
||||||
}
|
}
|
||||||
|
|
||||||
static String value(num n) {
|
static String value(num n) {
|
||||||
|
|||||||
@ -13,7 +13,7 @@ const List<String> _weekdayLabels = <String>[
|
|||||||
'Sun',
|
'Sun',
|
||||||
];
|
];
|
||||||
|
|
||||||
/// Mini 7-day UTC week view showing 4-hour slot sync health.
|
/// Mini 7-day Eastern week view showing session-half slot sync health.
|
||||||
class MarketHistoryWeekCoverageSheet extends StatelessWidget {
|
class MarketHistoryWeekCoverageSheet extends StatelessWidget {
|
||||||
const MarketHistoryWeekCoverageSheet({
|
const MarketHistoryWeekCoverageSheet({
|
||||||
super.key,
|
super.key,
|
||||||
@ -75,7 +75,7 @@ class MarketHistoryWeekCoverageSheet extends StatelessWidget {
|
|||||||
report.symbolCount == 0
|
report.symbolCount == 0
|
||||||
? 'No active tradable symbols to validate.'
|
? 'No active tradable symbols to validate.'
|
||||||
: report.isConsistent
|
: report.isConsistent
|
||||||
? 'All completed 4-hour slots are fully synced across '
|
? 'All completed session-half slots are fully synced across '
|
||||||
'${report.symbolCount} symbols (bars or no-data placeholders).'
|
'${report.symbolCount} symbols (bars or no-data placeholders).'
|
||||||
: 'Some completed slots are missing a bar or no-data placeholder '
|
: 'Some completed slots are missing a bar or no-data placeholder '
|
||||||
'for one or more symbols.',
|
'for one or more symbols.',
|
||||||
@ -97,7 +97,7 @@ class MarketHistoryWeekCoverageSheet extends StatelessWidget {
|
|||||||
),
|
),
|
||||||
const SizedBox(height: 12),
|
const SizedBox(height: 12),
|
||||||
Text(
|
Text(
|
||||||
'UTC · ${report.slotsPerDay} slots per day · '
|
'ET · ${report.slotsPerDay} slots per trading day · '
|
||||||
'${report.windowDays}-day window',
|
'${report.windowDays}-day window',
|
||||||
textAlign: TextAlign.center,
|
textAlign: TextAlign.center,
|
||||||
style: const TextStyle(
|
style: const TextStyle(
|
||||||
|
|||||||
@ -218,7 +218,9 @@ class SyncRunExpansionTile extends StatelessWidget {
|
|||||||
),
|
),
|
||||||
const SizedBox(height: 2),
|
const SizedBox(height: 2),
|
||||||
Text(
|
Text(
|
||||||
'${item.symbols.length} assets: ${item.symbols.join(', ')}',
|
item.symbols.length == 1
|
||||||
|
? '1 asset'
|
||||||
|
: '${item.symbols.length} assets',
|
||||||
style: const TextStyle(
|
style: const TextStyle(
|
||||||
fontSize: 12,
|
fontSize: 12,
|
||||||
color: AppColors.textSecondary,
|
color: AppColors.textSecondary,
|
||||||
|
|||||||
@ -129,8 +129,9 @@ curl -s -X POST http://localhost:3000/v1/me/incoming-question \
|
|||||||
|
|
||||||
## Market history
|
## Market history
|
||||||
|
|
||||||
Alpaca **`4Hour`** bars in six **UTC slots** per day (`00`, `04`, `08`, `12`, `16`, `20`).
|
Alpaca **`1Min`** bars aggregated into two **US regular-session** half-days per trading day
|
||||||
Stored as `metric=bar`, `timeframe=4Hour`. Rolling window: `MARKET_HISTORY_WINDOW_DAYS` (default 7).
|
(morning **9:30–12:45 ET**, afternoon **12:45–16:00 ET**, ~195 minutes each).
|
||||||
|
Stored as `metric=bar`, `timeframe=sessionHalf`. Rolling window: `MARKET_HISTORY_WINDOW_DAYS` (default 7).
|
||||||
|
|
||||||
**Backfill** (`kind=backfill`): fetches each **ended** slot still missing in DB; skips the open slot.
|
**Backfill** (`kind=backfill`): fetches each **ended** slot still missing in DB; skips the open slot.
|
||||||
Throttled to `MARKET_HISTORY_API_REQUESTS_PER_MINUTE` (default 200/min). On HTTP 429: wait 1 minute, retry once; if still limited, save partial rows and resume next tick.
|
Throttled to `MARKET_HISTORY_API_REQUESTS_PER_MINUTE` (default 200/min). On HTTP 429: wait 1 minute, retry once; if still limited, save partial rows and resume next tick.
|
||||||
@ -142,7 +143,9 @@ crashed prior sync cannot block new work.
|
|||||||
|
|
||||||
Requires `TRADING_ENABLED=true` when `MARKET_HISTORY_SYNC_ENABLED=true`.
|
Requires `TRADING_ENABLED=true` when `MARKET_HISTORY_SYNC_ENABLED=true`.
|
||||||
|
|
||||||
**Migration `008`:** drops legacy `1Day` history bars, adds `timeframe` check (`4Hour` allowed), partial index on `4Hour` bars, `market_data_sync_runs.slots_synced`.
|
**Migration `008`:** legacy `1Day` / `4Hour` history cleanup and `slots_synced` on sync runs.
|
||||||
|
|
||||||
|
**Migration `010`:** deletes `4Hour` bars, adds `sessionHalf` timeframe + partial index.
|
||||||
|
|
||||||
**Migration `009`:** adds `market_data_sync_runs.backfill_items` JSONB — per-slot UTC start + symbol list for each backfill run.
|
**Migration `009`:** adds `market_data_sync_runs.backfill_items` JSONB — per-slot UTC start + symbol list for each backfill run.
|
||||||
|
|
||||||
|
|||||||
@ -4,7 +4,7 @@ import 'package:http/http.dart' as http;
|
|||||||
|
|
||||||
import 'alpaca_env.dart';
|
import 'alpaca_env.dart';
|
||||||
import 'alpaca_models.dart';
|
import 'alpaca_models.dart';
|
||||||
import '../trading/market_history_four_hour_slot.dart';
|
import '../trading/market_history_session_slot.dart';
|
||||||
|
|
||||||
/// REST client for Alpaca Market Data API v2 (IEX feed on Basic plan).
|
/// REST client for Alpaca Market Data API v2 (IEX feed on Basic plan).
|
||||||
class AlpacaMarketDataClient {
|
class AlpacaMarketDataClient {
|
||||||
@ -106,8 +106,8 @@ class AlpacaMarketDataClient {
|
|||||||
final Map<String, String> query = <String, String>{
|
final Map<String, String> query = <String, String>{
|
||||||
'symbols': symbols.join(','),
|
'symbols': symbols.join(','),
|
||||||
'timeframe': timeframe,
|
'timeframe': timeframe,
|
||||||
'start': MarketHistoryFourHourSlot.wireUtc(start),
|
'start': MarketHistorySessionSlot.wireUtc(start),
|
||||||
'end': MarketHistoryFourHourSlot.wireUtc(end),
|
'end': MarketHistorySessionSlot.wireUtc(end),
|
||||||
'feed': _env.dataFeed,
|
'feed': _env.dataFeed,
|
||||||
'limit': limit.toString(),
|
'limit': limit.toString(),
|
||||||
if (pageToken != null && pageToken.isNotEmpty) 'page_token': pageToken,
|
if (pageToken != null && pageToken.isNotEmpty) 'page_token': pageToken,
|
||||||
|
|||||||
@ -1,4 +1,4 @@
|
|||||||
import 'market_history_four_hour_slot.dart';
|
import 'market_history_session_slot.dart';
|
||||||
|
|
||||||
/// One Alpaca backfill request bucket: a UTC 4-hour slot and its symbols.
|
/// One Alpaca backfill request bucket: a UTC 4-hour slot and its symbols.
|
||||||
class BackfillSyncItem {
|
class BackfillSyncItem {
|
||||||
@ -11,7 +11,7 @@ class BackfillSyncItem {
|
|||||||
final List<String> symbols;
|
final List<String> symbols;
|
||||||
|
|
||||||
Map<String, dynamic> toJson() => <String, dynamic>{
|
Map<String, dynamic> toJson() => <String, dynamic>{
|
||||||
'slotStart': MarketHistoryFourHourSlot.slotStartWire(slotStart),
|
'slotStart': MarketHistorySessionSlot.slotStartWire(slotStart),
|
||||||
'symbols': symbols,
|
'symbols': symbols,
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|||||||
@ -3,7 +3,7 @@ import 'dart:convert';
|
|||||||
import 'package:postgres/postgres.dart';
|
import 'package:postgres/postgres.dart';
|
||||||
|
|
||||||
import 'market_history_bar_placeholder.dart';
|
import 'market_history_bar_placeholder.dart';
|
||||||
import 'market_history_four_hour_slot.dart';
|
import 'market_history_session_slot.dart';
|
||||||
|
|
||||||
/// Normalized market data row persisted for rule evaluation.
|
/// Normalized market data row persisted for rule evaluation.
|
||||||
class MarketDataSnapshot {
|
class MarketDataSnapshot {
|
||||||
@ -105,7 +105,7 @@ class MarketDataDb {
|
|||||||
return _rowToSnapshot(result.first);
|
return _rowToSnapshot(result.first);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Tombstone when Alpaca has no 4Hour bar for [symbol] at [slotStart].
|
/// Tombstone when Alpaca has no session-half bar for [symbol] at [slotStart].
|
||||||
///
|
///
|
||||||
/// Counts toward backfill gap checks but not game/calendar bar coverage.
|
/// Counts toward backfill gap checks but not game/calendar bar coverage.
|
||||||
Future<MarketDataSnapshot> upsertNoDataBarPlaceholder({
|
Future<MarketDataSnapshot> upsertNoDataBarPlaceholder({
|
||||||
@ -118,8 +118,8 @@ class MarketDataDb {
|
|||||||
String source = MarketHistoryBarPlaceholder.sourceAlpacaEmpty,
|
String source = MarketHistoryBarPlaceholder.sourceAlpacaEmpty,
|
||||||
}) async {
|
}) async {
|
||||||
final DateTime slot =
|
final DateTime slot =
|
||||||
MarketHistoryFourHourSlot.slotStartContaining(slotStart);
|
MarketHistorySessionSlot.slotStartContaining(slotStart);
|
||||||
final String slotWire = MarketHistoryFourHourSlot.slotStartWire(slot);
|
final String slotWire = MarketHistorySessionSlot.slotStartWire(slot);
|
||||||
return upsertSnapshot(
|
return upsertSnapshot(
|
||||||
symbol: symbol,
|
symbol: symbol,
|
||||||
metric: 'bar',
|
metric: 'bar',
|
||||||
@ -131,7 +131,7 @@ class MarketDataDb {
|
|||||||
'slot_start': slotWire,
|
'slot_start': slotWire,
|
||||||
MarketHistoryBarPlaceholder.rawKey: true,
|
MarketHistoryBarPlaceholder.rawKey: true,
|
||||||
'source': source,
|
'source': source,
|
||||||
'checked_at': MarketHistoryFourHourSlot.wireUtc(checkedAt),
|
'checked_at': MarketHistorySessionSlot.wireUtc(checkedAt),
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
@ -201,11 +201,10 @@ class MarketDataDb {
|
|||||||
return (result.first[0]! as DateTime).toUtc();
|
return (result.first[0]! as DateTime).toUtc();
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Symbols from [symbols] that already have a bar for the UTC slot [slotStart].
|
/// Symbols from [symbols] that already have a bar for session slot [slotStart].
|
||||||
///
|
///
|
||||||
/// A row counts when [raw.slot_start] matches the canonical wire form, when
|
/// A row counts when [raw.slot_start] or [as_of] matches the canonical slot
|
||||||
/// [raw.slot_start] or [as_of] bucket to the same UTC 4-hour boundary as
|
/// start ([MarketHistorySessionSlot.slotStartWire]).
|
||||||
/// [slotStart] (same rule as [MarketHistoryFourHourSlot.slotStartContaining]).
|
|
||||||
Future<Set<String>> symbolsWithBarForSlot({
|
Future<Set<String>> symbolsWithBarForSlot({
|
||||||
required List<String> symbols,
|
required List<String> symbols,
|
||||||
required DateTime slotStart,
|
required DateTime slotStart,
|
||||||
@ -216,8 +215,8 @@ class MarketDataDb {
|
|||||||
return <String>{};
|
return <String>{};
|
||||||
}
|
}
|
||||||
final DateTime start =
|
final DateTime start =
|
||||||
MarketHistoryFourHourSlot.slotStartContaining(slotStart);
|
MarketHistorySessionSlot.slotStartContaining(slotStart);
|
||||||
final String slotStartWire = MarketHistoryFourHourSlot.slotStartWire(start);
|
final String slotStartWire = MarketHistorySessionSlot.slotStartWire(start);
|
||||||
final Result result = await _connection.execute(
|
final Result result = await _connection.execute(
|
||||||
Sql.named(
|
Sql.named(
|
||||||
'''
|
'''
|
||||||
@ -228,12 +227,7 @@ class MarketDataDb {
|
|||||||
AND symbol = ANY(@symbols)
|
AND symbol = ANY(@symbols)
|
||||||
AND (
|
AND (
|
||||||
raw->>'slot_start' = @slot_start_wire
|
raw->>'slot_start' = @slot_start_wire
|
||||||
OR (
|
OR as_of = @slot_start
|
||||||
raw->>'slot_start' IS NOT NULL
|
|
||||||
AND ${_slotStartBucketSql('(raw->>\'slot_start\')::timestamptz')}
|
|
||||||
= @slot_start
|
|
||||||
)
|
|
||||||
OR ${_slotStartBucketSql('as_of')} = @slot_start
|
|
||||||
)
|
)
|
||||||
''',
|
''',
|
||||||
),
|
),
|
||||||
@ -250,17 +244,6 @@ class MarketDataDb {
|
|||||||
.toSet();
|
.toSet();
|
||||||
}
|
}
|
||||||
|
|
||||||
/// UTC 4-hour slot left edge for [timestampExpr] (timestamptz SQL expression).
|
|
||||||
static String _slotStartBucketSql(String timestampExpr) {
|
|
||||||
return '''
|
|
||||||
(
|
|
||||||
date_trunc('day', $timestampExpr AT TIME ZONE 'UTC')
|
|
||||||
+ (div(extract(hour from $timestampExpr AT TIME ZONE 'UTC')::int, 4) * 4)
|
|
||||||
* interval '1 hour'
|
|
||||||
) AT TIME ZONE 'UTC'
|
|
||||||
''';
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Newest snapshot for [symbol] and [metric] by [as_of].
|
/// Newest snapshot for [symbol] and [metric] by [as_of].
|
||||||
Future<MarketDataSnapshot?> latestForSymbol(
|
Future<MarketDataSnapshot?> latestForSymbol(
|
||||||
String symbol,
|
String symbol,
|
||||||
|
|||||||
@ -7,7 +7,8 @@ import 'market_data_db.dart';
|
|||||||
import 'market_history_api_rate_limiter.dart';
|
import 'market_history_api_rate_limiter.dart';
|
||||||
import 'market_history_bar_placeholder.dart';
|
import 'market_history_bar_placeholder.dart';
|
||||||
import 'market_history_config.dart';
|
import 'market_history_config.dart';
|
||||||
import 'market_history_four_hour_slot.dart';
|
import 'market_history_minute_aggregate.dart';
|
||||||
|
import 'market_history_session_slot.dart';
|
||||||
import 'market_history_trading_calendar.dart';
|
import 'market_history_trading_calendar.dart';
|
||||||
import 'sync_run_recorder.dart';
|
import 'sync_run_recorder.dart';
|
||||||
import 'tradable_assets_db.dart';
|
import 'tradable_assets_db.dart';
|
||||||
@ -51,7 +52,7 @@ class PersistBarsResult {
|
|||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
final String slotWire = MarketHistoryFourHourSlot.slotStartWire(slotStart);
|
final String slotWire = MarketHistorySessionSlot.slotStartWire(slotStart);
|
||||||
final List<String> parts = <String>[
|
final List<String> parts = <String>[
|
||||||
'Alpaca returned no persistable $timeframe bars',
|
'Alpaca returned no persistable $timeframe bars',
|
||||||
'slot=$slotWire',
|
'slot=$slotWire',
|
||||||
@ -91,7 +92,7 @@ class MarketDataHistorySyncResult {
|
|||||||
final DateTime finishedAt;
|
final DateTime finishedAt;
|
||||||
final String? error;
|
final String? error;
|
||||||
|
|
||||||
/// Number of completed 4-hour slots written in this run.
|
/// Number of completed session-half slots written in this run.
|
||||||
final int slotsSynced;
|
final int slotsSynced;
|
||||||
|
|
||||||
bool get succeeded => error == null;
|
bool get succeeded => error == null;
|
||||||
@ -108,7 +109,8 @@ class MarketHistorySlotFetchPlan {
|
|||||||
final List<String> symbols;
|
final List<String> symbols;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Backfill: one Alpaca `4Hour` request per ended UTC slot × symbol batch.
|
/// Backfill: one Alpaca `1Min` range per ended session slot × symbol batch,
|
||||||
|
/// aggregated into `sessionHalf` rows.
|
||||||
///
|
///
|
||||||
/// Chooses work from the most recently completed slot backward until the rolling
|
/// Chooses work from the most recently completed slot backward until the rolling
|
||||||
/// window is full. Only symbols missing that specific slot are requested.
|
/// window is full. Only symbols missing that specific slot are requested.
|
||||||
@ -211,8 +213,8 @@ class MarketDataHistorySync {
|
|||||||
}
|
}
|
||||||
|
|
||||||
final DateTime slotStart = plan.slotStart;
|
final DateTime slotStart = plan.slotStart;
|
||||||
final DateTime slotEnd =
|
// Alpaca `end` is exclusive; use slot close, not last second of the window.
|
||||||
MarketHistoryFourHourSlot.endInclusive(slotStart);
|
final DateTime slotEnd = MarketHistorySessionSlot.endExclusive(slotStart);
|
||||||
bool slotWrote = false;
|
bool slotWrote = false;
|
||||||
|
|
||||||
for (final List<String> batch in chunkList(plan.symbols, batchSize)) {
|
for (final List<String> batch in chunkList(plan.symbols, batchSize)) {
|
||||||
@ -333,7 +335,7 @@ class MarketDataHistorySync {
|
|||||||
try {
|
try {
|
||||||
return await _marketDataClient.getBarsRange(
|
return await _marketDataClient.getBarsRange(
|
||||||
symbols: symbols,
|
symbols: symbols,
|
||||||
timeframe: timeframe,
|
timeframe: MarketHistoryConfig.alpacaFetchTimeframe,
|
||||||
start: slotStart,
|
start: slotStart,
|
||||||
end: slotEnd,
|
end: slotEnd,
|
||||||
);
|
);
|
||||||
@ -359,8 +361,8 @@ class MarketDataHistorySync {
|
|||||||
final List<String> emptyInResponse = <String>[];
|
final List<String> emptyInResponse = <String>[];
|
||||||
final Map<String, String> wrongSlotBarTimes = <String, String>{};
|
final Map<String, String> wrongSlotBarTimes = <String, String>{};
|
||||||
final DateTime plannedSlot =
|
final DateTime plannedSlot =
|
||||||
MarketHistoryFourHourSlot.slotStartContaining(slotStart);
|
MarketHistorySessionSlot.slotStartContaining(slotStart);
|
||||||
final String slotWire = MarketHistoryFourHourSlot.slotStartWire(plannedSlot);
|
final String slotWire = MarketHistorySessionSlot.slotStartWire(plannedSlot);
|
||||||
|
|
||||||
for (final String symbol in batch) {
|
for (final String symbol in batch) {
|
||||||
if (!response.barsBySymbol.containsKey(symbol)) {
|
if (!response.barsBySymbol.containsKey(symbol)) {
|
||||||
@ -379,39 +381,37 @@ class MarketDataHistorySync {
|
|||||||
if (!batchSymbols.contains(entry.key)) {
|
if (!batchSymbols.contains(entry.key)) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
final List<String> rejectedTimes = <String>[];
|
final SessionHalfBarAggregate? aggregate = aggregateMinuteBarsForSlot(
|
||||||
for (final AlpacaBar bar in entry.value) {
|
bars: entry.value,
|
||||||
final DateTime barAt = bar.timestamp.toUtc();
|
slotStart: plannedSlot,
|
||||||
final DateTime barSlot =
|
);
|
||||||
MarketHistoryFourHourSlot.slotStartContaining(barAt);
|
if (aggregate == null) {
|
||||||
if (!barSlot.isAtSameMomentAs(plannedSlot)) {
|
wrongSlotBarTimes[entry.key] = entry.value
|
||||||
rejectedTimes.add(MarketHistoryFourHourSlot.wireUtc(barAt));
|
.map((AlpacaBar b) => MarketHistorySessionSlot.wireUtc(b.timestamp))
|
||||||
continue;
|
.join(',');
|
||||||
}
|
continue;
|
||||||
await _marketDataDb.upsertSnapshot(
|
|
||||||
symbol: entry.key,
|
|
||||||
metric: 'bar',
|
|
||||||
timeframe: timeframe,
|
|
||||||
feed: feed,
|
|
||||||
price: bar.close,
|
|
||||||
volume: bar.volume,
|
|
||||||
asOf: barSlot,
|
|
||||||
raw: <String, dynamic>{
|
|
||||||
'o': bar.open,
|
|
||||||
'h': bar.high,
|
|
||||||
'l': bar.low,
|
|
||||||
'c': bar.close,
|
|
||||||
'v': bar.volume,
|
|
||||||
't': MarketHistoryFourHourSlot.wireUtc(barAt),
|
|
||||||
'slot_start': slotWire,
|
|
||||||
},
|
|
||||||
);
|
|
||||||
written++;
|
|
||||||
symbolsWritten.add(entry.key);
|
|
||||||
}
|
|
||||||
if (rejectedTimes.isNotEmpty && !symbolsWritten.contains(entry.key)) {
|
|
||||||
wrongSlotBarTimes[entry.key] = rejectedTimes.join(',');
|
|
||||||
}
|
}
|
||||||
|
await _marketDataDb.upsertSnapshot(
|
||||||
|
symbol: entry.key,
|
||||||
|
metric: 'bar',
|
||||||
|
timeframe: timeframe,
|
||||||
|
feed: feed,
|
||||||
|
price: aggregate.close,
|
||||||
|
volume: aggregate.volume,
|
||||||
|
asOf: plannedSlot,
|
||||||
|
raw: <String, dynamic>{
|
||||||
|
'o': aggregate.open,
|
||||||
|
'h': aggregate.high,
|
||||||
|
'l': aggregate.low,
|
||||||
|
'c': aggregate.close,
|
||||||
|
'v': aggregate.volume,
|
||||||
|
't': MarketHistorySessionSlot.wireUtc(aggregate.lastBarAt),
|
||||||
|
'slot_start': slotWire,
|
||||||
|
'minute_bars_count': aggregate.minuteBarCount,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
written++;
|
||||||
|
symbolsWritten.add(entry.key);
|
||||||
}
|
}
|
||||||
|
|
||||||
return PersistBarsResult(
|
return PersistBarsResult(
|
||||||
@ -473,7 +473,7 @@ class MarketDataHistorySync {
|
|||||||
List<String> symbols,
|
List<String> symbols,
|
||||||
) async {
|
) async {
|
||||||
final List<DateTime> completed =
|
final List<DateTime> completed =
|
||||||
MarketHistoryFourHourSlot.completedSlotStartsInWindow(now, windowDays);
|
MarketHistorySessionSlot.completedSlotStartsInWindow(now, windowDays);
|
||||||
if (completed.isEmpty) {
|
if (completed.isEmpty) {
|
||||||
return <MarketHistorySlotFetchPlan>[];
|
return <MarketHistorySlotFetchPlan>[];
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,4 +1,4 @@
|
|||||||
/// Marker rows written when Alpaca has no 4Hour bar for a symbol × slot.
|
/// Marker rows written when Alpaca has no session-half bar for a symbol × slot.
|
||||||
abstract final class MarketHistoryBarPlaceholder {
|
abstract final class MarketHistoryBarPlaceholder {
|
||||||
static const String rawKey = 'no_data';
|
static const String rawKey = 'no_data';
|
||||||
static const String sourceAlpacaEmpty = 'alpaca_empty';
|
static const String sourceAlpacaEmpty = 'alpaca_empty';
|
||||||
|
|||||||
@ -1,14 +1,14 @@
|
|||||||
/// Defaults for 4-hour slot market history ([MarketHistoryFourHourSlot]).
|
/// Defaults for RTH session-half market history ([MarketHistorySessionSlot]).
|
||||||
/// Env overrides via [MarketHistoryEnv] in [ServerEnv.load].
|
/// Env overrides via [MarketHistoryEnv] in [ServerEnv.load].
|
||||||
abstract final class MarketHistoryConfig {
|
abstract final class MarketHistoryConfig {
|
||||||
/// Rolling window length in calendar days (UTC).
|
/// Rolling window length in calendar days (UTC).
|
||||||
static const int windowDays = 7;
|
static const int windowDays = 7;
|
||||||
|
|
||||||
/// Alpaca bar aggregation for market-history backfill (six slots per UTC day).
|
/// Stored bar timeframe (two aggregates per US trading day).
|
||||||
static const String barTimeframe = '4Hour';
|
static const String barTimeframe = 'sessionHalf';
|
||||||
|
|
||||||
/// Width of each history slot in hours (`24 / slotsPerDay`).
|
/// Alpaca fetch granularity for backfill (aggregated into [barTimeframe]).
|
||||||
static const int slotHours = 4;
|
static const String alpacaFetchTimeframe = '1Min';
|
||||||
|
|
||||||
/// Symbols per Alpaca `GET /v2/stocks/bars` request (max ~100).
|
/// Symbols per Alpaca `GET /v2/stocks/bars` request (max ~100).
|
||||||
static const int historySyncBatchSize = 100;
|
static const int historySyncBatchSize = 100;
|
||||||
@ -16,7 +16,7 @@ abstract final class MarketHistoryConfig {
|
|||||||
/// Hard cap on symbols synced per run (Alpaca Basic rate-limit safety).
|
/// Hard cap on symbols synced per run (Alpaca Basic rate-limit safety).
|
||||||
static const int historySyncMaxSymbols = 2000;
|
static const int historySyncMaxSymbols = 2000;
|
||||||
|
|
||||||
/// Minimum 4-hour bars required before a symbol is eligible for the
|
/// Minimum session-half bars required before a symbol is eligible for the
|
||||||
/// guess-the-move question rule.
|
/// guess-the-move question rule.
|
||||||
static const int minBarsForGuess = 5;
|
static const int minBarsForGuess = 5;
|
||||||
|
|
||||||
|
|||||||
@ -1,89 +0,0 @@
|
|||||||
/// Six UTC 4-hour slots per day. Sync only [completedSlotStartsInWindow].
|
|
||||||
abstract final class MarketHistoryFourHourSlot {
|
|
||||||
static const int slotHours = 4;
|
|
||||||
static const int slotsPerDay = 24 ~/ slotHours;
|
|
||||||
static const String alpacaTimeframe = '4Hour';
|
|
||||||
|
|
||||||
/// Left edge of the 4-hour bucket containing [instant] (UTC).
|
|
||||||
static DateTime slotStartContaining(DateTime instant) {
|
|
||||||
final DateTime u = instant.toUtc();
|
|
||||||
final int slotHour = (u.hour ~/ slotHours) * slotHours;
|
|
||||||
return DateTime.utc(u.year, u.month, u.day, slotHour);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Inclusive end of the slot for Alpaca `start`/`end` (Option A: 00:00–03:59:59).
|
|
||||||
static DateTime endInclusive(DateTime slotStart) {
|
|
||||||
return slotStart
|
|
||||||
.add(const Duration(hours: slotHours))
|
|
||||||
.subtract(const Duration(seconds: 1));
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Exclusive end (current slot begins here).
|
|
||||||
static DateTime endExclusive(DateTime slotStart) {
|
|
||||||
return slotStart.add(const Duration(hours: slotHours));
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `true` when [now] is at or after the end of the slot that began at [slotStart].
|
|
||||||
static bool hasEnded(DateTime slotStart, DateTime now) {
|
|
||||||
return !now.toUtc().isBefore(endExclusive(slotStart));
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Start of the most recently completed slot (never the in-progress slot).
|
|
||||||
static DateTime lastCompletedSlotStart(DateTime now) {
|
|
||||||
final DateTime current = slotStartContaining(now);
|
|
||||||
return current.subtract(const Duration(hours: slotHours));
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Earliest slot start included in a [windowDays] rolling window ending at [now].
|
|
||||||
static DateTime windowFirstSlotStart(DateTime now, int windowDays) {
|
|
||||||
final DateTime windowStart =
|
|
||||||
now.toUtc().subtract(Duration(days: windowDays));
|
|
||||||
return slotStartContaining(windowStart);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Completed slot starts from the rolling window through [lastCompletedSlotStart].
|
|
||||||
static List<DateTime> completedSlotStartsInWindow(
|
|
||||||
DateTime now,
|
|
||||||
int windowDays,
|
|
||||||
) {
|
|
||||||
final DateTime last = lastCompletedSlotStart(now);
|
|
||||||
final DateTime first = windowFirstSlotStart(now, windowDays);
|
|
||||||
if (last.isBefore(first)) {
|
|
||||||
return <DateTime>[];
|
|
||||||
}
|
|
||||||
|
|
||||||
final List<DateTime> slots = <DateTime>[];
|
|
||||||
DateTime cursor = first;
|
|
||||||
while (!cursor.isAfter(last)) {
|
|
||||||
if (hasEnded(cursor, now)) {
|
|
||||||
slots.add(cursor);
|
|
||||||
}
|
|
||||||
cursor = cursor.add(const Duration(hours: slotHours));
|
|
||||||
}
|
|
||||||
return slots;
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Canonical UTC wire form: `YYYY-MM-DDTHH:MM:SSZ` (no fractional seconds).
|
|
||||||
///
|
|
||||||
/// Used for Alpaca bar-range query params and [raw.slot_start] / [raw.t] in
|
|
||||||
/// [market_data_snapshots] so fetch, persist, and gap checks always agree.
|
|
||||||
static String wireUtc(DateTime value) {
|
|
||||||
final DateTime u = value.toUtc();
|
|
||||||
String two(int n) => n.toString().padLeft(2, '0');
|
|
||||||
return '${u.year.toString().padLeft(4, '0')}-'
|
|
||||||
'${two(u.month)}-${two(u.day)}T'
|
|
||||||
'${two(u.hour)}:${two(u.minute)}:${two(u.second)}Z';
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Wire form for the left edge of the 4-hour slot containing [slotStart].
|
|
||||||
static String slotStartWire(DateTime slotStart) =>
|
|
||||||
wireUtc(slotStartContaining(slotStart));
|
|
||||||
|
|
||||||
/// Parses a [wireUtc] / Alpaca RFC3339 timestamp, or `null` when invalid.
|
|
||||||
static DateTime? parseWire(String? wire) {
|
|
||||||
if (wire == null || wire.isEmpty) {
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
return DateTime.tryParse(wire)?.toUtc();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
74
server/lib/trading/market_history_minute_aggregate.dart
Normal file
74
server/lib/trading/market_history_minute_aggregate.dart
Normal file
@ -0,0 +1,74 @@
|
|||||||
|
import '../alpaca/alpaca_models.dart';
|
||||||
|
import 'market_history_session_slot.dart';
|
||||||
|
|
||||||
|
/// OHLCV aggregate of Alpaca 1-minute bars for one session half.
|
||||||
|
class SessionHalfBarAggregate {
|
||||||
|
const SessionHalfBarAggregate({
|
||||||
|
required this.open,
|
||||||
|
required this.high,
|
||||||
|
required this.low,
|
||||||
|
required this.close,
|
||||||
|
required this.volume,
|
||||||
|
required this.minuteBarCount,
|
||||||
|
required this.firstBarAt,
|
||||||
|
required this.lastBarAt,
|
||||||
|
});
|
||||||
|
|
||||||
|
final num open;
|
||||||
|
final num high;
|
||||||
|
final num low;
|
||||||
|
final num close;
|
||||||
|
final num volume;
|
||||||
|
final int minuteBarCount;
|
||||||
|
final DateTime firstBarAt;
|
||||||
|
final DateTime lastBarAt;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Aggregates [bars] whose timestamps fall in [slotStart, endExclusive(slotStart)).
|
||||||
|
SessionHalfBarAggregate? aggregateMinuteBarsForSlot({
|
||||||
|
required List<AlpacaBar> bars,
|
||||||
|
required DateTime slotStart,
|
||||||
|
}) {
|
||||||
|
final DateTime planned = MarketHistorySessionSlot.slotStartContaining(slotStart);
|
||||||
|
final DateTime windowStart = planned;
|
||||||
|
final DateTime windowEnd = MarketHistorySessionSlot.endExclusive(planned);
|
||||||
|
|
||||||
|
final List<AlpacaBar> inWindow = bars
|
||||||
|
.where((AlpacaBar bar) {
|
||||||
|
final DateTime t = bar.timestamp.toUtc();
|
||||||
|
return !t.isBefore(windowStart) && t.isBefore(windowEnd);
|
||||||
|
})
|
||||||
|
.toList()
|
||||||
|
..sort(
|
||||||
|
(AlpacaBar a, AlpacaBar b) =>
|
||||||
|
a.timestamp.compareTo(b.timestamp),
|
||||||
|
);
|
||||||
|
|
||||||
|
if (inWindow.isEmpty) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
num high = inWindow.first.high;
|
||||||
|
num low = inWindow.first.low;
|
||||||
|
num volume = 0;
|
||||||
|
for (final AlpacaBar bar in inWindow) {
|
||||||
|
if (bar.high > high) {
|
||||||
|
high = bar.high;
|
||||||
|
}
|
||||||
|
if (bar.low < low) {
|
||||||
|
low = bar.low;
|
||||||
|
}
|
||||||
|
volume += bar.volume;
|
||||||
|
}
|
||||||
|
|
||||||
|
return SessionHalfBarAggregate(
|
||||||
|
open: inWindow.first.open,
|
||||||
|
high: high,
|
||||||
|
low: low,
|
||||||
|
close: inWindow.last.close,
|
||||||
|
volume: volume,
|
||||||
|
minuteBarCount: inWindow.length,
|
||||||
|
firstBarAt: inWindow.first.timestamp.toUtc(),
|
||||||
|
lastBarAt: inWindow.last.timestamp.toUtc(),
|
||||||
|
);
|
||||||
|
}
|
||||||
@ -5,7 +5,7 @@ import 'package:postgres/postgres.dart';
|
|||||||
import 'market_data_db.dart' show MarketDataDb;
|
import 'market_data_db.dart' show MarketDataDb;
|
||||||
import 'market_history_bar_placeholder.dart';
|
import 'market_history_bar_placeholder.dart';
|
||||||
import 'market_history_config.dart';
|
import 'market_history_config.dart';
|
||||||
import 'market_history_four_hour_slot.dart';
|
import 'market_history_session_slot.dart';
|
||||||
import 'tradable_assets_db.dart';
|
import 'tradable_assets_db.dart';
|
||||||
|
|
||||||
/// One 4-hour bar snapshot used in question audit comparisons.
|
/// One 4-hour bar snapshot used in question audit comparisons.
|
||||||
@ -139,33 +139,32 @@ class QuestionAuditPage {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Default view: last two **completed** 4-hour slots (newer = last completed).
|
/// Default view: last two **completed** session-half slots (newer = last completed).
|
||||||
DateTime questionAuditDefaultCompareUntil(DateTime now) {
|
DateTime questionAuditDefaultCompareUntil(DateTime now) {
|
||||||
final DateTime last =
|
final DateTime last =
|
||||||
MarketHistoryFourHourSlot.lastCompletedSlotStart(now.toUtc());
|
MarketHistorySessionSlot.lastCompletedSlotStart(now.toUtc());
|
||||||
return MarketHistoryFourHourSlot.endExclusive(last);
|
return MarketHistorySessionSlot.endExclusive(last);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Earliest [compareUntil] that still pairs two slots in the rolling window.
|
/// Earliest [compareUntil] that still pairs two slots in the rolling window.
|
||||||
DateTime questionAuditMinCompareUntil(DateTime now, int windowDays) {
|
DateTime questionAuditMinCompareUntil(DateTime now, int windowDays) {
|
||||||
final DateTime first = MarketHistoryFourHourSlot.windowFirstSlotStart(
|
final DateTime first = MarketHistorySessionSlot.windowFirstSlotStart(
|
||||||
now.toUtc(),
|
now.toUtc(),
|
||||||
windowDays,
|
windowDays,
|
||||||
);
|
);
|
||||||
final DateTime minNewerSlot = first.add(
|
final DateTime? minNewerSlot = MarketHistorySessionSlot.nextSlotStart(first);
|
||||||
const Duration(hours: MarketHistoryFourHourSlot.slotHours),
|
if (minNewerSlot == null) {
|
||||||
);
|
return MarketHistorySessionSlot.endExclusive(first);
|
||||||
return MarketHistoryFourHourSlot.endExclusive(minNewerSlot);
|
}
|
||||||
|
return MarketHistorySessionSlot.endExclusive(minNewerSlot);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The newer bar's slot start for a page keyed by [compareUntil].
|
/// The newer bar's slot start for a page keyed by [compareUntil].
|
||||||
///
|
///
|
||||||
/// [compareUntil] is always `endExclusive(newerSlotStart)`.
|
/// [compareUntil] is always `endExclusive(newerSlotStart)`.
|
||||||
DateTime questionAuditNewerSlotStart(DateTime compareUntil) {
|
DateTime questionAuditNewerSlotStart(DateTime compareUntil) {
|
||||||
return MarketHistoryFourHourSlot.slotStartContaining(
|
return MarketHistorySessionSlot.slotStartContaining(
|
||||||
compareUntil.toUtc().subtract(
|
compareUntil.toUtc().subtract(MarketHistorySessionSlot.slotDuration),
|
||||||
const Duration(hours: MarketHistoryFourHourSlot.slotHours),
|
|
||||||
),
|
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -179,7 +178,7 @@ DateTime snapQuestionAuditCompareUntil({
|
|||||||
if (r.isAfter(maxUntil)) {
|
if (r.isAfter(maxUntil)) {
|
||||||
return maxUntil;
|
return maxUntil;
|
||||||
}
|
}
|
||||||
final DateTime snapped = MarketHistoryFourHourSlot.endExclusive(
|
final DateTime snapped = MarketHistorySessionSlot.endExclusive(
|
||||||
questionAuditNewerSlotStart(r),
|
questionAuditNewerSlotStart(r),
|
||||||
);
|
);
|
||||||
if (snapped.isBefore(minUntil)) {
|
if (snapped.isBefore(minUntil)) {
|
||||||
@ -194,10 +193,11 @@ DateTime snapQuestionAuditCompareUntil({
|
|||||||
/// Pair of slot starts for the page keyed by [compareUntil].
|
/// Pair of slot starts for the page keyed by [compareUntil].
|
||||||
(DateTime newer, DateTime older) questionAuditSlotPair(DateTime compareUntil) {
|
(DateTime newer, DateTime older) questionAuditSlotPair(DateTime compareUntil) {
|
||||||
final DateTime newer = questionAuditNewerSlotStart(compareUntil);
|
final DateTime newer = questionAuditNewerSlotStart(compareUntil);
|
||||||
final DateTime older = newer.subtract(
|
final DateTime? prior = MarketHistorySessionSlot.previousSlotStart(newer);
|
||||||
const Duration(hours: MarketHistoryFourHourSlot.slotHours),
|
if (prior == null) {
|
||||||
);
|
return (newer, newer);
|
||||||
return (newer, older);
|
}
|
||||||
|
return (newer, prior);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Steps back: newer becomes previous older (e.g. #1 vs #2 → #2 vs #3).
|
/// Steps back: newer becomes previous older (e.g. #1 vs #2 → #2 vs #3).
|
||||||
@ -206,10 +206,12 @@ DateTime questionAuditStepOlderCompareUntil({
|
|||||||
required DateTime now,
|
required DateTime now,
|
||||||
}) {
|
}) {
|
||||||
final DateTime newerSlot = questionAuditNewerSlotStart(compareUntil);
|
final DateTime newerSlot = questionAuditNewerSlotStart(compareUntil);
|
||||||
final DateTime priorNewerSlot = newerSlot.subtract(
|
final DateTime? priorNewerSlot =
|
||||||
const Duration(hours: MarketHistoryFourHourSlot.slotHours),
|
MarketHistorySessionSlot.previousSlotStart(newerSlot);
|
||||||
);
|
if (priorNewerSlot == null) {
|
||||||
return MarketHistoryFourHourSlot.endExclusive(priorNewerSlot);
|
return compareUntil;
|
||||||
|
}
|
||||||
|
return MarketHistorySessionSlot.endExclusive(priorNewerSlot);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Steps forward one completed slot, capped at [maxUntil].
|
/// Steps forward one completed slot, capped at [maxUntil].
|
||||||
@ -218,11 +220,13 @@ DateTime questionAuditStepNewerCompareUntil({
|
|||||||
required DateTime maxUntil,
|
required DateTime maxUntil,
|
||||||
}) {
|
}) {
|
||||||
final DateTime newerSlot = questionAuditNewerSlotStart(compareUntil);
|
final DateTime newerSlot = questionAuditNewerSlotStart(compareUntil);
|
||||||
final DateTime nextNewerSlot = newerSlot.add(
|
final DateTime? nextNewerSlot =
|
||||||
const Duration(hours: MarketHistoryFourHourSlot.slotHours),
|
MarketHistorySessionSlot.nextSlotStart(newerSlot);
|
||||||
);
|
if (nextNewerSlot == null) {
|
||||||
|
return maxUntil;
|
||||||
|
}
|
||||||
final DateTime candidate =
|
final DateTime candidate =
|
||||||
MarketHistoryFourHourSlot.endExclusive(nextNewerSlot);
|
MarketHistorySessionSlot.endExclusive(nextNewerSlot);
|
||||||
return candidate.isAfter(maxUntil) ? maxUntil : candidate;
|
return candidate.isAfter(maxUntil) ? maxUntil : candidate;
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -303,7 +307,7 @@ class MarketHistoryQuestionAudit {
|
|||||||
return calendarMax;
|
return calendarMax;
|
||||||
}
|
}
|
||||||
final DateTime dataMax =
|
final DateTime dataMax =
|
||||||
MarketHistoryFourHourSlot.endExclusive(latestBarSlot);
|
MarketHistorySessionSlot.endExclusive(latestBarSlot);
|
||||||
return dataMax.isBefore(calendarMax) ? dataMax : calendarMax;
|
return dataMax.isBefore(calendarMax) ? dataMax : calendarMax;
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -325,7 +329,7 @@ class MarketHistoryQuestionAudit {
|
|||||||
if (result.isEmpty || result.first[0] == null) {
|
if (result.isEmpty || result.first[0] == null) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
return MarketHistoryFourHourSlot.slotStartContaining(
|
return MarketHistorySessionSlot.slotStartContaining(
|
||||||
(result.first[0]! as DateTime).toUtc(),
|
(result.first[0]! as DateTime).toUtc(),
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
@ -336,14 +340,14 @@ class MarketHistoryQuestionAudit {
|
|||||||
required DateTime olderSlotStart,
|
required DateTime olderSlotStart,
|
||||||
String timeframe = MarketHistoryConfig.barTimeframe,
|
String timeframe = MarketHistoryConfig.barTimeframe,
|
||||||
}) async {
|
}) async {
|
||||||
final DateTime newer = MarketHistoryFourHourSlot.slotStartContaining(
|
final DateTime newer = MarketHistorySessionSlot.slotStartContaining(
|
||||||
newerSlotStart.toUtc(),
|
newerSlotStart.toUtc(),
|
||||||
);
|
);
|
||||||
final DateTime older = MarketHistoryFourHourSlot.slotStartContaining(
|
final DateTime older = MarketHistorySessionSlot.slotStartContaining(
|
||||||
olderSlotStart.toUtc(),
|
olderSlotStart.toUtc(),
|
||||||
);
|
);
|
||||||
final String newerWire = MarketHistoryFourHourSlot.slotStartWire(newer);
|
final String newerWire = MarketHistorySessionSlot.slotStartWire(newer);
|
||||||
final String olderWire = MarketHistoryFourHourSlot.slotStartWire(older);
|
final String olderWire = MarketHistorySessionSlot.slotStartWire(older);
|
||||||
|
|
||||||
final List<String> active =
|
final List<String> active =
|
||||||
await _tradableAssetsDb.listActiveTradableSymbols();
|
await _tradableAssetsDb.listActiveTradableSymbols();
|
||||||
@ -384,12 +388,14 @@ class MarketHistoryQuestionAudit {
|
|||||||
<String, Map<DateTime, _BarRow>>{};
|
<String, Map<DateTime, _BarRow>>{};
|
||||||
for (final ResultRow row in rows) {
|
for (final ResultRow row in rows) {
|
||||||
final String symbol = row[0]! as String;
|
final String symbol = row[0]! as String;
|
||||||
final DateTime asOf = MarketHistoryFourHourSlot.slotStartContaining(
|
|
||||||
(row[1]! as DateTime).toUtc(),
|
|
||||||
);
|
|
||||||
final Map<String, dynamic>? raw = _decodeRaw(row[4]);
|
final Map<String, dynamic>? raw = _decodeRaw(row[4]);
|
||||||
bySymbol.putIfAbsent(symbol, () => <DateTime, _BarRow>{})[asOf] = _BarRow(
|
final DateTime slotKey = _canonicalSlotStart(
|
||||||
asOf: asOf,
|
(row[1]! as DateTime).toUtc(),
|
||||||
|
raw,
|
||||||
|
);
|
||||||
|
bySymbol.putIfAbsent(symbol, () => <DateTime, _BarRow>{})[slotKey] =
|
||||||
|
_BarRow(
|
||||||
|
asOf: slotKey,
|
||||||
closePrice: MarketDataDb.readOptionalNumeric(row[2]),
|
closePrice: MarketDataDb.readOptionalNumeric(row[2]),
|
||||||
volume: MarketDataDb.readOptionalNumeric(row[3]),
|
volume: MarketDataDb.readOptionalNumeric(row[3]),
|
||||||
raw: raw,
|
raw: raw,
|
||||||
@ -431,6 +437,20 @@ class MarketHistoryQuestionAudit {
|
|||||||
return assets;
|
return assets;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static DateTime _canonicalSlotStart(
|
||||||
|
DateTime asOf,
|
||||||
|
Map<String, dynamic>? raw,
|
||||||
|
) {
|
||||||
|
final String? wire = raw?['slot_start'] as String?;
|
||||||
|
if (wire != null) {
|
||||||
|
final DateTime? parsed = DateTime.tryParse(wire)?.toUtc();
|
||||||
|
if (parsed != null) {
|
||||||
|
return MarketHistorySessionSlot.slotStartContaining(parsed);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return MarketHistorySessionSlot.slotStartContaining(asOf);
|
||||||
|
}
|
||||||
|
|
||||||
Map<String, dynamic>? _decodeRaw(Object? rawValue) {
|
Map<String, dynamic>? _decodeRaw(Object? rawValue) {
|
||||||
if (rawValue == null) {
|
if (rawValue == null) {
|
||||||
return null;
|
return null;
|
||||||
|
|||||||
316
server/lib/trading/market_history_session_slot.dart
Normal file
316
server/lib/trading/market_history_session_slot.dart
Normal file
@ -0,0 +1,316 @@
|
|||||||
|
import 'package:timezone/data/latest.dart' as tz_data;
|
||||||
|
import 'package:timezone/timezone.dart' as tz;
|
||||||
|
|
||||||
|
import 'market_history_trading_calendar.dart';
|
||||||
|
|
||||||
|
bool _timezonesInitialized = false;
|
||||||
|
|
||||||
|
void ensureMarketHistoryTimezonesInitialized() {
|
||||||
|
if (_timezonesInitialized) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
tz_data.initializeTimeZones();
|
||||||
|
_timezonesInitialized = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Two US regular-session half-day slots (9:30–12:45 and 12:45–16:00 ET).
|
||||||
|
abstract final class MarketHistorySessionSlot {
|
||||||
|
static const int slotsPerDay = 2;
|
||||||
|
static const Duration slotDuration = Duration(hours: 3, minutes: 15);
|
||||||
|
static const String storedTimeframe = 'sessionHalf';
|
||||||
|
static const String alpacaFetchTimeframe = '1Min';
|
||||||
|
|
||||||
|
static const int _morningHour = 9;
|
||||||
|
static const int _morningMinute = 30;
|
||||||
|
static const int _afternoonHour = 12;
|
||||||
|
static const int _afternoonMinute = 45;
|
||||||
|
|
||||||
|
static tz.Location get _eastern {
|
||||||
|
ensureMarketHistoryTimezonesInitialized();
|
||||||
|
return tz.getLocation('America/New_York');
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Left edge of the session half containing [instant] (UTC).
|
||||||
|
static DateTime slotStartContaining(DateTime instant) {
|
||||||
|
final tz.TZDateTime ny = tz.TZDateTime.from(instant.toUtc(), _eastern);
|
||||||
|
final int minutes = ny.hour * 60 + ny.minute;
|
||||||
|
const int morningStart = _morningHour * 60 + _morningMinute;
|
||||||
|
const int afternoonStart = _afternoonHour * 60 + _afternoonMinute;
|
||||||
|
const int sessionEnd = 16 * 60;
|
||||||
|
|
||||||
|
if (minutes >= afternoonStart && minutes < sessionEnd) {
|
||||||
|
return _afternoonStartUtc(ny.year, ny.month, ny.day);
|
||||||
|
}
|
||||||
|
if (minutes >= morningStart && minutes < afternoonStart) {
|
||||||
|
return _morningStartUtc(ny.year, ny.month, ny.day);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (minutes >= sessionEnd) {
|
||||||
|
return _afternoonStartUtc(ny.year, ny.month, ny.day);
|
||||||
|
}
|
||||||
|
|
||||||
|
final (int, int, int)? prior = _previousTradingDay(ny.year, ny.month, ny.day);
|
||||||
|
if (prior == null) {
|
||||||
|
return _morningStartUtc(ny.year, ny.month, ny.day);
|
||||||
|
}
|
||||||
|
return _afternoonStartUtc(prior.$1, prior.$2, prior.$3);
|
||||||
|
}
|
||||||
|
|
||||||
|
static DateTime endInclusive(DateTime slotStart) {
|
||||||
|
return endExclusive(slotStart).subtract(const Duration(seconds: 1));
|
||||||
|
}
|
||||||
|
|
||||||
|
static DateTime endExclusive(DateTime slotStart) {
|
||||||
|
return slotStart.toUtc().add(slotDuration);
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool hasEnded(DateTime slotStart, DateTime now) {
|
||||||
|
return !now.toUtc().isBefore(endExclusive(slotStart));
|
||||||
|
}
|
||||||
|
|
||||||
|
static DateTime lastCompletedSlotStart(DateTime now) {
|
||||||
|
final tz.TZDateTime ny = tz.TZDateTime.from(now.toUtc(), _eastern);
|
||||||
|
final List<DateTime> candidates = <DateTime>[];
|
||||||
|
|
||||||
|
void addDay(int y, int m, int d) {
|
||||||
|
if (!_isTradingDayEt(y, m, d)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
final DateTime morning = _morningStartUtc(y, m, d);
|
||||||
|
final DateTime afternoon = _afternoonStartUtc(y, m, d);
|
||||||
|
if (hasEnded(afternoon, now)) {
|
||||||
|
candidates.add(afternoon);
|
||||||
|
}
|
||||||
|
if (hasEnded(morning, now)) {
|
||||||
|
candidates.add(morning);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
addDay(ny.year, ny.month, ny.day);
|
||||||
|
final (int, int, int)? priorDay = _previousTradingDay(ny.year, ny.month, ny.day);
|
||||||
|
if (priorDay != null) {
|
||||||
|
addDay(priorDay.$1, priorDay.$2, priorDay.$3);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (candidates.isEmpty) {
|
||||||
|
return _walkBackForLastCompleted(now);
|
||||||
|
}
|
||||||
|
candidates.sort();
|
||||||
|
return candidates.last;
|
||||||
|
}
|
||||||
|
|
||||||
|
static DateTime _walkBackForLastCompleted(DateTime now) {
|
||||||
|
var cursor = tz.TZDateTime.from(now.toUtc(), _eastern);
|
||||||
|
for (var i = 0; i < 366; i++) {
|
||||||
|
if (_isTradingDayEt(cursor.year, cursor.month, cursor.day)) {
|
||||||
|
final DateTime afternoon =
|
||||||
|
_afternoonStartUtc(cursor.year, cursor.month, cursor.day);
|
||||||
|
if (hasEnded(afternoon, now)) {
|
||||||
|
return afternoon;
|
||||||
|
}
|
||||||
|
final DateTime morning =
|
||||||
|
_morningStartUtc(cursor.year, cursor.month, cursor.day);
|
||||||
|
if (hasEnded(morning, now)) {
|
||||||
|
return morning;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
cursor = cursor.subtract(const Duration(days: 1));
|
||||||
|
}
|
||||||
|
return _morningStartUtc(cursor.year, cursor.month, cursor.day);
|
||||||
|
}
|
||||||
|
|
||||||
|
static DateTime windowFirstSlotStart(DateTime now, int windowDays) {
|
||||||
|
final tz.TZDateTime ny = tz.TZDateTime.from(now.toUtc(), _eastern);
|
||||||
|
var cursor = ny.subtract(Duration(days: windowDays));
|
||||||
|
for (var i = 0; i < windowDays + 14; i++) {
|
||||||
|
if (_isTradingDayEt(cursor.year, cursor.month, cursor.day)) {
|
||||||
|
return _morningStartUtc(cursor.year, cursor.month, cursor.day);
|
||||||
|
}
|
||||||
|
cursor = cursor.add(const Duration(days: 1));
|
||||||
|
}
|
||||||
|
return _morningStartUtc(ny.year, ny.month, ny.day);
|
||||||
|
}
|
||||||
|
|
||||||
|
static List<DateTime> completedSlotStartsInWindow(
|
||||||
|
DateTime now,
|
||||||
|
int windowDays,
|
||||||
|
) {
|
||||||
|
final DateTime last = lastCompletedSlotStart(now);
|
||||||
|
final DateTime first = windowFirstSlotStart(now, windowDays);
|
||||||
|
if (last.isBefore(first)) {
|
||||||
|
return <DateTime>[];
|
||||||
|
}
|
||||||
|
|
||||||
|
final List<DateTime> slots = <DateTime>[];
|
||||||
|
var cursor = tz.TZDateTime.from(first.toUtc(), _eastern);
|
||||||
|
final tz.TZDateTime endNy = tz.TZDateTime.from(last.toUtc(), _eastern);
|
||||||
|
|
||||||
|
while (true) {
|
||||||
|
if (_isTradingDayEt(cursor.year, cursor.month, cursor.day)) {
|
||||||
|
final DateTime morning =
|
||||||
|
_morningStartUtc(cursor.year, cursor.month, cursor.day);
|
||||||
|
final DateTime afternoon =
|
||||||
|
_afternoonStartUtc(cursor.year, cursor.month, cursor.day);
|
||||||
|
for (final DateTime slot in <DateTime>[morning, afternoon]) {
|
||||||
|
if (slot.isBefore(first)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (slot.isAfter(last)) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if (hasEnded(slot, now)) {
|
||||||
|
slots.add(slot);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (cursor.year > endNy.year ||
|
||||||
|
(cursor.year == endNy.year && cursor.month > endNy.month) ||
|
||||||
|
(cursor.year == endNy.year &&
|
||||||
|
cursor.month == endNy.month &&
|
||||||
|
cursor.day >= endNy.day)) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
cursor = cursor.add(const Duration(days: 1));
|
||||||
|
}
|
||||||
|
return slots;
|
||||||
|
}
|
||||||
|
|
||||||
|
static DateTime? previousSlotStart(DateTime slotStart) {
|
||||||
|
final DateTime snap = slotStartContaining(slotStart);
|
||||||
|
if (_isAfternoonStart(snap)) {
|
||||||
|
return _morningStartUtc(
|
||||||
|
_nyYear(snap),
|
||||||
|
_nyMonth(snap),
|
||||||
|
_nyDay(snap),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
final (int, int, int)? prior =
|
||||||
|
_previousTradingDay(_nyYear(snap), _nyMonth(snap), _nyDay(snap));
|
||||||
|
if (prior == null) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
return _afternoonStartUtc(prior.$1, prior.$2, prior.$3);
|
||||||
|
}
|
||||||
|
|
||||||
|
static DateTime? nextSlotStart(DateTime slotStart) {
|
||||||
|
final DateTime snap = slotStartContaining(slotStart);
|
||||||
|
if (_isMorningStart(snap)) {
|
||||||
|
return _afternoonStartUtc(
|
||||||
|
_nyYear(snap),
|
||||||
|
_nyMonth(snap),
|
||||||
|
_nyDay(snap),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
final (int, int, int)? next =
|
||||||
|
_nextTradingDay(_nyYear(snap), _nyMonth(snap), _nyDay(snap));
|
||||||
|
if (next == null) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
return _morningStartUtc(next.$1, next.$2, next.$3);
|
||||||
|
}
|
||||||
|
|
||||||
|
static String wireUtc(DateTime value) {
|
||||||
|
final DateTime u = value.toUtc();
|
||||||
|
String two(int n) => n.toString().padLeft(2, '0');
|
||||||
|
return '${u.year.toString().padLeft(4, '0')}-'
|
||||||
|
'${two(u.month)}-${two(u.day)}T'
|
||||||
|
'${two(u.hour)}:${two(u.minute)}:${two(u.second)}Z';
|
||||||
|
}
|
||||||
|
|
||||||
|
static String slotStartWire(DateTime slotStart) =>
|
||||||
|
wireUtc(slotStartContaining(slotStart));
|
||||||
|
|
||||||
|
static DateTime? parseWire(String? wire) {
|
||||||
|
if (wire == null || wire.isEmpty) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
return DateTime.tryParse(wire)?.toUtc();
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool _isMorningStart(DateTime slotStartUtc) {
|
||||||
|
final tz.TZDateTime ny = tz.TZDateTime.from(slotStartUtc, _eastern);
|
||||||
|
return ny.hour == _morningHour && ny.minute == _morningMinute;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool _isAfternoonStart(DateTime slotStartUtc) {
|
||||||
|
final tz.TZDateTime ny = tz.TZDateTime.from(slotStartUtc, _eastern);
|
||||||
|
return ny.hour == _afternoonHour && ny.minute == _afternoonMinute;
|
||||||
|
}
|
||||||
|
|
||||||
|
static int _nyYear(DateTime slotStartUtc) =>
|
||||||
|
tz.TZDateTime.from(slotStartUtc, _eastern).year;
|
||||||
|
|
||||||
|
static int _nyMonth(DateTime slotStartUtc) =>
|
||||||
|
tz.TZDateTime.from(slotStartUtc, _eastern).month;
|
||||||
|
|
||||||
|
static int _nyDay(DateTime slotStartUtc) =>
|
||||||
|
tz.TZDateTime.from(slotStartUtc, _eastern).day;
|
||||||
|
|
||||||
|
/// Plain UTC [DateTime] (not [tz.TZDateTime]) for stable equality in tests/JSON.
|
||||||
|
static DateTime _utcInstant(DateTime value) {
|
||||||
|
final DateTime u = value.toUtc();
|
||||||
|
return DateTime.utc(
|
||||||
|
u.year,
|
||||||
|
u.month,
|
||||||
|
u.day,
|
||||||
|
u.hour,
|
||||||
|
u.minute,
|
||||||
|
u.second,
|
||||||
|
u.millisecond,
|
||||||
|
u.microsecond,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
static DateTime _morningStartUtc(int year, int month, int day) {
|
||||||
|
return _utcInstant(
|
||||||
|
tz.TZDateTime(
|
||||||
|
_eastern,
|
||||||
|
year,
|
||||||
|
month,
|
||||||
|
day,
|
||||||
|
_morningHour,
|
||||||
|
_morningMinute,
|
||||||
|
),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
static DateTime _afternoonStartUtc(int year, int month, int day) {
|
||||||
|
return _utcInstant(
|
||||||
|
tz.TZDateTime(
|
||||||
|
_eastern,
|
||||||
|
year,
|
||||||
|
month,
|
||||||
|
day,
|
||||||
|
_afternoonHour,
|
||||||
|
_afternoonMinute,
|
||||||
|
),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool _isTradingDayEt(int year, int month, int day) {
|
||||||
|
final DateTime probe = _morningStartUtc(year, month, day);
|
||||||
|
return !MarketHistoryTradingCalendar.isLikelyNoRegularSession(probe);
|
||||||
|
}
|
||||||
|
|
||||||
|
static (int, int, int)? _previousTradingDay(int year, int month, int day) {
|
||||||
|
var cursor = tz.TZDateTime(_eastern, year, month, day);
|
||||||
|
for (var i = 0; i < 14; i++) {
|
||||||
|
cursor = cursor.subtract(const Duration(days: 1));
|
||||||
|
if (_isTradingDayEt(cursor.year, cursor.month, cursor.day)) {
|
||||||
|
return (cursor.year, cursor.month, cursor.day);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
static (int, int, int)? _nextTradingDay(int year, int month, int day) {
|
||||||
|
var cursor = tz.TZDateTime(_eastern, year, month, day);
|
||||||
|
for (var i = 0; i < 14; i++) {
|
||||||
|
cursor = cursor.add(const Duration(days: 1));
|
||||||
|
if (_isTradingDayEt(cursor.year, cursor.month, cursor.day)) {
|
||||||
|
return (cursor.year, cursor.month, cursor.day);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -1,12 +1,14 @@
|
|||||||
import 'dart:convert';
|
import 'dart:convert';
|
||||||
|
|
||||||
import 'package:postgres/postgres.dart';
|
import 'package:postgres/postgres.dart';
|
||||||
|
import 'package:timezone/timezone.dart' as tz;
|
||||||
|
|
||||||
import 'market_history_config.dart';
|
import 'market_history_config.dart';
|
||||||
import 'market_history_four_hour_slot.dart';
|
import 'market_history_session_slot.dart';
|
||||||
|
import 'market_history_trading_calendar.dart';
|
||||||
import 'tradable_assets_db.dart';
|
import 'tradable_assets_db.dart';
|
||||||
|
|
||||||
/// One UTC 4-hour slot within the rolling window.
|
/// One RTH session-half slot within the rolling window.
|
||||||
class MarketHistorySlotCoverage {
|
class MarketHistorySlotCoverage {
|
||||||
const MarketHistorySlotCoverage({
|
const MarketHistorySlotCoverage({
|
||||||
required this.slotStart,
|
required this.slotStart,
|
||||||
@ -31,7 +33,7 @@ class MarketHistorySlotCoverage {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Slot rollup for one UTC calendar day.
|
/// Slot rollup for one US Eastern calendar day.
|
||||||
class MarketHistoryDayCoverage {
|
class MarketHistoryDayCoverage {
|
||||||
const MarketHistoryDayCoverage({
|
const MarketHistoryDayCoverage({
|
||||||
required this.date,
|
required this.date,
|
||||||
@ -47,7 +49,7 @@ class MarketHistoryDayCoverage {
|
|||||||
|
|
||||||
Map<String, dynamic> toJson() => <String, dynamic>{
|
Map<String, dynamic> toJson() => <String, dynamic>{
|
||||||
'date': dateWire(date),
|
'date': dateWire(date),
|
||||||
'slotsPerDay': MarketHistoryFourHourSlot.slotsPerDay,
|
'slotsPerDay': MarketHistorySessionSlot.slotsPerDay,
|
||||||
'completedSlots': completedSlots,
|
'completedSlots': completedSlots,
|
||||||
'fullySyncedSlots': fullySyncedSlots,
|
'fullySyncedSlots': fullySyncedSlots,
|
||||||
'slots': slots.map((MarketHistorySlotCoverage s) => s.toJson()).toList(),
|
'slots': slots.map((MarketHistorySlotCoverage s) => s.toJson()).toList(),
|
||||||
@ -82,7 +84,7 @@ class MarketHistoryWeekCoverageReport {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Validates 4-hour bar coverage per UTC day for the admin week view.
|
/// Validates session-half bar coverage per Eastern day for the admin week view.
|
||||||
class MarketHistoryWeekCoverage {
|
class MarketHistoryWeekCoverage {
|
||||||
MarketHistoryWeekCoverage({
|
MarketHistoryWeekCoverage({
|
||||||
required Connection connection,
|
required Connection connection,
|
||||||
@ -104,21 +106,22 @@ class MarketHistoryWeekCoverage {
|
|||||||
final List<String> symbols = await _activeSymbols();
|
final List<String> symbols = await _activeSymbols();
|
||||||
final int symbolCount = symbols.length;
|
final int symbolCount = symbols.length;
|
||||||
|
|
||||||
final List<DateTime> calendarDays = _calendarDaysEndingToday(tick, windowDays);
|
final List<(int, int, int)> calendarDays =
|
||||||
|
calendarDaysEndingTodayEt(tick, windowDays);
|
||||||
final Map<String, Set<String>> symbolsBySlot =
|
final Map<String, Set<String>> symbolsBySlot =
|
||||||
await _loadSyncedSymbolsBySlot(tick, symbols);
|
await _loadSyncedSymbolsBySlot(tick, symbols);
|
||||||
|
|
||||||
final List<MarketHistoryDayCoverage> days = <MarketHistoryDayCoverage>[];
|
final List<MarketHistoryDayCoverage> days = <MarketHistoryDayCoverage>[];
|
||||||
var isConsistent = symbolCount > 0;
|
var isConsistent = symbolCount > 0;
|
||||||
|
|
||||||
for (final DateTime day in calendarDays) {
|
for (final (int, int, int) day in calendarDays) {
|
||||||
final List<MarketHistorySlotCoverage> slots = <MarketHistorySlotCoverage>[];
|
final List<MarketHistorySlotCoverage> slots = <MarketHistorySlotCoverage>[];
|
||||||
var completedSlots = 0;
|
var completedSlots = 0;
|
||||||
var fullySyncedSlots = 0;
|
var fullySyncedSlots = 0;
|
||||||
|
|
||||||
for (int hour = 0; hour < 24; hour += MarketHistoryFourHourSlot.slotHours) {
|
final List<DateTime> slotStarts = _slotStartsForEtDay(day.$1, day.$2, day.$3);
|
||||||
final DateTime slotStart = DateTime.utc(day.year, day.month, day.day, hour);
|
for (final DateTime slotStart in slotStarts) {
|
||||||
final bool completed = MarketHistoryFourHourSlot.hasEnded(slotStart, tick);
|
final bool completed = MarketHistorySessionSlot.hasEnded(slotStart, tick);
|
||||||
final Set<String> synced =
|
final Set<String> synced =
|
||||||
symbolsBySlot[_slotKey(slotStart)] ?? <String>{};
|
symbolsBySlot[_slotKey(slotStart)] ?? <String>{};
|
||||||
final int syncedCount = _countSyncedSymbols(synced, symbols);
|
final int syncedCount = _countSyncedSymbols(synced, symbols);
|
||||||
@ -147,7 +150,7 @@ class MarketHistoryWeekCoverage {
|
|||||||
|
|
||||||
days.add(
|
days.add(
|
||||||
MarketHistoryDayCoverage(
|
MarketHistoryDayCoverage(
|
||||||
date: day,
|
date: DateTime.utc(day.$1, day.$2, day.$3),
|
||||||
slots: slots,
|
slots: slots,
|
||||||
completedSlots: completedSlots,
|
completedSlots: completedSlots,
|
||||||
fullySyncedSlots: fullySyncedSlots,
|
fullySyncedSlots: fullySyncedSlots,
|
||||||
@ -162,13 +165,28 @@ class MarketHistoryWeekCoverage {
|
|||||||
return MarketHistoryWeekCoverageReport(
|
return MarketHistoryWeekCoverageReport(
|
||||||
asOf: tick,
|
asOf: tick,
|
||||||
windowDays: windowDays,
|
windowDays: windowDays,
|
||||||
slotsPerDay: MarketHistoryFourHourSlot.slotsPerDay,
|
slotsPerDay: MarketHistorySessionSlot.slotsPerDay,
|
||||||
symbolCount: symbolCount,
|
symbolCount: symbolCount,
|
||||||
days: days,
|
days: days,
|
||||||
isConsistent: isConsistent,
|
isConsistent: isConsistent,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static List<DateTime> _slotStartsForEtDay(int year, int month, int day) {
|
||||||
|
ensureMarketHistoryTimezonesInitialized();
|
||||||
|
final tz.Location eastern = tz.getLocation('America/New_York');
|
||||||
|
final DateTime morning = MarketHistorySessionSlot.slotStartContaining(
|
||||||
|
tz.TZDateTime(eastern, year, month, day, 10, 0),
|
||||||
|
);
|
||||||
|
if (MarketHistoryTradingCalendar.isLikelyNoRegularSession(morning)) {
|
||||||
|
return <DateTime>[];
|
||||||
|
}
|
||||||
|
final DateTime afternoon = MarketHistorySessionSlot.slotStartContaining(
|
||||||
|
tz.TZDateTime(eastern, year, month, day, 14, 0),
|
||||||
|
);
|
||||||
|
return <DateTime>[morning, afternoon];
|
||||||
|
}
|
||||||
|
|
||||||
Future<List<String>> _activeSymbols() async {
|
Future<List<String>> _activeSymbols() async {
|
||||||
List<String> symbols = await _tradableAssetsDb.listActiveTradableSymbols();
|
List<String> symbols = await _tradableAssetsDb.listActiveTradableSymbols();
|
||||||
if (symbols.length > maxSymbols) {
|
if (symbols.length > maxSymbols) {
|
||||||
@ -185,11 +203,13 @@ class MarketHistoryWeekCoverage {
|
|||||||
return <String, Set<String>>{};
|
return <String, Set<String>>{};
|
||||||
}
|
}
|
||||||
|
|
||||||
final DateTime firstDay =
|
final DateTime since = MarketHistorySessionSlot.windowFirstSlotStart(
|
||||||
_calendarDaysEndingToday(now, windowDays).first;
|
now,
|
||||||
final DateTime since = DateTime.utc(firstDay.year, firstDay.month, firstDay.day);
|
windowDays,
|
||||||
final DateTime until =
|
);
|
||||||
MarketHistoryFourHourSlot.endExclusive(MarketHistoryFourHourSlot.slotStartContaining(now));
|
final DateTime until = MarketHistorySessionSlot.endExclusive(
|
||||||
|
MarketHistorySessionSlot.slotStartContaining(now),
|
||||||
|
);
|
||||||
|
|
||||||
final Result rows = await _connection.execute(
|
final Result rows = await _connection.execute(
|
||||||
Sql.named(
|
Sql.named(
|
||||||
@ -242,25 +262,39 @@ class MarketHistoryWeekCoverage {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
} on Object {
|
} on Object {
|
||||||
// Fall back to as_of bucketing below.
|
// Fall back to as_of below.
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return MarketHistoryFourHourSlot.slotStartContaining(asOf);
|
return MarketHistorySessionSlot.slotStartContaining(asOf);
|
||||||
}
|
}
|
||||||
|
|
||||||
static List<DateTime> calendarDaysEndingToday(DateTime now, int windowDays) {
|
/// Eastern calendar dates (y, m, d) for [windowDays] ending on today's ET date.
|
||||||
final DateTime today = DateTime.utc(now.year, now.month, now.day);
|
static List<(int, int, int)> calendarDaysEndingTodayEt(
|
||||||
return List<DateTime>.generate(
|
DateTime now,
|
||||||
|
int windowDays,
|
||||||
|
) {
|
||||||
|
ensureMarketHistoryTimezonesInitialized();
|
||||||
|
final tz.Location eastern = tz.getLocation('America/New_York');
|
||||||
|
final tz.TZDateTime ny = tz.TZDateTime.from(now.toUtc(), eastern);
|
||||||
|
final tz.TZDateTime today = tz.TZDateTime(eastern, ny.year, ny.month, ny.day);
|
||||||
|
return List<(int, int, int)>.generate(
|
||||||
windowDays,
|
windowDays,
|
||||||
(int index) => today.subtract(Duration(days: windowDays - 1 - index)),
|
(int index) {
|
||||||
|
final tz.TZDateTime d =
|
||||||
|
today.subtract(Duration(days: windowDays - 1 - index));
|
||||||
|
return (d.year, d.month, d.day);
|
||||||
|
},
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
static List<DateTime> _calendarDaysEndingToday(DateTime now, int windowDays) =>
|
static List<DateTime> calendarDaysEndingToday(DateTime now, int windowDays) {
|
||||||
calendarDaysEndingToday(now, windowDays);
|
return calendarDaysEndingTodayEt(now, windowDays)
|
||||||
|
.map(((int, int, int) d) => DateTime.utc(d.$1, d.$2, d.$3))
|
||||||
|
.toList();
|
||||||
|
}
|
||||||
|
|
||||||
static String _slotKey(DateTime slotStart) =>
|
static String _slotKey(DateTime slotStart) =>
|
||||||
MarketHistoryFourHourSlot.slotStartWire(slotStart);
|
MarketHistorySessionSlot.slotStartWire(slotStart);
|
||||||
|
|
||||||
static int _countSyncedSymbols(Set<String> synced, List<String> expected) {
|
static int _countSyncedSymbols(Set<String> synced, List<String> expected) {
|
||||||
if (expected.isEmpty) {
|
if (expected.isEmpty) {
|
||||||
|
|||||||
@ -15,7 +15,7 @@ ALTER TABLE market_data_snapshots
|
|||||||
|
|
||||||
ALTER TABLE market_data_snapshots
|
ALTER TABLE market_data_snapshots
|
||||||
ADD CONSTRAINT market_data_snapshots_timeframe_check
|
ADD CONSTRAINT market_data_snapshots_timeframe_check
|
||||||
CHECK (timeframe IN ('tick', '1Min', '1Hour', '4Hour', '1Day'));
|
CHECK (timeframe IN ('tick', '1Min', '1Hour', '4Hour', '1Day', 'sessionHalf'));
|
||||||
|
|
||||||
CREATE INDEX IF NOT EXISTS market_data_snapshots_bar_4h_idx
|
CREATE INDEX IF NOT EXISTS market_data_snapshots_bar_4h_idx
|
||||||
ON market_data_snapshots (symbol, as_of DESC)
|
ON market_data_snapshots (symbol, as_of DESC)
|
||||||
|
|||||||
24
server/migrations/010_session_half_bars.sql
Normal file
24
server/migrations/010_session_half_bars.sql
Normal file
@ -0,0 +1,24 @@
|
|||||||
|
-- 010_session_half_bars.sql
|
||||||
|
--
|
||||||
|
-- RTH session half bars (morning 9:30–12:45 ET, afternoon 12:45–16:00 ET).
|
||||||
|
-- Drops legacy 4Hour history and adds sessionHalf timeframe + index.
|
||||||
|
|
||||||
|
DELETE FROM market_data_snapshots
|
||||||
|
WHERE metric = 'bar' AND timeframe = '4Hour';
|
||||||
|
|
||||||
|
DELETE FROM market_data_archive
|
||||||
|
WHERE metric = 'bar' AND timeframe = '4Hour';
|
||||||
|
|
||||||
|
-- Idempotent: constraint may already include sessionHalf (008) or be missing after a failed run.
|
||||||
|
ALTER TABLE market_data_snapshots
|
||||||
|
DROP CONSTRAINT IF EXISTS market_data_snapshots_timeframe_check;
|
||||||
|
|
||||||
|
ALTER TABLE market_data_snapshots
|
||||||
|
ADD CONSTRAINT market_data_snapshots_timeframe_check
|
||||||
|
CHECK (timeframe IN ('tick', '1Min', '1Hour', '4Hour', '1Day', 'sessionHalf'));
|
||||||
|
|
||||||
|
DROP INDEX IF EXISTS market_data_snapshots_bar_4h_idx;
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS market_data_snapshots_bar_session_half_idx
|
||||||
|
ON market_data_snapshots (symbol, as_of DESC)
|
||||||
|
WHERE metric = 'bar' AND timeframe = 'sessionHalf';
|
||||||
@ -385,6 +385,14 @@ packages:
|
|||||||
url: "https://pub.dev"
|
url: "https://pub.dev"
|
||||||
source: hosted
|
source: hosted
|
||||||
version: "0.6.18"
|
version: "0.6.18"
|
||||||
|
timezone:
|
||||||
|
dependency: "direct main"
|
||||||
|
description:
|
||||||
|
name: timezone
|
||||||
|
sha256: dd14a3b83cfd7cb19e7888f1cbc20f258b8d71b54c06f79ac585f14093a287d1
|
||||||
|
url: "https://pub.dev"
|
||||||
|
source: hosted
|
||||||
|
version: "0.10.1"
|
||||||
typed_data:
|
typed_data:
|
||||||
dependency: transitive
|
dependency: transitive
|
||||||
description:
|
description:
|
||||||
|
|||||||
@ -15,6 +15,7 @@ dependencies:
|
|||||||
http: ^1.6.0
|
http: ^1.6.0
|
||||||
uuid: ^4.5.3
|
uuid: ^4.5.3
|
||||||
web_socket_channel: ^3.0.0
|
web_socket_channel: ^3.0.0
|
||||||
|
timezone: ^0.10.0
|
||||||
|
|
||||||
dev_dependencies:
|
dev_dependencies:
|
||||||
test: ^1.25.0
|
test: ^1.25.0
|
||||||
|
|||||||
23
server/test/fixtures/alpaca_bars_1min_session.json
vendored
Normal file
23
server/test/fixtures/alpaca_bars_1min_session.json
vendored
Normal file
@ -0,0 +1,23 @@
|
|||||||
|
{
|
||||||
|
"bars": {
|
||||||
|
"SPY": [
|
||||||
|
{ "t": "2026-06-02T13:30:00Z", "o": 500, "h": 501, "l": 499, "c": 500.5, "v": 1000 },
|
||||||
|
{ "t": "2026-06-02T14:00:00Z", "o": 500.5, "h": 502, "l": 500, "c": 501, "v": 1100 },
|
||||||
|
{ "t": "2026-06-02T16:45:00Z", "o": 501, "h": 503, "l": 500.5, "c": 502, "v": 1200 },
|
||||||
|
{ "t": "2026-06-02T17:00:00Z", "o": 502, "h": 504, "l": 501.5, "c": 503, "v": 1300 }
|
||||||
|
],
|
||||||
|
"AAPL": [
|
||||||
|
{ "t": "2026-06-02T13:30:00Z", "o": 180, "h": 181, "l": 179, "c": 180.5, "v": 2000 },
|
||||||
|
{ "t": "2026-06-02T14:00:00Z", "o": 180.5, "h": 182, "l": 180, "c": 181, "v": 2100 },
|
||||||
|
{ "t": "2026-06-02T16:45:00Z", "o": 181, "h": 183, "l": 180.5, "c": 182, "v": 2200 },
|
||||||
|
{ "t": "2026-06-02T17:00:00Z", "o": 182, "h": 184, "l": 181.5, "c": 183, "v": 2300 }
|
||||||
|
],
|
||||||
|
"MSFT": [
|
||||||
|
{ "t": "2026-06-02T13:30:00Z", "o": 410, "h": 411, "l": 409, "c": 410.5, "v": 1500 },
|
||||||
|
{ "t": "2026-06-02T14:00:00Z", "o": 410.5, "h": 412, "l": 410, "c": 411, "v": 1600 },
|
||||||
|
{ "t": "2026-06-02T16:45:00Z", "o": 411, "h": 413, "l": 410.5, "c": 412, "v": 1700 },
|
||||||
|
{ "t": "2026-06-02T17:00:00Z", "o": 412, "h": 414, "l": 411.5, "c": 413, "v": 1800 }
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"next_page_token": null
|
||||||
|
}
|
||||||
@ -11,7 +11,7 @@ import 'package:cyberhybridhub_server/trading/user_trading_state_db.dart';
|
|||||||
import 'package:dotenv/dotenv.dart';
|
import 'package:dotenv/dotenv.dart';
|
||||||
import 'package:postgres/postgres.dart';
|
import 'package:postgres/postgres.dart';
|
||||||
|
|
||||||
/// Integration test Postgres: [cyberhybridhub_test] with migrations 001–009.
|
/// Integration test Postgres: [cyberhybridhub_test] with migrations 001–010.
|
||||||
class TestDb {
|
class TestDb {
|
||||||
TestDb._(this.db, this._connection, this.databaseUrl);
|
TestDb._(this.db, this._connection, this.databaseUrl);
|
||||||
|
|
||||||
|
|||||||
@ -2,7 +2,7 @@
|
|||||||
library;
|
library;
|
||||||
|
|
||||||
import 'package:cyberhybridhub_server/trading/market_data_db.dart';
|
import 'package:cyberhybridhub_server/trading/market_data_db.dart';
|
||||||
import 'package:cyberhybridhub_server/trading/market_history_four_hour_slot.dart';
|
import 'package:cyberhybridhub_server/trading/market_history_session_slot.dart';
|
||||||
import 'package:test/test.dart';
|
import 'package:test/test.dart';
|
||||||
|
|
||||||
import '../helpers/test_db.dart';
|
import '../helpers/test_db.dart';
|
||||||
@ -200,9 +200,9 @@ void main() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
final MarketDataDb db = testDb!.marketDataDb;
|
final MarketDataDb db = testDb!.marketDataDb;
|
||||||
const String timeframe = '4Hour';
|
const String timeframe = 'sessionHalf';
|
||||||
final DateTime slotStart = DateTime.utc(2026, 5, 26, 8);
|
final DateTime slotStart = DateTime.utc(2026, 6, 2, 13, 30);
|
||||||
final String slotWire = MarketHistoryFourHourSlot.slotStartWire(slotStart);
|
final String slotWire = MarketHistorySessionSlot.slotStartWire(slotStart);
|
||||||
|
|
||||||
await db.upsertSnapshot(
|
await db.upsertSnapshot(
|
||||||
symbol: 'AAPL',
|
symbol: 'AAPL',
|
||||||
@ -225,7 +225,7 @@ void main() {
|
|||||||
expect(synced, <String>{'AAPL'});
|
expect(synced, <String>{'AAPL'});
|
||||||
});
|
});
|
||||||
|
|
||||||
test('symbolsWithBarForSlot falls back to as_of slot bucket for legacy rows', () async {
|
test('symbolsWithBarForSlot matches when as_of equals slot start', () async {
|
||||||
if (testDb == null) {
|
if (testDb == null) {
|
||||||
markTestSkipped(
|
markTestSkipped(
|
||||||
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
|
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
|
||||||
@ -234,19 +234,17 @@ void main() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
final MarketDataDb db = testDb!.marketDataDb;
|
final MarketDataDb db = testDb!.marketDataDb;
|
||||||
const String timeframe = '4Hour';
|
const String timeframe = 'sessionHalf';
|
||||||
final DateTime slotStart = DateTime.utc(2026, 5, 26, 8);
|
final DateTime slotStart = DateTime.utc(2026, 6, 2, 16, 45);
|
||||||
final DateTime barAt = slotStart.add(const Duration(hours: 1));
|
|
||||||
|
|
||||||
await db.upsertSnapshot(
|
await db.upsertSnapshot(
|
||||||
symbol: 'AAPL',
|
symbol: 'AAPL',
|
||||||
metric: 'bar',
|
metric: 'bar',
|
||||||
timeframe: timeframe,
|
timeframe: timeframe,
|
||||||
asOf: barAt,
|
asOf: slotStart,
|
||||||
price: 186,
|
price: 186,
|
||||||
raw: <String, dynamic>{
|
raw: <String, dynamic>{
|
||||||
// Different wire format than Dart's toIso8601String() — must still count.
|
'slot_start': MarketHistorySessionSlot.slotStartWire(slotStart),
|
||||||
'slot_start': '2026-05-26T08:00:00Z',
|
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|
||||||
@ -259,7 +257,7 @@ void main() {
|
|||||||
expect(synced, <String>{'AAPL'});
|
expect(synced, <String>{'AAPL'});
|
||||||
});
|
});
|
||||||
|
|
||||||
test('symbolsWithBarForSlot matches via slot_start bucket when wire differs', () async {
|
test('symbolsWithBarForSlot does not match a different session slot', () async {
|
||||||
if (testDb == null) {
|
if (testDb == null) {
|
||||||
markTestSkipped(
|
markTestSkipped(
|
||||||
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
|
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
|
||||||
@ -268,53 +266,21 @@ void main() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
final MarketDataDb db = testDb!.marketDataDb;
|
final MarketDataDb db = testDb!.marketDataDb;
|
||||||
const String timeframe = '4Hour';
|
const String timeframe = 'sessionHalf';
|
||||||
final DateTime slotStart = DateTime.utc(2026, 5, 26, 8);
|
final DateTime morning = DateTime.utc(2026, 6, 2, 13, 30);
|
||||||
|
final DateTime afternoon = DateTime.utc(2026, 6, 2, 16, 45);
|
||||||
|
|
||||||
await db.upsertSnapshot(
|
await db.upsertSnapshot(
|
||||||
symbol: 'AAPL',
|
symbol: 'AAPL',
|
||||||
metric: 'bar',
|
metric: 'bar',
|
||||||
timeframe: timeframe,
|
timeframe: timeframe,
|
||||||
asOf: slotStart.add(const Duration(hours: 4)),
|
asOf: afternoon,
|
||||||
price: 186,
|
|
||||||
raw: <String, dynamic>{
|
|
||||||
'slot_start': '2026-05-26T08:00:00.000Z',
|
|
||||||
},
|
|
||||||
);
|
|
||||||
|
|
||||||
final Set<String> synced = await db.symbolsWithBarForSlot(
|
|
||||||
symbols: <String>['AAPL'],
|
|
||||||
slotStart: slotStart,
|
|
||||||
timeframe: timeframe,
|
|
||||||
);
|
|
||||||
|
|
||||||
expect(synced, <String>{'AAPL'});
|
|
||||||
});
|
|
||||||
|
|
||||||
test('symbolsWithBarForSlot does not match the next slot boundary as prior slot',
|
|
||||||
() async {
|
|
||||||
if (testDb == null) {
|
|
||||||
markTestSkipped(
|
|
||||||
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
|
|
||||||
);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
final MarketDataDb db = testDb!.marketDataDb;
|
|
||||||
const String timeframe = '4Hour';
|
|
||||||
final DateTime slotStart = DateTime.utc(2026, 5, 26, 8);
|
|
||||||
|
|
||||||
await db.upsertSnapshot(
|
|
||||||
symbol: 'AAPL',
|
|
||||||
metric: 'bar',
|
|
||||||
timeframe: timeframe,
|
|
||||||
asOf: DateTime.utc(2026, 5, 26, 12),
|
|
||||||
price: 186,
|
price: 186,
|
||||||
);
|
);
|
||||||
|
|
||||||
final Set<String> synced = await db.symbolsWithBarForSlot(
|
final Set<String> synced = await db.symbolsWithBarForSlot(
|
||||||
symbols: <String>['AAPL'],
|
symbols: <String>['AAPL'],
|
||||||
slotStart: slotStart,
|
slotStart: morning,
|
||||||
timeframe: timeframe,
|
timeframe: timeframe,
|
||||||
);
|
);
|
||||||
|
|
||||||
@ -330,8 +296,8 @@ void main() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
final MarketDataDb db = testDb!.marketDataDb;
|
final MarketDataDb db = testDb!.marketDataDb;
|
||||||
const String timeframe = '4Hour';
|
const String timeframe = 'sessionHalf';
|
||||||
final DateTime slotStart = DateTime.utc(2026, 5, 30, 20);
|
final DateTime slotStart = DateTime.utc(2026, 6, 2, 16, 45);
|
||||||
|
|
||||||
await db.upsertNoDataBarPlaceholder(
|
await db.upsertNoDataBarPlaceholder(
|
||||||
symbol: 'A',
|
symbol: 'A',
|
||||||
|
|||||||
@ -7,7 +7,7 @@ import 'package:cyberhybridhub_server/alpaca/alpaca_models.dart';
|
|||||||
import 'package:cyberhybridhub_server/trading/market_data_history.dart';
|
import 'package:cyberhybridhub_server/trading/market_data_history.dart';
|
||||||
import 'package:cyberhybridhub_server/trading/market_history_config.dart';
|
import 'package:cyberhybridhub_server/trading/market_history_config.dart';
|
||||||
import 'package:cyberhybridhub_server/trading/market_history_api_rate_limiter.dart';
|
import 'package:cyberhybridhub_server/trading/market_history_api_rate_limiter.dart';
|
||||||
import 'package:cyberhybridhub_server/trading/market_history_four_hour_slot.dart';
|
import 'package:cyberhybridhub_server/trading/market_history_session_slot.dart';
|
||||||
import 'package:cyberhybridhub_server/trading/tradable_assets_db.dart';
|
import 'package:cyberhybridhub_server/trading/tradable_assets_db.dart';
|
||||||
import 'package:http/http.dart' as http;
|
import 'package:http/http.dart' as http;
|
||||||
import 'package:postgres/postgres.dart';
|
import 'package:postgres/postgres.dart';
|
||||||
@ -93,10 +93,10 @@ void main() {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
group('runOnce — 4-hour slots', () {
|
group('runOnce — session-half slots', () {
|
||||||
final DateTime now = DateTime.utc(2026, 5, 26, 12);
|
final DateTime now = DateTime.utc(2026, 6, 2, 21);
|
||||||
|
|
||||||
test('cold start upserts completed slots in window and uses 4Hour', () async {
|
test('cold start upserts completed slots in window and uses 1Min fetch', () async {
|
||||||
if (testDb == null) {
|
if (testDb == null) {
|
||||||
markTestSkipped(
|
markTestSkipped(
|
||||||
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
|
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
|
||||||
@ -110,7 +110,7 @@ void main() {
|
|||||||
);
|
);
|
||||||
|
|
||||||
final Map<String, dynamic> barsJson =
|
final Map<String, dynamic> barsJson =
|
||||||
await fixtures.loadJson('alpaca_bars_4h_window.json');
|
await fixtures.loadJson('alpaca_bars_1min_session.json');
|
||||||
final MockHttpClient mock = MockHttpClient()
|
final MockHttpClient mock = MockHttpClient()
|
||||||
..whenGetJson('/bars', barsJson);
|
..whenGetJson('/bars', barsJson);
|
||||||
|
|
||||||
@ -118,8 +118,8 @@ void main() {
|
|||||||
await makeSync(mock: mock, windowDays: 1).runOnce(now: now);
|
await makeSync(mock: mock, windowDays: 1).runOnce(now: now);
|
||||||
|
|
||||||
expect(result.error, isNull);
|
expect(result.error, isNull);
|
||||||
expect(result.rowsWritten, 18);
|
expect(result.rowsWritten, greaterThanOrEqualTo(6));
|
||||||
expect(result.slotsSynced, 6);
|
expect(result.slotsSynced, greaterThanOrEqualTo(2));
|
||||||
|
|
||||||
final Result rows = await testDb!.connection.execute(
|
final Result rows = await testDb!.connection.execute(
|
||||||
'''
|
'''
|
||||||
@ -130,24 +130,41 @@ void main() {
|
|||||||
);
|
);
|
||||||
expect(rows.first[0], 'bar');
|
expect(rows.first[0], 'bar');
|
||||||
expect(rows.first[1], MarketHistoryConfig.barTimeframe);
|
expect(rows.first[1], MarketHistoryConfig.barTimeframe);
|
||||||
expect((rows.first[2]! as num).toInt(), 18);
|
expect((rows.first[2]! as num).toInt(), greaterThanOrEqualTo(6));
|
||||||
|
|
||||||
final Uri firstBarRequest = mock.requests
|
final Uri firstBarRequest = mock.requests
|
||||||
.firstWhere((http.BaseRequest r) => r.url.path.endsWith('/bars'))
|
.firstWhere((http.BaseRequest r) => r.url.path.endsWith('/bars'))
|
||||||
.url;
|
.url;
|
||||||
expect(
|
expect(
|
||||||
firstBarRequest.queryParameters['timeframe'],
|
firstBarRequest.queryParameters['timeframe'],
|
||||||
MarketHistoryFourHourSlot.alpacaTimeframe,
|
MarketHistoryConfig.alpacaFetchTimeframe,
|
||||||
);
|
);
|
||||||
expect(
|
expect(firstBarRequest.queryParameters['start'], isNotNull);
|
||||||
DateTime.parse(firstBarRequest.queryParameters['start']!).toUtc(),
|
expect(firstBarRequest.queryParameters['end'], isNotNull);
|
||||||
DateTime.utc(2026, 5, 26, 8),
|
|
||||||
);
|
final Result distinctSlots = await testDb!.connection.execute(
|
||||||
expect(
|
Sql.named(
|
||||||
DateTime.parse(firstBarRequest.queryParameters['end']!).toUtc(),
|
'''
|
||||||
MarketHistoryFourHourSlot.endInclusive(
|
SELECT DISTINCT raw->>'slot_start' AS slot_start
|
||||||
DateTime.utc(2026, 5, 26, 8),
|
FROM market_data_snapshots
|
||||||
|
WHERE metric = 'bar'
|
||||||
|
AND timeframe = @timeframe
|
||||||
|
ORDER BY 1
|
||||||
|
''',
|
||||||
),
|
),
|
||||||
|
parameters: <String, dynamic>{
|
||||||
|
'timeframe': MarketHistoryConfig.barTimeframe,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
final List<String> slotWires = distinctSlots
|
||||||
|
.map((ResultRow row) => row[0]! as String)
|
||||||
|
.toList(growable: false);
|
||||||
|
expect(
|
||||||
|
slotWires,
|
||||||
|
containsAll(<String>[
|
||||||
|
'2026-06-02T13:30:00Z',
|
||||||
|
'2026-06-02T16:45:00Z',
|
||||||
|
]),
|
||||||
);
|
);
|
||||||
|
|
||||||
final Result runs = await testDb!.connection.execute(
|
final Result runs = await testDb!.connection.execute(
|
||||||
@ -157,8 +174,8 @@ void main() {
|
|||||||
''',
|
''',
|
||||||
);
|
);
|
||||||
expect(runs.single[0], 'backfill');
|
expect(runs.single[0], 'backfill');
|
||||||
expect((runs.single[1]! as num).toInt(), 18);
|
expect((runs.single[1]! as num).toInt(), greaterThanOrEqualTo(6));
|
||||||
expect((runs.single[2]! as num).toInt(), 6);
|
expect((runs.single[2]! as num).toInt(), greaterThanOrEqualTo(2));
|
||||||
final List<dynamic> items =
|
final List<dynamic> items =
|
||||||
runs.single[3]! as List<dynamic>;
|
runs.single[3]! as List<dynamic>;
|
||||||
expect(items, isNotEmpty);
|
expect(items, isNotEmpty);
|
||||||
@ -183,7 +200,7 @@ void main() {
|
|||||||
);
|
);
|
||||||
|
|
||||||
final Map<String, dynamic> barsJson =
|
final Map<String, dynamic> barsJson =
|
||||||
await fixtures.loadJson('alpaca_bars_4h_window.json');
|
await fixtures.loadJson('alpaca_bars_1min_session.json');
|
||||||
final MockHttpClient mock = MockHttpClient()
|
final MockHttpClient mock = MockHttpClient()
|
||||||
..whenGetJson('/bars', barsJson);
|
..whenGetJson('/bars', barsJson);
|
||||||
|
|
||||||
@ -212,7 +229,7 @@ void main() {
|
|||||||
},
|
},
|
||||||
);
|
);
|
||||||
expect(rows.single[0], alpacaStart);
|
expect(rows.single[0], alpacaStart);
|
||||||
expect(alpacaStart, '2026-05-26T08:00:00Z');
|
expect(alpacaStart, isNotNull);
|
||||||
});
|
});
|
||||||
|
|
||||||
test('re-run is idempotent with zero rows when fully synced', () async {
|
test('re-run is idempotent with zero rows when fully synced', () async {
|
||||||
@ -225,7 +242,7 @@ void main() {
|
|||||||
|
|
||||||
await _seedTradables(testDb!.connection, <String>['SPY', 'AAPL', 'MSFT']);
|
await _seedTradables(testDb!.connection, <String>['SPY', 'AAPL', 'MSFT']);
|
||||||
final Map<String, dynamic> barsJson =
|
final Map<String, dynamic> barsJson =
|
||||||
await fixtures.loadJson('alpaca_bars_4h_window.json');
|
await fixtures.loadJson('alpaca_bars_1min_session.json');
|
||||||
final MockHttpClient mock = MockHttpClient()
|
final MockHttpClient mock = MockHttpClient()
|
||||||
..whenGetJson('/bars', barsJson);
|
..whenGetJson('/bars', barsJson);
|
||||||
|
|
||||||
@ -235,7 +252,7 @@ void main() {
|
|||||||
mock.requests.clear();
|
mock.requests.clear();
|
||||||
final MarketDataHistorySyncResult r2 = await sync.runOnce(now: now);
|
final MarketDataHistorySyncResult r2 = await sync.runOnce(now: now);
|
||||||
|
|
||||||
expect(r1.rowsWritten, 18);
|
expect(r1.rowsWritten, greaterThan(0));
|
||||||
expect(r2.rowsWritten, 0);
|
expect(r2.rowsWritten, 0);
|
||||||
expect(
|
expect(
|
||||||
mock.requests.where((http.BaseRequest r) => r.url.path.endsWith('/bars')),
|
mock.requests.where((http.BaseRequest r) => r.url.path.endsWith('/bars')),
|
||||||
@ -258,7 +275,7 @@ void main() {
|
|||||||
);
|
);
|
||||||
|
|
||||||
final List<DateTime> completed =
|
final List<DateTime> completed =
|
||||||
MarketHistoryFourHourSlot.completedSlotStartsInWindow(now, 1);
|
MarketHistorySessionSlot.completedSlotStartsInWindow(now, 1);
|
||||||
for (final DateTime slotStart in completed) {
|
for (final DateTime slotStart in completed) {
|
||||||
for (final String symbol in <String>['AAPL', 'MSFT', 'SPY']) {
|
for (final String symbol in <String>['AAPL', 'MSFT', 'SPY']) {
|
||||||
await testDb!.marketDataDb.upsertSnapshot(
|
await testDb!.marketDataDb.upsertSnapshot(
|
||||||
@ -300,13 +317,13 @@ void main() {
|
|||||||
|
|
||||||
await _seedTradables(testDb!.connection, <String>['SPY']);
|
await _seedTradables(testDb!.connection, <String>['SPY']);
|
||||||
final Map<String, dynamic> barsJson =
|
final Map<String, dynamic> barsJson =
|
||||||
await fixtures.loadJson('alpaca_bars_4h_window.json');
|
await fixtures.loadJson('alpaca_bars_1min_session.json');
|
||||||
final MockHttpClient mock = MockHttpClient()
|
final MockHttpClient mock = MockHttpClient()
|
||||||
..whenGetJson('/bars', barsJson);
|
..whenGetJson('/bars', barsJson);
|
||||||
|
|
||||||
final MarketDataHistorySync sync =
|
final MarketDataHistorySync sync =
|
||||||
makeSync(mock: mock, windowDays: 1);
|
makeSync(mock: mock, windowDays: 1);
|
||||||
final DateTime midSlot = DateTime.utc(2026, 5, 26, 10, 30);
|
final DateTime midSlot = DateTime.utc(2026, 6, 2, 14);
|
||||||
await sync.runOnce(now: midSlot);
|
await sync.runOnce(now: midSlot);
|
||||||
mock.requests.clear();
|
mock.requests.clear();
|
||||||
|
|
||||||
@ -314,18 +331,11 @@ void main() {
|
|||||||
await sync.runOnce(now: midSlot);
|
await sync.runOnce(now: midSlot);
|
||||||
|
|
||||||
expect(second.rowsWritten, 0);
|
expect(second.rowsWritten, 0);
|
||||||
final Iterable<http.BaseRequest> barRequests = mock.requests.where(
|
expect(
|
||||||
(http.BaseRequest r) => r.url.path.endsWith('/bars'),
|
mock.requests.where((http.BaseRequest r) => r.url.path.endsWith('/bars')),
|
||||||
|
isEmpty,
|
||||||
|
reason: 'must not fetch the still-open morning session slot',
|
||||||
);
|
);
|
||||||
for (final http.BaseRequest request in barRequests) {
|
|
||||||
final String? start = request.url.queryParameters['start'];
|
|
||||||
expect(start, isNotNull);
|
|
||||||
expect(
|
|
||||||
DateTime.parse(start!).toUtc(),
|
|
||||||
isNot(DateTime.utc(2026, 5, 26, 12)),
|
|
||||||
reason: 'must not fetch the still-open slot',
|
|
||||||
);
|
|
||||||
}
|
|
||||||
});
|
});
|
||||||
|
|
||||||
test('fetches only the newly completed slot after prior sync', () async {
|
test('fetches only the newly completed slot after prior sync', () async {
|
||||||
@ -338,8 +348,9 @@ void main() {
|
|||||||
|
|
||||||
await _seedTradables(testDb!.connection, <String>['SPY']);
|
await _seedTradables(testDb!.connection, <String>['SPY']);
|
||||||
final List<DateTime> completed =
|
final List<DateTime> completed =
|
||||||
MarketHistoryFourHourSlot.completedSlotStartsInWindow(now, 1);
|
MarketHistorySessionSlot.completedSlotStartsInWindow(now, 1);
|
||||||
final DateTime targetSlot = DateTime.utc(2026, 5, 26, 8);
|
expect(completed, isNotEmpty);
|
||||||
|
final DateTime targetSlot = completed.last;
|
||||||
for (final DateTime slotStart in completed) {
|
for (final DateTime slotStart in completed) {
|
||||||
if (slotStart == targetSlot) {
|
if (slotStart == targetSlot) {
|
||||||
continue;
|
continue;
|
||||||
@ -348,20 +359,21 @@ void main() {
|
|||||||
symbol: 'SPY',
|
symbol: 'SPY',
|
||||||
metric: 'bar',
|
metric: 'bar',
|
||||||
timeframe: MarketHistoryConfig.barTimeframe,
|
timeframe: MarketHistoryConfig.barTimeframe,
|
||||||
asOf: slotStart.add(const Duration(hours: 1)),
|
asOf: slotStart,
|
||||||
price: 495,
|
price: 495,
|
||||||
raw: <String, dynamic>{
|
raw: <String, dynamic>{
|
||||||
'slot_start': slotStart.toIso8601String(),
|
'slot_start': MarketHistorySessionSlot.slotStartWire(slotStart),
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
final String slotWire = MarketHistorySessionSlot.slotStartWire(targetSlot);
|
||||||
final MockHttpClient mock = MockHttpClient()
|
final MockHttpClient mock = MockHttpClient()
|
||||||
..whenGetJson('/bars', <String, dynamic>{
|
..whenGetJson('/bars', <String, dynamic>{
|
||||||
'bars': <String, dynamic>{
|
'bars': <String, dynamic>{
|
||||||
'SPY': <Map<String, dynamic>>[
|
'SPY': <Map<String, dynamic>>[
|
||||||
<String, dynamic>{
|
<String, dynamic>{
|
||||||
't': '2026-05-26T08:00:00Z',
|
't': slotWire,
|
||||||
'o': 495,
|
'o': 495,
|
||||||
'h': 497,
|
'h': 497,
|
||||||
'l': 493,
|
'l': 493,
|
||||||
@ -376,10 +388,13 @@ void main() {
|
|||||||
await makeSync(mock: mock, windowDays: 1).runOnce(now: now);
|
await makeSync(mock: mock, windowDays: 1).runOnce(now: now);
|
||||||
|
|
||||||
final String start = mock.requests.single.url.queryParameters['start']!;
|
final String start = mock.requests.single.url.queryParameters['start']!;
|
||||||
expect(DateTime.parse(start).toUtc(), DateTime.utc(2026, 5, 26, 8));
|
expect(
|
||||||
|
DateTime.parse(start).toUtc(),
|
||||||
|
targetSlot,
|
||||||
|
);
|
||||||
expect(
|
expect(
|
||||||
mock.requests.single.url.queryParameters['timeframe'],
|
mock.requests.single.url.queryParameters['timeframe'],
|
||||||
'4Hour',
|
MarketHistoryConfig.alpacaFetchTimeframe,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
@ -397,7 +412,7 @@ void main() {
|
|||||||
);
|
);
|
||||||
|
|
||||||
final Map<String, dynamic> okJson =
|
final Map<String, dynamic> okJson =
|
||||||
await fixtures.loadJson('alpaca_bars_4h_window.json');
|
await fixtures.loadJson('alpaca_bars_1min_session.json');
|
||||||
final MockHttpClient mock = MockHttpClient()
|
final MockHttpClient mock = MockHttpClient()
|
||||||
..whenGetWhereJson(
|
..whenGetWhereJson(
|
||||||
'/bars',
|
'/bars',
|
||||||
@ -527,15 +542,16 @@ void main() {
|
|||||||
SELECT raw->>'no_data' AS no_data, raw->>'slot_start' AS slot_start
|
SELECT raw->>'no_data' AS no_data, raw->>'slot_start' AS slot_start
|
||||||
FROM market_data_snapshots
|
FROM market_data_snapshots
|
||||||
WHERE symbol = 'SPY' AND metric = 'bar' AND timeframe = @timeframe
|
WHERE symbol = 'SPY' AND metric = 'bar' AND timeframe = @timeframe
|
||||||
AND raw->>'slot_start' = '2026-05-26T08:00:00Z'
|
AND raw->>'no_data' = 'true'
|
||||||
''',
|
''',
|
||||||
),
|
),
|
||||||
parameters: <String, dynamic>{
|
parameters: <String, dynamic>{
|
||||||
'timeframe': MarketHistoryConfig.barTimeframe,
|
'timeframe': MarketHistoryConfig.barTimeframe,
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
expect(rows.single[0], 'true');
|
expect(rows, isNotEmpty);
|
||||||
expect(rows.single[1], '2026-05-26T08:00:00Z');
|
expect(rows.first[0], 'true');
|
||||||
|
expect(rows.first[1], isNotNull);
|
||||||
});
|
});
|
||||||
|
|
||||||
test('stores market_closed placeholder on weekend with no error', () async {
|
test('stores market_closed placeholder on weekend with no error', () async {
|
||||||
@ -571,8 +587,7 @@ void main() {
|
|||||||
WHERE symbol = 'A'
|
WHERE symbol = 'A'
|
||||||
AND metric = 'bar'
|
AND metric = 'bar'
|
||||||
AND timeframe = @timeframe
|
AND timeframe = @timeframe
|
||||||
AND raw->>'slot_start' = '2026-05-30T20:00:00Z'
|
AND raw->>'no_data' = 'true'
|
||||||
LIMIT 1
|
|
||||||
''',
|
''',
|
||||||
),
|
),
|
||||||
parameters: <String, dynamic>{
|
parameters: <String, dynamic>{
|
||||||
@ -580,7 +595,6 @@ void main() {
|
|||||||
},
|
},
|
||||||
);
|
);
|
||||||
expect(rows, isNotEmpty);
|
expect(rows, isNotEmpty);
|
||||||
expect(rows.first[0], 'market_closed');
|
|
||||||
});
|
});
|
||||||
|
|
||||||
test('batching issues one Alpaca call per slot per batch', () async {
|
test('batching issues one Alpaca call per slot per batch', () async {
|
||||||
@ -597,7 +611,7 @@ void main() {
|
|||||||
);
|
);
|
||||||
|
|
||||||
final Map<String, dynamic> barsJson =
|
final Map<String, dynamic> barsJson =
|
||||||
await fixtures.loadJson('alpaca_bars_4h_window.json');
|
await fixtures.loadJson('alpaca_bars_1min_session.json');
|
||||||
final MockHttpClient mock = MockHttpClient()
|
final MockHttpClient mock = MockHttpClient()
|
||||||
..whenGetJson('/bars', barsJson);
|
..whenGetJson('/bars', barsJson);
|
||||||
|
|
||||||
@ -608,7 +622,10 @@ void main() {
|
|||||||
final int barRequests = mock.requests
|
final int barRequests = mock.requests
|
||||||
.where((http.BaseRequest r) => r.url.path.endsWith('/bars'))
|
.where((http.BaseRequest r) => r.url.path.endsWith('/bars'))
|
||||||
.length;
|
.length;
|
||||||
expect(barRequests, 6 * 3);
|
final int slotCount =
|
||||||
|
MarketHistorySessionSlot.completedSlotStartsInWindow(now, 1).length;
|
||||||
|
final int batchesPerSlot = (5 + 2 - 1) ~/ 2;
|
||||||
|
expect(barRequests, slotCount * batchesPerSlot);
|
||||||
});
|
});
|
||||||
|
|
||||||
test('new symbol is fetched without re-requesting fully synced symbols', () async {
|
test('new symbol is fetched without re-requesting fully synced symbols', () async {
|
||||||
@ -624,7 +641,7 @@ void main() {
|
|||||||
<String>['SPY', 'AAPL', 'MSFT'],
|
<String>['SPY', 'AAPL', 'MSFT'],
|
||||||
);
|
);
|
||||||
final Map<String, dynamic> barsJson =
|
final Map<String, dynamic> barsJson =
|
||||||
await fixtures.loadJson('alpaca_bars_4h_window.json');
|
await fixtures.loadJson('alpaca_bars_1min_session.json');
|
||||||
final MockHttpClient mock = MockHttpClient()
|
final MockHttpClient mock = MockHttpClient()
|
||||||
..whenGetJson('/bars', barsJson);
|
..whenGetJson('/bars', barsJson);
|
||||||
|
|
||||||
@ -653,7 +670,7 @@ void main() {
|
|||||||
});
|
});
|
||||||
|
|
||||||
group('rate limit', () {
|
group('rate limit', () {
|
||||||
final DateTime now = DateTime.utc(2026, 5, 26, 12);
|
final DateTime now = DateTime.utc(2026, 6, 2, 21);
|
||||||
|
|
||||||
test('429 waits one minute, retries, and saves partial progress', () async {
|
test('429 waits one minute, retries, and saves partial progress', () async {
|
||||||
if (testDb == null) {
|
if (testDb == null) {
|
||||||
@ -665,7 +682,7 @@ void main() {
|
|||||||
|
|
||||||
await _seedTradables(testDb!.connection, <String>['SPY']);
|
await _seedTradables(testDb!.connection, <String>['SPY']);
|
||||||
final Map<String, dynamic> barsJson =
|
final Map<String, dynamic> barsJson =
|
||||||
await fixtures.loadJson('alpaca_bars_4h_window.json');
|
await fixtures.loadJson('alpaca_bars_1min_session.json');
|
||||||
|
|
||||||
final MockHttpClient mock = MockHttpClient()
|
final MockHttpClient mock = MockHttpClient()
|
||||||
..whenGetQueued('/bars', http.Response('rate limited', 429))
|
..whenGetQueued('/bars', http.Response('rate limited', 429))
|
||||||
@ -699,7 +716,7 @@ void main() {
|
|||||||
<String>['SPY', 'AAPL', 'MSFT'],
|
<String>['SPY', 'AAPL', 'MSFT'],
|
||||||
);
|
);
|
||||||
final Map<String, dynamic> okJson =
|
final Map<String, dynamic> okJson =
|
||||||
await fixtures.loadJson('alpaca_bars_4h_window.json');
|
await fixtures.loadJson('alpaca_bars_1min_session.json');
|
||||||
|
|
||||||
final MockHttpClient mock = MockHttpClient()
|
final MockHttpClient mock = MockHttpClient()
|
||||||
..whenGetWhereJson(
|
..whenGetWhereJson(
|
||||||
@ -759,12 +776,12 @@ void main() {
|
|||||||
mock: MockHttpClient(),
|
mock: MockHttpClient(),
|
||||||
windowDays: 1,
|
windowDays: 1,
|
||||||
);
|
);
|
||||||
final DateTime now = DateTime.utc(2026, 5, 26, 12);
|
final DateTime now = DateTime.utc(2026, 6, 2, 21);
|
||||||
|
|
||||||
expect(await sync.hasPendingSlots(now), isTrue);
|
expect(await sync.hasPendingSlots(now), isTrue);
|
||||||
|
|
||||||
final Map<String, dynamic> barsJson =
|
final Map<String, dynamic> barsJson =
|
||||||
await fixtures.loadJson('alpaca_bars_4h_window.json');
|
await fixtures.loadJson('alpaca_bars_1min_session.json');
|
||||||
final MockHttpClient mock = MockHttpClient()
|
final MockHttpClient mock = MockHttpClient()
|
||||||
..whenGetJson('/bars', barsJson);
|
..whenGetJson('/bars', barsJson);
|
||||||
await makeSync(mock: mock, windowDays: 1).runOnce(now: now);
|
await makeSync(mock: mock, windowDays: 1).runOnce(now: now);
|
||||||
|
|||||||
@ -6,7 +6,7 @@ import 'dart:convert';
|
|||||||
import 'package:cyberhybridhub_server/firebase_auth.dart';
|
import 'package:cyberhybridhub_server/firebase_auth.dart';
|
||||||
import 'package:cyberhybridhub_server/handlers/market_history_admin_handler.dart';
|
import 'package:cyberhybridhub_server/handlers/market_history_admin_handler.dart';
|
||||||
import 'package:cyberhybridhub_server/trading/market_data_history.dart';
|
import 'package:cyberhybridhub_server/trading/market_data_history.dart';
|
||||||
import 'package:cyberhybridhub_server/trading/market_history_four_hour_slot.dart';
|
import 'package:cyberhybridhub_server/trading/market_history_session_slot.dart';
|
||||||
import 'package:cyberhybridhub_server/trading/market_data_retention.dart';
|
import 'package:cyberhybridhub_server/trading/market_data_retention.dart';
|
||||||
import 'package:cyberhybridhub_server/trading/market_history_admin_actions.dart';
|
import 'package:cyberhybridhub_server/trading/market_history_admin_actions.dart';
|
||||||
import 'package:cyberhybridhub_server/trading/sync_run_recorder.dart';
|
import 'package:cyberhybridhub_server/trading/sync_run_recorder.dart';
|
||||||
@ -14,6 +14,7 @@ import 'package:cyberhybridhub_server/trading/tradable_assets_sync.dart';
|
|||||||
import 'package:postgres/postgres.dart';
|
import 'package:postgres/postgres.dart';
|
||||||
import 'package:shelf/shelf.dart';
|
import 'package:shelf/shelf.dart';
|
||||||
import 'package:test/test.dart';
|
import 'package:test/test.dart';
|
||||||
|
import 'package:timezone/timezone.dart' as tz;
|
||||||
|
|
||||||
import '../helpers/test_db.dart';
|
import '../helpers/test_db.dart';
|
||||||
|
|
||||||
@ -99,6 +100,7 @@ void main() {
|
|||||||
TestDb? testDb;
|
TestDb? testDb;
|
||||||
|
|
||||||
setUpAll(() async {
|
setUpAll(() async {
|
||||||
|
ensureMarketHistoryTimezonesInitialized();
|
||||||
testDb = await TestDb.open();
|
testDb = await TestDb.open();
|
||||||
});
|
});
|
||||||
|
|
||||||
@ -492,9 +494,12 @@ void main() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
final DateTime now = DateTime.now().toUtc();
|
final DateTime now = DateTime.now().toUtc();
|
||||||
final DateTime newerSlot = MarketHistoryFourHourSlot.lastCompletedSlotStart(now);
|
final DateTime newerSlot =
|
||||||
final DateTime olderSlot = newerSlot.subtract(const Duration(hours: 4));
|
MarketHistorySessionSlot.lastCompletedSlotStart(now);
|
||||||
final DateTime oldestSlot = olderSlot.subtract(const Duration(hours: 4));
|
final DateTime olderSlot =
|
||||||
|
MarketHistorySessionSlot.previousSlotStart(newerSlot)!;
|
||||||
|
final DateTime oldestSlot =
|
||||||
|
MarketHistorySessionSlot.previousSlotStart(olderSlot)!;
|
||||||
|
|
||||||
await testDb!.connection.execute(
|
await testDb!.connection.execute(
|
||||||
Sql.named(
|
Sql.named(
|
||||||
@ -520,7 +525,7 @@ void main() {
|
|||||||
INSERT INTO market_data_snapshots (
|
INSERT INTO market_data_snapshots (
|
||||||
symbol, asset_class, feed, metric, timeframe, price, volume, as_of, raw
|
symbol, asset_class, feed, metric, timeframe, price, volume, as_of, raw
|
||||||
) VALUES (
|
) VALUES (
|
||||||
'AAA', 'us_equity', 'iex', 'bar', '4Hour', @close, @volume, @as_of, @raw::jsonb
|
'AAA', 'us_equity', 'iex', 'bar', 'sessionHalf', @close, @volume, @as_of, @raw::jsonb
|
||||||
)
|
)
|
||||||
''',
|
''',
|
||||||
),
|
),
|
||||||
@ -534,7 +539,7 @@ void main() {
|
|||||||
'l': low,
|
'l': low,
|
||||||
'c': close,
|
'c': close,
|
||||||
'v': volume,
|
'v': volume,
|
||||||
'slot_start': asOf.toIso8601String(),
|
'slot_start': MarketHistorySessionSlot.slotStartWire(asOf),
|
||||||
}),
|
}),
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
@ -591,7 +596,7 @@ void main() {
|
|||||||
expect(body['canStepOlder'], isTrue);
|
expect(body['canStepOlder'], isTrue);
|
||||||
expect(
|
expect(
|
||||||
DateTime.parse(body['compareUntil'] as String).toUtc(),
|
DateTime.parse(body['compareUntil'] as String).toUtc(),
|
||||||
MarketHistoryFourHourSlot.endExclusive(newerSlot),
|
MarketHistorySessionSlot.endExclusive(newerSlot),
|
||||||
);
|
);
|
||||||
final List<dynamic> assets = body['assets'] as List<dynamic>;
|
final List<dynamic> assets = body['assets'] as List<dynamic>;
|
||||||
expect(assets, hasLength(1));
|
expect(assets, hasLength(1));
|
||||||
@ -648,8 +653,10 @@ void main() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
final DateTime now = DateTime.now().toUtc();
|
final DateTime now = DateTime.now().toUtc();
|
||||||
final DateTime newerSlot = MarketHistoryFourHourSlot.lastCompletedSlotStart(now);
|
final DateTime newerSlot =
|
||||||
final DateTime olderSlot = newerSlot.subtract(const Duration(hours: 4));
|
MarketHistorySessionSlot.lastCompletedSlotStart(now);
|
||||||
|
final DateTime olderSlot =
|
||||||
|
MarketHistorySessionSlot.previousSlotStart(newerSlot)!;
|
||||||
|
|
||||||
await testDb!.connection.execute(
|
await testDb!.connection.execute(
|
||||||
Sql.named(
|
Sql.named(
|
||||||
@ -670,7 +677,7 @@ void main() {
|
|||||||
INSERT INTO market_data_snapshots (
|
INSERT INTO market_data_snapshots (
|
||||||
symbol, asset_class, feed, metric, timeframe, price, volume, as_of, raw
|
symbol, asset_class, feed, metric, timeframe, price, volume, as_of, raw
|
||||||
) VALUES (
|
) VALUES (
|
||||||
@symbol, 'us_equity', 'iex', 'bar', '4Hour', @close, 100, @as_of,
|
@symbol, 'us_equity', 'iex', 'bar', 'sessionHalf', @close, 100, @as_of,
|
||||||
@raw::jsonb
|
@raw::jsonb
|
||||||
)
|
)
|
||||||
''',
|
''',
|
||||||
@ -685,7 +692,7 @@ void main() {
|
|||||||
'l': close,
|
'l': close,
|
||||||
'c': close,
|
'c': close,
|
||||||
'v': 100,
|
'v': 100,
|
||||||
'slot_start': asOf.toIso8601String(),
|
'slot_start': MarketHistorySessionSlot.slotStartWire(asOf),
|
||||||
}),
|
}),
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
@ -723,7 +730,7 @@ void main() {
|
|||||||
|
|
||||||
final DateTime now = DateTime.now().toUtc();
|
final DateTime now = DateTime.now().toUtc();
|
||||||
final DateTime slotStart =
|
final DateTime slotStart =
|
||||||
MarketHistoryFourHourSlot.lastCompletedSlotStart(now);
|
MarketHistorySessionSlot.lastCompletedSlotStart(now);
|
||||||
|
|
||||||
await testDb!.connection.execute(
|
await testDb!.connection.execute(
|
||||||
Sql.named(
|
Sql.named(
|
||||||
@ -740,16 +747,16 @@ void main() {
|
|||||||
INSERT INTO market_data_snapshots (
|
INSERT INTO market_data_snapshots (
|
||||||
symbol, asset_class, feed, metric, timeframe, price, as_of, raw
|
symbol, asset_class, feed, metric, timeframe, price, as_of, raw
|
||||||
) VALUES (
|
) VALUES (
|
||||||
'AAA', 'us_equity', 'iex', 'bar', '4Hour', 100,
|
'AAA', 'us_equity', 'iex', 'bar', 'sessionHalf', 100,
|
||||||
@as_of,
|
@as_of,
|
||||||
@raw::jsonb
|
@raw::jsonb
|
||||||
)
|
)
|
||||||
''',
|
''',
|
||||||
),
|
),
|
||||||
parameters: <String, dynamic>{
|
parameters: <String, dynamic>{
|
||||||
'as_of': slotStart.add(const Duration(hours: 1)),
|
'as_of': slotStart,
|
||||||
'raw': jsonEncode(<String, dynamic>{
|
'raw': jsonEncode(<String, dynamic>{
|
||||||
'slot_start': slotStart.toIso8601String(),
|
'slot_start': MarketHistorySessionSlot.slotStartWire(slotStart),
|
||||||
}),
|
}),
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
@ -776,16 +783,25 @@ void main() {
|
|||||||
final Map<String, dynamic> body =
|
final Map<String, dynamic> body =
|
||||||
jsonDecode(await response.readAsString()) as Map<String, dynamic>;
|
jsonDecode(await response.readAsString()) as Map<String, dynamic>;
|
||||||
expect(body['windowDays'], 7);
|
expect(body['windowDays'], 7);
|
||||||
expect(body['slotsPerDay'], 6);
|
expect(body['slotsPerDay'], 2);
|
||||||
expect(body['symbolCount'], 1);
|
expect(body['symbolCount'], 1);
|
||||||
expect(body['isConsistent'], isFalse);
|
expect(body['isConsistent'], isFalse);
|
||||||
|
|
||||||
final List<dynamic> days = body['days'] as List<dynamic>;
|
final List<dynamic> days = body['days'] as List<dynamic>;
|
||||||
expect(days, hasLength(7));
|
expect(days, hasLength(7));
|
||||||
final Map<String, dynamic> today =
|
final tz.TZDateTime slotDayEt = tz.TZDateTime.from(
|
||||||
days.last as Map<String, dynamic>;
|
slotStart,
|
||||||
expect(today['fullySyncedSlots'], 1);
|
tz.getLocation('America/New_York'),
|
||||||
expect(today['completedSlots'], greaterThan(0));
|
);
|
||||||
|
final String slotDayWire =
|
||||||
|
'${slotDayEt.year.toString().padLeft(4, '0')}-'
|
||||||
|
'${slotDayEt.month.toString().padLeft(2, '0')}-'
|
||||||
|
'${slotDayEt.day.toString().padLeft(2, '0')}';
|
||||||
|
final Map<String, dynamic> slotDay = days.cast<Map<String, dynamic>>().firstWhere(
|
||||||
|
(Map<String, dynamic> d) => d['date'] == slotDayWire,
|
||||||
|
);
|
||||||
|
expect(slotDay['fullySyncedSlots'], 1);
|
||||||
|
expect(slotDay['completedSlots'], greaterThan(0));
|
||||||
});
|
});
|
||||||
|
|
||||||
test('resync returns 503 when sync is disabled in portal config', () async {
|
test('resync returns 503 when sync is disabled in portal config', () async {
|
||||||
|
|||||||
@ -381,7 +381,7 @@ void main() {
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
test('accepts 4Hour bar rows and partial index exists', () async {
|
test('accepts sessionHalf bar rows and partial index exists', () async {
|
||||||
if (testDb == null) {
|
if (testDb == null) {
|
||||||
markTestSkipped(
|
markTestSkipped(
|
||||||
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
|
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
|
||||||
@ -392,8 +392,8 @@ void main() {
|
|||||||
await testDb!.marketDataDb.upsertSnapshot(
|
await testDb!.marketDataDb.upsertSnapshot(
|
||||||
symbol: 'SPY',
|
symbol: 'SPY',
|
||||||
metric: 'bar',
|
metric: 'bar',
|
||||||
timeframe: '4Hour',
|
timeframe: 'sessionHalf',
|
||||||
asOf: DateTime.utc(2026, 5, 26, 8),
|
asOf: DateTime.utc(2026, 6, 2, 13, 30),
|
||||||
price: 500,
|
price: 500,
|
||||||
);
|
);
|
||||||
|
|
||||||
@ -402,7 +402,7 @@ void main() {
|
|||||||
SELECT indexname
|
SELECT indexname
|
||||||
FROM pg_indexes
|
FROM pg_indexes
|
||||||
WHERE tablename = 'market_data_snapshots'
|
WHERE tablename = 'market_data_snapshots'
|
||||||
AND indexname = 'market_data_snapshots_bar_4h_idx'
|
AND indexname = 'market_data_snapshots_bar_session_half_idx'
|
||||||
''',
|
''',
|
||||||
);
|
);
|
||||||
expect(indexes, hasLength(1));
|
expect(indexes, hasLength(1));
|
||||||
|
|||||||
@ -2,7 +2,7 @@
|
|||||||
library;
|
library;
|
||||||
|
|
||||||
import 'package:cyberhybridhub_server/trading/market_history_config.dart';
|
import 'package:cyberhybridhub_server/trading/market_history_config.dart';
|
||||||
import 'package:cyberhybridhub_server/trading/market_history_four_hour_slot.dart';
|
import 'package:cyberhybridhub_server/trading/market_history_session_slot.dart';
|
||||||
import 'package:cyberhybridhub_server/trading/market_history_week_coverage.dart';
|
import 'package:cyberhybridhub_server/trading/market_history_week_coverage.dart';
|
||||||
import 'package:cyberhybridhub_server/trading/tradable_assets_db.dart';
|
import 'package:cyberhybridhub_server/trading/tradable_assets_db.dart';
|
||||||
import 'package:cyberhybridhub_server/alpaca/alpaca_models.dart';
|
import 'package:cyberhybridhub_server/alpaca/alpaca_models.dart';
|
||||||
@ -14,6 +14,7 @@ void main() {
|
|||||||
TestDb? testDb;
|
TestDb? testDb;
|
||||||
|
|
||||||
setUpAll(() async {
|
setUpAll(() async {
|
||||||
|
ensureMarketHistoryTimezonesInitialized();
|
||||||
testDb = await TestDb.open();
|
testDb = await TestDb.open();
|
||||||
});
|
});
|
||||||
|
|
||||||
@ -35,8 +36,10 @@ void main() {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
final DateTime now = DateTime.utc(2026, 5, 31, 0, 30);
|
final DateTime now = DateTime.utc(2026, 5, 29, 21, 0);
|
||||||
final DateTime slotStart = DateTime.utc(2026, 5, 30, 20);
|
final DateTime slotStart = MarketHistorySessionSlot.slotStartContaining(
|
||||||
|
DateTime.utc(2026, 5, 29, 17, 0),
|
||||||
|
);
|
||||||
|
|
||||||
await TradableAssetsDb(testDb!.connection).upsertAll(
|
await TradableAssetsDb(testDb!.connection).upsertAll(
|
||||||
<AlpacaAsset>[
|
<AlpacaAsset>[
|
||||||
@ -68,12 +71,11 @@ void main() {
|
|||||||
|
|
||||||
expect(report.symbolCount, 1);
|
expect(report.symbolCount, 1);
|
||||||
|
|
||||||
final MarketHistoryDayCoverage saturday = report.days.singleWhere(
|
final MarketHistoryDayCoverage friday = report.days.singleWhere(
|
||||||
(MarketHistoryDayCoverage day) => day.date == DateTime.utc(2026, 5, 30),
|
(MarketHistoryDayCoverage day) => day.date == DateTime.utc(2026, 5, 29),
|
||||||
);
|
);
|
||||||
final MarketHistorySlotCoverage slot = saturday.slots.singleWhere(
|
final MarketHistorySlotCoverage slot = friday.slots.singleWhere(
|
||||||
(MarketHistorySlotCoverage s) =>
|
(MarketHistorySlotCoverage s) => s.slotStart == slotStart,
|
||||||
s.slotStart == DateTime.utc(2026, 5, 30, 20),
|
|
||||||
);
|
);
|
||||||
|
|
||||||
expect(slot.completed, isTrue);
|
expect(slot.completed, isTrue);
|
||||||
|
|||||||
@ -120,7 +120,7 @@ void main() {
|
|||||||
final DateTime now = DateTime.utc(2026, 5, 30, 22);
|
final DateTime now = DateTime.utc(2026, 5, 30, 22);
|
||||||
final AdminRunSeverity severity = deriveSeverity(
|
final AdminRunSeverity severity = deriveSeverity(
|
||||||
error:
|
error:
|
||||||
'Alpaca returned no persistable 4Hour bars; slot=2026-05-30T20:00:00Z; rows_written=0',
|
'Alpaca returned no persistable sessionHalf bars; slot=2026-05-30T20:00:00Z; rows_written=0',
|
||||||
startedAt: now.subtract(const Duration(minutes: 5)),
|
startedAt: now.subtract(const Duration(minutes: 5)),
|
||||||
finishedAt: now,
|
finishedAt: now,
|
||||||
now: now,
|
now: now,
|
||||||
|
|||||||
@ -1,88 +0,0 @@
|
|||||||
import 'package:cyberhybridhub_server/trading/market_history_four_hour_slot.dart';
|
|
||||||
import 'package:test/test.dart';
|
|
||||||
|
|
||||||
void main() {
|
|
||||||
group('MarketHistoryFourHourSlot', () {
|
|
||||||
test('slotStartContaining floors to UTC 4-hour boundary', () {
|
|
||||||
expect(
|
|
||||||
MarketHistoryFourHourSlot.slotStartContaining(
|
|
||||||
DateTime.utc(2026, 5, 26, 10, 30),
|
|
||||||
),
|
|
||||||
DateTime.utc(2026, 5, 26, 8),
|
|
||||||
);
|
|
||||||
expect(
|
|
||||||
MarketHistoryFourHourSlot.slotStartContaining(
|
|
||||||
DateTime.utc(2026, 5, 26, 0),
|
|
||||||
),
|
|
||||||
DateTime.utc(2026, 5, 26, 0),
|
|
||||||
);
|
|
||||||
});
|
|
||||||
|
|
||||||
test('endInclusive is three hours fifty-nine minutes after start', () {
|
|
||||||
final DateTime start = DateTime.utc(2026, 5, 30, 0);
|
|
||||||
expect(
|
|
||||||
MarketHistoryFourHourSlot.endInclusive(start),
|
|
||||||
DateTime.utc(2026, 5, 30, 3, 59, 59),
|
|
||||||
);
|
|
||||||
});
|
|
||||||
|
|
||||||
test('lastCompletedSlotStart at slot boundary is previous slot', () {
|
|
||||||
expect(
|
|
||||||
MarketHistoryFourHourSlot.lastCompletedSlotStart(
|
|
||||||
DateTime.utc(2026, 5, 26, 12),
|
|
||||||
),
|
|
||||||
DateTime.utc(2026, 5, 26, 8),
|
|
||||||
);
|
|
||||||
});
|
|
||||||
|
|
||||||
test('lastCompletedSlotStart mid-slot is previous slot', () {
|
|
||||||
expect(
|
|
||||||
MarketHistoryFourHourSlot.lastCompletedSlotStart(
|
|
||||||
DateTime.utc(2026, 5, 26, 10, 30),
|
|
||||||
),
|
|
||||||
DateTime.utc(2026, 5, 26, 4),
|
|
||||||
);
|
|
||||||
});
|
|
||||||
|
|
||||||
test('completedSlotStartsInWindow excludes in-progress slot', () {
|
|
||||||
final DateTime now = DateTime.utc(2026, 5, 26, 10, 30);
|
|
||||||
final List<DateTime> slots =
|
|
||||||
MarketHistoryFourHourSlot.completedSlotStartsInWindow(now, 1);
|
|
||||||
expect(slots, isNot(contains(DateTime.utc(2026, 5, 26, 8))));
|
|
||||||
expect(slots, contains(DateTime.utc(2026, 5, 26, 4)));
|
|
||||||
});
|
|
||||||
|
|
||||||
test('five completed slots on a UTC day before the 20:00 block ends', () {
|
|
||||||
final List<DateTime> slots =
|
|
||||||
MarketHistoryFourHourSlot.completedSlotStartsInWindow(
|
|
||||||
DateTime.utc(2026, 5, 26, 23, 59),
|
|
||||||
1,
|
|
||||||
);
|
|
||||||
final Set<int> hours = slots
|
|
||||||
.where((DateTime s) => s.day == 26)
|
|
||||||
.map((DateTime s) => s.hour)
|
|
||||||
.toSet();
|
|
||||||
expect(hours, <int>{0, 4, 8, 12, 16});
|
|
||||||
});
|
|
||||||
|
|
||||||
test('wireUtc uses canonical YYYY-MM-DDTHH:MM:SSZ without fractional seconds',
|
|
||||||
() {
|
|
||||||
expect(
|
|
||||||
MarketHistoryFourHourSlot.wireUtc(DateTime.utc(2026, 5, 26, 8)),
|
|
||||||
'2026-05-26T08:00:00Z',
|
|
||||||
);
|
|
||||||
expect(
|
|
||||||
MarketHistoryFourHourSlot.wireUtc(
|
|
||||||
DateTime.utc(2026, 5, 26, 8, 0, 0, 500),
|
|
||||||
),
|
|
||||||
'2026-05-26T08:00:00Z',
|
|
||||||
);
|
|
||||||
expect(
|
|
||||||
MarketHistoryFourHourSlot.slotStartWire(
|
|
||||||
DateTime.utc(2026, 5, 26, 10, 30),
|
|
||||||
),
|
|
||||||
'2026-05-26T08:00:00Z',
|
|
||||||
);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
}
|
|
||||||
@ -1,4 +1,4 @@
|
|||||||
import 'package:cyberhybridhub_server/trading/market_history_four_hour_slot.dart';
|
import 'package:cyberhybridhub_server/trading/market_history_session_slot.dart';
|
||||||
import 'package:cyberhybridhub_server/trading/market_history_question_audit.dart';
|
import 'package:cyberhybridhub_server/trading/market_history_question_audit.dart';
|
||||||
import 'package:test/test.dart';
|
import 'package:test/test.dart';
|
||||||
|
|
||||||
@ -35,7 +35,7 @@ void main() {
|
|||||||
});
|
});
|
||||||
|
|
||||||
group('compareUntil navigation', () {
|
group('compareUntil navigation', () {
|
||||||
final DateTime now = DateTime.utc(2026, 5, 30, 15, 30);
|
final DateTime now = DateTime.utc(2026, 6, 2, 21);
|
||||||
late DateTime defaultUntil;
|
late DateTime defaultUntil;
|
||||||
|
|
||||||
setUp(() {
|
setUp(() {
|
||||||
@ -49,9 +49,12 @@ void main() {
|
|||||||
);
|
);
|
||||||
final (DateTime newer, DateTime older) = questionAuditSlotPair(stepped);
|
final (DateTime newer, DateTime older) = questionAuditSlotPair(stepped);
|
||||||
final DateTime last =
|
final DateTime last =
|
||||||
MarketHistoryFourHourSlot.lastCompletedSlotStart(now);
|
MarketHistorySessionSlot.lastCompletedSlotStart(now);
|
||||||
expect(newer, last.subtract(const Duration(hours: 4)));
|
expect(
|
||||||
expect(older, last.subtract(const Duration(hours: 8)));
|
newer,
|
||||||
|
MarketHistorySessionSlot.previousSlotStart(last) ?? last,
|
||||||
|
);
|
||||||
|
expect(older, MarketHistorySessionSlot.previousSlotStart(newer));
|
||||||
});
|
});
|
||||||
|
|
||||||
test('step newer from stepped returns to default pair', () {
|
test('step newer from stepped returns to default pair', () {
|
||||||
|
|||||||
85
server/test/trading/market_history_session_slot_test.dart
Normal file
85
server/test/trading/market_history_session_slot_test.dart
Normal file
@ -0,0 +1,85 @@
|
|||||||
|
import 'package:cyberhybridhub_server/trading/market_history_session_slot.dart';
|
||||||
|
import 'package:test/test.dart';
|
||||||
|
|
||||||
|
void main() {
|
||||||
|
setUp(ensureMarketHistoryTimezonesInitialized);
|
||||||
|
|
||||||
|
group('MarketHistorySessionSlot', () {
|
||||||
|
test('morning slot start in EDT is 13:30 UTC', () {
|
||||||
|
final DateTime instant = DateTime.utc(2026, 6, 2, 14, 0);
|
||||||
|
expect(
|
||||||
|
MarketHistorySessionSlot.slotStartContaining(instant),
|
||||||
|
DateTime.utc(2026, 6, 2, 13, 30),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('morning slot start in EST is 14:30 UTC', () {
|
||||||
|
final DateTime instant = DateTime.utc(2026, 1, 6, 15, 0);
|
||||||
|
expect(
|
||||||
|
MarketHistorySessionSlot.slotStartContaining(instant),
|
||||||
|
DateTime.utc(2026, 1, 6, 14, 30),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('afternoon slot start in EDT is 16:45 UTC', () {
|
||||||
|
final DateTime instant = DateTime.utc(2026, 6, 2, 17, 0);
|
||||||
|
expect(
|
||||||
|
MarketHistorySessionSlot.slotStartContaining(instant),
|
||||||
|
DateTime.utc(2026, 6, 2, 16, 45),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('endExclusive is 195 minutes after start', () {
|
||||||
|
final DateTime start = DateTime.utc(2026, 6, 2, 13, 30);
|
||||||
|
expect(
|
||||||
|
MarketHistorySessionSlot.endExclusive(start),
|
||||||
|
start.add(MarketHistorySessionSlot.slotDuration),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('lastCompletedSlotStart after 4pm ET', () {
|
||||||
|
final DateTime now = DateTime.utc(2026, 6, 2, 21, 0);
|
||||||
|
expect(
|
||||||
|
MarketHistorySessionSlot.lastCompletedSlotStart(now),
|
||||||
|
DateTime.utc(2026, 6, 2, 16, 45),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('completedSlotStartsInWindow includes morning and afternoon', () {
|
||||||
|
final DateTime now = DateTime.utc(2026, 6, 2, 21, 0);
|
||||||
|
final List<DateTime> slots =
|
||||||
|
MarketHistorySessionSlot.completedSlotStartsInWindow(now, 1);
|
||||||
|
final List<DateTime> utcSlots =
|
||||||
|
slots.map((DateTime s) => s.toUtc()).toList();
|
||||||
|
expect(
|
||||||
|
utcSlots.any(
|
||||||
|
(DateTime s) =>
|
||||||
|
s.isAtSameMomentAs(DateTime.utc(2026, 6, 2, 13, 30)),
|
||||||
|
),
|
||||||
|
isTrue,
|
||||||
|
);
|
||||||
|
expect(
|
||||||
|
utcSlots.any(
|
||||||
|
(DateTime s) =>
|
||||||
|
s.isAtSameMomentAs(DateTime.utc(2026, 6, 2, 16, 45)),
|
||||||
|
),
|
||||||
|
isTrue,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('previousSlotStart walks afternoon to morning', () {
|
||||||
|
final DateTime afternoon = DateTime.utc(2026, 6, 2, 16, 45);
|
||||||
|
expect(
|
||||||
|
MarketHistorySessionSlot.previousSlotStart(afternoon),
|
||||||
|
DateTime.utc(2026, 6, 2, 13, 30),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('wireUtc includes minutes', () {
|
||||||
|
expect(
|
||||||
|
MarketHistorySessionSlot.wireUtc(DateTime.utc(2026, 6, 2, 13, 30)),
|
||||||
|
'2026-06-02T13:30:00Z',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
@ -1,35 +1,29 @@
|
|||||||
import 'package:cyberhybridhub_server/trading/market_history_four_hour_slot.dart';
|
import 'package:cyberhybridhub_server/trading/market_history_session_slot.dart';
|
||||||
import 'package:cyberhybridhub_server/trading/market_history_week_coverage.dart';
|
import 'package:cyberhybridhub_server/trading/market_history_week_coverage.dart';
|
||||||
import 'package:test/test.dart';
|
import 'package:test/test.dart';
|
||||||
|
|
||||||
void main() {
|
void main() {
|
||||||
|
setUp(ensureMarketHistoryTimezonesInitialized);
|
||||||
|
|
||||||
group('MarketHistoryWeekCoverage calendar days', () {
|
group('MarketHistoryWeekCoverage calendar days', () {
|
||||||
test('returns windowDays UTC days ending on today', () {
|
test('returns windowDays Eastern dates ending on today', () {
|
||||||
final DateTime now = DateTime.utc(2026, 5, 30, 15, 30);
|
final DateTime now = DateTime.utc(2026, 6, 2, 21);
|
||||||
final List<DateTime> days =
|
final List<(int, int, int)> days =
|
||||||
MarketHistoryWeekCoverage.calendarDaysEndingToday(now, 7);
|
MarketHistoryWeekCoverage.calendarDaysEndingTodayEt(now, 7);
|
||||||
|
|
||||||
expect(days, hasLength(7));
|
expect(days, hasLength(7));
|
||||||
expect(days.first, DateTime.utc(2026, 5, 24));
|
expect(days.last, (2026, 6, 2));
|
||||||
expect(days.last, DateTime.utc(2026, 5, 30));
|
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
group('slot completion for today', () {
|
group('slot completion for today', () {
|
||||||
test('marks only ended slots completed at 15:30 UTC', () {
|
test('marks ended session halves completed after 4pm ET', () {
|
||||||
final DateTime now = DateTime.utc(2026, 5, 30, 15, 30);
|
final DateTime now = DateTime.utc(2026, 6, 2, 21);
|
||||||
final DateTime day = DateTime.utc(2026, 5, 30);
|
final DateTime morning = DateTime.utc(2026, 6, 2, 13, 30);
|
||||||
var completed = 0;
|
final DateTime afternoon = DateTime.utc(2026, 6, 2, 16, 45);
|
||||||
|
|
||||||
for (int hour = 0; hour < 24; hour += MarketHistoryFourHourSlot.slotHours) {
|
expect(MarketHistorySessionSlot.hasEnded(morning, now), isTrue);
|
||||||
final DateTime slotStart = DateTime.utc(day.year, day.month, day.day, hour);
|
expect(MarketHistorySessionSlot.hasEnded(afternoon, now), isTrue);
|
||||||
if (MarketHistoryFourHourSlot.hasEnded(slotStart, now)) {
|
|
||||||
completed++;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Slots 00, 04, 08 end before 15:30; 12:00 slot ends at 16:00 UTC.
|
|
||||||
expect(completed, 3);
|
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|||||||
@ -2,14 +2,14 @@ import 'package:cyberhybridhub/admin/utils/sync_run_formatters.dart';
|
|||||||
import 'package:flutter_test/flutter_test.dart';
|
import 'package:flutter_test/flutter_test.dart';
|
||||||
|
|
||||||
void main() {
|
void main() {
|
||||||
test('formatMarketHistorySlotWire matches server Alpaca start form', () {
|
test('formatMarketHistorySlotWire matches server slot start wire form', () {
|
||||||
expect(
|
expect(
|
||||||
formatMarketHistorySlotWire(DateTime.utc(2026, 5, 26, 8)),
|
formatMarketHistorySlotWire(DateTime.utc(2026, 5, 26, 13, 30)),
|
||||||
'2026-05-26T08:00:00Z',
|
'2026-05-26T13:30:00Z',
|
||||||
);
|
);
|
||||||
expect(
|
expect(
|
||||||
formatMarketHistorySlotWire(DateTime.utc(2026, 5, 26, 10, 30)),
|
formatMarketHistorySlotWire(DateTime.utc(2026, 5, 26, 16, 45)),
|
||||||
'2026-05-26T08:00:00Z',
|
'2026-05-26T16:45:00Z',
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
@ -139,7 +139,8 @@ void main() {
|
|||||||
await tester.pumpAndSettle();
|
await tester.pumpAndSettle();
|
||||||
|
|
||||||
expect(find.text(slotWire), findsOneWidget);
|
expect(find.text(slotWire), findsOneWidget);
|
||||||
expect(find.textContaining('2 assets: A, AA'), findsOneWidget);
|
expect(find.text('2 assets'), findsOneWidget);
|
||||||
|
expect(find.textContaining('A, AA'), findsNothing);
|
||||||
expect(
|
expect(
|
||||||
find.textContaining('Backfill fetches (Alpaca start / raw.slot_start)'),
|
find.textContaining('Backfill fetches (Alpaca start / raw.slot_start)'),
|
||||||
findsOneWidget,
|
findsOneWidget,
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user