Pretty close

This commit is contained in:
Nathan Anderson 2026-05-31 11:17:12 -05:00
parent 8278f93a34
commit 3af1e31fac
124 changed files with 17034 additions and 150 deletions

65
.github/workflows/admin-portal.yml vendored Normal file
View File

@ -0,0 +1,65 @@
name: Admin portal tests
on:
push:
paths:
- 'lib/admin/**'
- 'test/admin/**'
- 'server/lib/handlers/market_history_admin_handler.dart'
- 'server/lib/trading/market_history_admin_*.dart'
- 'server/test/trading/market_history_admin_*.dart'
- 'server/test/integration/market_history_admin_risk_test.dart'
- 'server/test/helpers/admin_sync_run_fixtures.dart'
- 'scripts/test-admin-portal.sh'
- '.github/workflows/admin-portal.yml'
pull_request:
paths:
- 'lib/admin/**'
- 'test/admin/**'
- 'server/lib/handlers/market_history_admin_handler.dart'
- 'server/lib/trading/market_history_admin_*.dart'
- 'server/test/trading/market_history_admin_*.dart'
- 'server/test/integration/market_history_admin_risk_test.dart'
- 'server/test/helpers/admin_sync_run_fixtures.dart'
- 'scripts/test-admin-portal.sh'
- '.github/workflows/admin-portal.yml'
jobs:
test:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: postgres
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
env:
TEST_DATABASE_URL: postgres://postgres:postgres@localhost:5432/postgres
steps:
- uses: actions/checkout@v4
- uses: dart-lang/setup-dart@v1
- uses: subosito/flutter-action@v2
with:
channel: stable
- name: Run admin portal test suite
env:
TZ: UTC
run: bash scripts/test-admin-portal.sh
- name: Enforce admin portal coverage thresholds
env:
TZ: UTC
run: bash scripts/check-admin-portal-coverage.sh

461
FLUTTER-ADMIN-PORTAL.md Normal file
View File

@ -0,0 +1,461 @@
# Flutter Admin Portal — Development Plan
**Status:** Planning
**Related:** `TODO.md` §8 (operational CLIs), §8.5 (admin API), rolling market-history pipeline (§1§7)
**Audience:** Mobile/web Flutter developers, API maintainers, operators
---
## 1. Purpose
Build an **admin-only** area in the Flutter app that shows a **live audit log** of periodic market-history operations:
- **Universe sync** (`kind=universe`) — tradable asset refresh
- **Historical backfill** (`kind=backfill`) — ended **4-hour UTC slots** (`4Hour` bars) into `market_data_snapshots`; open slot not fetched yet
- **Retention / cleanup** (`kind=cleanup`) — prune (or archive-then-delete) expired rows
Operators use it to confirm the rolling 7-day window is healthy, see when cleanup ran, and **notice Alpaca/API failures or rate limits immediately** without reading server logs or SQL.
This plan **replaces the need for §8 shell CLIs** for day-to-day product use. CLIs remain optional for CI/SRE (see §12).
---
## 2. Goals and non-goals
### Goals
| # | Goal |
|---|------|
| G1 | Chronological log: **newest events at the top**; scroll down for older history |
| G2 | **Pin unresolved failures** (API errors, rate limits, partial batch failures) at the top until a **successful retry** for the same `kind` (and optionally same error “thread”) |
| G3 | **Expandable list rows** — collapsed summary; expanded shows structured detail **without** raw JSON payloads (no bar blobs, no `tradable_assets.raw`) |
| G4 | For API failures, expanded view shows **full operator-relevant error text** (HTTP status, Alpaca message body summary, batch scope) |
| G5 | Optional: **on-demand** “Run sync” / “Run cleanup” from the portal (API-triggered), with new rows appearing in the log |
| G6 | Reuse existing auth (Firebase ID token → API) and app theme/navigation patterns |
### Non-goals (v1)
- End-user (non-admin) access to sync logs
- Displaying raw `market_data_snapshots` or per-symbol bar tables
- Full log export / SIEM integration (future)
- Replacing the background worker; scheduler still runs automatically when `MARKET_HISTORY_SYNC_ENABLED=true`
- Building §8 `bin/sync_market_history.dart` / `bin/cleanup_market_history.dart` (defer unless ops requests them)
---
## 3. Source of truth (already in Postgres)
All events are recorded in **`market_data_sync_runs`** (migration `005_market_history.sql`):
| Column | Meaning |
|--------|---------|
| `id` | Stable row id |
| `kind` | `universe` \| `backfill` \| `cleanup` |
| `started_at` | UTC start |
| `finished_at` | UTC end (null if crash mid-run — treat as in-progress) |
| `rows_written` | Upserted rows (universe/backfill) |
| `rows_removed` | Deleted rows (cleanup) |
| `error` | `NULL` = success; non-null = failure or **partial** failure message |
**Kinds map to UI labels:**
| `kind` | UI title | Typical success summary |
|--------|----------|-------------------------|
| `universe` | Asset universe sync | N assets refreshed |
| `backfill` | 4-hour slot backfill (ended slots only) | N bar rows written |
| `cleanup` | Data retention cleanup | N rows removed |
**Error shapes today (no schema change required for v1):**
- Thrown exception → `error` is `e.toString()` (e.g. `AlpacaMarketDataException: getBarsRange rate limited: 429 …`)
- Partial backfill → `error` may list batch failures while `rows_written > 0` (`SyncRunCounts.error` in `market_data_history.dart`)
**v1 recommendation:** Parse `error` text client-side for keywords (`429`, `rate`, `Alpaca`, `batch`) to set severity and pin rules. **v2 (optional):** add `error_code`, `http_status`, `severity` columns or a JSON `detail` object on insert (server change).
---
## 4. List ordering and “pin until resolved”
### 4.1 Default sort
Primary: **`started_at` DESC** (most recent at top).
### 4.2 Pin rule (failure priority)
An event is **pinned** when:
1. `error IS NOT NULL` **or** `finished_at IS NULL` (stuck in-progress > N minutes → show as warning), **and**
2. There is **no later successful run** of the **same `kind`** with `error IS NULL` and `finished_at IS NOT NULL` **after** this rows `started_at`.
**Pinned block** appears **above** the chronological list (fixed section or visually distinct header: “Needs attention”).
When a new successful run for that `kind` completes, the older failed row **unpins** and appears only in normal history order.
### 4.3 Rate limits and API errors
Treat as **high severity** when `error` matches (case-insensitive):
- `429`, `rate limit`, `rate limited`
- `AlpacaMarketDataException`, `AlpacaAssetsException`, `AlpacaTradingException`
- HTTP `5xx` patterns in message
Pinned rate-limit rows use a **warning/error** color and icon until the pin rule clears.
### 4.4 Partial success (backfill)
If `rows_written > 0` **and** `error != null`:
- Collapsed: **“Partial success”** badge + counts
- Expanded: show `rows_written`, full `error` text (batch list), **do not** show raw bar JSON
- Pin until next **fully successful** backfill (`error` null) **or** product decision: pin until error null even if partial (stricter — **recommended for ops**)
---
## 5. Backend API (prerequisite)
No admin market-history endpoints exist today. Implement **before** Flutter UI (or in parallel with mock API).
### 5.1 Auth model
| Approach | Recommendation |
|----------|----------------|
| Firebase + allowlist | `ADMIN_FIREBASE_UIDS=uid1,uid2` in server env; middleware rejects others with `403` |
| Custom claim | `admin: true` on Firebase token; server verifies claim |
| Separate admin API key | Avoid for mobile; poor UX |
Match existing pattern: `Authorization: Bearer <Firebase ID token>` like `QuestionsApiService`.
**Do not** expose these under `/v1/me/...` without admin checks — use `/v1/admin/...` prefix.
### 5.2 Endpoints (v1)
#### `GET /v1/admin/market-history/sync-runs`
Query params:
| Param | Default | Description |
|-------|---------|-------------|
| `limit` | `50` | Page size |
| `before` | — | Cursor: `id` or `started_at` of oldest item in current page (for infinite scroll) |
| `kind` | — | Optional filter: `universe`, `backfill`, `cleanup` |
Response:
```json
{
"runs": [
{
"id": 123,
"kind": "cleanup",
"startedAt": "2026-05-26T10:00:00Z",
"finishedAt": "2026-05-26T10:00:05Z",
"rowsWritten": 0,
"rowsRemoved": 4200,
"error": null,
"severity": "ok",
"status": "success",
"durationMs": 5000,
"summary": "Removed 4,200 expired snapshots"
}
],
"pinned": [
{
"id": 120,
"kind": "backfill",
"error": "getBarsRange rate limited: 429 ...",
"severity": "rate_limit",
"status": "failed",
"summary": "Backfill partial: 12,000 rows; rate limited"
}
],
"nextBefore": "2026-05-25T08:00:00Z"
}
```
Server computes `pinned[]` with the rule in §4.2.
Server computes `severity`: `ok` \| `warning` \| `error` \| `rate_limit` from `error` text + `finished_at`.
**Never** include `raw` JSON from snapshots or assets in this API.
#### `GET /v1/admin/market-history/sync-runs/{id}`
Single run for expanded row refresh (optional if list payload is enough).
#### `POST /v1/admin/market-data/resync` (optional v1.1)
Runs universe + slot backfill (ended `4Hour` slots in window, not the open slot). Returns `{ "runIds": […] }` (202). `windowDays` on cleanup only today.
#### `POST /v1/admin/market-data/cleanup` (optional v1.1)
Query: `windowDays`, `archive=true|false`.
Runs `MarketDataRetention.run(archive: …)` once.
### 5.3 Implementation notes (server)
- New handler: `server/lib/handlers/market_history_admin_handler.dart`
- Mount only when `ADMIN_PORTAL_ENABLED=true` (or reuse `TRADING_DEV_ENDPOINTS_ENABLED`**prefer dedicated flag**)
- Reuse `TradableAssetsSync`, `MarketDataHistorySync`, `MarketDataRetention` — same as `bin/server.dart` scheduler wiring
- List query: indexed `ORDER BY started_at DESC` (index exists: `market_data_sync_runs_kind_started_idx`)
### 5.4 Error detail for API failures (v2 enhancement)
When recording sync failures, optionally persist structured fields:
```json
{
"httpStatus": 429,
"endpoint": "GET /v2/stocks/bars",
"message": "...",
"batchSymbols": ["AAPL", "MSFT"]
}
```
Store in new `detail JSONB` column or append to `error` as delimited text for v1.
---
## 6. Flutter architecture
### 6.1 Entry and routing
| Item | Proposal |
|------|----------|
| Route | `/admin/market-history` or dedicated `AdminPortalScreen` pushed from a hidden entry |
| Gate | `AdminGate` widget: calls `GET /v1/admin/market-history/sync-runs?limit=1``403` → “Not authorized” |
| Discovery | Long-press app logo, `kDebugMode` menu item, or profile flag `isAdmin` from `GET /v1/me/profile` extension |
Do **not** show admin nav to all signed-in users.
### 6.2 Layering (match existing app)
```
lib/
admin/
models/
sync_run_event.dart # DTO from API
services/
market_history_admin_api.dart # HTTP + auth headers (mirror QuestionsApiService)
repositories/
sync_run_log_repository.dart # paging, pin merge, refresh
screens/
market_history_log_screen.dart
widgets/
sync_run_expansion_tile.dart
pinned_alerts_section.dart
sync_run_status_chip.dart
```
- **API service:** `apiBaseUrl` from `lib/config/api_config.dart`, token via `AuthService.instance.getIdToken()`
- **State:** `Listenable`/`ChangeNotifier` or `flutter_riverpod` if project adopts it later; v1 can use `StatefulWidget` + repository like profile sync
### 6.3 `SyncRunEvent` model (client)
Fields from API plus derived:
- `isPinned` (from `pinned` array or client recompute)
- `Severity` enum: `ok`, `warning`, `error`, `rateLimit`
- `displayTitle` from `kind`
- `collapsedSubtitle` — one line: time ago + summary
- `expandedSections``List<DetailSection>` title/body pairs (no raw JSON)
### 6.4 Expanded row content (templates)
**Success — cleanup**
- Started / finished (local timezone)
- Duration
- Rows removed
- Archive mode on/off (if API adds flag later)
**Success — backfill**
- Rows written
- Window days (from env or API meta)
- “No errors”
**Failure / rate limit**
- Full `error` string (selectable text)
- Parsed HTTP status if regex finds it
- Kind + run id (for support tickets)
- Hint: “Will retry on next scheduled run” or button **Retry now** if POST endpoint exists
**Partial backfill**
- Rows written (success count)
- Error block (batch failures)
- Do **not** render `raw` from bars
**In progress**
- Started at; spinner if `finished_at == null` and started &lt; 30m ago
- Stale run warning if &gt; 30m without `finished_at`
---
## 7. UI specification
### 7.1 Screen layout
```
┌─────────────────────────────────────┐
│ ← Market history log [↻] │ AppBar: refresh
├─────────────────────────────────────┤
│ NEEDS ATTENTION (0n) │ Pinned section (hidden if empty)
│ ⚠ Backfill — Rate limited 2h ago │ ExpansionTile, error styling
│ ✗ Cleanup — failed 1d ago │
├─────────────────────────────────────┤
│ HISTORY │ Section header
│ ✓ Cleanup — 4.2k removed 10:00 │ Newest first
│ ✓ Backfill — 98k written 09:00 │
│ ✓ Universe — 8.2k assets 08:00 │
│ ... │ Infinite scroll
└─────────────────────────────────────┘
```
### 7.2 Collapsed row
- Leading icon by `severity` / `kind`
- Title: `{Kind label} — {short status}`
- Trailing: relative time (`2h ago`)
- Optional chips: `rows_written`, `rows_removed`
### 7.3 Expanded row
- `ExpansionTile` children: labeled rows (not monospace dumps)
- **Selectable** `SelectableText` for full error message
- Copy-to-clipboard button for error block
### 7.4 Refresh and pagination
| Action | Behavior |
|--------|----------|
| Pull-to-refresh | Reload first page + recompute pins |
| Scroll end | `before=` cursor load next page, **append** to history (not pinned) |
| Auto-refresh | Optional 60s timer while screen visible |
### 7.5 On-demand actions (v1.1)
AppBar overflow menu:
- **Run backfill now**`POST resync` → show in-progress row → refresh
- **Run cleanup now**`POST cleanup` → confirm dialog (destructive) → refresh
Disable buttons while any `kind` has in-progress run (`finished_at == null`).
---
## 8. Phased delivery
### Phase A — Read-only log (MVP)
| Track | Work |
|-------|------|
| Server | `GET sync-runs` + admin auth + `pinned` computation |
| Flutter | Models, API service, log screen, expansion tiles, pull-to-refresh |
| QA | Seed `market_data_sync_runs` in test DB; verify pin/unpin when new success inserted |
**Exit:** Operator can view retention/cleanup/backfill history on device/emulator.
### Phase B — Failure UX polish
- Severity icons/colors, rate-limit copy, partial-success template
- Copy error, relative timestamps, empty/loading/error states
- Widget tests for sort/pin merge logic
### Phase C — On-demand triggers
- `POST resync`, `POST cleanup`
- Flutter buttons + confirmation + in-progress state
### Phase D — Structured errors (optional)
- Server writes `detail JSONB`; Flutter renders HTTP status, endpoint, batch symbols without parsing free text
---
## 9. Testing
### Server
| Test | Description |
|------|-------------|
| Integration | `test/integration/market_history_admin_handler_test.dart` — 403 non-admin, 200 list shape, pinned logic |
| Fixture rows | Success cleanup, failed backfill 429, partial backfill |
### Flutter
| Test | Description |
|------|-------------|
| Unit | `sync_run_log_repository_test.dart` — merge pinned + chronological, pin clearance |
| Unit | `sync_run_event_test.dart` — severity from error string |
| Widget | Expansion tile shows/hides detail; pinned section visibility |
| Integration | Mock HTTP → screen goldens or pump tests |
### Manual
1. Enable worker sync; wait for 3 kinds in log
2. Simulate 429 (mock Alpaca or test env throttle)
3. Confirm pin → run successful backfill → pin clears
---
## 10. Security and compliance
- Admin routes **only** with verified admin identity
- Logs may contain Alpaca error bodies — **no PII**, but treat as internal ops data
- Do not log API keys; ensure `error` column never stores secrets (review `SyncRunRecorder` callers)
- CORS: same as existing API (`apiCorsHeaders`)
- Production: `ADMIN_PORTAL_ENABLED=false` by default
---
## 11. Configuration (align with §7 env)
Document in `server/README.md` when implemented:
| Env var | Purpose |
|---------|---------|
| `ADMIN_PORTAL_ENABLED` | Mount `/v1/admin/market-history/*` |
| `ADMIN_FIREBASE_UIDS` | Comma-separated allowlist |
| `MARKET_HISTORY_*` | Window, sync cadence (worker); shown in expanded row as read-only meta |
---
## 12. Relationship to §8 CLIs
| Capability | Admin portal | §8 CLI |
|------------|----------------|--------|
| View sync/cleanup log | Yes (primary) | No |
| On-demand sync/cleanup | Yes (Phase C API) | Yes (planned) |
| CI / headless / no UI | No | Yes |
| Ops SSH without app | No | Yes |
**Recommendation:** Implement **portal + admin API** first; defer §8 CLIs unless SRE requests them.
---
## 13. Open decisions
| # | Question | Default proposal |
|---|----------|------------------|
| 1 | Admin discovery in app? | Debug-only entry + allowlist |
| 2 | Pin partial backfill with `rows_written > 0`? | Yes, until `error` null |
| 3 | Separate web-only admin build? | Single app, gated route (v1) |
| 4 | Real-time updates? | Pull-to-refresh only (v1); WebSocket later |
| 5 | `finished_at` null stale threshold? | 30 minutes → warning row |
---
## 14. Acceptance criteria
- [ ] Admin user sees **newest** history events at top of main list
- [ ] Failed/rate-limited runs appear in **Needs attention** until same `kind` succeeds later
- [ ] Expand row shows operational detail **without** raw market-data JSON
- [ ] API error expanded view shows **complete** `error` text from server
- [ ] Non-admin receives 403 and no data
- [ ] Pull-to-refresh and pagination load additional older events
- [ ] (Phase C) On-demand cleanup/sync creates visible new log entries
---
*Document version: 1.0 — Flutter admin portal for `market_data_sync_runs` audit log.*

307
FLUTTER-TDD-PLAN.md Normal file
View File

@ -0,0 +1,307 @@
# Flutter Admin Portal TDD Plan
**Scope:** Validate the functionality defined in `FLUTTER-ADMIN-PORTAL.md` using test-first development across server API + Flutter UI/repository layers.
**Primary objective:** Ship an admin portal that reliably shows retention/cleanup/backfill history, prioritizes unresolved failures/rate limits, and presents concise expandable details.
---
## 1. Test strategy at a glance
Use a layered pyramid so core behavior is proven at the cheapest layer first:
| Layer | Purpose | Tools |
|---|---|---|
| Unit (server) | pin/severity/status logic, query mapping | `dart test` |
| Integration (server) | auth, DB query shape/order, endpoint behavior | `dart test` + Postgres test DB |
| Unit (Flutter) | DTO mapping, repository merge, parsing, formatting | `flutter test` |
| Widget (Flutter) | pinned section, newest-first list, expansion details, pagination interactions | `flutter test` |
| Optional e2e/manual | confidence pass with real API + seeded rows | local runbook |
---
## 2. Coverage targets (reasonable)
Aim for practical coverage without over-testing visual polish:
- **Server admin handler package:** >= **85% line coverage**
- **Server pin/severity helper logic:** >= **95% line coverage**
- **Flutter admin repository/models:** >= **90% line coverage**
- **Flutter admin widgets/screens:** >= **75% line coverage**
- **Critical behavior cases:** **100% scenario coverage** for:
- unresolved errors pinned
- unpin after successful retry by same `kind`
- newest-first ordering
- expanded full error text display
- no raw data fields rendered
---
## 3. Test data model fixtures
Create reusable fixture builders for `market_data_sync_runs` rows:
```text
run(id, kind, startedAt, finishedAt, rowsWritten, rowsRemoved, error)
```
Named fixture sets:
1. `all_success_recent_first`
2. `rate_limit_unresolved`
3. `failed_then_success_same_kind`
4. `partial_backfill_error`
5. `in_progress_stale`
6. `mixed_kinds_mixed_outcomes`
Keep fixtures concise; avoid raw bar payloads by design.
---
## 4. Server TDD plan
Target files (planned):
- `server/lib/handlers/market_history_admin_handler.dart`
- `server/lib/trading/market_history_admin_query.dart` (or equivalent helper)
- `server/test/integration/market_history_admin_handler_test.dart`
- `server/test/trading/market_history_admin_logic_test.dart`
### 4.1 RED: pin/severity/status logic unit tests
Write tests before implementation:
1. **Pinned unresolved failure**
- failed `backfill` with no later success => pinned
2. **Unpin after retry success**
- failed `backfill` then later successful `backfill` => old failure not pinned
3. **Pinned is per-kind**
- `cleanup` success does not clear `backfill` failure pin
4. **Rate limit classification**
- `error` containing `429` or `rate limited` => `severity=rate_limit`
5. **Partial success classification**
- `rows_written>0 && error!=null` => `status=partial`
6. **In-progress classification**
- `finished_at==null` recent => `status=in_progress`
- stale threshold exceeded => `severity=warning/error` (per chosen policy)
7. **Newest-first ordering**
- runs sorted by `started_at DESC`
### 4.2 GREEN: minimal implementation
Implement pure helper functions first:
- `deriveSeverity(error, finishedAt, startedAt, now)`
- `deriveStatus(error, finishedAt, rowsWritten, rowsRemoved)`
- `computePinned(runs)`
- `toSummary(run)`
Keep these deterministic and injectable with `clock` for testability.
### 4.3 RED: handler integration tests
`market_history_admin_handler_test.dart`:
1. **403 for non-admin token**
2. **200 + shape for admin**
3. **`runs` newest-first**
4. **`pinned` contains unresolved rate-limit row**
5. **`pinned` clears when later success seeded**
6. **`limit` respected**
7. **pagination via `before`**
8. **`kind` filter**
9. **response excludes raw fields** (no `raw`, no snapshot payload)
### 4.4 GREEN: handler + routing
Implement route:
- `GET /v1/admin/market-history/sync-runs`
Wire auth middleware with admin allowlist/claim gate.
### 4.5 REFACTOR
- Move query + transform logic out of handler into service/query class.
- Ensure all SQL is parameterized.
- Add response DTO class to reduce map-shape drift.
---
## 5. Flutter TDD plan
Planned files:
- `lib/admin/models/sync_run_event.dart`
- `lib/admin/services/market_history_admin_api.dart`
- `lib/admin/repositories/sync_run_log_repository.dart`
- `lib/admin/screens/market_history_log_screen.dart`
- `lib/admin/widgets/sync_run_expansion_tile.dart`
Tests:
- `test/admin/models/sync_run_event_test.dart`
- `test/admin/repositories/sync_run_log_repository_test.dart`
- `test/admin/services/market_history_admin_api_test.dart`
- `test/admin/widgets/sync_run_expansion_tile_test.dart`
- `test/admin/screens/market_history_log_screen_test.dart`
### 5.1 RED: model + repository unit tests
1. Parse API response into domain model
2. Severity/status mapping from API and fallback parsing
3. Merge pinned + history sections
4. Keep pinned visible above chronological list
5. Append paged results without reordering newer rows
6. Deduplicate by `id` during refresh
7. Error state handling (`401/403/500/network`)
### 5.2 GREEN: implement model/repository
- Implement immutable model objects
- Implement repository APIs:
- `loadInitial()`
- `refresh()`
- `loadMore()`
- Include simple in-memory cache for current session
### 5.3 RED: service HTTP tests
Using mocked HTTP client:
1. sends bearer token header
2. sends `limit`, `before`, `kind` query params
3. handles non-200 responses
4. parse malformed payload guardrails
### 5.4 GREEN: service implementation
Mirror existing app API style (`apiBaseUrl`, auth token retrieval, robust parse).
### 5.5 RED: widget/screen tests
1. **Pinned section shown** when pinned rows exist
2. **Pinned section hidden** when none exist
3. **Newest row at top** in history list
4. **Expansion tile details**
- shows full message text
- does not show raw data fields
5. **Severity styling** (icon/chip text present)
6. **Pull-to-refresh triggers repository refresh**
7. **Scroll-to-end triggers `loadMore()`**
8. **Loading, empty, and error states render correctly**
### 5.6 GREEN: implement UI
- Build screen with two sections:
- `Needs attention` (pinned)
- `History` (paged, newest-first)
- Use `ExpansionTile` rows with concise collapsed summary.
### 5.7 REFACTOR
- Extract common chips/icons/date formatters
- Keep widget tree shallow for testability
- Avoid business logic in widget build methods
---
## 6. API-trigger actions (optional phase C) TDD
If adding on-demand actions:
Server tests:
- `POST /v1/admin/market-data/resync` returns `202` and creates run rows
- `POST /v1/admin/market-data/cleanup` with `archive` flag routes correctly
- auth + invalid input tests
Flutter tests:
- retry button visible for pinned failures
- confirm dialog for cleanup
- action triggers refresh and shows new in-progress/success rows
This phase can be deferred without affecting MVP log validation.
---
## 7. Quality gates and CI commands
### Server
```bash
cd server && dart test test/trading/market_history_admin_logic_test.dart
cd server && dart test test/integration/market_history_admin_handler_test.dart
```
### Flutter
```bash
flutter test test/admin/models/
flutter test test/admin/repositories/
flutter test test/admin/services/
flutter test test/admin/widgets/
flutter test test/admin/screens/
```
### Coverage checks (recommended)
Server:
```bash
cd server && dart test --coverage=coverage
```
Flutter:
```bash
flutter test --coverage
```
Add CI thresholds for the targets in Section 2.
---
## 8. Risk-driven test checklist
High-priority cases to lock down early:
- [x] Pin logic regression when multiple kinds interleave
- [x] Rate-limit text variants (`429`, `rate limit`, capitalization differences)
- [x] Very long error message expansion rendering
- [x] Pagination duplicates/holes on refresh + load-more race
- [x] Timezone display mismatch (`started_at` UTC vs local display)
- [x] Empty payload safety (no crashes)
---
## 9. Definition of done
The admin portal feature is considered validated when:
1. All MVP tests listed above pass.
2. Coverage targets in Section 2 are met.
3. Automated acceptance tests (`test/admin/acceptance/`) cover:
- newest-first order,
- pinned unresolved issues,
- expandable full error detail,
- no raw payload details in UI,
- cleanup archive toggle when enabled.
4. CI includes these test suites and fails on regression.
Manual sanity pass (live server + seeded rows) remains recommended before production rollout.
---
## 10. Suggested implementation order
1. Server pure logic tests + implementation ✅
2. Server handler integration tests + endpoint ✅
3. Flutter models/repository/service tests + implementation ✅
4. Flutter widget/screen tests + UI implementation ✅
5. Optional trigger actions phase ✅
6. Risk checklist (§8) + acceptance tests (§9) ✅
This order keeps risky logic proven first and gives UI teams a stable API contract early.

View File

@ -0,0 +1,3 @@
abstract final class AdminRoutes {
static const String marketHistoryLog = '/admin/market-history';
}

View File

@ -0,0 +1,30 @@
import '../utils/sync_run_formatters.dart';
class BackfillSyncItem {
const BackfillSyncItem({
required this.slotStart,
required this.symbols,
this.slotStartWire,
});
final DateTime slotStart;
final List<String> symbols;
/// Exact `slotStart` string from the sync-run API (Alpaca `start` param).
final String? slotStartWire;
/// Wire form for DB spot checks: `raw->>'slot_start'` and Alpaca `start`.
String get fetchSlotStartWire =>
slotStartWire ?? formatMarketHistorySlotWire(slotStart);
factory BackfillSyncItem.fromJson(Map<String, dynamic> json) {
final String slotStartRaw = json['slotStart'] as String;
return BackfillSyncItem(
slotStart: DateTime.parse(slotStartRaw).toUtc(),
slotStartWire: slotStartRaw,
symbols: (json['symbols'] as List<dynamic>? ?? <dynamic>[])
.map((dynamic value) => value.toString())
.toList(growable: false),
);
}
}

View File

@ -0,0 +1,30 @@
class MarketHistoryAdminConfig {
const MarketHistoryAdminConfig({
required this.archiveEnabled,
required this.windowDays,
required this.retentionDays,
required this.syncEnabled,
});
final bool archiveEnabled;
final int windowDays;
final int retentionDays;
final bool syncEnabled;
factory MarketHistoryAdminConfig.fromJson(Map<String, dynamic>? json) {
if (json == null) {
return const MarketHistoryAdminConfig(
archiveEnabled: false,
windowDays: 7,
retentionDays: 7,
syncEnabled: false,
);
}
return MarketHistoryAdminConfig(
archiveEnabled: json['archiveEnabled'] as bool? ?? false,
windowDays: (json['windowDays'] as num?)?.toInt() ?? 7,
retentionDays: (json['retentionDays'] as num?)?.toInt() ?? 7,
syncEnabled: json['syncEnabled'] as bool? ?? false,
);
}
}

View File

@ -0,0 +1,96 @@
class MarketHistorySlotCoverage {
const MarketHistorySlotCoverage({
required this.slotStart,
required this.completed,
required this.fullySynced,
required this.syncedSymbolCount,
required this.expectedSymbolCount,
});
final DateTime slotStart;
final bool completed;
final bool fullySynced;
final int syncedSymbolCount;
final int expectedSymbolCount;
factory MarketHistorySlotCoverage.fromJson(Map<String, dynamic> json) {
return MarketHistorySlotCoverage(
slotStart: DateTime.parse(json['slotStart'] as String).toUtc(),
completed: json['completed'] as bool? ?? false,
fullySynced: json['fullySynced'] as bool? ?? false,
syncedSymbolCount: (json['syncedSymbolCount'] as num?)?.toInt() ?? 0,
expectedSymbolCount: (json['expectedSymbolCount'] as num?)?.toInt() ?? 0,
);
}
}
class MarketHistoryDayCoverage {
const MarketHistoryDayCoverage({
required this.date,
required this.slotsPerDay,
required this.completedSlots,
required this.fullySyncedSlots,
required this.slots,
});
final DateTime date;
final int slotsPerDay;
final int completedSlots;
final int fullySyncedSlots;
final List<MarketHistorySlotCoverage> slots;
factory MarketHistoryDayCoverage.fromJson(Map<String, dynamic> json) {
final List<dynamic> rawSlots =
json['slots'] as List<dynamic>? ?? <dynamic>[];
return MarketHistoryDayCoverage(
date: DateTime.parse('${json['date']}T00:00:00Z').toUtc(),
slotsPerDay: (json['slotsPerDay'] as num?)?.toInt() ?? 6,
completedSlots: (json['completedSlots'] as num?)?.toInt() ?? 0,
fullySyncedSlots: (json['fullySyncedSlots'] as num?)?.toInt() ?? 0,
slots: rawSlots
.map(
(dynamic item) => MarketHistorySlotCoverage.fromJson(
item as Map<String, dynamic>,
),
)
.toList(growable: false),
);
}
}
class MarketHistoryWeekCoverageReport {
const MarketHistoryWeekCoverageReport({
required this.asOf,
required this.windowDays,
required this.slotsPerDay,
required this.symbolCount,
required this.isConsistent,
required this.days,
});
final DateTime asOf;
final int windowDays;
final int slotsPerDay;
final int symbolCount;
final bool isConsistent;
final List<MarketHistoryDayCoverage> days;
factory MarketHistoryWeekCoverageReport.fromJson(Map<String, dynamic> json) {
final List<dynamic> rawDays =
json['days'] as List<dynamic>? ?? <dynamic>[];
return MarketHistoryWeekCoverageReport(
asOf: DateTime.parse(json['asOf'] as String).toUtc(),
windowDays: (json['windowDays'] as num?)?.toInt() ?? 7,
slotsPerDay: (json['slotsPerDay'] as num?)?.toInt() ?? 6,
symbolCount: (json['symbolCount'] as num?)?.toInt() ?? 0,
isConsistent: json['isConsistent'] as bool? ?? false,
days: rawDays
.map(
(dynamic item) => MarketHistoryDayCoverage.fromJson(
item as Map<String, dynamic>,
),
)
.toList(growable: false),
);
}
}

View File

@ -0,0 +1,119 @@
class QuestionAuditBarSlot {
const QuestionAuditBarSlot({
required this.asOf,
required this.avgPrice,
required this.volume,
this.open,
this.high,
this.low,
this.close,
});
final DateTime asOf;
final num? open;
final num? high;
final num? low;
final num? close;
final num avgPrice;
final num volume;
factory QuestionAuditBarSlot.fromJson(Map<String, dynamic> json) {
return QuestionAuditBarSlot(
asOf: DateTime.parse(json['asOf']! as String).toUtc(),
open: json['open'] as num?,
high: json['high'] as num?,
low: json['low'] as num?,
close: json['close'] as num?,
avgPrice: json['avgPrice'] as num,
volume: json['volume'] as num,
);
}
}
class QuestionAuditAsset {
const QuestionAuditAsset({
required this.symbol,
required this.priceDelta,
required this.volumeDelta,
required this.olderSlot,
required this.newerSlot,
});
final String symbol;
final num priceDelta;
final num volumeDelta;
final QuestionAuditBarSlot olderSlot;
final QuestionAuditBarSlot newerSlot;
factory QuestionAuditAsset.fromJson(Map<String, dynamic> json) {
return QuestionAuditAsset(
symbol: json['symbol']! as String,
priceDelta: json['priceDelta'] as num,
volumeDelta: json['volumeDelta'] as num,
olderSlot: QuestionAuditBarSlot.fromJson(
json['olderSlot']! as Map<String, dynamic>,
),
newerSlot: QuestionAuditBarSlot.fromJson(
json['newerSlot']! as Map<String, dynamic>,
),
);
}
}
class QuestionAuditReport {
const QuestionAuditReport({
required this.compareUntil,
required this.newerSlotStart,
required this.olderSlotStart,
required this.windowDays,
required this.assets,
required this.canStepOlder,
required this.canStepNewer,
this.stepOlderCompareUntil,
this.stepNewerCompareUntil,
});
final DateTime compareUntil;
final DateTime newerSlotStart;
final DateTime olderSlotStart;
final int windowDays;
final List<QuestionAuditAsset> assets;
final bool canStepOlder;
final bool canStepNewer;
final DateTime? stepOlderCompareUntil;
final DateTime? stepNewerCompareUntil;
factory QuestionAuditReport.fromJson(Map<String, dynamic> json) {
final List<dynamic> raw = json['assets'] as List<dynamic>? ?? <dynamic>[];
final DateTime compareUntil = DateTime.parse(
(json['compareUntil'] ?? json['asOf'])! as String,
).toUtc();
return QuestionAuditReport(
compareUntil: compareUntil,
newerSlotStart: json['newerSlotStart'] == null
? compareUntil.subtract(const Duration(hours: 4))
: DateTime.parse(json['newerSlotStart']! as String).toUtc(),
olderSlotStart: json['olderSlotStart'] == null
? compareUntil.subtract(const Duration(hours: 8))
: DateTime.parse(json['olderSlotStart']! as String).toUtc(),
windowDays: (json['windowDays'] as num).toInt(),
canStepOlder: json['canStepOlder'] as bool? ?? false,
canStepNewer: json['canStepNewer'] as bool? ?? false,
stepOlderCompareUntil: _parseOptionalUtc(json['stepOlderCompareUntil']),
stepNewerCompareUntil: _parseOptionalUtc(json['stepNewerCompareUntil']),
assets: raw
.map(
(dynamic item) =>
QuestionAuditAsset.fromJson(item as Map<String, dynamic>),
)
.toList(growable: false),
);
}
}
DateTime? _parseOptionalUtc(dynamic raw) {
if (raw == null) {
return null;
}
return DateTime.parse(raw as String).toUtc();
}

View File

@ -0,0 +1,191 @@
import 'market_history_admin_config.dart';
import 'backfill_sync_item.dart';
enum SyncRunSeverity { ok, warning, error, rateLimit }
enum SyncRunStatus { success, failed, partial, inProgress }
class SyncRunEvent {
const SyncRunEvent({
required this.id,
required this.kind,
required this.startedAt,
required this.finishedAt,
required this.rowsWritten,
required this.rowsRemoved,
required this.error,
required this.severity,
required this.status,
required this.durationMs,
required this.summary,
this.backfillItems = const <BackfillSyncItem>[],
});
final int id;
final String kind;
final DateTime startedAt;
final DateTime? finishedAt;
final int rowsWritten;
final int rowsRemoved;
final String? error;
final SyncRunSeverity severity;
final SyncRunStatus status;
final int? durationMs;
final String summary;
final List<BackfillSyncItem> backfillItems;
String get displayTitle {
switch (kind) {
case 'universe':
return 'Asset universe sync';
case 'backfill':
return 'Market history backfill';
case 'cleanup':
return 'Data retention cleanup';
default:
return kind;
}
}
factory SyncRunEvent.fromJson(Map<String, dynamic> json) {
return SyncRunEvent(
id: (json['id'] as num).toInt(),
kind: json['kind'] as String,
startedAt: DateTime.parse(json['startedAt'] as String).toUtc(),
finishedAt: (json['finishedAt'] as String?) == null
? null
: DateTime.parse(json['finishedAt'] as String).toUtc(),
rowsWritten: (json['rowsWritten'] as num?)?.toInt() ?? 0,
rowsRemoved: (json['rowsRemoved'] as num?)?.toInt() ?? 0,
error: json['error'] as String?,
severity: _parseSeverity(
json['severity'] as String?,
json['error'] as String?,
(json['finishedAt'] as String?) == null,
),
status: _parseStatus(
json['status'] as String?,
json['error'] as String?,
(json['finishedAt'] as String?) == null,
(json['rowsWritten'] as num?)?.toInt() ?? 0,
),
durationMs: (json['durationMs'] as num?)?.toInt(),
summary: (json['summary'] as String?)?.trim().isNotEmpty == true
? (json['summary'] as String)
: _fallbackSummary(
kind: json['kind'] as String,
rowsWritten: (json['rowsWritten'] as num?)?.toInt() ?? 0,
rowsRemoved: (json['rowsRemoved'] as num?)?.toInt() ?? 0,
error: json['error'] as String?,
),
backfillItems: _parseBackfillItems(json['backfillItems']),
);
}
static List<BackfillSyncItem> _parseBackfillItems(dynamic raw) {
if (raw is! List<dynamic>) {
return const <BackfillSyncItem>[];
}
return raw
.whereType<Map<String, dynamic>>()
.map(BackfillSyncItem.fromJson)
.toList(growable: false);
}
static SyncRunSeverity _parseSeverity(
String? wire,
String? error,
bool inProgress,
) {
switch (wire) {
case 'ok':
return SyncRunSeverity.ok;
case 'warning':
return SyncRunSeverity.warning;
case 'error':
return SyncRunSeverity.error;
case 'rate_limit':
return SyncRunSeverity.rateLimit;
default:
final String normalized = (error ?? '').toLowerCase();
if (normalized.contains('429') ||
normalized.contains('rate limit') ||
normalized.contains('rate limited')) {
return SyncRunSeverity.rateLimit;
}
if (error != null && error.trim().isNotEmpty) {
return SyncRunSeverity.error;
}
if (inProgress) {
return SyncRunSeverity.warning;
}
return SyncRunSeverity.ok;
}
}
static SyncRunStatus _parseStatus(
String? wire,
String? error,
bool inProgress,
int rowsWritten,
) {
switch (wire) {
case 'success':
return SyncRunStatus.success;
case 'failed':
return SyncRunStatus.failed;
case 'partial':
return SyncRunStatus.partial;
case 'in_progress':
return SyncRunStatus.inProgress;
default:
if (inProgress) {
return SyncRunStatus.inProgress;
}
if (error != null && error.trim().isNotEmpty) {
return rowsWritten > 0 ? SyncRunStatus.partial : SyncRunStatus.failed;
}
return SyncRunStatus.success;
}
}
static String _fallbackSummary({
required String kind,
required int rowsWritten,
required int rowsRemoved,
required String? error,
}) {
if (error != null && error.trim().isNotEmpty && rowsWritten > 0) {
return 'Partial success: $rowsWritten rows written';
}
if (error != null && error.trim().isNotEmpty) {
return 'Run failed';
}
if (kind == 'cleanup') {
return '$rowsRemoved rows removed';
}
if (kind == 'universe') {
return '$rowsWritten assets refreshed';
}
return '$rowsWritten bar rows written';
}
}
class SyncRunLogPage {
const SyncRunLogPage({
required this.runs,
required this.pinned,
required this.nextBefore,
this.config = const MarketHistoryAdminConfig(
archiveEnabled: false,
windowDays: 7,
retentionDays: 7,
syncEnabled: false,
),
});
final List<SyncRunEvent> runs;
final List<SyncRunEvent> pinned;
final DateTime? nextBefore;
final MarketHistoryAdminConfig config;
}

View File

@ -0,0 +1,179 @@
import '../models/market_history_admin_config.dart';
import '../models/sync_run_event.dart';
import '../services/market_history_admin_api.dart';
abstract class SyncRunLogController {
SyncRunLogRepositoryState get state;
Future<SyncRunLogRepositoryState> loadInitial({int limit = 50});
Future<SyncRunLogRepositoryState> refresh({int limit = 50});
Future<SyncRunLogRepositoryState> loadMore({int limit = 50});
Future<SyncRunLogRepositoryState> triggerResync();
Future<SyncRunLogRepositoryState> triggerCleanup({bool archive = false});
}
class SyncRunLogRepositoryState {
const SyncRunLogRepositoryState({
required this.pinned,
required this.history,
required this.nextBefore,
required this.isLoading,
required this.errorMessage,
this.config = const MarketHistoryAdminConfig(
archiveEnabled: false,
windowDays: 7,
retentionDays: 7,
syncEnabled: false,
),
});
final List<SyncRunEvent> pinned;
final List<SyncRunEvent> history;
final DateTime? nextBefore;
final bool isLoading;
final String? errorMessage;
final MarketHistoryAdminConfig config;
bool get hasMore => nextBefore != null;
bool get hasInProgressRun {
bool inProgress(SyncRunEvent event) =>
event.status == SyncRunStatus.inProgress;
return pinned.any(inProgress) || history.any(inProgress);
}
}
class SyncRunLogRepository implements SyncRunLogController {
SyncRunLogRepository({required MarketHistoryAdminApi api}) : _api = api;
final MarketHistoryAdminApi _api;
List<SyncRunEvent> _pinned = <SyncRunEvent>[];
List<SyncRunEvent> _history = <SyncRunEvent>[];
DateTime? _nextBefore;
bool _isLoading = false;
String? _errorMessage;
MarketHistoryAdminConfig _config = const MarketHistoryAdminConfig(
archiveEnabled: false,
windowDays: 7,
retentionDays: 7,
syncEnabled: false,
);
SyncRunLogRepositoryState get state => SyncRunLogRepositoryState(
pinned: List<SyncRunEvent>.unmodifiable(_pinned),
history: List<SyncRunEvent>.unmodifiable(_history),
nextBefore: _nextBefore,
isLoading: _isLoading,
errorMessage: _errorMessage,
config: _config,
);
Future<SyncRunLogRepositoryState> loadInitial({int limit = 50}) async {
_isLoading = true;
_errorMessage = null;
try {
final SyncRunLogPage page = await _api.fetchSyncRuns(limit: limit);
_pinned = _dedupeById(page.pinned);
_history = _dedupeById(_sortedNewestFirst(page.runs));
_nextBefore = page.nextBefore;
_config = page.config;
} on MarketHistoryAdminApiException catch (e) {
_errorMessage = e.message;
} catch (e) {
_errorMessage = e.toString();
} finally {
_isLoading = false;
}
return state;
}
Future<SyncRunLogRepositoryState> refresh({int limit = 50}) async {
return loadInitial(limit: limit);
}
Future<SyncRunLogRepositoryState> loadMore({int limit = 50}) async {
if (_isLoading || _nextBefore == null) {
return state;
}
_isLoading = true;
_errorMessage = null;
try {
final SyncRunLogPage page = await _api.fetchSyncRuns(
limit: limit,
before: _nextBefore,
);
_pinned = _dedupeById(page.pinned);
final List<SyncRunEvent> merged = <SyncRunEvent>[
..._history,
...page.runs,
];
_history = _dedupeById(_sortedNewestFirst(merged));
_nextBefore = page.nextBefore;
_config = page.config;
} on MarketHistoryAdminApiException catch (e) {
_errorMessage = e.message;
} catch (e) {
_errorMessage = e.toString();
} finally {
_isLoading = false;
}
return state;
}
@override
Future<SyncRunLogRepositoryState> triggerResync() async {
_isLoading = true;
_errorMessage = null;
try {
await _api.triggerResync();
return refresh();
} on MarketHistoryAdminApiException catch (e) {
_errorMessage = e.message;
_isLoading = false;
return state;
} catch (e) {
_errorMessage = e.toString();
_isLoading = false;
return state;
}
}
@override
Future<SyncRunLogRepositoryState> triggerCleanup({bool archive = false}) async {
_isLoading = true;
_errorMessage = null;
try {
await _api.triggerCleanup(archive: archive);
return refresh();
} on MarketHistoryAdminApiException catch (e) {
_errorMessage = e.message;
_isLoading = false;
return state;
} catch (e) {
_errorMessage = e.toString();
_isLoading = false;
return state;
}
}
static List<SyncRunEvent> _dedupeById(List<SyncRunEvent> events) {
final Map<int, SyncRunEvent> byId = <int, SyncRunEvent>{};
for (final SyncRunEvent event in events) {
byId[event.id] = event;
}
return byId.values.toList();
}
static List<SyncRunEvent> _sortedNewestFirst(List<SyncRunEvent> events) {
final List<SyncRunEvent> sorted = List<SyncRunEvent>.from(events);
sorted.sort(
(SyncRunEvent a, SyncRunEvent b) => b.startedAt.compareTo(a.startedAt),
);
return sorted;
}
}

View File

@ -0,0 +1,501 @@
import 'dart:async';
import 'package:flutter/material.dart';
import '../../theme/app_theme.dart';
import '../models/sync_run_event.dart';
import '../repositories/sync_run_log_repository.dart';
import '../services/market_history_admin_api.dart';
import '../widgets/market_history_question_audit_sheet.dart';
import '../widgets/market_history_week_coverage_sheet.dart';
import '../widgets/sync_run_expansion_tile.dart';
class MarketHistoryLogScreen extends StatefulWidget {
const MarketHistoryLogScreen({
super.key,
required this.controller,
this.coverageApi,
this.now,
this.autoRefreshInterval = const Duration(seconds: 60),
});
final SyncRunLogController controller;
final MarketHistoryAdminApi? coverageApi;
final DateTime? now;
final Duration? autoRefreshInterval;
@override
State<MarketHistoryLogScreen> createState() => _MarketHistoryLogScreenState();
}
class _MarketHistoryLogScreenState extends State<MarketHistoryLogScreen> {
final ScrollController _scrollController = ScrollController();
Timer? _autoRefreshTimer;
SyncRunLogRepositoryState _state = const SyncRunLogRepositoryState(
pinned: <SyncRunEvent>[],
history: <SyncRunEvent>[],
nextBefore: null,
isLoading: true,
errorMessage: null,
);
bool _initialLoaded = false;
bool _loadingMore = false;
bool _triggerInFlight = false;
bool _weekCoverageLoading = false;
bool _questionAuditLoading = false;
MarketHistoryAdminApi get _coverageApi =>
widget.coverageApi ?? MarketHistoryAdminApi();
@override
void initState() {
super.initState();
_scrollController.addListener(_onScroll);
_loadInitial();
_startAutoRefresh();
}
void _startAutoRefresh() {
final Duration? interval = widget.autoRefreshInterval;
if (interval == null) {
return;
}
_autoRefreshTimer = Timer.periodic(interval, (_) {
if (!mounted || _state.isLoading || _triggerInFlight) {
return;
}
_refresh();
});
}
@override
void dispose() {
_autoRefreshTimer?.cancel();
_scrollController.dispose();
super.dispose();
}
Future<void> _loadInitial() async {
final SyncRunLogRepositoryState next =
await widget.controller.loadInitial();
if (!mounted) {
return;
}
setState(() {
_state = next;
_initialLoaded = true;
});
}
Future<void> _refresh() async {
final SyncRunLogRepositoryState next = await widget.controller.refresh();
if (!mounted) {
return;
}
setState(() => _state = next);
}
Future<void> _loadMore() async {
if (_loadingMore || !_state.hasMore || _state.isLoading) {
return;
}
_loadingMore = true;
final SyncRunLogRepositoryState next = await widget.controller.loadMore();
if (!mounted) {
return;
}
setState(() {
_state = next;
_loadingMore = false;
});
}
Future<void> _runResync() async {
if (_triggerInFlight || _state.hasInProgressRun) {
return;
}
setState(() => _triggerInFlight = true);
final SyncRunLogRepositoryState next =
await widget.controller.triggerResync();
if (!mounted) {
return;
}
setState(() {
_state = next;
_triggerInFlight = false;
});
if (next.errorMessage != null) {
_showSnack(next.errorMessage!);
}
}
Future<void> _confirmAndRunCleanup() async {
if (_triggerInFlight || _state.hasInProgressRun) {
return;
}
bool archive = false;
final bool? confirmed = await showDialog<bool>(
context: context,
builder: (BuildContext context) {
return StatefulBuilder(
builder: (BuildContext context, StateSetter setDialogState) {
return AlertDialog(
key: const Key('cleanup-confirm-dialog'),
title: const Text('Run cleanup now?'),
content: Column(
mainAxisSize: MainAxisSize.min,
crossAxisAlignment: CrossAxisAlignment.start,
children: <Widget>[
Text(
'This removes expired market history snapshots older than '
'${_state.config.retentionDays} days. Continue?',
),
if (_state.config.archiveEnabled) ...<Widget>[
const SizedBox(height: 12),
CheckboxListTile(
key: const Key('cleanup-archive-toggle'),
contentPadding: EdgeInsets.zero,
title: const Text('Archive before delete'),
value: archive,
onChanged: (bool? value) {
setDialogState(() => archive = value ?? false);
},
controlAffinity: ListTileControlAffinity.leading,
),
],
],
),
actions: <Widget>[
TextButton(
key: const Key('cleanup-cancel'),
onPressed: () => Navigator.of(context).pop(false),
child: const Text('Cancel'),
),
FilledButton(
key: const Key('cleanup-confirm'),
onPressed: () => Navigator.of(context).pop(true),
child: const Text('Run cleanup'),
),
],
);
},
);
},
);
if (confirmed != true || !mounted) {
return;
}
setState(() => _triggerInFlight = true);
final SyncRunLogRepositoryState next =
await widget.controller.triggerCleanup(archive: archive);
if (!mounted) {
return;
}
setState(() {
_state = next;
_triggerInFlight = false;
});
if (next.errorMessage != null) {
_showSnack(next.errorMessage!);
}
}
void _showSnack(String message) {
ScaffoldMessenger.of(context).showSnackBar(
SnackBar(content: Text(message)),
);
}
Future<void> _showQuestionAudit() async {
if (_questionAuditLoading) {
return;
}
setState(() => _questionAuditLoading = true);
try {
if (!mounted) {
return;
}
await MarketHistoryQuestionAuditSheet.show(
context,
api: _coverageApi,
);
} on MarketHistoryAdminApiException catch (e) {
if (mounted) {
_showSnack('Could not load question audit (${e.statusCode ?? 'error'})');
}
} catch (e) {
if (mounted) {
_showSnack('Could not load question audit');
}
} finally {
if (mounted) {
setState(() => _questionAuditLoading = false);
}
}
}
Future<void> _showWeekCoverage() async {
if (_weekCoverageLoading) {
return;
}
setState(() => _weekCoverageLoading = true);
try {
final report = await _coverageApi.fetchWeekCoverage();
if (!mounted) {
return;
}
await MarketHistoryWeekCoverageSheet.show(context, report: report);
} on MarketHistoryAdminApiException catch (e) {
if (mounted) {
_showSnack('Could not load week coverage (${e.statusCode ?? 'error'})');
}
} catch (e) {
if (mounted) {
_showSnack('Could not load week coverage');
}
} finally {
if (mounted) {
setState(() => _weekCoverageLoading = false);
}
}
}
void _onScroll() {
if (!_scrollController.hasClients) {
return;
}
const double threshold = 120;
final ScrollPosition position = _scrollController.position;
if (position.pixels >= position.maxScrollExtent - threshold) {
_loadMore();
}
}
bool get _actionsDisabled =>
!_state.config.syncEnabled ||
_state.isLoading ||
_triggerInFlight ||
_state.hasInProgressRun;
@override
Widget build(BuildContext context) {
return Scaffold(
appBar: AppBar(
title: const Text('Market history log'),
actions: <Widget>[
IconButton(
key: const Key('question-audit-button'),
tooltip: 'Audit prospective question data',
onPressed: _questionAuditLoading ? null : _showQuestionAudit,
icon: const Icon(Icons.help_outline),
),
IconButton(
key: const Key('week-coverage-button'),
tooltip: '7-day sync calendar',
onPressed: _weekCoverageLoading ? null : _showWeekCoverage,
icon: const Icon(Icons.calendar_month_outlined),
),
PopupMenuButton<String>(
key: const Key('actions-menu'),
enabled: !_actionsDisabled,
onSelected: (String value) {
switch (value) {
case 'resync':
_runResync();
case 'cleanup':
_confirmAndRunCleanup();
}
},
itemBuilder: (BuildContext context) => <PopupMenuEntry<String>>[
const PopupMenuItem<String>(
key: Key('action-resync'),
value: 'resync',
child: Text('Run backfill now'),
),
const PopupMenuItem<String>(
key: Key('action-cleanup'),
value: 'cleanup',
child: Text('Run cleanup now'),
),
],
),
IconButton(
key: const Key('refresh-button'),
tooltip: 'Refresh',
onPressed: _state.isLoading ? null : _refresh,
icon: const Icon(Icons.refresh),
),
],
),
body: _buildBody(),
);
}
Widget _buildBody() {
if (!_initialLoaded && _state.isLoading) {
return const Center(
key: Key('loading-indicator'),
child: CircularProgressIndicator(color: AppColors.accent),
);
}
if (_state.errorMessage != null &&
_state.pinned.isEmpty &&
_state.history.isEmpty) {
return Center(
key: const Key('error-state'),
child: Padding(
padding: const EdgeInsets.all(24),
child: Column(
mainAxisSize: MainAxisSize.min,
children: <Widget>[
const Icon(Icons.error_outline, color: Colors.redAccent, size: 40),
const SizedBox(height: 12),
Text(
_state.errorMessage!,
textAlign: TextAlign.center,
style: const TextStyle(color: AppColors.textSecondary),
),
const SizedBox(height: 12),
FilledButton(onPressed: _refresh, child: const Text('Retry')),
],
),
),
);
}
if (_state.pinned.isEmpty && _state.history.isEmpty) {
return Column(
children: <Widget>[
if (!_state.config.syncEnabled) const _SyncDisabledBanner(),
const Expanded(
child: Center(
key: Key('empty-state'),
child: Text(
'No sync runs yet.',
style: TextStyle(color: AppColors.textSecondary),
),
),
),
],
);
}
return RefreshIndicator(
key: const Key('refresh-indicator'),
onRefresh: _refresh,
color: AppColors.accent,
child: CustomScrollView(
controller: _scrollController,
physics: const AlwaysScrollableScrollPhysics(),
slivers: <Widget>[
if (!_state.config.syncEnabled)
const SliverToBoxAdapter(child: _SyncDisabledBanner()),
if (_state.pinned.isNotEmpty) ...<Widget>[
const SliverToBoxAdapter(
child: Padding(
key: Key('pinned-section'),
padding: EdgeInsets.fromLTRB(16, 16, 16, 8),
child: Text(
'NEEDS ATTENTION',
style: TextStyle(
fontSize: 12,
fontWeight: FontWeight.w700,
letterSpacing: 0.8,
color: Colors.amber,
),
),
),
),
SliverList(
delegate: SliverChildBuilderDelegate(
(BuildContext context, int index) {
final SyncRunEvent event = _state.pinned[index];
return SyncRunExpansionTile(
event: event,
now: widget.now,
emphasizeError: true,
onRetry: _actionsDisabled ? null : _runResync,
);
},
childCount: _state.pinned.length,
),
),
],
const SliverToBoxAdapter(
child: Padding(
key: Key('history-section'),
padding: EdgeInsets.fromLTRB(16, 16, 16, 8),
child: Text(
'HISTORY',
style: TextStyle(
fontSize: 12,
fontWeight: FontWeight.w700,
letterSpacing: 0.8,
color: AppColors.textSecondary,
),
),
),
),
SliverList(
delegate: SliverChildBuilderDelegate(
(BuildContext context, int index) {
final SyncRunEvent event = _state.history[index];
return SyncRunExpansionTile(
event: event,
now: widget.now,
);
},
childCount: _state.history.length,
),
),
if (_state.isLoading || _loadingMore || _triggerInFlight)
const SliverToBoxAdapter(
child: Padding(
padding: EdgeInsets.all(16),
child: Center(
child: CircularProgressIndicator(color: AppColors.accent),
),
),
),
],
),
);
}
}
/// Convenience factory for production use.
MarketHistoryLogScreen marketHistoryLogScreen() {
return MarketHistoryLogScreen(
controller: SyncRunLogRepository(
api: MarketHistoryAdminApi(),
),
);
}
class _SyncDisabledBanner extends StatelessWidget {
const _SyncDisabledBanner();
@override
Widget build(BuildContext context) {
return Container(
key: const Key('sync-disabled-banner'),
width: double.infinity,
margin: const EdgeInsets.fromLTRB(16, 12, 16, 0),
padding: const EdgeInsets.symmetric(horizontal: 12, vertical: 10),
decoration: BoxDecoration(
color: Colors.amber.withValues(alpha: 0.12),
borderRadius: BorderRadius.circular(8),
border: Border.all(color: Colors.amber.withValues(alpha: 0.45)),
),
child: const Text(
'Market history sync is disabled (MARKET_HISTORY_SYNC_ENABLED=false). '
'Log and calendar views are read-only.',
style: TextStyle(fontSize: 12, color: AppColors.textSecondary),
),
);
}
}

View File

@ -0,0 +1,42 @@
import 'package:flutter/foundation.dart';
import 'market_history_admin_api.dart';
enum AdminAccessStatus {
unknown,
loading,
authorized,
forbidden,
unauthorized,
error,
}
/// Probes admin API access for home-screen discovery (allowlist-backed).
class AdminAccessService {
AdminAccessService._();
static final AdminAccessService instance = AdminAccessService._();
final ValueNotifier<AdminAccessStatus> status =
ValueNotifier<AdminAccessStatus>(AdminAccessStatus.unknown);
Future<void> refresh({MarketHistoryAdminApi? api}) async {
status.value = AdminAccessStatus.loading;
try {
await (api ?? MarketHistoryAdminApi()).fetchSyncRuns(limit: 1);
status.value = AdminAccessStatus.authorized;
} on MarketHistoryAdminApiException catch (e) {
status.value = switch (e.statusCode) {
401 => AdminAccessStatus.unauthorized,
403 => AdminAccessStatus.forbidden,
_ => AdminAccessStatus.error,
};
} catch (_) {
status.value = AdminAccessStatus.error;
}
}
void reset() {
status.value = AdminAccessStatus.unknown;
}
}

View File

@ -0,0 +1,233 @@
import 'dart:convert';
import 'package:http/http.dart' as http;
import '../../config/api_config.dart';
import '../../services/auth_service.dart';
import '../models/market_history_admin_config.dart';
import '../models/market_history_week_coverage.dart';
import '../models/question_audit_asset.dart';
import '../models/sync_run_event.dart';
typedef AdminTokenProvider = Future<String?> Function();
class MarketHistoryAdminApiException implements Exception {
MarketHistoryAdminApiException(this.message, {this.statusCode});
final String message;
final int? statusCode;
@override
String toString() => 'MarketHistoryAdminApiException($statusCode): $message';
}
class AdminTriggerResponse {
const AdminTriggerResponse({required this.runIds});
final List<int> runIds;
factory AdminTriggerResponse.fromJson(Map<String, dynamic> json) {
final List<dynamic> raw = json['runIds'] as List<dynamic>? ?? <dynamic>[];
return AdminTriggerResponse(
runIds: raw.map((dynamic id) => (id as num).toInt()).toList(),
);
}
}
class MarketHistoryAdminApi {
MarketHistoryAdminApi({
http.Client? client,
String? baseUrl,
AdminTokenProvider? tokenProvider,
}) : _client = client ?? http.Client(),
_baseUrl = baseUrl ?? apiBaseUrl,
_tokenProvider = tokenProvider ?? AuthService.instance.getIdToken;
final http.Client _client;
final String _baseUrl;
final AdminTokenProvider _tokenProvider;
Future<SyncRunLogPage> fetchSyncRuns({
int limit = 50,
DateTime? before,
String? kind,
}) async {
final String? token = await _tokenProvider();
if (token == null || token.isEmpty) {
throw MarketHistoryAdminApiException('Missing auth token');
}
final Map<String, String> query = <String, String>{'limit': '$limit'};
if (before != null) {
query['before'] = before.toUtc().toIso8601String();
}
if (kind != null && kind.isNotEmpty) {
query['kind'] = kind;
}
final Uri uri = Uri.parse('$_baseUrl/v1/admin/market-history/sync-runs')
.replace(queryParameters: query);
final http.Response response = await _client.get(
uri,
headers: <String, String>{
'Authorization': 'Bearer $token',
'Content-Type': 'application/json',
'Accept': 'application/json',
},
);
if (response.statusCode != 200) {
throw MarketHistoryAdminApiException(
response.body,
statusCode: response.statusCode,
);
}
final Map<String, dynamic> body =
jsonDecode(response.body) as Map<String, dynamic>;
final List<SyncRunEvent> runs = _parseEventList(body['runs'], 'runs');
final List<SyncRunEvent> pinned = _parseEventList(body['pinned'], 'pinned');
final String? nextBeforeRaw = body['nextBefore'] as String?;
final MarketHistoryAdminConfig config = MarketHistoryAdminConfig.fromJson(
body['config'] as Map<String, dynamic>?,
);
return SyncRunLogPage(
runs: runs,
pinned: pinned,
nextBefore: nextBeforeRaw == null ? null : DateTime.parse(nextBeforeRaw).toUtc(),
config: config,
);
}
Future<QuestionAuditReport> fetchQuestionAudit({DateTime? asOf}) async {
final String? token = await _tokenProvider();
if (token == null || token.isEmpty) {
throw MarketHistoryAdminApiException('Missing auth token');
}
final Map<String, String> query = <String, String>{};
if (asOf != null) {
query['asOf'] = asOf.toUtc().toIso8601String();
}
final Uri uri = Uri.parse('$_baseUrl/v1/admin/market-history/question-audit')
.replace(queryParameters: query.isEmpty ? null : query);
final http.Response response = await _client.get(
uri,
headers: <String, String>{
'Authorization': 'Bearer $token',
'Content-Type': 'application/json',
'Accept': 'application/json',
},
);
if (response.statusCode != 200) {
throw MarketHistoryAdminApiException(
response.body,
statusCode: response.statusCode,
);
}
final Map<String, dynamic> body =
jsonDecode(response.body) as Map<String, dynamic>;
return QuestionAuditReport.fromJson(body);
}
Future<MarketHistoryWeekCoverageReport> fetchWeekCoverage() async {
final String? token = await _tokenProvider();
if (token == null || token.isEmpty) {
throw MarketHistoryAdminApiException('Missing auth token');
}
final Uri uri =
Uri.parse('$_baseUrl/v1/admin/market-history/week-coverage');
final http.Response response = await _client.get(
uri,
headers: <String, String>{
'Authorization': 'Bearer $token',
'Content-Type': 'application/json',
'Accept': 'application/json',
},
);
if (response.statusCode != 200) {
throw MarketHistoryAdminApiException(
response.body,
statusCode: response.statusCode,
);
}
final Map<String, dynamic> body =
jsonDecode(response.body) as Map<String, dynamic>;
return MarketHistoryWeekCoverageReport.fromJson(body);
}
static List<SyncRunEvent> _parseEventList(dynamic raw, String field) {
if (raw == null) {
return <SyncRunEvent>[];
}
if (raw is! List<dynamic>) {
throw MarketHistoryAdminApiException(
'Invalid sync-runs payload: $field must be a list',
);
}
final List<SyncRunEvent> events = <SyncRunEvent>[];
for (int i = 0; i < raw.length; i++) {
final dynamic item = raw[i];
if (item is! Map<String, dynamic>) {
throw MarketHistoryAdminApiException(
'Invalid sync-runs payload: $field[$i] must be an object',
);
}
try {
events.add(SyncRunEvent.fromJson(item));
} catch (e) {
throw MarketHistoryAdminApiException(
'Invalid sync-runs payload: $field[$i] $e',
);
}
}
return events;
}
Future<AdminTriggerResponse> triggerResync() async {
return _postTrigger('/v1/admin/market-data/resync');
}
Future<AdminTriggerResponse> triggerCleanup({bool archive = false}) async {
final Uri uri = Uri.parse('$_baseUrl/v1/admin/market-data/cleanup').replace(
queryParameters: archive ? <String, String>{'archive': 'true'} : null,
);
return _postTriggerUri(uri);
}
Future<AdminTriggerResponse> _postTrigger(String path) {
return _postTriggerUri(Uri.parse('$_baseUrl$path'));
}
Future<AdminTriggerResponse> _postTriggerUri(Uri uri) async {
final String? token = await _tokenProvider();
if (token == null || token.isEmpty) {
throw MarketHistoryAdminApiException('Missing auth token');
}
final http.Response response = await _client.post(
uri,
headers: <String, String>{
'Authorization': 'Bearer $token',
'Content-Type': 'application/json',
'Accept': 'application/json',
},
);
if (response.statusCode != 202) {
throw MarketHistoryAdminApiException(
response.body,
statusCode: response.statusCode,
);
}
final Map<String, dynamic> body =
jsonDecode(response.body) as Map<String, dynamic>;
return AdminTriggerResponse.fromJson(body);
}
}

View File

@ -0,0 +1,101 @@
import 'package:flutter/material.dart';
import '../../theme/app_theme.dart';
import '../models/sync_run_event.dart';
String formatRelativeTime(DateTime startedAt, {DateTime? now}) {
final DateTime reference = (now ?? DateTime.now()).toUtc();
final Duration delta = reference.difference(startedAt.toUtc());
if (delta.inMinutes < 1) {
return 'just now';
}
if (delta.inHours < 1) {
return '${delta.inMinutes}m ago';
}
if (delta.inDays < 1) {
return '${delta.inHours}h ago';
}
return '${delta.inDays}d ago';
}
String formatMarketHistorySlotWire(DateTime value) {
final DateTime utc = value.toUtc();
final int slotHour = (utc.hour ~/ 4) * 4;
final DateTime slotStart = DateTime.utc(
utc.year,
utc.month,
utc.day,
slotHour,
);
String two(int n) => n.toString().padLeft(2, '0');
return '${slotStart.year.toString().padLeft(4, '0')}-'
'${two(slotStart.month)}-${two(slotStart.day)}T'
'${two(slotStart.hour)}:${two(slotStart.minute)}:${two(slotStart.second)}Z';
}
String formatUtcTimestamp(DateTime? value) {
if (value == null) {
return '';
}
final DateTime utc = value.toUtc();
final String hour = utc.hour.toString().padLeft(2, '0');
final String minute = utc.minute.toString().padLeft(2, '0');
return '${utc.year}-${utc.month.toString().padLeft(2, '0')}-'
'${utc.day.toString().padLeft(2, '0')} $hour:$minute UTC';
}
String formatLocalTimestamp(DateTime? value) {
if (value == null) {
return '';
}
final DateTime local = value.toLocal();
final String hour = local.hour.toString().padLeft(2, '0');
final String minute = local.minute.toString().padLeft(2, '0');
return '${local.year}-${local.month.toString().padLeft(2, '0')}-'
'${local.day.toString().padLeft(2, '0')} $hour:$minute';
}
String formatDurationMs(int? durationMs) {
if (durationMs == null) {
return '';
}
if (durationMs < 1000) {
return '${durationMs}ms';
}
return '${(durationMs / 1000).toStringAsFixed(1)}s';
}
String shortStatusLabel(SyncRunEvent event) {
return switch (event.status) {
SyncRunStatus.success => 'Success',
SyncRunStatus.failed => 'Failed',
SyncRunStatus.partial => 'Partial success',
SyncRunStatus.inProgress => 'In progress',
};
}
IconData severityIcon(SyncRunSeverity severity) {
return switch (severity) {
SyncRunSeverity.ok => Icons.check_circle_outline,
SyncRunSeverity.warning => Icons.schedule,
SyncRunSeverity.error => Icons.error_outline,
SyncRunSeverity.rateLimit => Icons.speed,
};
}
Color severityColor(SyncRunSeverity severity) {
return switch (severity) {
SyncRunSeverity.ok => AppColors.success,
SyncRunSeverity.warning => Colors.orange,
SyncRunSeverity.error => const Color(0xFFF87171),
SyncRunSeverity.rateLimit => Colors.amber,
};
}
String? parseHttpStatus(String? error) {
if (error == null || error.trim().isEmpty) {
return null;
}
final RegExpMatch? match = RegExp(r'\b(\d{3})\b').firstMatch(error);
return match?.group(1);
}

View File

@ -0,0 +1,43 @@
import 'package:flutter/material.dart';
import '../admin_routes.dart';
import '../services/admin_access_service.dart';
import '../services/market_history_admin_api.dart';
/// App bar icon shown when the signed-in user's Firebase UID is on the admin allowlist.
class AdminAppBarAction extends StatefulWidget {
const AdminAppBarAction({super.key, this.api});
final MarketHistoryAdminApi? api;
@override
State<AdminAppBarAction> createState() => _AdminAppBarActionState();
}
class _AdminAppBarActionState extends State<AdminAppBarAction> {
@override
void initState() {
super.initState();
AdminAccessService.instance.refresh(api: widget.api);
}
@override
Widget build(BuildContext context) {
return ValueListenableBuilder<AdminAccessStatus>(
valueListenable: AdminAccessService.instance.status,
builder: (BuildContext context, AdminAccessStatus access, Widget? child) {
if (access != AdminAccessStatus.authorized) {
return const SizedBox.shrink();
}
return IconButton(
key: const Key('admin-app-bar-action'),
tooltip: 'Admin portal',
onPressed: () {
Navigator.of(context).pushNamed(AdminRoutes.marketHistoryLog);
},
icon: const Icon(Icons.admin_panel_settings_outlined),
);
},
);
}
}

View File

@ -0,0 +1,144 @@
import 'package:flutter/material.dart';
import '../../theme/app_theme.dart';
import '../services/market_history_admin_api.dart';
enum AdminGateStatus { loading, authorized, forbidden, unauthorized, error }
/// Probes admin API access before showing [child].
///
/// Calls `GET /v1/admin/market-history/sync-runs?limit=1`.
/// Returns 403 not authorized UI; 200 [child].
class AdminGate extends StatefulWidget {
const AdminGate({
super.key,
required this.child,
this.api,
});
final Widget child;
final MarketHistoryAdminApi? api;
@override
State<AdminGate> createState() => _AdminGateState();
}
class _AdminGateState extends State<AdminGate> {
AdminGateStatus _status = AdminGateStatus.loading;
String? _errorMessage;
@override
void initState() {
super.initState();
_probe();
}
Future<void> _probe() async {
final MarketHistoryAdminApi api = widget.api ?? MarketHistoryAdminApi();
try {
await api.fetchSyncRuns(limit: 1);
if (!mounted) {
return;
}
setState(() => _status = AdminGateStatus.authorized);
} on MarketHistoryAdminApiException catch (e) {
if (!mounted) {
return;
}
setState(() {
_errorMessage = e.message;
_status = switch (e.statusCode) {
401 => AdminGateStatus.unauthorized,
403 => AdminGateStatus.forbidden,
_ => AdminGateStatus.error,
};
});
} catch (e) {
if (!mounted) {
return;
}
setState(() {
_status = AdminGateStatus.error;
_errorMessage = e.toString();
});
}
}
@override
Widget build(BuildContext context) {
return switch (_status) {
AdminGateStatus.loading => const Scaffold(
key: Key('admin-gate-loading'),
body: Center(
child: CircularProgressIndicator(color: AppColors.accent),
),
),
AdminGateStatus.authorized => widget.child,
AdminGateStatus.forbidden => _messageScaffold(
key: const Key('admin-gate-forbidden'),
title: 'Not authorized',
message: 'Your account does not have admin access.',
icon: Icons.lock_outline,
),
AdminGateStatus.unauthorized => _messageScaffold(
key: const Key('admin-gate-unauthorized'),
title: 'Sign in required',
message: 'Sign in again to access the admin portal.',
icon: Icons.login,
),
AdminGateStatus.error => _messageScaffold(
key: const Key('admin-gate-error'),
title: 'Admin check failed',
message: _errorMessage ?? 'Unknown error',
icon: Icons.error_outline,
showRetry: true,
),
};
}
Widget _messageScaffold({
required Key key,
required String title,
required String message,
required IconData icon,
bool showRetry = false,
}) {
return Scaffold(
key: key,
appBar: AppBar(title: const Text('Admin portal')),
body: Center(
child: Padding(
padding: const EdgeInsets.all(24),
child: Column(
mainAxisSize: MainAxisSize.min,
children: <Widget>[
Icon(icon, size: 40, color: AppColors.textSecondary),
const SizedBox(height: 12),
Text(
title,
style: Theme.of(context).textTheme.titleLarge,
textAlign: TextAlign.center,
),
const SizedBox(height: 8),
Text(
message,
style: const TextStyle(color: AppColors.textSecondary),
textAlign: TextAlign.center,
),
if (showRetry) ...<Widget>[
const SizedBox(height: 16),
FilledButton(
onPressed: () {
setState(() => _status = AdminGateStatus.loading);
_probe();
},
child: const Text('Retry'),
),
],
],
),
),
),
);
}
}

View File

@ -0,0 +1,573 @@
import 'package:flutter/material.dart';
import '../../theme/app_theme.dart';
import '../models/question_audit_asset.dart';
import '../services/market_history_admin_api.dart';
/// Scrollable audit of last-two 4-hour bar price and volume deltas per symbol.
class MarketHistoryQuestionAuditSheet extends StatefulWidget {
const MarketHistoryQuestionAuditSheet({
super.key,
required this.api,
});
final MarketHistoryAdminApi api;
static Future<void> show(
BuildContext context, {
required MarketHistoryAdminApi api,
}) {
return showDialog<void>(
context: context,
builder: (BuildContext context) {
return Dialog(
key: const Key('question-audit-dialog'),
insetPadding: const EdgeInsets.symmetric(horizontal: 16, vertical: 24),
child: ConstrainedBox(
constraints: BoxConstraints(
maxWidth: 480,
maxHeight: MediaQuery.sizeOf(context).height * 0.85,
),
child: MarketHistoryQuestionAuditSheet(api: api),
),
);
},
);
}
@override
State<MarketHistoryQuestionAuditSheet> createState() =>
_MarketHistoryQuestionAuditSheetState();
}
class _MarketHistoryQuestionAuditSheetState
extends State<MarketHistoryQuestionAuditSheet> {
QuestionAuditReport? _report;
bool _loading = true;
String? _errorMessage;
@override
void initState() {
super.initState();
_load();
}
Future<void> _load({DateTime? compareUntil}) async {
setState(() {
_loading = true;
_errorMessage = null;
});
try {
final QuestionAuditReport report =
await widget.api.fetchQuestionAudit(asOf: compareUntil);
if (!mounted) {
return;
}
setState(() {
_report = report;
_loading = false;
});
} on MarketHistoryAdminApiException catch (e) {
if (!mounted) {
return;
}
setState(() {
_loading = false;
_errorMessage = e.message;
});
} catch (e) {
if (!mounted) {
return;
}
setState(() {
_loading = false;
_errorMessage = e.toString();
});
}
}
Future<void> _stepOlder() async {
final QuestionAuditReport? report = _report;
final DateTime? next = report?.stepOlderCompareUntil;
if (report == null || !report.canStepOlder || next == null || _loading) {
return;
}
await _load(compareUntil: next);
}
Future<void> _stepNewer() async {
final QuestionAuditReport? report = _report;
final DateTime? next = report?.stepNewerCompareUntil;
if (report == null || !report.canStepNewer || next == null || _loading) {
return;
}
await _load(compareUntil: next);
}
@override
Widget build(BuildContext context) {
final QuestionAuditReport? report = _report;
return Padding(
padding: const EdgeInsets.fromLTRB(16, 12, 8, 16),
child: Column(
crossAxisAlignment: CrossAxisAlignment.stretch,
children: <Widget>[
Row(
children: <Widget>[
const Icon(Icons.help_outline, color: AppColors.accent, size: 22),
const SizedBox(width: 8),
const Expanded(
child: Text(
'Prospective question data',
style: TextStyle(
fontSize: 18,
fontWeight: FontWeight.w700,
color: AppColors.textPrimary,
),
),
),
IconButton(
key: const Key('question-audit-close'),
tooltip: 'Close',
onPressed: () => Navigator.of(context).pop(),
icon: const Icon(Icons.close),
),
],
),
_SymbolCountNav(
newerSlotStart: report?.newerSlotStart,
olderSlotStart: report?.olderSlotStart,
symbolCount: report?.assets.length ?? 0,
canStepOlder: report?.canStepOlder ?? false,
canStepNewer: report?.canStepNewer ?? false,
loading: _loading,
onStepOlder: _stepOlder,
onStepNewer: _stepNewer,
),
const SizedBox(height: 12),
Expanded(child: _buildBody(report)),
],
),
);
}
Widget _buildBody(QuestionAuditReport? report) {
if (_loading && report == null) {
return const Center(
key: Key('question-audit-loading'),
child: CircularProgressIndicator(color: AppColors.accent),
);
}
if (_errorMessage != null && report == null) {
return Center(
key: const Key('question-audit-error'),
child: Padding(
padding: const EdgeInsets.all(16),
child: Column(
mainAxisSize: MainAxisSize.min,
children: <Widget>[
Text(
_errorMessage!,
textAlign: TextAlign.center,
style: const TextStyle(color: AppColors.textSecondary),
),
const SizedBox(height: 12),
FilledButton(
onPressed: _load,
child: const Text('Retry'),
),
],
),
),
);
}
if (report == null) {
return const SizedBox.shrink();
}
if (report.assets.isEmpty) {
return const Center(
child: Text(
'No symbols with bars for both slots in this range.',
textAlign: TextAlign.center,
style: TextStyle(color: AppColors.textSecondary),
),
);
}
return Stack(
children: <Widget>[
ListView.separated(
key: ValueKey<String>(
'question-audit-list-${report.compareUntil.toIso8601String()}',
),
itemCount: report.assets.length,
separatorBuilder: (_, __) => const SizedBox(height: 8),
itemBuilder: (BuildContext context, int index) {
return _AssetTile(asset: report.assets[index]);
},
),
if (_loading)
const Positioned(
top: 0,
left: 0,
right: 0,
child: LinearProgressIndicator(
color: AppColors.accent,
minHeight: 2,
),
),
],
);
}
}
class _SymbolCountNav extends StatelessWidget {
const _SymbolCountNav({
required this.newerSlotStart,
required this.olderSlotStart,
required this.symbolCount,
required this.canStepOlder,
required this.canStepNewer,
required this.loading,
required this.onStepOlder,
required this.onStepNewer,
});
final DateTime? newerSlotStart;
final DateTime? olderSlotStart;
final int symbolCount;
final bool canStepOlder;
final bool canStepNewer;
final bool loading;
final VoidCallback onStepOlder;
final VoidCallback onStepNewer;
@override
Widget build(BuildContext context) {
final String label = symbolCount == 1 ? '1 symbol' : '$symbolCount symbols';
final String? slotRange = newerSlotStart == null || olderSlotStart == null
? null
: _AuditFormat.compareSlotRange(
older: olderSlotStart!,
newer: newerSlotStart!,
);
return Row(
children: <Widget>[
IconButton(
key: const Key('question-audit-step-older'),
tooltip: 'Earlier 4-hour slot',
onPressed: canStepOlder && !loading ? onStepOlder : null,
icon: const Icon(Icons.chevron_left),
),
Expanded(
child: Column(
mainAxisSize: MainAxisSize.min,
children: <Widget>[
if (slotRange != null)
Text(
slotRange,
key: const Key('question-audit-slot-range'),
textAlign: TextAlign.center,
style: const TextStyle(
fontSize: 12,
fontWeight: FontWeight.w600,
color: AppColors.accent,
),
),
if (slotRange != null) const SizedBox(height: 2),
Text(
label,
key: const Key('question-audit-symbol-count'),
textAlign: TextAlign.center,
style: const TextStyle(
fontSize: 15,
fontWeight: FontWeight.w600,
color: AppColors.textPrimary,
),
),
],
),
),
IconButton(
key: const Key('question-audit-step-newer'),
tooltip: 'Later 4-hour slot',
onPressed: canStepNewer && !loading ? onStepNewer : null,
icon: const Icon(Icons.chevron_right),
),
],
);
}
}
class _AssetTile extends StatefulWidget {
const _AssetTile({required this.asset});
final QuestionAuditAsset asset;
@override
State<_AssetTile> createState() => _AssetTileState();
}
class _AssetTileState extends State<_AssetTile> {
bool _expanded = false;
QuestionAuditAsset get asset => widget.asset;
@override
Widget build(BuildContext context) {
return Material(
key: Key('question-audit-tile-${asset.symbol}'),
color: AppColors.surfaceElevated,
shape: RoundedRectangleBorder(
borderRadius: BorderRadius.circular(8),
side: BorderSide(color: AppColors.textSecondary.withValues(alpha: 0.2)),
),
clipBehavior: Clip.antiAlias,
child: InkWell(
onTap: () => setState(() => _expanded = !_expanded),
child: Padding(
padding: const EdgeInsets.symmetric(horizontal: 12, vertical: 10),
child: Column(
crossAxisAlignment: CrossAxisAlignment.stretch,
children: <Widget>[
Stack(
alignment: Alignment.center,
children: <Widget>[
Align(
alignment: Alignment.centerLeft,
child: Row(
mainAxisSize: MainAxisSize.min,
children: <Widget>[
Icon(
_expanded
? Icons.expand_less
: Icons.expand_more,
size: 18,
color: AppColors.textSecondary,
),
const SizedBox(width: 4),
Text(
asset.symbol,
style: const TextStyle(
fontSize: 14,
fontWeight: FontWeight.w700,
color: AppColors.textPrimary,
),
),
],
),
),
Column(
mainAxisSize: MainAxisSize.min,
children: <Widget>[
Text(
'P:${_AuditFormat.delta(asset.priceDelta)}',
style: TextStyle(
fontSize: 15,
fontWeight: FontWeight.w600,
color: _AuditFormat.deltaColor(asset.priceDelta),
),
),
const SizedBox(height: 2),
Text(
'V:${_AuditFormat.delta(asset.volumeDelta)}',
style: TextStyle(
fontSize: 15,
fontWeight: FontWeight.w600,
color: _AuditFormat.deltaColor(asset.volumeDelta),
),
),
],
),
],
),
if (_expanded) ...<Widget>[
const SizedBox(height: 12),
const Divider(height: 1),
const SizedBox(height: 10),
IntrinsicHeight(
child: Row(
crossAxisAlignment: CrossAxisAlignment.stretch,
children: <Widget>[
Expanded(
child: _SlotPanel(
key: Key('question-audit-slot-older-${asset.symbol}'),
label: 'Older',
slot: asset.olderSlot,
),
),
const SizedBox(width: 8),
Container(
width: 1,
color: AppColors.textSecondary.withValues(alpha: 0.25),
),
const SizedBox(width: 8),
Expanded(
child: _SlotPanel(
key: Key('question-audit-slot-newer-${asset.symbol}'),
label: 'Newer',
slot: asset.newerSlot,
),
),
],
),
),
],
],
),
),
),
);
}
}
class _SlotPanel extends StatelessWidget {
const _SlotPanel({
super.key,
required this.label,
required this.slot,
});
final String label;
final QuestionAuditBarSlot slot;
@override
Widget build(BuildContext context) {
return Column(
crossAxisAlignment: CrossAxisAlignment.stretch,
children: <Widget>[
Text(
label,
textAlign: TextAlign.center,
style: const TextStyle(
fontSize: 11,
fontWeight: FontWeight.w700,
letterSpacing: 0.6,
color: AppColors.accent,
),
),
const SizedBox(height: 4),
Text(
_AuditFormat.slotTime(slot.asOf),
textAlign: TextAlign.center,
style: const TextStyle(
fontSize: 10,
color: AppColors.textSecondary,
),
),
const SizedBox(height: 8),
_SlotRow(label: 'O', value: slot.open),
_SlotRow(label: 'H', value: slot.high),
_SlotRow(label: 'L', value: slot.low),
_SlotRow(label: 'C', value: slot.close),
_SlotRow(label: 'Avg', value: slot.avgPrice, emphasize: true),
_SlotRow(label: 'Vol', value: slot.volume),
],
);
}
}
class _SlotRow extends StatelessWidget {
const _SlotRow({
required this.label,
required this.value,
this.emphasize = false,
});
final String label;
final num? value;
final bool emphasize;
@override
Widget build(BuildContext context) {
return Padding(
padding: const EdgeInsets.symmetric(vertical: 2),
child: Row(
children: <Widget>[
SizedBox(
width: 28,
child: Text(
label,
style: const TextStyle(
fontSize: 11,
color: AppColors.textSecondary,
),
),
),
Expanded(
child: Text(
value == null ? '' : _AuditFormat.value(value!),
textAlign: TextAlign.right,
style: TextStyle(
fontSize: emphasize ? 13 : 12,
fontWeight: emphasize ? FontWeight.w700 : FontWeight.w500,
color: AppColors.textPrimary,
),
),
),
],
),
);
}
}
abstract final class _AuditFormat {
/// Older newer UTC 4-hour slot starts being compared.
static String compareSlotRange({
required DateTime older,
required DateTime newer,
}) {
return '${_slotLabel(older)} ${_slotLabel(newer)} UTC';
}
static String _slotLabel(DateTime asOf) {
final DateTime utc = asOf.toUtc();
final String month = utc.month.toString().padLeft(2, '0');
final String day = utc.day.toString().padLeft(2, '0');
final String hour = utc.hour.toString().padLeft(2, '0');
return '$month/$day $hour:00';
}
static String slotTime(DateTime asOf) {
final DateTime utc = asOf.toUtc();
final String month = utc.month.toString().padLeft(2, '0');
final String day = utc.day.toString().padLeft(2, '0');
final String hour = utc.hour.toString().padLeft(2, '0');
return '$month/$day ${hour}:00 UTC';
}
static String value(num n) {
if (n == n.roundToDouble() && n.abs() < 1e12) {
return n.round().toString();
}
return n.toStringAsFixed(2);
}
static Color deltaColor(num delta) {
if (delta > 0) {
return AppColors.success;
}
if (delta < 0) {
return const Color(0xFFF87171);
}
return AppColors.textPrimary;
}
static String delta(num value) {
final num rounded = value.abs() >= 1000
? (value * 100).round() / 100
: (value * 10000).round() / 10000;
final String text = rounded == rounded.roundToDouble()
? rounded.round().toString()
: rounded.toStringAsFixed(2);
if (value > 0) {
return '+$text';
}
return text;
}
}

View File

@ -0,0 +1,208 @@
import 'package:flutter/material.dart';
import '../../theme/app_theme.dart';
import '../models/market_history_week_coverage.dart';
const List<String> _weekdayLabels = <String>[
'Mon',
'Tue',
'Wed',
'Thu',
'Fri',
'Sat',
'Sun',
];
/// Mini 7-day UTC week view showing 4-hour slot sync health.
class MarketHistoryWeekCoverageSheet extends StatelessWidget {
const MarketHistoryWeekCoverageSheet({
super.key,
required this.report,
});
final MarketHistoryWeekCoverageReport report;
static Future<void> show(
BuildContext context, {
required MarketHistoryWeekCoverageReport report,
}) {
return showDialog<void>(
context: context,
builder: (BuildContext context) {
return Dialog(
key: const Key('week-coverage-dialog'),
insetPadding: const EdgeInsets.symmetric(horizontal: 16, vertical: 24),
child: ConstrainedBox(
constraints: const BoxConstraints(maxWidth: 560),
child: MarketHistoryWeekCoverageSheet(report: report),
),
);
},
);
}
@override
Widget build(BuildContext context) {
return Padding(
padding: const EdgeInsets.fromLTRB(16, 12, 8, 16),
child: Column(
mainAxisSize: MainAxisSize.min,
crossAxisAlignment: CrossAxisAlignment.stretch,
children: <Widget>[
Row(
children: <Widget>[
const Icon(Icons.calendar_month, color: AppColors.accent, size: 22),
const SizedBox(width: 8),
const Expanded(
child: Text(
'7-day sync health',
style: TextStyle(
fontSize: 18,
fontWeight: FontWeight.w700,
color: AppColors.textPrimary,
),
),
),
IconButton(
key: const Key('week-coverage-close'),
tooltip: 'Close',
onPressed: () => Navigator.of(context).pop(),
icon: const Icon(Icons.close),
),
],
),
Text(
report.symbolCount == 0
? 'No active tradable symbols to validate.'
: report.isConsistent
? 'All completed 4-hour slots are fully synced across '
'${report.symbolCount} symbols (bars or no-data placeholders).'
: 'Some completed slots are missing a bar or no-data placeholder '
'for one or more symbols.',
style: const TextStyle(
fontSize: 13,
color: AppColors.textSecondary,
),
),
const SizedBox(height: 16),
Row(
crossAxisAlignment: CrossAxisAlignment.start,
children: report.days
.map(
(MarketHistoryDayCoverage day) => Expanded(
child: _DayColumn(day: day),
),
)
.toList(growable: false),
),
const SizedBox(height: 12),
Text(
'UTC · ${report.slotsPerDay} slots per day · '
'${report.windowDays}-day window',
textAlign: TextAlign.center,
style: const TextStyle(
fontSize: 11,
color: AppColors.textSecondary,
),
),
],
),
);
}
}
class _DayColumn extends StatelessWidget {
const _DayColumn({required this.day});
final MarketHistoryDayCoverage day;
@override
Widget build(BuildContext context) {
final int weekdayIndex = day.date.weekday - DateTime.monday;
final String weekday = _weekdayLabels[weekdayIndex];
final String dateLabel = '${day.date.month}/${day.date.day}';
final int denominator =
day.completedSlots > 0 ? day.completedSlots : day.slotsPerDay;
return Padding(
padding: const EdgeInsets.symmetric(horizontal: 2),
child: Column(
children: <Widget>[
Text(
weekday,
style: const TextStyle(
fontSize: 11,
fontWeight: FontWeight.w600,
color: AppColors.textSecondary,
),
),
Text(
dateLabel,
style: const TextStyle(
fontSize: 10,
color: AppColors.textSecondary,
),
),
const SizedBox(height: 8),
...day.slots.map(_SlotDot.new),
const SizedBox(height: 8),
Text(
'${day.fullySyncedSlots}/$denominator',
style: TextStyle(
fontSize: 11,
fontWeight: FontWeight.w700,
color: day.fullySyncedSlots == denominator && day.completedSlots > 0
? AppColors.success
: day.fullySyncedSlots < day.completedSlots
? Colors.amber
: AppColors.textPrimary,
),
),
const SizedBox(height: 2),
Text(
'synced',
style: const TextStyle(
fontSize: 9,
color: AppColors.textSecondary,
),
),
],
),
);
}
}
class _SlotDot extends StatelessWidget {
const _SlotDot(this.slot);
final MarketHistorySlotCoverage slot;
@override
Widget build(BuildContext context) {
final Color color;
if (!slot.completed) {
color = AppColors.surface;
} else if (slot.fullySynced) {
color = AppColors.success;
} else if (slot.syncedSymbolCount > 0) {
color = Colors.amber;
} else {
color = Colors.redAccent;
}
return Container(
key: Key('slot-${slot.slotStart.toIso8601String()}'),
width: 10,
height: 10,
margin: const EdgeInsets.symmetric(vertical: 2),
decoration: BoxDecoration(
color: slot.completed ? color : Colors.transparent,
shape: BoxShape.circle,
border: Border.all(
color: slot.completed ? color : AppColors.textSecondary.withValues(alpha: 0.35),
width: 1.2,
),
),
);
}
}

View File

@ -0,0 +1,261 @@
import 'package:flutter/material.dart';
import 'package:flutter/services.dart';
import '../../theme/app_theme.dart';
import '../models/backfill_sync_item.dart';
import '../models/sync_run_event.dart';
import '../utils/sync_run_formatters.dart';
import 'sync_run_status_chip.dart';
class SyncRunExpansionTile extends StatelessWidget {
const SyncRunExpansionTile({
super.key,
required this.event,
this.now,
this.emphasizeError = false,
this.onRetry,
});
final SyncRunEvent event;
final DateTime? now;
final bool emphasizeError;
final VoidCallback? onRetry;
@override
Widget build(BuildContext context) {
final Color iconColor = severityColor(event.severity);
return Card(
key: Key('sync-run-${event.id}'),
color: emphasizeError
? AppColors.surfaceElevated
: AppColors.surface,
margin: const EdgeInsets.symmetric(horizontal: 16, vertical: 4),
child: Theme(
data: Theme.of(context).copyWith(dividerColor: Colors.transparent),
child: ExpansionTile(
leading: Icon(severityIcon(event.severity), color: iconColor),
title: Text(
'${event.displayTitle}${shortStatusLabel(event)}',
style: const TextStyle(
fontSize: 14,
fontWeight: FontWeight.w600,
color: AppColors.textPrimary,
),
),
subtitle: Text(
event.summary,
style: const TextStyle(fontSize: 12, color: AppColors.textSecondary),
),
trailing: Text(
formatRelativeTime(event.startedAt, now: now),
style: const TextStyle(fontSize: 12, color: AppColors.textSecondary),
),
children: <Widget>[
Padding(
padding: const EdgeInsets.fromLTRB(16, 0, 16, 16),
child: Column(
crossAxisAlignment: CrossAxisAlignment.start,
children: <Widget>[
SyncRunStatusChip(event: event),
const SizedBox(height: 12),
..._detailRows(context),
],
),
),
],
),
),
);
}
List<Widget> _detailRows(BuildContext context) {
final List<Widget> rows = <Widget>[
_detailRow('Run ID', '${event.id}'),
_detailRow('Kind', event.kind),
_detailRow('Started', formatLocalTimestamp(event.startedAt)),
_detailRow('Finished', formatLocalTimestamp(event.finishedAt)),
_detailRow('Duration', formatDurationMs(event.durationMs)),
];
if (event.rowsWritten > 0) {
rows.add(_detailRow('Rows written', '${event.rowsWritten}'));
}
if (event.rowsRemoved > 0) {
rows.add(_detailRow('Rows removed', '${event.rowsRemoved}'));
}
if (event.kind == 'backfill' && event.backfillItems.isNotEmpty) {
rows.add(
const Padding(
padding: EdgeInsets.only(top: 8, bottom: 4),
child: Text(
'Backfill fetches (Alpaca start / raw.slot_start)',
style: TextStyle(
fontSize: 12,
fontWeight: FontWeight.w600,
color: AppColors.textSecondary,
),
),
),
);
for (final BackfillSyncItem item in event.backfillItems) {
rows.add(_backfillFetchRow(item));
}
}
if (event.status == SyncRunStatus.success && event.error == null) {
rows.add(_detailRow('Errors', 'No errors'));
}
if (event.error != null && event.error!.trim().isNotEmpty) {
final String? httpStatus = parseHttpStatus(event.error);
if (httpStatus != null) {
rows.add(_detailRow('HTTP status', httpStatus));
}
rows.add(
Padding(
padding: const EdgeInsets.only(top: 8),
child: Column(
crossAxisAlignment: CrossAxisAlignment.start,
children: <Widget>[
const Text(
'Error',
style: TextStyle(
fontSize: 12,
fontWeight: FontWeight.w600,
color: AppColors.textSecondary,
),
),
const SizedBox(height: 4),
ConstrainedBox(
constraints: const BoxConstraints(maxHeight: 240),
child: SingleChildScrollView(
child: SelectableText(
event.error!,
key: Key('sync-run-error-${event.id}'),
style: const TextStyle(
fontSize: 13,
color: AppColors.textPrimary,
),
),
),
),
TextButton.icon(
onPressed: () {
Clipboard.setData(ClipboardData(text: event.error!));
ScaffoldMessenger.of(context).showSnackBar(
const SnackBar(content: Text('Error copied')),
);
},
icon: const Icon(Icons.copy, size: 16),
label: const Text('Copy error'),
),
if (event.severity == SyncRunSeverity.rateLimit ||
event.status == SyncRunStatus.failed)
const Text(
'Will retry on next scheduled run.',
style: TextStyle(
fontSize: 12,
color: AppColors.textSecondary,
),
),
if (onRetry != null &&
(event.status == SyncRunStatus.failed ||
event.status == SyncRunStatus.partial ||
event.severity == SyncRunSeverity.rateLimit))
TextButton.icon(
key: Key('retry-run-${event.id}'),
onPressed: onRetry,
icon: const Icon(Icons.replay, size: 16),
label: const Text('Retry now'),
),
],
),
),
);
}
if (event.status == SyncRunStatus.inProgress) {
rows.add(
const Padding(
padding: EdgeInsets.only(top: 8),
child: Row(
children: <Widget>[
SizedBox(
width: 16,
height: 16,
child: CircularProgressIndicator(strokeWidth: 2),
),
SizedBox(width: 8),
Text(
'Run in progress…',
style: TextStyle(fontSize: 12, color: AppColors.textSecondary),
),
],
),
),
);
}
return rows;
}
Widget _backfillFetchRow(BackfillSyncItem item) {
final String wire = item.fetchSlotStartWire;
return Padding(
padding: const EdgeInsets.only(bottom: 8),
child: Column(
crossAxisAlignment: CrossAxisAlignment.start,
children: <Widget>[
SelectableText(
wire,
key: Key('backfill-slot-$wire'),
style: const TextStyle(
fontSize: 12,
fontFamily: 'monospace',
color: AppColors.textPrimary,
),
),
const SizedBox(height: 2),
Text(
'${item.symbols.length} assets: ${item.symbols.join(', ')}',
style: const TextStyle(
fontSize: 12,
color: AppColors.textSecondary,
),
),
],
),
);
}
Widget _detailRow(String label, String value) {
return Padding(
padding: const EdgeInsets.only(bottom: 6),
child: Row(
crossAxisAlignment: CrossAxisAlignment.start,
children: <Widget>[
SizedBox(
width: 110,
child: Text(
label,
style: const TextStyle(
fontSize: 12,
color: AppColors.textSecondary,
),
),
),
Expanded(
child: Text(
value,
style: const TextStyle(
fontSize: 12,
color: AppColors.textPrimary,
),
),
),
],
),
);
}
}

View File

@ -0,0 +1,27 @@
import 'package:flutter/material.dart';
import '../models/sync_run_event.dart';
import '../utils/sync_run_formatters.dart';
class SyncRunStatusChip extends StatelessWidget {
const SyncRunStatusChip({super.key, required this.event});
final SyncRunEvent event;
@override
Widget build(BuildContext context) {
final Color color = severityColor(event.severity);
return Container(
padding: const EdgeInsets.symmetric(horizontal: 8, vertical: 2),
decoration: BoxDecoration(
color: color.withValues(alpha: 0.15),
borderRadius: BorderRadius.circular(12),
border: Border.all(color: color.withValues(alpha: 0.5)),
),
child: Text(
shortStatusLabel(event),
style: TextStyle(color: color, fontSize: 12, fontWeight: FontWeight.w600),
),
);
}
}

View File

@ -1,5 +1,8 @@
import 'package:flutter/material.dart'; import 'package:flutter/material.dart';
import 'admin/admin_routes.dart';
import 'admin/screens/market_history_log_screen.dart';
import 'admin/widgets/admin_gate.dart';
import 'bootstrap.dart'; import 'bootstrap.dart';
import 'models/app_user.dart'; import 'models/app_user.dart';
import 'screens/home_screen.dart'; import 'screens/home_screen.dart';
@ -24,6 +27,17 @@ class CyberHybridHubApp extends StatelessWidget {
debugShowCheckedModeBanner: false, debugShowCheckedModeBanner: false,
theme: buildAppTheme(), theme: buildAppTheme(),
home: const AuthGate(), home: const AuthGate(),
onGenerateRoute: (RouteSettings settings) {
if (settings.name == AdminRoutes.marketHistoryLog) {
return MaterialPageRoute<void>(
settings: settings,
builder: (BuildContext context) => AdminGate(
child: marketHistoryLogScreen(),
),
);
}
return null;
},
); );
} }
} }

View File

@ -1,5 +1,6 @@
import 'package:flutter/material.dart'; import 'package:flutter/material.dart';
import '../admin/widgets/admin_app_bar_action.dart';
import '../models/app_user.dart'; import '../models/app_user.dart';
import '../models/incoming_question.dart'; import '../models/incoming_question.dart';
import '../models/sync_result.dart'; import '../models/sync_result.dart';
@ -8,6 +9,7 @@ import '../repositories/user_profile_repository.dart';
import '../services/auth_service.dart'; import '../services/auth_service.dart';
import '../services/questions_hub_service.dart'; import '../services/questions_hub_service.dart';
import '../theme/app_theme.dart'; import '../theme/app_theme.dart';
import '../widgets/profile_avatar.dart';
import '../widgets/swipe_question_tile.dart'; import '../widgets/swipe_question_tile.dart';
class HomeScreen extends StatelessWidget { class HomeScreen extends StatelessWidget {
@ -63,6 +65,7 @@ class HomeScreen extends StatelessWidget {
}, },
), ),
actions: <Widget>[ actions: <Widget>[
const AdminAppBarAction(),
IconButton( IconButton(
onPressed: () => UserProfileRepository.instance.sync(), onPressed: () => UserProfileRepository.instance.sync(),
tooltip: 'Sync profile', tooltip: 'Sync profile',
@ -155,15 +158,9 @@ class HomeScreen extends StatelessWidget {
), ),
child: Column( child: Column(
children: <Widget>[ children: <Widget>[
if (photoUrl != null) ProfileAvatar(
CircleAvatar( photoUrl: photoUrl,
radius: 36, radius: 36,
backgroundImage: NetworkImage(photoUrl),
)
else
const CircleAvatar(
radius: 36,
child: Icon(Icons.person, size: 36),
), ),
const SizedBox(height: 16), const SizedBox(height: 16),
Text( Text(

View File

@ -1,6 +1,7 @@
import 'package:flutter/foundation.dart' import 'package:flutter/foundation.dart'
show TargetPlatform, defaultTargetPlatform, kIsWeb; show TargetPlatform, defaultTargetPlatform, kIsWeb;
import '../admin/services/admin_access_service.dart';
import '../models/app_user.dart'; import '../models/app_user.dart';
import 'auth_service_firebase.dart'; import 'auth_service_firebase.dart';
import 'auth_service_linux.dart'; import 'auth_service_linux.dart';
@ -52,6 +53,7 @@ class AuthService {
} else { } else {
await AuthServiceFirebase.instance.signOut(); await AuthServiceFirebase.instance.signOut();
} }
AdminAccessService.instance.reset();
} }
/// Firebase ID token for API requests. Returns null when signed out. /// Firebase ID token for API requests. Returns null when signed out.

View File

@ -0,0 +1,81 @@
import 'package:flutter/material.dart';
import '../theme/app_theme.dart';
/// User profile photo with a graceful fallback when the URL fails (e.g.
/// Google `lh3.googleusercontent.com` returning HTTP 429).
class ProfileAvatar extends StatelessWidget {
const ProfileAvatar({
super.key,
this.photoUrl,
this.radius = 36,
});
final String? photoUrl;
final double radius;
@override
Widget build(BuildContext context) {
final String? url = photoUrl?.trim();
if (url == null || url.isEmpty) {
return _fallbackAvatar();
}
final double size = radius * 2;
return ClipOval(
child: SizedBox(
width: size,
height: size,
child: Image.network(
url,
width: size,
height: size,
fit: BoxFit.cover,
// Prevents NetworkImageLoadException from bubbling to the console
// when Google rate-limits profile photo URLs.
errorBuilder: (_, Object error, StackTrace? stackTrace) {
return _fallbackContents();
},
loadingBuilder: (
BuildContext context,
Widget child,
ImageChunkEvent? progress,
) {
if (progress == null) {
return child;
}
return ColoredBox(
color: AppColors.surfaceElevated,
child: Center(
child: SizedBox(
width: radius * 0.55,
height: radius * 0.55,
child: CircularProgressIndicator(
strokeWidth: 2,
color: AppColors.accent.withValues(alpha: 0.7),
),
),
),
);
},
),
),
);
}
Widget _fallbackAvatar() {
return CircleAvatar(
radius: radius,
backgroundColor: AppColors.surfaceElevated,
child: _fallbackContents(),
);
}
Widget _fallbackContents() {
return Icon(
Icons.person,
size: radius,
color: AppColors.accent.withValues(alpha: 0.85),
);
}
}

View File

@ -1,6 +1,8 @@
import 'dart:async'; import 'dart:async';
import 'dart:math' as math;
import 'package:flutter/material.dart'; import 'package:flutter/material.dart';
import 'package:flutter/services.dart';
import '../guid/guid_glyph_shape.dart'; import '../guid/guid_glyph_shape.dart';
import '../theme/app_theme.dart'; import '../theme/app_theme.dart';
@ -26,19 +28,66 @@ class SwipeQuestionTile extends StatefulWidget {
State<SwipeQuestionTile> createState() => _SwipeQuestionTileState(); State<SwipeQuestionTile> createState() => _SwipeQuestionTileState();
} }
class _SwipeQuestionTileState extends State<SwipeQuestionTile> { class _SwipeQuestionTileState extends State<SwipeQuestionTile>
with SingleTickerProviderStateMixin {
double _dragOffset = 0; double _dragOffset = 0;
double _verticalOffset = 0; double _verticalOffset = 0;
bool _acting = false; bool _acting = false;
int _lastSnappedValue = 0;
late final AnimationController _snapController;
static const double _swipeThreshold = 96; static const double _swipeThreshold = 96;
static const double _maxVerticalDrag = 120; static const double _glyphSize = 80;
static const double _sliderMin = -10; static const double _trackEdgeInset = 8;
static const double _sliderMax = 10; static const int _sliderMin = -10;
static const int _sliderMax = 10;
/// Swipe up +10, swipe down -10 (linear between). /// Updated each build from the tile height so ±10 reaches near the track edges.
double get _sliderValue => double _maxVerticalDrag = 120;
(_verticalOffset / _maxVerticalDrag * _sliderMax).clamp(_sliderMin, _sliderMax);
@override
void initState() {
super.initState();
_snapController = AnimationController(
vsync: this,
duration: const Duration(milliseconds: 140),
);
}
@override
void dispose() {
_snapController.dispose();
super.dispose();
}
/// Swipe up +10, swipe down -10; snaps to whole numbers.
int get _snappedSliderValue =>
(_clampedVerticalOffset / _maxVerticalDrag * _sliderMax)
.round()
.clamp(_sliderMin, _sliderMax);
double get _clampedVerticalOffset =>
_verticalOffset.clamp(-_maxVerticalDrag, _maxVerticalDrag);
double get _snappedVerticalOffset =>
_snappedSliderValue / _sliderMax * _maxVerticalDrag;
void _maybeTriggerSnapFeedback(int snapped) {
if (snapped == _lastSnappedValue) {
return;
}
_lastSnappedValue = snapped;
unawaited(_snapController.forward(from: 0));
HapticFeedback.selectionClick();
}
({double shakeX, double scale}) _snapMotion(double t) {
final double damp = 1 - t;
return (
shakeX: math.sin(t * math.pi * 6) * 5 * damp,
scale: 1 + 0.035 * math.sin(t * math.pi) * damp,
);
}
Future<void> _releaseDrag() async { Future<void> _releaseDrag() async {
if (_acting || widget.busy) { if (_acting || widget.busy) {
@ -51,7 +100,7 @@ class _SwipeQuestionTileState extends State<SwipeQuestionTile> {
_acting = true; _acting = true;
_dragOffset = MediaQuery.sizeOf(context).width; _dragOffset = MediaQuery.sizeOf(context).width;
}); });
await widget.onSwipeRight(_sliderValue); await widget.onSwipeRight(_snappedSliderValue);
} else if (_dragOffset < -_swipeThreshold) { } else if (_dragOffset < -_swipeThreshold) {
setState(() { setState(() {
_acting = true; _acting = true;
@ -76,6 +125,18 @@ class _SwipeQuestionTileState extends State<SwipeQuestionTile> {
return LayoutBuilder( return LayoutBuilder(
builder: (BuildContext context, BoxConstraints constraints) { builder: (BuildContext context, BoxConstraints constraints) {
// Container vertical padding (24×2) + track insets (8×2).
const double outerVerticalPadding = 48;
const double trackVerticalInset = 16;
final double innerHeight =
math.max(constraints.maxHeight - outerVerticalPadding, 172);
final double trackHeight =
math.max(innerHeight - trackVerticalInset, _glyphSize);
_maxVerticalDrag = math.max(
(trackHeight / 2) - (_glyphSize / 2) - _trackEdgeInset,
40,
);
return Stack( return Stack(
alignment: Alignment.center, alignment: Alignment.center,
children: <Widget>[ children: <Widget>[
@ -101,13 +162,27 @@ class _SwipeQuestionTileState extends State<SwipeQuestionTile> {
), ),
Transform.translate( Transform.translate(
offset: Offset(_dragOffset, 0), offset: Offset(_dragOffset, 0),
child: AnimatedBuilder(
animation: _snapController,
builder: (BuildContext context, Widget? child) {
final ({double shakeX, double scale}) motion =
_snapMotion(_snapController.value);
return Transform.translate(
offset: Offset(motion.shakeX, 0),
child: Transform.scale(
scale: motion.scale,
child: child,
),
);
},
child: GestureDetector( child: GestureDetector(
onHorizontalDragUpdate: widget.busy || _acting onHorizontalDragUpdate: widget.busy || _acting
? null ? null
: (DragUpdateDetails details) { : (DragUpdateDetails details) {
setState(() { setState(() {
_dragOffset += details.delta.dx; _dragOffset += details.delta.dx;
_dragOffset = _dragOffset.clamp(-width * 0.55, width * 0.55); _dragOffset =
_dragOffset.clamp(-width * 0.55, width * 0.55);
}); });
}, },
onHorizontalDragEnd: widget.busy || _acting onHorizontalDragEnd: widget.busy || _acting
@ -132,8 +207,8 @@ class _SwipeQuestionTileState extends State<SwipeQuestionTile> {
alignment: Alignment.center, alignment: Alignment.center,
children: <Widget>[ children: <Widget>[
Positioned( Positioned(
top: 20, top: 8,
bottom: 20, bottom: 8,
left: constraints.maxWidth * 0.22, left: constraints.maxWidth * 0.22,
right: constraints.maxWidth * 0.22, right: constraints.maxWidth * 0.22,
child: DecoratedBox( child: DecoratedBox(
@ -148,8 +223,8 @@ class _SwipeQuestionTileState extends State<SwipeQuestionTile> {
), ),
), ),
Positioned( Positioned(
top: 24, top: 8,
bottom: 24, bottom: 8,
left: constraints.maxWidth * 0.22, left: constraints.maxWidth * 0.22,
right: constraints.maxWidth * 0.22, right: constraints.maxWidth * 0.22,
child: GestureDetector( child: GestureDetector(
@ -163,14 +238,18 @@ class _SwipeQuestionTileState extends State<SwipeQuestionTile> {
-_maxVerticalDrag, -_maxVerticalDrag,
_maxVerticalDrag, _maxVerticalDrag,
); );
_maybeTriggerSnapFeedback(
_snappedSliderValue,
);
}); });
}, },
child: Center( child: Center(
child: Transform.translate( child: Transform.translate(
offset: Offset(0, -_verticalOffset), offset: Offset(0, -_snappedVerticalOffset),
child: QuestionGuidGlyph( child: QuestionGuidGlyph(
guid: widget.questionId, guid: widget.questionId,
displayValue: _sliderValue, size: _glyphSize,
displayValue: _snappedSliderValue,
), ),
), ),
), ),
@ -182,6 +261,7 @@ class _SwipeQuestionTileState extends State<SwipeQuestionTile> {
), ),
), ),
), ),
),
if (widget.busy) if (widget.busy)
const Positioned.fill( const Positioned.fill(
child: ColoredBox( child: ColoredBox(

View File

@ -0,0 +1,29 @@
#!/usr/bin/env bash
# Collect coverage for admin portal packages (Section 2 targets are advisory).
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
echo "==> Server coverage"
cd "$ROOT/server"
rm -rf coverage
dart test \
test/trading/market_history_admin_logic_test.dart \
test/trading/market_history_admin_actions_test.dart \
test/integration/market_history_admin_handler_test.dart \
--coverage=coverage
dart pub global activate coverage 2>/dev/null || true
if command -v format_coverage >/dev/null 2>&1; then
format_coverage --lcov --in=coverage --out=coverage/lcov.info --report-on=lib
echo "Server lcov: server/coverage/lcov.info"
else
echo "Install coverage package for lcov: dart pub global activate coverage"
fi
echo "==> Flutter coverage"
cd "$ROOT"
rm -rf coverage
flutter test test/admin/ --coverage
echo "Flutter lcov: coverage/lcov.info"
echo "Review lcov files against FLUTTER-TDD-PLAN.md Section 2 targets."

View File

@ -0,0 +1,8 @@
#!/usr/bin/env bash
# Generate coverage and enforce FLUTTER-TDD-PLAN.md Section 2 thresholds.
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
bash "$ROOT/scripts/admin-portal-coverage.sh"
dart "$ROOT/scripts/check_admin_portal_coverage.dart"

View File

@ -0,0 +1,131 @@
import 'dart:io';
/// Enforces FLUTTER-TDD-PLAN.md Section 2 line-coverage targets for admin portal files.
void main() {
final String root = _repoRoot();
final List<_CoverageGate> gates = <_CoverageGate>[
_CoverageGate(
label: 'server admin logic',
lcovPath: '$root/server/coverage/lcov.info',
pathContains: 'market_history_admin_logic.dart',
minLinePercent: 95,
),
_CoverageGate(
label: 'server admin handler',
lcovPath: '$root/server/coverage/lcov.info',
pathContains: 'market_history_admin_handler.dart',
minLinePercent: 85,
),
_CoverageGate(
label: 'server admin actions',
lcovPath: '$root/server/coverage/lcov.info',
pathContains: 'market_history_admin_actions.dart',
minLinePercent: 85,
),
_CoverageGate(
label: 'flutter admin models',
lcovPath: '$root/coverage/lcov.info',
pathContains: 'lib/admin/models/',
minLinePercent: 90,
),
_CoverageGate(
label: 'flutter admin repositories',
lcovPath: '$root/coverage/lcov.info',
pathContains: 'lib/admin/repositories/',
minLinePercent: 90,
),
_CoverageGate(
label: 'flutter admin widgets',
lcovPath: '$root/coverage/lcov.info',
pathContains: 'lib/admin/widgets/',
minLinePercent: 75,
),
_CoverageGate(
label: 'flutter admin screens',
lcovPath: '$root/coverage/lcov.info',
pathContains: 'lib/admin/screens/',
minLinePercent: 75,
),
];
final List<String> failures = <String>[];
for (final _CoverageGate gate in gates) {
final File file = File(gate.lcovPath);
if (!file.existsSync()) {
failures.add('${gate.label}: missing ${gate.lcovPath}');
continue;
}
final double pct = _lineCoveragePercent(
file.readAsLinesSync(),
gate.pathContains,
);
stdout.writeln(
'${gate.label}: ${pct.toStringAsFixed(1)}% (min ${gate.minLinePercent}%)',
);
if (pct + 0.05 < gate.minLinePercent) {
failures.add(
'${gate.label}: ${pct.toStringAsFixed(1)}% < ${gate.minLinePercent}%',
);
}
}
if (failures.isNotEmpty) {
stderr.writeln('Admin portal coverage gates failed:');
for (final String failure in failures) {
stderr.writeln(' - $failure');
}
exit(1);
}
}
String _repoRoot() {
final String script = Platform.script.toFilePath();
return File(script).parent.parent.path;
}
double _lineCoveragePercent(List<String> lines, String pathContains) {
double total = 0;
double hit = 0;
String? currentFile;
for (final String line in lines) {
if (line.startsWith('SF:')) {
currentFile = line.substring(3);
continue;
}
if (line == 'end_of_record') {
currentFile = null;
continue;
}
if (currentFile == null || !currentFile.contains(pathContains)) {
continue;
}
if (line.startsWith('DA:')) {
final List<String> parts = line.substring(3).split(',');
if (parts.length != 2) {
continue;
}
total++;
if (int.tryParse(parts[1]) != 0) {
hit++;
}
}
}
if (total == 0) {
return 0;
}
return hit * 100 / total;
}
class _CoverageGate {
const _CoverageGate({
required this.label,
required this.lcovPath,
required this.pathContains,
required this.minLinePercent,
});
final String label;
final String lcovPath;
final String pathContains;
final int minLinePercent;
}

View File

@ -0,0 +1,30 @@
#!/usr/bin/env bash
# Copies flutter.js.map next to build/web/flutter_bootstrap.js so Firefox/
# Chrome DevTools stop 404ing on "flutter.js.map" after `flutter build web`.
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
OUT_DIR="${1:-$ROOT/build/web}"
if [ ! -d "$OUT_DIR" ]; then
echo "copy_flutter_js_source_map: output dir not found: $OUT_DIR" >&2
exit 1
fi
if ! command -v flutter >/dev/null 2>&1; then
echo "copy_flutter_js_source_map: flutter not on PATH" >&2
exit 1
fi
FLUTTER_BIN="$(command -v flutter)"
SDK_ROOT="$(cd "$(dirname "$FLUTTER_BIN")/.." && pwd)"
MAP_SRC="$SDK_ROOT/bin/cache/flutter_web_sdk/flutter_js/flutter.js.map"
MAP_DST="$OUT_DIR/flutter.js.map"
if [ ! -f "$MAP_SRC" ]; then
echo "copy_flutter_js_source_map: source map missing at $MAP_SRC" >&2
exit 1
fi
cp "$MAP_SRC" "$MAP_DST"
echo "copy_flutter_js_source_map: installed $MAP_DST"

26
scripts/test-admin-portal.sh Executable file
View File

@ -0,0 +1,26 @@
#!/usr/bin/env bash
# Run Flutter Admin Portal test suites (server + Flutter).
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
echo "==> Server admin portal tests"
cd "$ROOT/server"
if [[ -f .env ]]; then
set -a
# shellcheck source=/dev/null
source .env
set +a
fi
dart pub get
dart test \
test/trading/market_history_admin_logic_test.dart \
test/trading/market_history_admin_actions_test.dart \
test/integration/market_history_admin_handler_test.dart \
test/integration/market_history_admin_risk_test.dart
echo "==> Flutter admin portal tests"
cd "$ROOT"
flutter test test/admin/
echo "Admin portal tests passed."

View File

@ -43,7 +43,7 @@ dart pub get
dart test dart test
``` ```
Integration tests apply migrations `001``004` on `cyberhybridhub_test` and truncate Integration tests apply migrations `001``009` on `cyberhybridhub_test` and truncate
trading tables between cases. Optional override: `TEST_DATABASE_URL`. trading tables between cases. Optional override: `TEST_DATABASE_URL`.
## Endpoints ## Endpoints
@ -127,6 +127,84 @@ curl -s -X POST http://localhost:3000/v1/me/incoming-question \
-d '{"text":"What is your preferred contact method?"}' -d '{"text":"What is your preferred contact method?"}'
``` ```
## Market history
Alpaca **`4Hour`** bars in six **UTC slots** per day (`00`, `04`, `08`, `12`, `16`, `20`).
Stored as `metric=bar`, `timeframe=4Hour`. Rolling window: `MARKET_HISTORY_WINDOW_DAYS` (default 7).
**Backfill** (`kind=backfill`): fetches each **ended** slot still missing in DB; skips the open slot.
Throttled to `MARKET_HISTORY_API_REQUESTS_PER_MINUTE` (default 200/min). On HTTP 429: wait 1 minute, retry once; if still limited, save partial rows and resume next tick.
Runs when `hasPendingSlots` is true (worker tick). One bars request per slot × symbol batch.
Before each worker or admin pipeline, orphaned `market_data_sync_runs` rows with
`finished_at IS NULL` are closed (stale rows first, then any remaining orphans) so a
crashed prior sync cannot block new work.
**Universe** / **cleanup** keep hour-based cadence. `guess_weekly_move` reads these bars.
Requires `TRADING_ENABLED=true` when `MARKET_HISTORY_SYNC_ENABLED=true`.
**Migration `008`:** drops legacy `1Day` history bars, adds `timeframe` check (`4Hour` allowed), partial index on `4Hour` bars, `market_data_sync_runs.slots_synced`.
**Migration `009`:** adds `market_data_sync_runs.backfill_items` JSONB — per-slot UTC start + symbol list for each backfill run.
| Env var | Default | Purpose |
|---------|---------|---------|
| `MARKET_HISTORY_SYNC_ENABLED` | `false` | Worker: universe → backfill → cleanup |
| `MARKET_HISTORY_WINDOW_DAYS` | `7` | Oldest slot start to retain / backfill |
| `MARKET_HISTORY_RETENTION_DAYS` | `7` | Delete `as_of` older than this |
| `MARKET_HISTORY_ARCHIVE_ENABLED` | `false` | Archive before delete |
| `MARKET_UNIVERSE_REFRESH_HOURS` | `24` | Min hours between universe syncs |
| `MARKET_HISTORY_SYNC_HOURS` | `24` | Unused for backfill (slot-gated); fallback if no slot gate |
| `MARKET_HISTORY_CLEANUP_HOURS` | `24` | Min hours between cleanups |
| `MARKET_HISTORY_SYNC_HOUR_UTC` | *(unset)* | UTC hour floor; same-day cap applies to universe/cleanup only |
| `HISTORY_SYNC_BATCH_SIZE` | `50` | Symbols per bars request |
| `HISTORY_SYNC_MAX_SYMBOLS` | `2000` | Max symbols per backfill run |
| `MARKET_HISTORY_API_REQUESTS_PER_MINUTE` | `200` | Max Alpaca HTTP calls per rolling minute during backfill |
| `MARKET_HISTORY_SYNC_STALE_MINUTES` | `30` | Abort in-progress sync rows older than this before a new pipeline |
| `MIN_BARS_FOR_GUESS` | `5` | Min 4-hour bars for guess eligibility |
| `GUESS_COOLDOWN_HOURS` | `24` | Per-symbol guess cooldown |
## Admin portal (market history log)
Read-only audit log and on-demand triggers for universe sync, backfill, and
retention cleanup. Mounted when `ADMIN_PORTAL_ENABLED=true`.
| Env var | Default | Purpose |
|---------|---------|---------|
| `ADMIN_PORTAL_ENABLED` | `false` | Mount `/v1/admin/market-history/*` and `/v1/admin/market-data/*` |
| `ADMIN_FIREBASE_UIDS` | *(empty)* | Comma-separated Firebase UIDs allowed to call admin routes |
Requires `TRADING_ENABLED=true`, Alpaca credentials, and
`QUESTION_PIPELINE_TEST_MODE=false` for on-demand resync/cleanup when
`MARKET_HISTORY_SYNC_ENABLED=true`. When sync is disabled, the admin log and
week-coverage calendar remain available read-only; resync/cleanup return `503`.
Scheduled worker sync also requires `MARKET_HISTORY_SYNC_ENABLED=true`.
| Method | Path | Auth |
|--------|------|------|
| `GET` | `/v1/admin/market-history/sync-runs` | Admin Firebase UID |
| `POST` | `/v1/admin/market-data/resync` | Admin Firebase UID |
| `POST` | `/v1/admin/market-data/cleanup?archive=true\|false` | Admin Firebase UID |
`sync-runs` includes optional `config` when the server has market-history env loaded:
```json
{ "archiveEnabled": true, "windowDays": 7, "retentionDays": 7, "syncEnabled": true }
```
Flutter uses `config.archiveEnabled` to show the archive checkbox in the cleanup
confirm dialog.
### Admin portal tests
From the repo root:
```bash
./scripts/test-admin-portal.sh # server + Flutter admin suites
./scripts/check-admin-portal-coverage.sh # coverage + Section 2 thresholds
```
CI: `.github/workflows/admin-portal.yml` runs both on admin-related changes.
## Flutter client ## Flutter client
Run the app with the API URL (defaults to `http://localhost:3000`): Run the app with the API URL (defaults to `http://localhost:3000`):

View File

@ -3,12 +3,15 @@ import 'dart:io';
import 'package:shelf/shelf.dart'; import 'package:shelf/shelf.dart';
import 'package:shelf/shelf_io.dart' as shelf_io; import 'package:shelf/shelf_io.dart' as shelf_io;
import '../lib/alpaca/alpaca_assets_client.dart';
import '../lib/alpaca/alpaca_market_data_client.dart'; import '../lib/alpaca/alpaca_market_data_client.dart';
import '../lib/alpaca/alpaca_trading_client.dart'; import '../lib/alpaca/alpaca_trading_client.dart';
import '../lib/db.dart'; import '../lib/db.dart';
import '../lib/env.dart'; import '../lib/env.dart';
import '../lib/market_history_env.dart';
import '../lib/firebase_auth.dart'; import '../lib/firebase_auth.dart';
import '../lib/handlers/incoming_question_handler.dart'; import '../lib/handlers/incoming_question_handler.dart';
import '../lib/handlers/market_history_admin_handler.dart';
import '../lib/handlers/profile_handler.dart'; import '../lib/handlers/profile_handler.dart';
import '../lib/handlers/questions_handler.dart'; import '../lib/handlers/questions_handler.dart';
import '../lib/handlers/questions_hub_handler.dart'; import '../lib/handlers/questions_hub_handler.dart';
@ -18,7 +21,14 @@ import '../lib/question_service.dart';
import '../lib/questions_db.dart'; import '../lib/questions_db.dart';
import '../lib/trading/guardrails.dart'; import '../lib/trading/guardrails.dart';
import '../lib/trading/market_data_db.dart'; import '../lib/trading/market_data_db.dart';
import '../lib/trading/market_data_history.dart';
import '../lib/trading/market_history_api_rate_limiter.dart';
import '../lib/trading/market_history_query.dart';
import '../lib/trading/market_data_ingest.dart'; import '../lib/trading/market_data_ingest.dart';
import '../lib/trading/market_data_retention.dart';
import '../lib/trading/market_history_admin_actions.dart';
import '../lib/trading/tradable_assets_db.dart';
import '../lib/trading/tradable_assets_sync.dart';
import '../lib/trading/trade_actuator.dart'; import '../lib/trading/trade_actuator.dart';
import '../lib/trading/trade_orders_db.dart'; import '../lib/trading/trade_orders_db.dart';
import '../lib/trading/trading_config_db.dart'; import '../lib/trading/trading_config_db.dart';
@ -26,6 +36,8 @@ import '../lib/trading/trading_dev_actions.dart';
import '../lib/trading/trading_orchestrator.dart'; import '../lib/trading/trading_orchestrator.dart';
import '../lib/trading/trading_pipeline.dart'; import '../lib/trading/trading_pipeline.dart';
import '../lib/trading/user_trading_state_db.dart'; import '../lib/trading/user_trading_state_db.dart';
import '../lib/workers/market_history_scheduler.dart';
import '../lib/workers/market_history_scheduler_config.dart';
import '../lib/workers/question_background_worker.dart'; import '../lib/workers/question_background_worker.dart';
Future<void> main() async { Future<void> main() async {
@ -57,6 +69,8 @@ Future<void> main() async {
TradingPipeline? tradingPipeline; TradingPipeline? tradingPipeline;
TradingOrchestrator? tradingOrchestrator; TradingOrchestrator? tradingOrchestrator;
TradingDevActions? tradingDevActions; TradingDevActions? tradingDevActions;
MarketHistoryScheduler? marketHistoryScheduler;
MarketHistoryAdminActions? marketHistoryAdminActions;
AlpacaMarketDataClient? alpacaMarketDataClient; AlpacaMarketDataClient? alpacaMarketDataClient;
AlpacaTradingClient? alpacaTradingClient; AlpacaTradingClient? alpacaTradingClient;
if (env.tradingEnabled) { if (env.tradingEnabled) {
@ -66,12 +80,15 @@ Future<void> main() async {
final UserTradingStateDb tradingStateDb = final UserTradingStateDb tradingStateDb =
UserTradingStateDb(db.connection); UserTradingStateDb(db.connection);
final MarketHistoryEnv mh = env.marketHistory;
tradingPipeline = TradingPipeline( tradingPipeline = TradingPipeline(
questionsDb: questionsDb, questionsDb: questionsDb,
questionService: questionService, questionService: questionService,
marketDataDb: marketDataDb, marketDataDb: marketDataDb,
tradingConfigDb: tradingConfigDb, tradingConfigDb: tradingConfigDb,
tradingStateDb: tradingStateDb, tradingStateDb: tradingStateDb,
marketHistoryQuery: MarketHistoryQuery(connection: db.connection),
marketHistoryEnv: mh,
guardrails: Guardrails(allowLive: env.alpaca.allowLive), guardrails: Guardrails(allowLive: env.alpaca.allowLive),
); );
@ -118,6 +135,79 @@ Future<void> main() async {
tradingPipeline: tradingPipeline, tradingPipeline: tradingPipeline,
); );
} }
if (useRealAlpaca &&
(env.marketHistorySyncEnabled || env.adminPortalEnabled)) {
final TradableAssetsDb tradableAssetsDb =
TradableAssetsDb(db.connection);
final AlpacaAssetsClient assetsClient = AlpacaAssetsClient(env: env.alpaca);
final MarketHistoryApiRateLimiter historyRateLimiter =
MarketHistoryApiRateLimiter(
requestsPerMinute: mh.apiRequestsPerMinute,
);
final AlpacaMarketDataClient historyMarketDataClient =
AlpacaMarketDataClient(env: env.alpaca);
final TradableAssetsSync tradableAssetsSync = TradableAssetsSync(
assetsClient: assetsClient,
assetsDb: tradableAssetsDb,
connection: db.connection,
);
final MarketDataHistorySync historySync = MarketDataHistorySync(
marketDataClient: historyMarketDataClient,
apiRequestsPerMinute: mh.apiRequestsPerMinute,
tradableAssetsDb: tradableAssetsDb,
marketDataDb: marketDataDb,
connection: db.connection,
batchSize: mh.historySyncBatchSize,
maxSymbols: mh.historySyncMaxSymbols,
windowDays: mh.windowDays,
);
final MarketDataRetention retention = MarketDataRetention(
connection: db.connection,
windowDays: mh.retentionDays,
);
if (env.marketHistorySyncEnabled) {
marketHistoryScheduler = MarketHistoryScheduler(
connection: db.connection,
config: MarketHistorySchedulerConfig(
universeRefreshHours: mh.universeRefreshHours,
historySyncHours: mh.historySyncHours,
cleanupHours: mh.cleanupHours,
syncHourUtc: mh.syncHourUtc,
staleSyncRunMinutes: mh.staleSyncRunMinutes,
),
runUniverse: (DateTime now) => tradableAssetsSync.runOnce(now: now),
runBackfill: (DateTime now) => historySync.runOnce(now: now),
backfillIsDue: historySync.hasPendingSlots,
runCleanup: (DateTime now) =>
retention.run(archive: mh.archiveEnabled, now: now),
);
}
if (env.adminPortalEnabled) {
if (env.marketHistorySyncEnabled) {
marketHistoryAdminActions = MarketHistoryAdminActions(
connection: db.connection,
runUniverse: (DateTime now) => tradableAssetsSync.runOnce(now: now),
runBackfill: (DateTime now) => historySync.runOnce(now: now),
runCleanup: (DateTime now, bool archive, int windowDays) =>
retention.run(archive: archive, now: now, windowDays: windowDays),
defaultArchiveEnabled: mh.archiveEnabled,
defaultWindowDays: mh.windowDays,
);
} else {
stderr.writeln(
'Admin portal on-demand sync/cleanup disabled: '
'MARKET_HISTORY_SYNC_ENABLED=false.',
);
}
}
} else if (env.adminPortalEnabled) {
stderr.writeln(
'Admin portal on-demand sync/cleanup unavailable: requires '
'TRADING_ENABLED=true, Alpaca credentials, and '
'QUESTION_PIPELINE_TEST_MODE=false.',
);
}
} }
final QuestionPipeline questionPipeline = QuestionPipeline( final QuestionPipeline questionPipeline = QuestionPipeline(
@ -132,6 +222,7 @@ Future<void> main() async {
pipeline: questionPipeline, pipeline: questionPipeline,
interval: Duration(seconds: env.questionWorkerIntervalSeconds), interval: Duration(seconds: env.questionWorkerIntervalSeconds),
tradingOrchestrator: tradingOrchestrator, tradingOrchestrator: tradingOrchestrator,
marketHistoryScheduler: marketHistoryScheduler,
); );
backgroundWorker.start(); backgroundWorker.start();
} }
@ -153,6 +244,20 @@ Future<void> main() async {
final Handler? tradingDev = tradingDevActions == null final Handler? tradingDev = tradingDevActions == null
? null ? null
: tradingDevHandler(auth: auth, devActions: tradingDevActions); : tradingDevHandler(auth: auth, devActions: tradingDevActions);
final Handler? marketHistoryAdmin = env.adminPortalEnabled
? marketHistoryAdminHandler(
auth: auth,
connection: db.connection,
adminFirebaseUids: env.adminFirebaseUids,
actions: marketHistoryAdminActions,
portalConfig: MarketHistoryAdminPortalConfig(
archiveEnabled: env.marketHistory.archiveEnabled,
windowDays: env.marketHistory.windowDays,
retentionDays: env.marketHistory.retentionDays,
syncEnabled: env.marketHistorySyncEnabled,
),
)
: null;
final Handler handler = Pipeline() final Handler handler = Pipeline()
.addMiddleware(logRequests()) .addMiddleware(logRequests())
@ -167,6 +272,9 @@ Future<void> main() async {
if (tradingDev != null && path.startsWith(tradingDevBasePath)) { if (tradingDev != null && path.startsWith(tradingDevBasePath)) {
return tradingDev(request); return tradingDev(request);
} }
if (marketHistoryAdmin != null && path.startsWith('/v1/admin')) {
return marketHistoryAdmin(request);
}
if (path.startsWith(questionsBasePath)) { if (path.startsWith(questionsBasePath)) {
return questions(request); return questions(request);
} }

View File

@ -0,0 +1,82 @@
import 'dart:convert';
import 'package:http/http.dart' as http;
import 'alpaca_env.dart';
import 'alpaca_models.dart';
/// REST client for Alpaca's Trading-API `/v2/assets` endpoint.
///
/// The asset universe lives behind the trading host, NOT the data host
/// see [AlpacaEnv.tradingBaseUrl]. We treat it as read-only here; the
/// tradable_assets sync (§2.2) is the only writer to the DB.
class AlpacaAssetsClient {
AlpacaAssetsClient({
required AlpacaEnv env,
http.Client? httpClient,
}) : _env = env,
_client = httpClient ?? http.Client();
final AlpacaEnv _env;
final http.Client _client;
/// `GET ${tradingBaseUrl}/v2/assets?status=active&asset_class=us_equity`.
///
/// Filters to `tradable=true` are applied **server-side** by the caller
/// (the asset-universe sync) so the client stays a thin wrapper and the
/// audit trail in [tradable_assets] still records inactive symbols when
/// they later disappear.
Future<List<AlpacaAsset>> listActiveTradable() async {
_env.requireCredentials();
final Uri uri = Uri.parse('${_env.tradingBaseUrl}/v2/assets').replace(
queryParameters: <String, String>{
'status': 'active',
'asset_class': 'us_equity',
},
);
final http.Response response;
try {
response = await _client.get(uri, headers: _env.authHeaders);
} on http.ClientException catch (e) {
throw AlpacaAssetsException(
'GET /v2/assets transport error: ${e.message}',
);
}
if (response.statusCode != 200) {
throw AlpacaAssetsException(
'GET /v2/assets failed: '
'${response.statusCode} ${response.body}',
);
}
final dynamic decoded = jsonDecode(response.body);
if (decoded is! List) {
throw AlpacaAssetsException(
'GET /v2/assets returned non-list body: ${response.body}',
);
}
return decoded
.whereType<Map>()
.map((Map<dynamic, dynamic> m) =>
AlpacaAsset.fromJson(Map<String, dynamic>.from(m)))
.toList(growable: false);
}
void close() => _client.close();
}
/// Thrown by [AlpacaAssetsClient] when an upstream HTTP call fails.
///
/// The message always includes the HTTP status code (when applicable) and
/// the response body so the caller's audit row can capture it verbatim.
class AlpacaAssetsException implements Exception {
AlpacaAssetsException(this.message);
final String message;
@override
String toString() => message;
}

View File

@ -24,6 +24,16 @@ class AlpacaEnv {
bool get hasCredentials => bool get hasCredentials =>
apiKeyId.isNotEmpty && apiSecretKey.isNotEmpty; apiKeyId.isNotEmpty && apiSecretKey.isNotEmpty;
/// HTTP headers that authenticate every Alpaca REST call.
///
/// Shared across `AlpacaMarketDataClient`, `AlpacaTradingClient`, and
/// `AlpacaAssetsClient`; spread it (`...env.authHeaders`) and add
/// content-type/accept where the request body needs them.
Map<String, String> get authHeaders => <String, String>{
'APCA-API-KEY-ID': apiKeyId,
'APCA-API-SECRET-KEY': apiSecretKey,
};
bool get isPaperUrl => bool get isPaperUrl =>
tradingBaseUrl.contains('paper-api') || tradingBaseUrl.contains('paper-api') ||
!tradingBaseUrl.contains(liveTradingHost); !tradingBaseUrl.contains(liveTradingHost);

View File

@ -4,22 +4,28 @@ import 'package:http/http.dart' as http;
import 'alpaca_env.dart'; import 'alpaca_env.dart';
import 'alpaca_models.dart'; import 'alpaca_models.dart';
import '../trading/market_history_four_hour_slot.dart';
/// REST client for Alpaca Market Data API v2 (IEX feed on Basic plan). /// REST client for Alpaca Market Data API v2 (IEX feed on Basic plan).
class AlpacaMarketDataClient { class AlpacaMarketDataClient {
AlpacaMarketDataClient({ AlpacaMarketDataClient({
required AlpacaEnv env, required AlpacaEnv env,
http.Client? httpClient, http.Client? httpClient,
Future<void> Function()? beforeHttpRequest,
}) : _env = env, }) : _env = env,
_client = httpClient ?? http.Client(); _client = httpClient ?? http.Client(),
_beforeHttpRequest = beforeHttpRequest;
final AlpacaEnv _env; final AlpacaEnv _env;
final http.Client _client; final http.Client _client;
final Future<void> Function()? _beforeHttpRequest;
Map<String, String> get _headers => <String, String>{ Future<void> _throttle() async {
'APCA-API-KEY-ID': _env.apiKeyId, final Future<void> Function()? hook = _beforeHttpRequest;
'APCA-API-SECRET-KEY': _env.apiSecretKey, if (hook != null) {
}; await hook();
}
}
/// `GET /v2/stocks/{symbol}/trades/latest` /// `GET /v2/stocks/{symbol}/trades/latest`
Future<AlpacaLatestTradeResponse> getLatestTrade(String symbol) async { Future<AlpacaLatestTradeResponse> getLatestTrade(String symbol) async {
@ -28,7 +34,9 @@ class AlpacaMarketDataClient {
'${_env.dataBaseUrl}/v2/stocks/${Uri.encodeComponent(symbol)}/trades/latest', '${_env.dataBaseUrl}/v2/stocks/${Uri.encodeComponent(symbol)}/trades/latest',
).replace(queryParameters: <String, String>{'feed': _env.dataFeed}); ).replace(queryParameters: <String, String>{'feed': _env.dataFeed});
final http.Response response = await _client.get(uri, headers: _headers); await _throttle();
final http.Response response =
await _client.get(uri, headers: _env.authHeaders);
if (response.statusCode != 200) { if (response.statusCode != 200) {
throw AlpacaMarketDataException( throw AlpacaMarketDataException(
'getLatestTrade($symbol) failed: ${response.statusCode} ${response.body}', 'getLatestTrade($symbol) failed: ${response.statusCode} ${response.body}',
@ -58,7 +66,9 @@ class AlpacaMarketDataClient {
}, },
); );
final http.Response response = await _client.get(uri, headers: _headers); await _throttle();
final http.Response response =
await _client.get(uri, headers: _env.authHeaders);
if (response.statusCode != 200) { if (response.statusCode != 200) {
throw AlpacaMarketDataException( throw AlpacaMarketDataException(
'getDailyBars failed: ${response.statusCode} ${response.body}', 'getDailyBars failed: ${response.statusCode} ${response.body}',
@ -69,13 +79,89 @@ class AlpacaMarketDataClient {
); );
} }
/// `GET /v2/stocks/bars` over a time range with pagination.
///
/// Follows `next_page_token` until exhausted or [maxPages] is reached.
/// HTTP 429 responses throw [AlpacaMarketDataException] with `rate` in
/// the message so callers can back off.
Future<AlpacaBarsResponse> getBarsRange({
required List<String> symbols,
required String timeframe,
required DateTime start,
required DateTime end,
int maxPages = 20,
int limit = 10000,
}) async {
_env.requireCredentials();
if (symbols.isEmpty) {
return AlpacaBarsResponse(barsBySymbol: <String, List<AlpacaBar>>{});
}
AlpacaBarsResponse merged =
AlpacaBarsResponse(barsBySymbol: <String, List<AlpacaBar>>{});
String? pageToken;
int pagesFetched = 0;
while (pagesFetched < maxPages) {
final Map<String, String> query = <String, String>{
'symbols': symbols.join(','),
'timeframe': timeframe,
'start': MarketHistoryFourHourSlot.wireUtc(start),
'end': MarketHistoryFourHourSlot.wireUtc(end),
'feed': _env.dataFeed,
'limit': limit.toString(),
if (pageToken != null && pageToken.isNotEmpty) 'page_token': pageToken,
};
final Uri uri = Uri.parse('${_env.dataBaseUrl}/v2/stocks/bars')
.replace(queryParameters: query);
await _throttle();
final http.Response response =
await _client.get(uri, headers: _env.authHeaders);
if (response.statusCode == 429) {
throw AlpacaMarketDataException.rateLimited(
'getBarsRange rate limited: ${response.statusCode} ${response.body}',
);
}
if (response.statusCode != 200) {
throw AlpacaMarketDataException(
'getBarsRange failed: ${response.statusCode} ${response.body}',
);
}
final Map<String, dynamic> decoded =
jsonDecode(response.body) as Map<String, dynamic>;
final AlpacaBarsResponse page = AlpacaBarsResponse.fromJson(decoded);
merged = merged.merge(page);
pagesFetched++;
pageToken = page.nextPageToken;
if (pageToken == null || pageToken.isEmpty) {
break;
}
}
return merged;
}
void close() => _client.close(); void close() => _client.close();
} }
class AlpacaMarketDataException implements Exception { class AlpacaMarketDataException implements Exception {
AlpacaMarketDataException(this.message); AlpacaMarketDataException(this.message, {this.statusCode});
AlpacaMarketDataException.rateLimited(this.message)
: statusCode = 429;
final String message; final String message;
final int? statusCode;
bool get isRateLimited =>
statusCode == 429 ||
message.toLowerCase().contains('rate limited') ||
message.contains('429');
@override @override
String toString() => message; String toString() => message;

View File

@ -1,3 +1,46 @@
/// Alpaca `/v2/assets` row the catalog entry for one tradable instrument.
///
/// The API is documented at
/// https://docs.alpaca.markets/reference/get-v2-assets-1; we keep [raw] so
/// downstream code can persist the full payload as JSONB without losing
/// any fields the model doesn't yet expose.
class AlpacaAsset {
AlpacaAsset({
required this.symbol,
required this.assetClass,
required this.status,
required this.tradable,
required this.fractionable,
this.exchange,
this.name,
this.raw,
});
final String symbol;
final String assetClass;
final String? exchange;
final String? name;
final String status;
final bool tradable;
final bool fractionable;
final Map<String, dynamic>? raw;
factory AlpacaAsset.fromJson(Map<String, dynamic> json) {
return AlpacaAsset(
symbol: json['symbol']! as String,
// Alpaca's payload calls the field `class` (a Dart reserved word),
// so we have to read it dynamically rather than via a typed getter.
assetClass: (json['class'] as String?) ?? 'us_equity',
exchange: json['exchange'] as String?,
name: json['name'] as String?,
status: (json['status'] as String?) ?? 'active',
tradable: (json['tradable'] as bool?) ?? false,
fractionable: (json['fractionable'] as bool?) ?? false,
raw: json,
);
}
}
/// Alpaca v2 latest-trade response: `{ "symbol", "trade" }`. /// Alpaca v2 latest-trade response: `{ "symbol", "trade" }`.
class AlpacaLatestTradeResponse { class AlpacaLatestTradeResponse {
AlpacaLatestTradeResponse({required this.symbol, required this.trade}); AlpacaLatestTradeResponse({required this.symbol, required this.trade});
@ -74,9 +117,13 @@ class AlpacaBar {
/// Multi-symbol bars response: `{ "bars": { "SPY": [ ... ] } }`. /// Multi-symbol bars response: `{ "bars": { "SPY": [ ... ] } }`.
class AlpacaBarsResponse { class AlpacaBarsResponse {
AlpacaBarsResponse({required this.barsBySymbol}); AlpacaBarsResponse({
required this.barsBySymbol,
this.nextPageToken,
});
final Map<String, List<AlpacaBar>> barsBySymbol; final Map<String, List<AlpacaBar>> barsBySymbol;
final String? nextPageToken;
factory AlpacaBarsResponse.fromJson(Map<String, dynamic> json) { factory AlpacaBarsResponse.fromJson(Map<String, dynamic> json) {
final Map<String, dynamic> rawBars = final Map<String, dynamic> rawBars =
@ -90,7 +137,28 @@ class AlpacaBarsResponse {
AlpacaBar.fromJson(Map<String, dynamic>.from(m))) AlpacaBar.fromJson(Map<String, dynamic>.from(m)))
.toList(); .toList();
} }
return AlpacaBarsResponse(barsBySymbol: parsed); return AlpacaBarsResponse(
barsBySymbol: parsed,
nextPageToken: json['next_page_token'] as String?,
);
}
/// Combines [other] into this response, appending bars per symbol and
/// sorting each symbol's series by timestamp ascending.
AlpacaBarsResponse merge(AlpacaBarsResponse other) {
final Map<String, List<AlpacaBar>> merged =
Map<String, List<AlpacaBar>>.from(barsBySymbol);
for (final MapEntry<String, List<AlpacaBar>> entry
in other.barsBySymbol.entries) {
final List<AlpacaBar> existing =
List<AlpacaBar>.from(merged[entry.key] ?? <AlpacaBar>[]);
existing.addAll(entry.value);
existing.sort(
(AlpacaBar a, AlpacaBar b) => a.timestamp.compareTo(b.timestamp),
);
merged[entry.key] = existing;
}
return AlpacaBarsResponse(barsBySymbol: merged);
} }
AlpacaBar? latestBar(String symbol) { AlpacaBar? latestBar(String symbol) {

View File

@ -20,8 +20,7 @@ class AlpacaTradingClient {
final http.Client _client; final http.Client _client;
Map<String, String> get _headers => <String, String>{ Map<String, String> get _headers => <String, String>{
'APCA-API-KEY-ID': _env.apiKeyId, ..._env.authHeaders,
'APCA-API-SECRET-KEY': _env.apiSecretKey,
'Content-Type': 'application/json', 'Content-Type': 'application/json',
'Accept': 'application/json', 'Accept': 'application/json',
}; };

View File

@ -1,6 +1,7 @@
import 'dart:io'; import 'dart:io';
import 'alpaca/alpaca_env.dart'; import 'alpaca/alpaca_env.dart';
import 'market_history_env.dart';
import 'package:dotenv/dotenv.dart'; import 'package:dotenv/dotenv.dart';
class ServerEnv { class ServerEnv {
@ -15,6 +16,9 @@ class ServerEnv {
required this.tradingWorkerIngestEnabled, required this.tradingWorkerIngestEnabled,
required this.tradingWorkerEvalEnabled, required this.tradingWorkerEvalEnabled,
required this.tradingDevEndpointsEnabled, required this.tradingDevEndpointsEnabled,
required this.adminPortalEnabled,
required this.adminFirebaseUids,
required this.marketHistory,
required this.alpaca, required this.alpaca,
}); });
@ -31,8 +35,22 @@ class ServerEnv {
/// Mounts dev-only endpoints under `/v1/me/trading/dev/*` (e.g. `force-fire`). /// Mounts dev-only endpoints under `/v1/me/trading/dev/*` (e.g. `force-fire`).
/// Default false never enable in production. /// Default false never enable in production.
final bool tradingDevEndpointsEnabled; final bool tradingDevEndpointsEnabled;
final bool adminPortalEnabled;
final Set<String> adminFirebaseUids;
/// Rolling market-history sync, retention, and guess-rule settings.
final MarketHistoryEnv marketHistory;
/// Shorthand for [MarketHistoryEnv.syncEnabled].
bool get marketHistorySyncEnabled => marketHistory.syncEnabled;
final AlpacaEnv alpaca; final AlpacaEnv alpaca;
/// Validates cross-flag constraints after [load].
void assertConsistent() {
marketHistory.assertConsistent(tradingEnabled: tradingEnabled);
}
static ServerEnv load() { static ServerEnv load() {
final DotEnv env = DotEnv(includePlatformEnvironment: true) final DotEnv env = DotEnv(includePlatformEnvironment: true)
..load(['.env']); ..load(['.env']);
@ -80,10 +98,41 @@ class ServerEnv {
final bool tradingDevEndpointsEnabled = final bool tradingDevEndpointsEnabled =
(env['TRADING_DEV_ENDPOINTS_ENABLED'] ?? 'false').toLowerCase() == (env['TRADING_DEV_ENDPOINTS_ENABLED'] ?? 'false').toLowerCase() ==
'true'; 'true';
final bool adminPortalEnabled =
(env['ADMIN_PORTAL_ENABLED'] ?? 'false').toLowerCase() == 'true';
final Set<String> adminFirebaseUids = (env['ADMIN_FIREBASE_UIDS'] ?? '')
.split(',')
.map((String value) => value.trim())
.where((String value) => value.isNotEmpty)
.toSet();
final Map<String, String> stringEnv =
Map<String, String>.from(Platform.environment);
const List<String> marketHistoryKeys = <String>[
'MARKET_HISTORY_SYNC_ENABLED',
'MARKET_HISTORY_WINDOW_DAYS',
'MARKET_HISTORY_RETENTION_DAYS',
'MARKET_HISTORY_ARCHIVE_ENABLED',
'MARKET_UNIVERSE_REFRESH_HOURS',
'MARKET_HISTORY_SYNC_HOURS',
'MARKET_HISTORY_CLEANUP_HOURS',
'MARKET_HISTORY_SYNC_HOUR_UTC',
'HISTORY_SYNC_BATCH_SIZE',
'HISTORY_SYNC_MAX_SYMBOLS',
'MIN_BARS_FOR_GUESS',
'GUESS_COOLDOWN_HOURS',
];
for (final String key in marketHistoryKeys) {
final String? value = env[key];
if (value != null && value.isNotEmpty) {
stringEnv[key] = value;
}
}
final MarketHistoryEnv marketHistory =
MarketHistoryEnv.fromMap(stringEnv);
final AlpacaEnv alpaca = AlpacaEnv.fromMap(envMap)..assertPaperOnly(); final AlpacaEnv alpaca = AlpacaEnv.fromMap(envMap)..assertPaperOnly();
return ServerEnv._( final ServerEnv loaded = ServerEnv._(
databaseUrl: databaseUrl, databaseUrl: databaseUrl,
port: port, port: port,
firebaseWebApiKey: apiKey, firebaseWebApiKey: apiKey,
@ -94,7 +143,12 @@ class ServerEnv {
tradingWorkerIngestEnabled: tradingWorkerIngestEnabled, tradingWorkerIngestEnabled: tradingWorkerIngestEnabled,
tradingWorkerEvalEnabled: tradingWorkerEvalEnabled, tradingWorkerEvalEnabled: tradingWorkerEvalEnabled,
tradingDevEndpointsEnabled: tradingDevEndpointsEnabled, tradingDevEndpointsEnabled: tradingDevEndpointsEnabled,
adminPortalEnabled: adminPortalEnabled,
adminFirebaseUids: adminFirebaseUids,
marketHistory: marketHistory,
alpaca: alpaca, alpaca: alpaca,
); );
loaded.assertConsistent();
return loaded;
} }
} }

View File

@ -0,0 +1,369 @@
import 'dart:convert';
import 'dart:io';
import 'package:postgres/postgres.dart';
import 'package:shelf/shelf.dart';
import 'package:shelf_router/shelf_router.dart';
import '../cors_headers.dart';
import '../firebase_auth.dart';
import '../trading/backfill_sync_item.dart';
import '../trading/market_history_admin_actions.dart';
import '../trading/market_history_admin_logic.dart';
import '../trading/market_history_config.dart';
import '../trading/market_history_question_audit.dart';
import '../trading/market_history_week_coverage.dart';
import '../trading/tradable_assets_db.dart';
const String marketHistoryAdminBasePath = '/v1/admin/market-history';
const String marketHistoryAdminDataBasePath = '/v1/admin/market-data';
class MarketHistoryAdminPortalConfig {
const MarketHistoryAdminPortalConfig({
required this.archiveEnabled,
required this.windowDays,
required this.retentionDays,
required this.syncEnabled,
});
final bool archiveEnabled;
final int windowDays;
final int retentionDays;
final bool syncEnabled;
Map<String, dynamic> toJson() => <String, dynamic>{
'archiveEnabled': archiveEnabled,
'windowDays': windowDays,
'retentionDays': retentionDays,
'syncEnabled': syncEnabled,
};
}
Handler marketHistoryAdminHandler({
required FirebaseAuthVerifier auth,
required Connection connection,
required Set<String> adminFirebaseUids,
MarketHistoryAdminActions? actions,
MarketHistoryAdminPortalConfig? portalConfig,
}) {
final Router router = Router();
router.get('$marketHistoryAdminBasePath/sync-runs', (Request request) async {
final String? firebaseUid = await _verifyAdmin(auth, request, adminFirebaseUids);
if (firebaseUid == null) {
return _authFailure(auth, request, adminFirebaseUids);
}
final Uri uri = request.requestedUri;
final int limit = _parseLimit(uri.queryParameters['limit']);
final DateTime? before = _parseBefore(uri.queryParameters['before']);
final String? kind = _parseKind(uri.queryParameters['kind']);
try {
final StringBuffer sql = StringBuffer(
'''
SELECT id, kind, started_at, finished_at, rows_written, rows_removed,
slots_synced, backfill_items, error
FROM market_data_sync_runs
WHERE 1=1
''',
);
final Map<String, dynamic> params = <String, dynamic>{'limit': limit};
if (kind != null) {
sql.write(' AND kind = @kind');
params['kind'] = kind;
}
if (before != null) {
sql.write(' AND started_at < @before');
params['before'] = before.toUtc();
}
sql.write(' ORDER BY started_at DESC, id DESC LIMIT @limit');
final Result result = await connection.execute(
Sql.named(sql.toString()),
parameters: params,
);
final List<AdminSyncRunRecord> allRuns = result.map(_toRecord).toList();
final List<AdminSyncRunRecord> pinned = computePinned(allRuns);
final Set<int> pinnedIds = pinned.map((AdminSyncRunRecord r) => r.id).toSet();
final List<AdminSyncRunRecord> history = allRuns
.where((AdminSyncRunRecord r) => !pinnedIds.contains(r.id))
.toList();
final DateTime now = DateTime.now().toUtc();
final DateTime? nextBefore =
allRuns.isEmpty ? null : allRuns.last.startedAt.toUtc();
return _jsonResponse(200, <String, dynamic>{
'runs': history.map((AdminSyncRunRecord run) => _toJson(run, now)).toList(),
'pinned': pinned.map((AdminSyncRunRecord run) => _toJson(run, now)).toList(),
'nextBefore': nextBefore?.toIso8601String(),
if (portalConfig != null) 'config': portalConfig.toJson(),
});
} catch (e, st) {
stderr.writeln('market history admin sync-runs failed: $e\n$st');
return _jsonResponse(500, <String, dynamic>{'error': 'Internal error'});
}
});
router.get('$marketHistoryAdminBasePath/question-audit', (Request request) async {
final String? firebaseUid = await _verifyAdmin(auth, request, adminFirebaseUids);
if (firebaseUid == null) {
return _authFailure(auth, request, adminFirebaseUids);
}
try {
final int windowDays =
portalConfig?.windowDays ?? MarketHistoryConfig.windowDays;
final MarketHistoryQuestionAudit audit = MarketHistoryQuestionAudit(
connection: connection,
);
final DateTime now = DateTime.now().toUtc();
final DateTime? compareUntil =
_parseBefore(request.requestedUri.queryParameters['asOf']);
final QuestionAuditPage page = await audit.page(
now: now,
compareUntil: compareUntil,
windowDays: windowDays,
);
return _jsonResponse(200, page.toJson());
} catch (e, st) {
stderr.writeln('market history admin question-audit failed: $e\n$st');
return _jsonResponse(500, <String, dynamic>{'error': 'Internal error'});
}
});
router.get('$marketHistoryAdminBasePath/week-coverage', (Request request) async {
final String? firebaseUid = await _verifyAdmin(auth, request, adminFirebaseUids);
if (firebaseUid == null) {
return _authFailure(auth, request, adminFirebaseUids);
}
try {
final int windowDays =
portalConfig?.windowDays ?? MarketHistoryConfig.windowDays;
final MarketHistoryWeekCoverage coverage = MarketHistoryWeekCoverage(
connection: connection,
tradableAssetsDb: TradableAssetsDb(connection),
windowDays: windowDays,
);
final MarketHistoryWeekCoverageReport report = await coverage.compute();
return _jsonResponse(200, report.toJson());
} catch (e, st) {
stderr.writeln('market history admin week-coverage failed: $e\n$st');
return _jsonResponse(500, <String, dynamic>{'error': 'Internal error'});
}
});
router.post('$marketHistoryAdminDataBasePath/resync', (Request request) async {
final String? firebaseUid = await _verifyAdmin(auth, request, adminFirebaseUids);
if (firebaseUid == null) {
return _authFailure(auth, request, adminFirebaseUids);
}
if (actions == null) {
return _jsonResponse(
503,
<String, dynamic>{
'error': portalConfig?.syncEnabled == false
? 'Market history sync is disabled (MARKET_HISTORY_SYNC_ENABLED=false)'
: 'Market history actions unavailable',
},
);
}
try {
final AdminTriggerResult result = await actions.resync();
return _jsonResponse(202, result.toJson());
} on StateError catch (e) {
return _jsonResponse(409, <String, dynamic>{'error': e.message});
} catch (e, st) {
stderr.writeln('market history admin resync failed: $e\n$st');
return _jsonResponse(500, <String, dynamic>{'error': 'Internal error'});
}
});
router.post('$marketHistoryAdminDataBasePath/cleanup', (Request request) async {
final String? firebaseUid = await _verifyAdmin(auth, request, adminFirebaseUids);
if (firebaseUid == null) {
return _authFailure(auth, request, adminFirebaseUids);
}
if (actions == null) {
return _jsonResponse(
503,
<String, dynamic>{
'error': portalConfig?.syncEnabled == false
? 'Market history sync is disabled (MARKET_HISTORY_SYNC_ENABLED=false)'
: 'Market history actions unavailable',
},
);
}
final Uri uri = request.requestedUri;
final bool archive = _parseBool(uri.queryParameters['archive']);
final int? windowDays = _parseOptionalPositiveInt(uri.queryParameters['windowDays']);
try {
final AdminTriggerResult result = await actions.cleanup(
archive: archive,
windowDays: windowDays,
);
return _jsonResponse(202, result.toJson());
} on StateError catch (e) {
return _jsonResponse(409, <String, dynamic>{'error': e.message});
} catch (e, st) {
stderr.writeln('market history admin cleanup failed: $e\n$st');
return _jsonResponse(500, <String, dynamic>{'error': 'Internal error'});
}
});
return (Request request) async {
if (request.method == 'OPTIONS' && request.requestedUri.path.startsWith('/v1/admin')) {
return Response.ok('', headers: apiCorsHeaders());
}
final Response response = await router.call(request);
if (response.statusCode != 404) {
return response;
}
return _jsonResponse(404, <String, dynamic>{'error': 'Not found'});
};
}
Future<Response> _authFailure(
FirebaseAuthVerifier auth,
Request request,
Set<String> adminFirebaseUids,
) async {
final String? firebaseUid = await _verify(auth, request);
if (firebaseUid == null) {
return _jsonResponse(401, <String, dynamic>{'error': 'Unauthorized'});
}
return _jsonResponse(403, <String, dynamic>{'error': 'Forbidden'});
}
Future<String?> _verifyAdmin(
FirebaseAuthVerifier auth,
Request request,
Set<String> adminFirebaseUids,
) async {
final String? firebaseUid = await _verify(auth, request);
if (firebaseUid == null || !adminFirebaseUids.contains(firebaseUid)) {
return null;
}
return firebaseUid;
}
AdminSyncRunRecord _toRecord(ResultRow row) {
return AdminSyncRunRecord(
id: (row[0]! as num).toInt(),
kind: row[1]! as String,
startedAt: (row[2]! as DateTime).toUtc(),
finishedAt: (row[3] as DateTime?)?.toUtc(),
rowsWritten: (row[4]! as num).toInt(),
rowsRemoved: (row[5]! as num).toInt(),
slotsSynced: (row[6]! as num).toInt(),
backfillItems: BackfillSyncItem.listFromJson(row[7]),
error: row[8] as String?,
);
}
Map<String, dynamic> _toJson(AdminSyncRunRecord run, DateTime now) {
final AdminRunSeverity severity = deriveSeverity(
error: run.error,
startedAt: run.startedAt,
finishedAt: run.finishedAt,
now: now,
rowsWritten: run.rowsWritten,
);
final AdminRunStatus status = deriveStatus(
error: run.error,
finishedAt: run.finishedAt,
rowsWritten: run.rowsWritten,
rowsRemoved: run.rowsRemoved,
);
final String? displayError = effectiveRunError(run);
return <String, dynamic>{
'id': run.id,
'kind': run.kind,
'startedAt': run.startedAt.toIso8601String(),
'finishedAt': run.finishedAt?.toIso8601String(),
'rowsWritten': run.rowsWritten,
'rowsRemoved': run.rowsRemoved,
'slotsSynced': run.slotsSynced,
if (run.backfillItems.isNotEmpty)
'backfillItems':
run.backfillItems.map((BackfillSyncItem item) => item.toJson()).toList(),
'error': displayError,
'severity': severity.wireValue,
'status': status.wireValue,
'durationMs': run.finishedAt == null
? null
: run.finishedAt!.difference(run.startedAt).inMilliseconds,
'summary': toSummary(run),
};
}
int _parseLimit(String? raw) {
final int parsed = int.tryParse(raw ?? '') ?? 50;
if (parsed < 1) {
return 1;
}
if (parsed > 200) {
return 200;
}
return parsed;
}
DateTime? _parseBefore(String? raw) {
if (raw == null || raw.trim().isEmpty) {
return null;
}
return DateTime.tryParse(raw)?.toUtc();
}
String? _parseKind(String? raw) {
if (raw == null || raw.trim().isEmpty) {
return null;
}
const Set<String> allowed = <String>{'universe', 'backfill', 'cleanup'};
final String kind = raw.trim();
if (!allowed.contains(kind)) {
return null;
}
return kind;
}
bool _parseBool(String? raw) {
if (raw == null || raw.trim().isEmpty) {
return false;
}
return raw.toLowerCase() == 'true';
}
int? _parseOptionalPositiveInt(String? raw) {
if (raw == null || raw.trim().isEmpty) {
return null;
}
final int? parsed = int.tryParse(raw);
if (parsed == null || parsed <= 0) {
return null;
}
return parsed;
}
Future<String?> _verify(FirebaseAuthVerifier auth, Request request) {
return auth.verifyBearerToken(
request.headers['Authorization'] ?? request.headers['authorization'],
);
}
Response _jsonResponse(int status, Map<String, dynamic> body) {
return Response(
status,
body: jsonEncode(body),
headers: <String, String>{
...apiCorsHeaders(),
'Content-Type': 'application/json',
},
);
}

View File

@ -0,0 +1,153 @@
/// Market-history pipeline settings loaded from env (§7).
class MarketHistoryEnv {
MarketHistoryEnv({
required this.syncEnabled,
required this.windowDays,
required this.retentionDays,
required this.archiveEnabled,
required this.universeRefreshHours,
required this.historySyncHours,
required this.cleanupHours,
required this.syncHourUtc,
required this.historySyncBatchSize,
required this.historySyncMaxSymbols,
required this.minBarsForGuess,
required this.guessCooldownHours,
required this.apiRequestsPerMinute,
required this.staleSyncRunMinutes,
});
final bool syncEnabled;
final int windowDays;
final int retentionDays;
final bool archiveEnabled;
final int universeRefreshHours;
final int historySyncHours;
final int cleanupHours;
/// UTC hour (023) for optional daily alignment, or `null` when unset.
final int? syncHourUtc;
final int historySyncBatchSize;
final int historySyncMaxSymbols;
final int minBarsForGuess;
final int guessCooldownHours;
final int apiRequestsPerMinute;
final int staleSyncRunMinutes;
static MarketHistoryEnv fromMap(Map<String, String> env) {
final bool syncEnabled =
(env['MARKET_HISTORY_SYNC_ENABLED'] ?? 'false').toLowerCase() == 'true';
final int windowDays =
_positiveInt(env['MARKET_HISTORY_WINDOW_DAYS'], defaultValue: 7, name: 'MARKET_HISTORY_WINDOW_DAYS');
final int retentionDays = _positiveInt(
env['MARKET_HISTORY_RETENTION_DAYS'],
defaultValue: 7,
name: 'MARKET_HISTORY_RETENTION_DAYS',
);
final bool archiveEnabled =
(env['MARKET_HISTORY_ARCHIVE_ENABLED'] ?? 'false').toLowerCase() ==
'true';
final int universeRefreshHours = _positiveInt(
env['MARKET_UNIVERSE_REFRESH_HOURS'],
defaultValue: 24,
name: 'MARKET_UNIVERSE_REFRESH_HOURS',
);
final int historySyncHours = _positiveInt(
env['MARKET_HISTORY_SYNC_HOURS'],
defaultValue: 24,
name: 'MARKET_HISTORY_SYNC_HOURS',
);
final int cleanupHours = _positiveInt(
env['MARKET_HISTORY_CLEANUP_HOURS'],
defaultValue: 24,
name: 'MARKET_HISTORY_CLEANUP_HOURS',
);
final int? syncHourUtc = _optionalSyncHourUtc(env['MARKET_HISTORY_SYNC_HOUR_UTC']);
final int historySyncBatchSize = _positiveInt(
env['HISTORY_SYNC_BATCH_SIZE'],
defaultValue: 50,
name: 'HISTORY_SYNC_BATCH_SIZE',
);
final int historySyncMaxSymbols = _positiveInt(
env['HISTORY_SYNC_MAX_SYMBOLS'],
defaultValue: 2000,
name: 'HISTORY_SYNC_MAX_SYMBOLS',
);
final int minBarsForGuess = _positiveInt(
env['MIN_BARS_FOR_GUESS'],
defaultValue: 5,
name: 'MIN_BARS_FOR_GUESS',
);
final int guessCooldownHours = _positiveInt(
env['GUESS_COOLDOWN_HOURS'],
defaultValue: 24,
name: 'GUESS_COOLDOWN_HOURS',
);
final int apiRequestsPerMinute = _positiveInt(
env['MARKET_HISTORY_API_REQUESTS_PER_MINUTE'],
defaultValue: 200,
name: 'MARKET_HISTORY_API_REQUESTS_PER_MINUTE',
);
final int staleSyncRunMinutes = _positiveInt(
env['MARKET_HISTORY_SYNC_STALE_MINUTES'],
defaultValue: 30,
name: 'MARKET_HISTORY_SYNC_STALE_MINUTES',
);
return MarketHistoryEnv(
syncEnabled: syncEnabled,
windowDays: windowDays,
retentionDays: retentionDays,
archiveEnabled: archiveEnabled,
universeRefreshHours: universeRefreshHours,
historySyncHours: historySyncHours,
cleanupHours: cleanupHours,
syncHourUtc: syncHourUtc,
historySyncBatchSize: historySyncBatchSize,
historySyncMaxSymbols: historySyncMaxSymbols,
minBarsForGuess: minBarsForGuess,
guessCooldownHours: guessCooldownHours,
apiRequestsPerMinute: apiRequestsPerMinute,
staleSyncRunMinutes: staleSyncRunMinutes,
);
}
/// Fails fast when market-history flags conflict with trading gates.
void assertConsistent({required bool tradingEnabled}) {
if (syncEnabled && !tradingEnabled) {
throw StateError(
'MARKET_HISTORY_SYNC_ENABLED=true requires TRADING_ENABLED=true',
);
}
}
static int _positiveInt(
String? raw, {
required int defaultValue,
required String name,
}) {
if (raw == null || raw.trim().isEmpty) {
return defaultValue;
}
final int? parsed = int.tryParse(raw.trim());
if (parsed == null || parsed <= 0) {
throw ArgumentError.value(raw, name, 'must be a positive integer');
}
return parsed;
}
static int? _optionalSyncHourUtc(String? raw) {
if (raw == null || raw.trim().isEmpty) {
return null;
}
final int? parsed = int.tryParse(raw.trim());
if (parsed == null || parsed < 0 || parsed > 23) {
throw ArgumentError.value(
raw,
'MARKET_HISTORY_SYNC_HOUR_UTC',
'must be an integer from 0 to 23',
);
}
return parsed;
}
}

View File

@ -46,6 +46,9 @@ abstract final class TradingPhases {
/// User said +10; order staged in pending_orders for the actuator. /// User said +10; order staged in pending_orders for the actuator.
static const String submitOrder = 'submit_order'; static const String submitOrder = 'submit_order';
/// Guess-the-move question awaiting +10/-10 (scoring only, no order).
static const String awaitAnswer = 'await_answer';
/// Outcome recorded; rule returns to [idle] after cooldown rolls off. /// Outcome recorded; rule returns to [idle] after cooldown rolls off.
static const String done = 'done'; static const String done = 'done';
} }

View File

@ -39,6 +39,7 @@ class QuestionService {
String? sourceTag, String? sourceTag,
String? pipelineKey, String? pipelineKey,
String? pipelineStep, String? pipelineStep,
Map<String, dynamic>? metadata,
}) async { }) async {
final Map<String, dynamic> question = await _questionsDb.createQuestion( final Map<String, dynamic> question = await _questionsDb.createQuestion(
assignedUserId: assignedUserId, assignedUserId: assignedUserId,
@ -47,6 +48,7 @@ class QuestionService {
sourceTag: sourceTag, sourceTag: sourceTag,
pipelineKey: pipelineKey, pipelineKey: pipelineKey,
pipelineStep: pipelineStep, pipelineStep: pipelineStep,
metadata: metadata,
); );
final int unansweredCount = final int unansweredCount =
await _questionsDb.countUnansweredQuestions(assignedUserId); await _questionsDb.countUnansweredQuestions(assignedUserId);

View File

@ -33,7 +33,8 @@ class QuestionsDb {
Sql.named( Sql.named(
''' '''
SELECT id, assigned_user_id, question_text, user_response, correct_answer, SELECT id, assigned_user_id, question_text, user_response, correct_answer,
created_at, modified_at, source_tag, pipeline_key, pipeline_step created_at, modified_at, source_tag, pipeline_key, pipeline_step,
metadata
FROM questions FROM questions
WHERE assigned_user_id = @uid AND user_response IS NULL WHERE assigned_user_id = @uid AND user_response IS NULL
ORDER BY created_at ASC ORDER BY created_at ASC
@ -56,7 +57,8 @@ class QuestionsDb {
Sql.named( Sql.named(
''' '''
SELECT id, assigned_user_id, question_text, user_response, correct_answer, SELECT id, assigned_user_id, question_text, user_response, correct_answer,
created_at, modified_at, source_tag, pipeline_key, pipeline_step created_at, modified_at, source_tag, pipeline_key, pipeline_step,
metadata
FROM questions FROM questions
WHERE assigned_user_id = @uid AND user_response IS NULL WHERE assigned_user_id = @uid AND user_response IS NULL
ORDER BY created_at ASC ORDER BY created_at ASC
@ -83,7 +85,8 @@ class QuestionsDb {
AND assigned_user_id = @uid AND assigned_user_id = @uid
AND user_response IS NULL AND user_response IS NULL
RETURNING id, assigned_user_id, question_text, user_response, correct_answer, RETURNING id, assigned_user_id, question_text, user_response, correct_answer,
created_at, modified_at, source_tag, pipeline_key, pipeline_step created_at, modified_at, source_tag, pipeline_key, pipeline_step,
metadata
''', ''',
), ),
parameters: <String, dynamic>{ parameters: <String, dynamic>{
@ -192,7 +195,7 @@ class QuestionsDb {
AND q.user_response IS NULL AND q.user_response IS NULL
RETURNING q.id, q.assigned_user_id, q.question_text, q.user_response, RETURNING q.id, q.assigned_user_id, q.question_text, q.user_response,
q.correct_answer, q.created_at, q.modified_at, q.correct_answer, q.created_at, q.modified_at,
q.source_tag, q.pipeline_key, q.pipeline_step q.source_tag, q.pipeline_key, q.pipeline_step, q.metadata
''', ''',
), ),
parameters: <String, dynamic>{ parameters: <String, dynamic>{
@ -283,6 +286,7 @@ class QuestionsDb {
String? sourceTag, String? sourceTag,
String? pipelineKey, String? pipelineKey,
String? pipelineStep, String? pipelineStep,
Map<String, dynamic>? metadata,
}) async { }) async {
await ensureUserExists(assignedUserId); await ensureUserExists(assignedUserId);
final String id = _uuid.v4(); final String id = _uuid.v4();
@ -293,10 +297,12 @@ class QuestionsDb {
''' '''
INSERT INTO questions ( INSERT INTO questions (
id, assigned_user_id, question_text, user_response, correct_answer, id, assigned_user_id, question_text, user_response, correct_answer,
created_at, modified_at, source_tag, pipeline_key, pipeline_step created_at, modified_at, source_tag, pipeline_key, pipeline_step,
metadata
) VALUES ( ) VALUES (
@id::uuid, @assigned_user_id, @question_text, NULL, @correct_answer, @id::uuid, @assigned_user_id, @question_text, NULL, @correct_answer,
@created_at, @modified_at, @source_tag, @pipeline_key, @pipeline_step @created_at, @modified_at, @source_tag, @pipeline_key, @pipeline_step,
@metadata::jsonb
) )
''', ''',
), ),
@ -310,6 +316,7 @@ class QuestionsDb {
'source_tag': sourceTag, 'source_tag': sourceTag,
'pipeline_key': pipelineKey, 'pipeline_key': pipelineKey,
'pipeline_step': pipelineStep, 'pipeline_step': pipelineStep,
'metadata': jsonEncode(metadata ?? <String, dynamic>{}),
}, },
); );
@ -324,6 +331,7 @@ class QuestionsDb {
sourceTag: sourceTag, sourceTag: sourceTag,
pipelineKey: pipelineKey, pipelineKey: pipelineKey,
pipelineStep: pipelineStep, pipelineStep: pipelineStep,
metadata: metadata ?? <String, dynamic>{},
); );
} }
@ -357,9 +365,23 @@ class QuestionsDb {
sourceTag: row.length > 7 ? row[7] as String? : null, sourceTag: row.length > 7 ? row[7] as String? : null,
pipelineKey: row.length > 8 ? row[8] as String? : null, pipelineKey: row.length > 8 ? row[8] as String? : null,
pipelineStep: row.length > 9 ? row[9] as String? : null, pipelineStep: row.length > 9 ? row[9] as String? : null,
metadata: _readJsonMap(row.length > 10 ? row[10] : null),
); );
} }
Map<String, dynamic> _readJsonMap(Object? value) {
if (value is Map<String, dynamic>) {
return value;
}
if (value is Map) {
return Map<String, dynamic>.from(value);
}
if (value == null) {
return <String, dynamic>{};
}
return jsonDecode(value.toString()) as Map<String, dynamic>;
}
/// Postgres NUMERIC columns may decode as [String] or [num]. /// Postgres NUMERIC columns may decode as [String] or [num].
static num _readNumeric(Object? value) { static num _readNumeric(Object? value) {
if (value == null) { if (value == null) {
@ -392,6 +414,7 @@ class QuestionsDb {
String? sourceTag, String? sourceTag,
String? pipelineKey, String? pipelineKey,
String? pipelineStep, String? pipelineStep,
Map<String, dynamic>? metadata,
}) { }) {
return <String, dynamic>{ return <String, dynamic>{
'id': id, 'id': id,
@ -404,6 +427,7 @@ class QuestionsDb {
if (sourceTag != null) 'sourceTag': sourceTag, if (sourceTag != null) 'sourceTag': sourceTag,
if (pipelineKey != null) 'pipelineKey': pipelineKey, if (pipelineKey != null) 'pipelineKey': pipelineKey,
if (pipelineStep != null) 'pipelineStep': pipelineStep, if (pipelineStep != null) 'pipelineStep': pipelineStep,
if (metadata != null && metadata.isNotEmpty) 'metadata': metadata,
}; };
} }
} }

View File

@ -0,0 +1,59 @@
import 'market_history_four_hour_slot.dart';
/// One Alpaca backfill request bucket: a UTC 4-hour slot and its symbols.
class BackfillSyncItem {
const BackfillSyncItem({
required this.slotStart,
required this.symbols,
});
final DateTime slotStart;
final List<String> symbols;
Map<String, dynamic> toJson() => <String, dynamic>{
'slotStart': MarketHistoryFourHourSlot.slotStartWire(slotStart),
'symbols': symbols,
};
static BackfillSyncItem? tryFromJson(dynamic raw) {
if (raw is! Map<String, dynamic>) {
return null;
}
final String? slotStartRaw = raw['slotStart'] as String?;
if (slotStartRaw == null) {
return null;
}
final DateTime? slotStart = DateTime.tryParse(slotStartRaw)?.toUtc();
if (slotStart == null) {
return null;
}
final List<dynamic> symbolRaw =
raw['symbols'] as List<dynamic>? ?? <dynamic>[];
final List<String> symbols = symbolRaw
.map((dynamic value) => value.toString())
.where((String value) => value.isNotEmpty)
.toList(growable: false);
return BackfillSyncItem(slotStart: slotStart, symbols: symbols);
}
static List<BackfillSyncItem> listFromJson(dynamic raw) {
if (raw == null) {
return <BackfillSyncItem>[];
}
if (raw is! List<dynamic>) {
return <BackfillSyncItem>[];
}
final List<BackfillSyncItem> items = <BackfillSyncItem>[];
for (final dynamic entry in raw) {
final BackfillSyncItem? item = tryFromJson(entry);
if (item != null) {
items.add(item);
}
}
return items;
}
static List<Map<String, dynamic>> encodeList(List<BackfillSyncItem> items) {
return items.map((BackfillSyncItem item) => item.toJson()).toList();
}
}

View File

@ -2,6 +2,9 @@ import 'dart:convert';
import 'package:postgres/postgres.dart'; import 'package:postgres/postgres.dart';
import 'market_history_bar_placeholder.dart';
import 'market_history_four_hour_slot.dart';
/// Normalized market data row persisted for rule evaluation. /// Normalized market data row persisted for rule evaluation.
class MarketDataSnapshot { class MarketDataSnapshot {
MarketDataSnapshot({ MarketDataSnapshot({
@ -41,6 +44,32 @@ class MarketDataDb {
required DateTime asOf, required DateTime asOf,
String assetClass = 'us_equity', String assetClass = 'us_equity',
String feed = 'iex', String feed = 'iex',
String timeframe = 'tick',
num? price,
num? volume,
Map<String, dynamic>? raw,
}) {
return upsertSnapshot(
symbol: symbol,
metric: metric,
asOf: asOf,
assetClass: assetClass,
feed: feed,
timeframe: timeframe,
price: price,
volume: volume,
raw: raw,
);
}
/// Idempotent write keyed by `(symbol, metric, timeframe, as_of)`.
Future<MarketDataSnapshot> upsertSnapshot({
required String symbol,
required String metric,
required DateTime asOf,
String assetClass = 'us_equity',
String feed = 'iex',
String timeframe = 'tick',
num? price, num? price,
num? volume, num? volume,
Map<String, dynamic>? raw, Map<String, dynamic>? raw,
@ -49,10 +78,15 @@ class MarketDataDb {
Sql.named( Sql.named(
''' '''
INSERT INTO market_data_snapshots ( INSERT INTO market_data_snapshots (
symbol, asset_class, feed, metric, price, volume, as_of, raw symbol, asset_class, feed, metric, timeframe, price, volume, as_of, raw
) VALUES ( ) VALUES (
@symbol, @asset_class, @feed, @metric, @price, @volume, @as_of, @raw::jsonb @symbol, @asset_class, @feed, @metric, @timeframe,
@price, @volume, @as_of, @raw::jsonb
) )
ON CONFLICT (symbol, metric, timeframe, as_of) DO UPDATE SET
price = EXCLUDED.price,
volume = EXCLUDED.volume,
raw = EXCLUDED.raw
RETURNING id, symbol, asset_class, feed, metric, price, volume, as_of, raw, created_at RETURNING id, symbol, asset_class, feed, metric, price, volume, as_of, raw, created_at
''', ''',
), ),
@ -61,6 +95,7 @@ class MarketDataDb {
'asset_class': assetClass, 'asset_class': assetClass,
'feed': feed, 'feed': feed,
'metric': metric, 'metric': metric,
'timeframe': timeframe,
'price': price, 'price': price,
'volume': volume, 'volume': volume,
'as_of': asOf.toUtc(), 'as_of': asOf.toUtc(),
@ -70,6 +105,162 @@ class MarketDataDb {
return _rowToSnapshot(result.first); return _rowToSnapshot(result.first);
} }
/// Tombstone when Alpaca has no 4Hour bar for [symbol] at [slotStart].
///
/// Counts toward backfill gap checks but not game/calendar bar coverage.
Future<MarketDataSnapshot> upsertNoDataBarPlaceholder({
required String symbol,
required DateTime slotStart,
required String timeframe,
required DateTime checkedAt,
String assetClass = 'us_equity',
String feed = 'iex',
String source = MarketHistoryBarPlaceholder.sourceAlpacaEmpty,
}) async {
final DateTime slot =
MarketHistoryFourHourSlot.slotStartContaining(slotStart);
final String slotWire = MarketHistoryFourHourSlot.slotStartWire(slot);
return upsertSnapshot(
symbol: symbol,
metric: 'bar',
timeframe: timeframe,
asOf: slot,
assetClass: assetClass,
feed: feed,
raw: <String, dynamic>{
'slot_start': slotWire,
MarketHistoryBarPlaceholder.rawKey: true,
'source': source,
'checked_at': MarketHistoryFourHourSlot.wireUtc(checkedAt),
},
);
}
/// Daily (or intraday) bars for [symbol] in [`since`, `until`).
Future<List<MarketDataSnapshot>> barsForSymbol({
required String symbol,
required String timeframe,
required DateTime since,
required DateTime until,
String metric = 'bar',
}) async {
final Result result = await _connection.execute(
Sql.named(
'''
SELECT id, symbol, asset_class, feed, metric, price, volume, as_of, raw, created_at
FROM market_data_snapshots
WHERE symbol = @symbol
AND metric = @metric
AND timeframe = @timeframe
AND as_of >= @since
AND as_of < @until
ORDER BY as_of ASC
''',
),
parameters: <String, dynamic>{
'symbol': symbol,
'metric': metric,
'timeframe': timeframe,
'since': since.toUtc(),
'until': until.toUtc(),
},
);
if (result.isEmpty) {
return <MarketDataSnapshot>[];
}
return result.map(_rowToSnapshot).toList(growable: false);
}
/// Newest `as_of` for historical bars, or `null` on cold start.
Future<DateTime?> latestSyncedAsOf(
String symbol,
String timeframe, {
String metric = 'bar',
}) async {
final Result result = await _connection.execute(
Sql.named(
'''
SELECT as_of
FROM market_data_snapshots
WHERE symbol = @symbol
AND metric = @metric
AND timeframe = @timeframe
ORDER BY as_of DESC
LIMIT 1
''',
),
parameters: <String, dynamic>{
'symbol': symbol,
'metric': metric,
'timeframe': timeframe,
},
);
if (result.isEmpty) {
return null;
}
return (result.first[0]! as DateTime).toUtc();
}
/// Symbols from [symbols] that already have a bar for the UTC slot [slotStart].
///
/// A row counts when [raw.slot_start] matches the canonical wire form, when
/// [raw.slot_start] or [as_of] bucket to the same UTC 4-hour boundary as
/// [slotStart] (same rule as [MarketHistoryFourHourSlot.slotStartContaining]).
Future<Set<String>> symbolsWithBarForSlot({
required List<String> symbols,
required DateTime slotStart,
required String timeframe,
String metric = 'bar',
}) async {
if (symbols.isEmpty) {
return <String>{};
}
final DateTime start =
MarketHistoryFourHourSlot.slotStartContaining(slotStart);
final String slotStartWire = MarketHistoryFourHourSlot.slotStartWire(start);
final Result result = await _connection.execute(
Sql.named(
'''
SELECT DISTINCT symbol
FROM market_data_snapshots
WHERE metric = @metric
AND timeframe = @timeframe
AND symbol = ANY(@symbols)
AND (
raw->>'slot_start' = @slot_start_wire
OR (
raw->>'slot_start' IS NOT NULL
AND ${_slotStartBucketSql('(raw->>\'slot_start\')::timestamptz')}
= @slot_start
)
OR ${_slotStartBucketSql('as_of')} = @slot_start
)
''',
),
parameters: <String, dynamic>{
'metric': metric,
'timeframe': timeframe,
'symbols': symbols,
'slot_start_wire': slotStartWire,
'slot_start': start,
},
);
return result
.map((ResultRow row) => row[0]! as String)
.toSet();
}
/// UTC 4-hour slot left edge for [timestampExpr] (timestamptz SQL expression).
static String _slotStartBucketSql(String timestampExpr) {
return '''
(
date_trunc('day', $timestampExpr AT TIME ZONE 'UTC')
+ (div(extract(hour from $timestampExpr AT TIME ZONE 'UTC')::int, 4) * 4)
* interval '1 hour'
) AT TIME ZONE 'UTC'
''';
}
/// Newest snapshot for [symbol] and [metric] by [as_of]. /// Newest snapshot for [symbol] and [metric] by [as_of].
Future<MarketDataSnapshot?> latestForSymbol( Future<MarketDataSnapshot?> latestForSymbol(
String symbol, String symbol,
@ -111,15 +302,15 @@ class MarketDataDb {
assetClass: row[2]! as String, assetClass: row[2]! as String,
feed: row[3]! as String, feed: row[3]! as String,
metric: row[4]! as String, metric: row[4]! as String,
price: _readOptionalNumeric(row[5]), price: readOptionalNumeric(row[5]),
volume: _readOptionalNumeric(row[6]), volume: readOptionalNumeric(row[6]),
asOf: (row[7]! as DateTime).toUtc(), asOf: (row[7]! as DateTime).toUtc(),
raw: raw, raw: raw,
createdAt: (row[9]! as DateTime).toUtc(), createdAt: (row[9]! as DateTime).toUtc(),
); );
} }
static num? _readOptionalNumeric(Object? value) { static num? readOptionalNumeric(Object? value) {
if (value == null) { if (value == null) {
return null; return null;
} }

View File

@ -0,0 +1,518 @@
import 'package:postgres/postgres.dart';
import '../alpaca/alpaca_market_data_client.dart';
import '../alpaca/alpaca_models.dart';
import 'backfill_sync_item.dart';
import 'market_data_db.dart';
import 'market_history_api_rate_limiter.dart';
import 'market_history_bar_placeholder.dart';
import 'market_history_config.dart';
import 'market_history_four_hour_slot.dart';
import 'market_history_trading_calendar.dart';
import 'sync_run_recorder.dart';
import 'tradable_assets_db.dart';
/// Result of persisting one Alpaca bars batch for a single slot.
class PersistBarsResult {
const PersistBarsResult({
required this.written,
required this.symbolsWritten,
required this.notInResponse,
required this.emptyInResponse,
required this.wrongSlotBarTimes,
});
final int written;
final Set<String> symbolsWritten;
final List<String> notInResponse;
final List<String> emptyInResponse;
/// Symbol comma-separated bar [t] values outside the requested slot.
final Map<String, String> wrongSlotBarTimes;
/// Full error when [requested] symbols were fetched but not all persisted
/// and [placeholdersWritten] did not cover the remainder.
String? errorIfIncomplete({
required List<String> requested,
required DateTime slotStart,
required String timeframe,
int placeholdersWritten = 0,
}) {
if (requested.isEmpty) {
return null;
}
final List<String> unpersisted = requested
.where((String symbol) => !symbolsWritten.contains(symbol))
.toList(growable: false);
if (unpersisted.isEmpty) {
return null;
}
if (placeholdersWritten >= unpersisted.length) {
return null;
}
final String slotWire = MarketHistoryFourHourSlot.slotStartWire(slotStart);
final List<String> parts = <String>[
'Alpaca returned no persistable $timeframe bars',
'slot=$slotWire',
'requested=${requested.join(",")}',
'unpersisted=${unpersisted.join(",")}',
];
if (notInResponse.isNotEmpty) {
parts.add('missing_from_response=${notInResponse.join(",")}');
}
if (emptyInResponse.isNotEmpty) {
parts.add('empty_bar_series=${emptyInResponse.join(",")}');
}
if (wrongSlotBarTimes.isNotEmpty) {
parts.add(
'wrong_slot_bars=${wrongSlotBarTimes.entries.map((MapEntry<String, String> e) => '${e.key}:${e.value}').join("|")}',
);
}
if (written == 0) {
parts.add('rows_written=0');
}
return parts.join('; ');
}
}
/// Outcome of one [MarketDataHistorySync.runOnce] invocation.
class MarketDataHistorySyncResult {
MarketDataHistorySyncResult({
required this.rowsWritten,
required this.startedAt,
required this.finishedAt,
this.error,
this.slotsSynced = 0,
});
final int rowsWritten;
final DateTime startedAt;
final DateTime finishedAt;
final String? error;
/// Number of completed 4-hour slots written in this run.
final int slotsSynced;
bool get succeeded => error == null;
}
/// One ended UTC slot and the symbols still missing a bar for that slot.
class MarketHistorySlotFetchPlan {
const MarketHistorySlotFetchPlan({
required this.slotStart,
required this.symbols,
});
final DateTime slotStart;
final List<String> symbols;
}
/// Backfill: one Alpaca `4Hour` request per ended UTC slot × symbol batch.
///
/// Chooses work from the most recently completed slot backward until the rolling
/// window is full. Only symbols missing that specific slot are requested.
///
/// Throttles to [apiRequestsPerMinute]. On HTTP 429, waits [rateLimitCooldown],
/// retries once, then stops the run so partial rows are kept for the next tick.
class MarketDataHistorySync {
MarketDataHistorySync({
required AlpacaMarketDataClient marketDataClient,
required TradableAssetsDb tradableAssetsDb,
required MarketDataDb marketDataDb,
required Connection connection,
this.batchSize = MarketHistoryConfig.historySyncBatchSize,
this.maxSymbols = MarketHistoryConfig.historySyncMaxSymbols,
this.windowDays = MarketHistoryConfig.windowDays,
this.timeframe = MarketHistoryConfig.barTimeframe,
this.feed = 'iex',
int? apiRequestsPerMinute,
MarketHistoryApiRateLimiter? rateLimiter,
Duration rateLimitCooldown = MarketHistoryConfig.rateLimitCooldown,
Future<void> Function(Duration delay)? sleep,
}) : _marketDataClient = marketDataClient,
_tradableAssetsDb = tradableAssetsDb,
_marketDataDb = marketDataDb,
_recorder = SyncRunRecorder(connection),
_rateLimiter = rateLimiter ??
MarketHistoryApiRateLimiter(
requestsPerMinute:
apiRequestsPerMinute ?? MarketHistoryConfig.apiRequestsPerMinute,
),
_rateLimitCooldown = rateLimitCooldown,
_sleep = sleep ?? Future<void>.delayed;
final AlpacaMarketDataClient _marketDataClient;
final TradableAssetsDb _tradableAssetsDb;
final MarketDataDb _marketDataDb;
final SyncRunRecorder _recorder;
final MarketHistoryApiRateLimiter _rateLimiter;
final Duration _rateLimitCooldown;
final Future<void> Function(Duration delay) _sleep;
final int batchSize;
final int maxSymbols;
final int windowDays;
final String timeframe;
final String feed;
static const String kind = 'backfill';
Future<MarketDataHistorySyncResult> runOnce({DateTime? now}) async {
final DateTime tick = (now ?? DateTime.now()).toUtc();
final SyncRunOutcome outcome = await _recorder.record(
kind,
() => _syncBody(tick),
now: tick,
);
return MarketDataHistorySyncResult(
rowsWritten: outcome.rowsWritten,
startedAt: outcome.startedAt,
finishedAt: outcome.finishedAt,
error: outcome.error,
slotsSynced: outcome.slotsSynced ?? 0,
);
}
/// Whether any symbol is missing a completed slot in the rolling window.
Future<bool> hasPendingSlots(DateTime now) async {
final List<String> symbols = await _activeSymbols();
if (symbols.isEmpty) {
return false;
}
final List<MarketHistorySlotFetchPlan> plans =
await _pendingSlotFetchPlans(now, symbols);
return plans.isNotEmpty;
}
Future<SyncRunCounts> _syncBody(DateTime now) async {
List<String> symbols = await _activeSymbols();
if (symbols.isEmpty) {
return const SyncRunCounts();
}
final List<MarketHistorySlotFetchPlan> fetchPlans =
await _pendingSlotFetchPlans(now, symbols);
if (fetchPlans.isEmpty) {
return const SyncRunCounts();
}
int rowsWritten = 0;
int slotsSynced = 0;
final List<String> batchErrors = <String>[];
final Map<DateTime, Set<String>> backfillItemsBySlot = <DateTime, Set<String>>{};
bool stopForRateLimit = false;
for (final MarketHistorySlotFetchPlan plan in fetchPlans) {
if (stopForRateLimit) {
break;
}
final DateTime slotStart = plan.slotStart;
final DateTime slotEnd =
MarketHistoryFourHourSlot.endInclusive(slotStart);
bool slotWrote = false;
for (final List<String> batch in chunkList(plan.symbols, batchSize)) {
if (stopForRateLimit) {
break;
}
final Set<String> alreadySynced =
await _marketDataDb.symbolsWithBarForSlot(
symbols: batch,
slotStart: slotStart,
timeframe: timeframe,
);
final List<String> toFetch = batch
.where((String symbol) => !alreadySynced.contains(symbol))
.toList(growable: false);
if (toFetch.isEmpty) {
continue;
}
backfillItemsBySlot
.putIfAbsent(slotStart, () => <String>{})
.addAll(toFetch);
try {
final AlpacaBarsResponse response = await _fetchBarsWithRateLimitRetry(
symbols: toFetch,
slotStart: slotStart,
slotEnd: slotEnd,
);
final PersistBarsResult persisted = await _persistBars(
response: response,
batch: toFetch,
slotStart: slotStart,
);
rowsWritten += persisted.written;
final int placeholders = await _writeNoDataPlaceholders(
symbols: toFetch,
persisted: persisted,
slotStart: slotStart,
checkedAt: now,
);
rowsWritten += placeholders;
if (persisted.written > 0 || placeholders > 0) {
slotWrote = true;
}
final String? emptyDataError = persisted.errorIfIncomplete(
requested: toFetch,
slotStart: slotStart,
timeframe: timeframe,
placeholdersWritten: placeholders,
);
if (emptyDataError != null &&
!_suppressEmptyMarketError(
slotStart: slotStart,
placeholdersWritten: placeholders,
)) {
batchErrors.add(emptyDataError);
}
} on AlpacaMarketDataException catch (e) {
if (e.isRateLimited) {
batchErrors.add(
'rate limited after ${_rateLimitCooldown.inSeconds}s cooldown; '
'partial sync saved ($rowsWritten rows); resume on next run',
);
stopForRateLimit = true;
break;
}
batchErrors.add(
'slot ${slotStart.toIso8601String()} batch ${toFetch.join(",")}: $e',
);
} on Object catch (e) {
batchErrors.add(
'slot ${slotStart.toIso8601String()} batch ${toFetch.join(",")}: $e',
);
}
}
if (slotWrote) {
slotsSynced++;
}
}
return SyncRunCounts(
rowsWritten: rowsWritten,
error: batchErrors.isEmpty ? null : batchErrors.join('; '),
slotsSynced: slotsSynced,
backfillItems: _backfillItemsFromMap(backfillItemsBySlot),
);
}
static List<BackfillSyncItem> _backfillItemsFromMap(
Map<DateTime, Set<String>> itemsBySlot,
) {
if (itemsBySlot.isEmpty) {
return <BackfillSyncItem>[];
}
final List<DateTime> slotStarts = itemsBySlot.keys.toList()
..sort((DateTime a, DateTime b) => b.compareTo(a));
return slotStarts
.map(
(DateTime slotStart) => BackfillSyncItem(
slotStart: slotStart,
symbols: itemsBySlot[slotStart]!.toList()..sort(),
),
)
.toList(growable: false);
}
Future<AlpacaBarsResponse> _fetchBarsWithRateLimitRetry({
required List<String> symbols,
required DateTime slotStart,
required DateTime slotEnd,
}) async {
for (int attempt = 0; attempt < 2; attempt++) {
await _rateLimiter.acquire();
try {
return await _marketDataClient.getBarsRange(
symbols: symbols,
timeframe: timeframe,
start: slotStart,
end: slotEnd,
);
} on AlpacaMarketDataException catch (e) {
if (!e.isRateLimited || attempt == 1) {
rethrow;
}
await _sleep(_rateLimitCooldown);
}
}
throw StateError('unreachable fetch retry loop');
}
Future<PersistBarsResult> _persistBars({
required AlpacaBarsResponse response,
required List<String> batch,
required DateTime slotStart,
}) async {
int written = 0;
final Set<String> batchSymbols = batch.toSet();
final Set<String> symbolsWritten = <String>{};
final List<String> notInResponse = <String>[];
final List<String> emptyInResponse = <String>[];
final Map<String, String> wrongSlotBarTimes = <String, String>{};
final DateTime plannedSlot =
MarketHistoryFourHourSlot.slotStartContaining(slotStart);
final String slotWire = MarketHistoryFourHourSlot.slotStartWire(plannedSlot);
for (final String symbol in batch) {
if (!response.barsBySymbol.containsKey(symbol)) {
notInResponse.add(symbol);
continue;
}
final List<AlpacaBar> bars = response.barsBySymbol[symbol]!;
if (bars.isEmpty) {
emptyInResponse.add(symbol);
continue;
}
}
for (final MapEntry<String, List<AlpacaBar>> entry
in response.barsBySymbol.entries) {
if (!batchSymbols.contains(entry.key)) {
continue;
}
final List<String> rejectedTimes = <String>[];
for (final AlpacaBar bar in entry.value) {
final DateTime barAt = bar.timestamp.toUtc();
final DateTime barSlot =
MarketHistoryFourHourSlot.slotStartContaining(barAt);
if (!barSlot.isAtSameMomentAs(plannedSlot)) {
rejectedTimes.add(MarketHistoryFourHourSlot.wireUtc(barAt));
continue;
}
await _marketDataDb.upsertSnapshot(
symbol: entry.key,
metric: 'bar',
timeframe: timeframe,
feed: feed,
price: bar.close,
volume: bar.volume,
asOf: barSlot,
raw: <String, dynamic>{
'o': bar.open,
'h': bar.high,
'l': bar.low,
'c': bar.close,
'v': bar.volume,
't': MarketHistoryFourHourSlot.wireUtc(barAt),
'slot_start': slotWire,
},
);
written++;
symbolsWritten.add(entry.key);
}
if (rejectedTimes.isNotEmpty && !symbolsWritten.contains(entry.key)) {
wrongSlotBarTimes[entry.key] = rejectedTimes.join(',');
}
}
return PersistBarsResult(
written: written,
symbolsWritten: symbolsWritten,
notInResponse: notInResponse,
emptyInResponse: emptyInResponse,
wrongSlotBarTimes: wrongSlotBarTimes,
);
}
Future<int> _writeNoDataPlaceholders({
required List<String> symbols,
required PersistBarsResult persisted,
required DateTime slotStart,
required DateTime checkedAt,
}) async {
final List<String> needPlaceholder = symbols
.where((String symbol) => !persisted.symbolsWritten.contains(symbol))
.toList(growable: false);
if (needPlaceholder.isEmpty) {
return 0;
}
for (final String symbol in needPlaceholder) {
await _marketDataDb.upsertNoDataBarPlaceholder(
symbol: symbol,
slotStart: slotStart,
timeframe: timeframe,
feed: feed,
checkedAt: checkedAt,
source: MarketHistoryTradingCalendar.isLikelyNoRegularSession(slotStart)
? MarketHistoryBarPlaceholder.sourceMarketClosed
: MarketHistoryBarPlaceholder.sourceAlpacaEmpty,
);
}
return needPlaceholder.length;
}
/// Suppress error on weekends/holidays when no-data placeholders were stored.
static bool _suppressEmptyMarketError({
required DateTime slotStart,
required int placeholdersWritten,
}) {
return placeholdersWritten > 0 &&
MarketHistoryTradingCalendar.isLikelyNoRegularSession(slotStart);
}
Future<List<String>> _activeSymbols() async {
List<String> symbols = await _tradableAssetsDb.listActiveTradableSymbols();
if (symbols.length > maxSymbols) {
symbols = symbols.sublist(0, maxSymbols);
}
return symbols;
}
/// Completed slots newest-first; each plan lists symbols missing that slot.
Future<List<MarketHistorySlotFetchPlan>> _pendingSlotFetchPlans(
DateTime now,
List<String> symbols,
) async {
final List<DateTime> completed =
MarketHistoryFourHourSlot.completedSlotStartsInWindow(now, windowDays);
if (completed.isEmpty) {
return <MarketHistorySlotFetchPlan>[];
}
final List<MarketHistorySlotFetchPlan> plans = <MarketHistorySlotFetchPlan>[];
for (final DateTime slotStart in completed.reversed) {
final Set<String> synced = await _marketDataDb.symbolsWithBarForSlot(
symbols: symbols,
slotStart: slotStart,
timeframe: timeframe,
);
final List<String> missing = symbols
.where((String symbol) => !synced.contains(symbol))
.toList(growable: false);
if (missing.isNotEmpty) {
plans.add(
MarketHistorySlotFetchPlan(
slotStart: slotStart,
symbols: missing,
),
);
}
}
return plans;
}
}
/// Splits [items] into consecutive chunks of at most [size].
List<List<T>> chunkList<T>(List<T> items, int size) {
if (size <= 0) {
throw ArgumentError.value(size, 'size', 'must be positive');
}
if (items.isEmpty) {
return <List<T>>[];
}
final List<List<T>> chunks = <List<T>>[];
for (int i = 0; i < items.length; i += size) {
final int end = i + size > items.length ? items.length : i + size;
chunks.add(items.sublist(i, end));
}
return chunks;
}

View File

@ -92,11 +92,12 @@ class MarketDataIngest {
final AlpacaLatestTradeResponse latest = final AlpacaLatestTradeResponse latest =
await _alpacaClient.getLatestTrade(symbol); await _alpacaClient.getLatestTrade(symbol);
_httpRequests++; _httpRequests++;
await _marketDataDb.insertSnapshot( await _marketDataDb.upsertSnapshot(
symbol: symbol, symbol: symbol,
assetClass: input.assetClass, assetClass: input.assetClass,
feed: input.feed, feed: input.feed,
metric: 'last_trade', metric: 'last_trade',
timeframe: 'tick',
price: latest.trade.price, price: latest.trade.price,
volume: latest.trade.size, volume: latest.trade.size,
asOf: latest.trade.timestamp, asOf: latest.trade.timestamp,
@ -115,11 +116,12 @@ class MarketDataIngest {
if (input.metrics.contains('daily_bar')) { if (input.metrics.contains('daily_bar')) {
final AlpacaBar? bar = barsResponse.latestBar(symbol); final AlpacaBar? bar = barsResponse.latestBar(symbol);
if (bar != null) { if (bar != null) {
await _marketDataDb.insertSnapshot( await _marketDataDb.upsertSnapshot(
symbol: symbol, symbol: symbol,
assetClass: input.assetClass, assetClass: input.assetClass,
feed: input.feed, feed: input.feed,
metric: 'daily_bar', metric: 'daily_bar',
timeframe: 'tick',
price: bar.close, price: bar.close,
volume: bar.volume, volume: bar.volume,
asOf: bar.timestamp, asOf: bar.timestamp,
@ -135,11 +137,12 @@ class MarketDataIngest {
if (input.metrics.contains('prev_close')) { if (input.metrics.contains('prev_close')) {
final AlpacaBar? prev = barsResponse.previousDailyBar(symbol); final AlpacaBar? prev = barsResponse.previousDailyBar(symbol);
if (prev != null) { if (prev != null) {
await _marketDataDb.insertSnapshot( await _marketDataDb.upsertSnapshot(
symbol: symbol, symbol: symbol,
assetClass: input.assetClass, assetClass: input.assetClass,
feed: input.feed, feed: input.feed,
metric: 'prev_close', metric: 'prev_close',
timeframe: 'tick',
price: prev.close, price: prev.close,
volume: prev.volume, volume: prev.volume,
asOf: prev.timestamp, asOf: prev.timestamp,

View File

@ -0,0 +1,166 @@
import 'package:postgres/postgres.dart';
import 'market_history_config.dart';
import 'sync_run_recorder.dart';
/// Outcome of a [MarketDataRetention] cleanup pass.
class MarketDataRetentionResult {
MarketDataRetentionResult({
required this.rowsRemoved,
required this.startedAt,
required this.finishedAt,
this.error,
});
final int rowsRemoved;
final DateTime startedAt;
final DateTime finishedAt;
final String? error;
bool get succeeded => error == null;
}
/// Prunes [market_data_snapshots] older than the rolling window.
///
/// Phase 1 ([runCleanup]): hard-delete in batches.
/// Phase 2 ([runArchiveAndCleanup]): copy expired rows into
/// [market_data_archive] inside the same transaction, then delete.
class MarketDataRetention {
MarketDataRetention({
required Connection connection,
this.windowDays = MarketHistoryConfig.windowDays,
this.batchSize = MarketHistoryConfig.retentionBatchSize,
void Function(String sql)? onExecute,
}) : _connection = connection,
_recorder = SyncRunRecorder(connection),
_onExecute = onExecute;
final Connection _connection;
final SyncRunRecorder _recorder;
final void Function(String sql)? _onExecute;
final int windowDays;
final int batchSize;
static const String kind = 'cleanup';
/// Hard-delete rows with `as_of` older than [windowDays].
Future<MarketDataRetentionResult> runCleanup({DateTime? now}) {
return run(archive: false, now: now);
}
/// Archive-then-delete for rows older than [windowDays].
Future<MarketDataRetentionResult> runArchiveAndCleanup({DateTime? now}) {
return run(archive: true, now: now);
}
/// Dispatches to hard-delete or archive mode.
Future<MarketDataRetentionResult> run({
bool archive = false,
DateTime? now,
int? windowDays,
}) async {
final DateTime tick = (now ?? DateTime.now()).toUtc();
final int effectiveWindow = windowDays ?? this.windowDays;
final SyncRunOutcome outcome = await _recorder.record(
kind,
() => _cleanupBody(
now: tick,
windowDays: effectiveWindow,
archive: archive,
),
now: tick,
);
return MarketDataRetentionResult(
rowsRemoved: outcome.rowsRemoved,
startedAt: outcome.startedAt,
finishedAt: outcome.finishedAt,
error: outcome.error,
);
}
Future<SyncRunCounts> _cleanupBody({
required DateTime now,
required int windowDays,
required bool archive,
}) async {
final DateTime cutoff = now.subtract(Duration(days: windowDays));
int totalRemoved = 0;
while (true) {
final int removed = archive
? await _archiveAndDeleteBatch(cutoff)
: await _deleteBatch(cutoff);
if (removed == 0) {
break;
}
totalRemoved += removed;
}
return SyncRunCounts(rowsRemoved: totalRemoved);
}
Future<int> _deleteBatch(DateTime cutoff) async {
const String sql = '''
WITH doomed AS (
SELECT id
FROM market_data_snapshots
WHERE as_of < @cutoff
LIMIT @batch_size
)
DELETE FROM market_data_snapshots
WHERE id IN (SELECT id FROM doomed)
RETURNING id
''';
_onExecute?.call(sql);
final Result result = await _connection.execute(
Sql.named(sql),
parameters: <String, dynamic>{
'cutoff': cutoff,
'batch_size': batchSize,
},
);
return result.length;
}
Future<int> _archiveAndDeleteBatch(DateTime cutoff) async {
const String sql = '''
WITH doomed AS (
SELECT id, symbol, asset_class, feed, metric, timeframe,
price, volume, as_of, raw
FROM market_data_snapshots
WHERE as_of < @cutoff
LIMIT @batch_size
),
inserted AS (
INSERT INTO market_data_archive (
symbol, asset_class, feed, metric, timeframe,
price, volume, as_of, raw, archived_at
)
SELECT symbol, asset_class, feed, metric, timeframe,
price, volume, as_of, raw, now()
FROM doomed
RETURNING id
)
DELETE FROM market_data_snapshots m
USING doomed d
WHERE m.id = d.id
RETURNING m.id
''';
int removed = 0;
await _connection.runTx((TxSession tx) async {
_onExecute?.call(sql);
final Result archived = await tx.execute(
Sql.named(sql),
parameters: <String, dynamic>{
'cutoff': cutoff,
'batch_size': batchSize,
},
);
removed = archived.length;
});
return removed;
}
}

View File

@ -0,0 +1,107 @@
import 'package:postgres/postgres.dart';
import 'sync_run_recorder.dart';
typedef AdminRunStage = Future<void> Function(DateTime now);
class AdminTriggerResult {
AdminTriggerResult({required this.runIds});
final List<int> runIds;
Map<String, dynamic> toJson() => <String, dynamic>{'runIds': runIds};
}
/// On-demand market-history operations for the admin portal.
class MarketHistoryAdminActions {
MarketHistoryAdminActions({
required Connection connection,
required AdminRunStage runUniverse,
required AdminRunStage runBackfill,
required Future<void> Function(DateTime now, bool archive, int windowDays)
runCleanup,
this.defaultArchiveEnabled = false,
this.defaultWindowDays = 7,
}) : _connection = connection,
_recorder = SyncRunRecorder(connection),
_runUniverse = runUniverse,
_runBackfill = runBackfill,
_runCleanup = runCleanup;
final Connection _connection;
final SyncRunRecorder _recorder;
final AdminRunStage _runUniverse;
final AdminRunStage _runBackfill;
final Future<void> Function(DateTime now, bool archive, int windowDays)
_runCleanup;
final bool defaultArchiveEnabled;
final int defaultWindowDays;
Future<bool> hasInProgressRun() async {
final Result rows = await _connection.execute(
'''
SELECT 1
FROM market_data_sync_runs
WHERE finished_at IS NULL
LIMIT 1
''',
);
return rows.isNotEmpty;
}
Future<void> _abortPreviousSyncRuns(DateTime tick) async {
await _recorder.abortAllInProgressRuns(
now: tick,
message: 'aborted: superseded by admin trigger',
);
}
Future<AdminTriggerResult> resync({DateTime? now}) async {
final DateTime tick = (now ?? DateTime.now()).toUtc();
await _abortPreviousSyncRuns(tick);
final int beforeMaxId = await _maxRunId();
await _runUniverse(tick);
await _runBackfill(tick);
final List<int> runIds = await _runIdsAfter(beforeMaxId);
return AdminTriggerResult(runIds: runIds);
}
Future<AdminTriggerResult> cleanup({
DateTime? now,
bool? archive,
int? windowDays,
}) async {
final DateTime tick = (now ?? DateTime.now()).toUtc();
await _abortPreviousSyncRuns(tick);
final int beforeMaxId = await _maxRunId();
await _runCleanup(
tick,
archive ?? defaultArchiveEnabled,
windowDays ?? defaultWindowDays,
);
final List<int> runIds = await _runIdsAfter(beforeMaxId);
return AdminTriggerResult(runIds: runIds);
}
Future<int> _maxRunId() async {
final Result rows = await _connection.execute(
'SELECT COALESCE(MAX(id), 0) FROM market_data_sync_runs',
);
return (rows.first[0]! as num).toInt();
}
Future<List<int>> _runIdsAfter(int afterId) async {
final Result rows = await _connection.execute(
Sql.named(
'''
SELECT id
FROM market_data_sync_runs
WHERE id > @after_id
ORDER BY id ASC
''',
),
parameters: <String, dynamic>{'after_id': afterId},
);
return rows.map((ResultRow row) => (row[0]! as num).toInt()).toList();
}
}

View File

@ -0,0 +1,180 @@
import 'backfill_sync_item.dart';
/// Errors that only mean Alpaca had no bars but placeholders were stored.
bool isBenignEmptyMarketError(String? error, {required int rowsWritten}) {
if (error == null || error.trim().isEmpty || rowsWritten <= 0) {
return false;
}
final String normalized = error.toLowerCase();
if (normalized.contains('rate limited') || normalized.contains('429')) {
return false;
}
return normalized.contains('alpaca returned no persistable');
}
String? effectiveRunError(AdminSyncRunRecord run) {
if (isBenignEmptyMarketError(run.error, rowsWritten: run.rowsWritten)) {
return null;
}
return run.error;
}
enum AdminRunSeverity {
ok('ok'),
warning('warning'),
error('error'),
rateLimit('rate_limit');
const AdminRunSeverity(this.wireValue);
final String wireValue;
}
enum AdminRunStatus {
success('success'),
failed('failed'),
partial('partial'),
inProgress('in_progress');
const AdminRunStatus(this.wireValue);
final String wireValue;
}
class AdminSyncRunRecord {
const AdminSyncRunRecord({
required this.id,
required this.kind,
required this.startedAt,
required this.finishedAt,
required this.rowsWritten,
required this.rowsRemoved,
required this.slotsSynced,
required this.backfillItems,
required this.error,
});
final int id;
final String kind;
final DateTime startedAt;
final DateTime? finishedAt;
final int rowsWritten;
final int rowsRemoved;
final int slotsSynced;
final List<BackfillSyncItem> backfillItems;
final String? error;
}
AdminRunSeverity deriveSeverity({
required String? error,
required DateTime startedAt,
required DateTime? finishedAt,
required DateTime now,
Duration staleThreshold = const Duration(minutes: 30),
int rowsWritten = 0,
}) {
final String? effectiveError = error;
if (effectiveError != null && effectiveError.trim().isNotEmpty) {
if (isBenignEmptyMarketError(effectiveError, rowsWritten: rowsWritten)) {
return AdminRunSeverity.ok;
}
final String normalized = effectiveError.toLowerCase();
final bool looksRateLimited =
normalized.contains('429') ||
normalized.contains('rate limit') ||
normalized.contains('rate limited');
final bool looksAlpacaError =
normalized.contains('alpacamarketdataexception') ||
normalized.contains('alpacaassetsexception') ||
normalized.contains('alpacatradingexception');
final bool looks5xx = RegExp(r'\b5\d\d\b').hasMatch(normalized);
if (looksRateLimited || looksAlpacaError || looks5xx) {
return AdminRunSeverity.rateLimit;
}
return AdminRunSeverity.error;
}
if (finishedAt == null) {
final bool stale = now.toUtc().difference(startedAt.toUtc()) >= staleThreshold;
return stale ? AdminRunSeverity.warning : AdminRunSeverity.warning;
}
return AdminRunSeverity.ok;
}
AdminRunStatus deriveStatus({
required String? error,
required DateTime? finishedAt,
required int rowsWritten,
required int rowsRemoved,
}) {
if (finishedAt == null) {
return AdminRunStatus.inProgress;
}
if (isBenignEmptyMarketError(error, rowsWritten: rowsWritten)) {
return AdminRunStatus.success;
}
if (error != null && error.trim().isNotEmpty) {
if (rowsWritten > 0) {
return AdminRunStatus.partial;
}
return AdminRunStatus.failed;
}
return AdminRunStatus.success;
}
List<AdminSyncRunRecord> sortNewestFirst(Iterable<AdminSyncRunRecord> runs) {
final List<AdminSyncRunRecord> sorted = List<AdminSyncRunRecord>.from(runs);
sorted.sort(
(AdminSyncRunRecord a, AdminSyncRunRecord b) =>
b.startedAt.compareTo(a.startedAt),
);
return sorted;
}
List<AdminSyncRunRecord> computePinned(Iterable<AdminSyncRunRecord> runs) {
final List<AdminSyncRunRecord> sorted = sortNewestFirst(runs);
final Set<String> resolvedKinds = <String>{};
final List<AdminSyncRunRecord> pinned = <AdminSyncRunRecord>[];
for (final AdminSyncRunRecord run in sorted) {
final String? error = effectiveRunError(run);
final bool unresolved =
(error != null && error.trim().isNotEmpty) || run.finishedAt == null;
if (!unresolved) {
resolvedKinds.add(run.kind);
continue;
}
if (!resolvedKinds.contains(run.kind)) {
pinned.add(run);
}
}
return pinned;
}
String toSummary(AdminSyncRunRecord run) {
final String? error = effectiveRunError(run);
switch (run.kind) {
case 'universe':
return '${run.rowsWritten} assets refreshed';
case 'backfill':
final String slotNote =
run.slotsSynced > 0 ? '${run.slotsSynced} slots, ' : '';
if (error != null && run.rowsWritten > 0) {
return 'Backfill partial: $slotNote${run.rowsWritten} rows written';
}
if (error != null) {
return 'Backfill failed';
}
if (isBenignEmptyMarketError(run.error, rowsWritten: run.rowsWritten)) {
return '$slotNote${run.rowsWritten} no-data placeholders stored';
}
if (run.slotsSynced > 0) {
return '$slotNote${run.rowsWritten} bar rows written';
}
return '${run.rowsWritten} bar rows written';
case 'cleanup':
return '${run.rowsRemoved} rows removed';
default:
return error == null ? 'Run completed' : 'Run failed';
}
}

View File

@ -0,0 +1,51 @@
/// Throttles Alpaca market-data HTTP calls to a max count per rolling minute.
class MarketHistoryApiRateLimiter {
MarketHistoryApiRateLimiter({
required int requestsPerMinute,
DateTime Function()? clock,
Future<void> Function(Duration delay)? sleep,
}) : _requestsPerMinute = requestsPerMinute,
_clock = clock ?? DateTime.now,
_sleep = sleep ?? Future<void>.delayed;
final int _requestsPerMinute;
final DateTime Function() _clock;
final Future<void> Function(Duration delay) _sleep;
final List<DateTime> _requestTimes = <DateTime>[];
/// Blocks until another request is allowed under [requestsPerMinute].
Future<void> acquire() async {
if (_requestsPerMinute <= 0) {
throw ArgumentError.value(
_requestsPerMinute,
'requestsPerMinute',
'must be positive',
);
}
while (true) {
final DateTime now = _clock().toUtc();
_dropOlderThan(now.subtract(const Duration(minutes: 1)));
if (_requestTimes.length < _requestsPerMinute) {
_requestTimes.add(now);
return;
}
final DateTime oldest = _requestTimes.first;
final Duration wait = oldest
.add(const Duration(minutes: 1))
.difference(now);
if (wait > Duration.zero) {
await _sleep(wait);
}
}
}
void _dropOlderThan(DateTime cutoff) {
while (_requestTimes.isNotEmpty && !_requestTimes.first.isAfter(cutoff)) {
_requestTimes.removeAt(0);
}
}
}

View File

@ -0,0 +1,13 @@
/// Marker rows written when Alpaca has no 4Hour bar for a symbol × slot.
abstract final class MarketHistoryBarPlaceholder {
static const String rawKey = 'no_data';
static const String sourceAlpacaEmpty = 'alpaca_empty';
static const String sourceMarketClosed = 'market_closed';
static bool isPlaceholder(Map<String, dynamic>? raw) =>
raw?[rawKey] == true;
/// SQL predicate: row is a real bar, not a no-data tombstone.
static const String sqlExcludePlaceholders =
"(raw->>'$rawKey' IS NULL OR raw->>'$rawKey' <> 'true')";
}

View File

@ -0,0 +1,34 @@
/// Defaults for 4-hour slot market history ([MarketHistoryFourHourSlot]).
/// Env overrides via [MarketHistoryEnv] in [ServerEnv.load].
abstract final class MarketHistoryConfig {
/// Rolling window length in calendar days (UTC).
static const int windowDays = 7;
/// Alpaca bar aggregation for market-history backfill (six slots per UTC day).
static const String barTimeframe = '4Hour';
/// Width of each history slot in hours (`24 / slotsPerDay`).
static const int slotHours = 4;
/// Symbols per Alpaca `GET /v2/stocks/bars` request (max ~100).
static const int historySyncBatchSize = 100;
/// Hard cap on symbols synced per run (Alpaca Basic rate-limit safety).
static const int historySyncMaxSymbols = 2000;
/// Minimum 4-hour bars required before a symbol is eligible for the
/// guess-the-move question rule.
static const int minBarsForGuess = 5;
/// Hours before the same symbol can fire another guess question.
static const int guessCooldownHours = 24;
/// Rows deleted per cleanup loop iteration (Postgres batched DELETE).
static const int retentionBatchSize = 5000;
/// Max Alpaca HTTP calls per rolling minute during history backfill.
static const int apiRequestsPerMinute = 200;
/// Wait after HTTP 429 before retrying the same bars request.
static const Duration rateLimitCooldown = Duration(minutes: 1);
}

View File

@ -0,0 +1,89 @@
/// Six UTC 4-hour slots per day. Sync only [completedSlotStartsInWindow].
abstract final class MarketHistoryFourHourSlot {
static const int slotHours = 4;
static const int slotsPerDay = 24 ~/ slotHours;
static const String alpacaTimeframe = '4Hour';
/// Left edge of the 4-hour bucket containing [instant] (UTC).
static DateTime slotStartContaining(DateTime instant) {
final DateTime u = instant.toUtc();
final int slotHour = (u.hour ~/ slotHours) * slotHours;
return DateTime.utc(u.year, u.month, u.day, slotHour);
}
/// Inclusive end of the slot for Alpaca `start`/`end` (Option A: 00:0003:59:59).
static DateTime endInclusive(DateTime slotStart) {
return slotStart
.add(const Duration(hours: slotHours))
.subtract(const Duration(seconds: 1));
}
/// Exclusive end (current slot begins here).
static DateTime endExclusive(DateTime slotStart) {
return slotStart.add(const Duration(hours: slotHours));
}
/// `true` when [now] is at or after the end of the slot that began at [slotStart].
static bool hasEnded(DateTime slotStart, DateTime now) {
return !now.toUtc().isBefore(endExclusive(slotStart));
}
/// Start of the most recently completed slot (never the in-progress slot).
static DateTime lastCompletedSlotStart(DateTime now) {
final DateTime current = slotStartContaining(now);
return current.subtract(const Duration(hours: slotHours));
}
/// Earliest slot start included in a [windowDays] rolling window ending at [now].
static DateTime windowFirstSlotStart(DateTime now, int windowDays) {
final DateTime windowStart =
now.toUtc().subtract(Duration(days: windowDays));
return slotStartContaining(windowStart);
}
/// Completed slot starts from the rolling window through [lastCompletedSlotStart].
static List<DateTime> completedSlotStartsInWindow(
DateTime now,
int windowDays,
) {
final DateTime last = lastCompletedSlotStart(now);
final DateTime first = windowFirstSlotStart(now, windowDays);
if (last.isBefore(first)) {
return <DateTime>[];
}
final List<DateTime> slots = <DateTime>[];
DateTime cursor = first;
while (!cursor.isAfter(last)) {
if (hasEnded(cursor, now)) {
slots.add(cursor);
}
cursor = cursor.add(const Duration(hours: slotHours));
}
return slots;
}
/// Canonical UTC wire form: `YYYY-MM-DDTHH:MM:SSZ` (no fractional seconds).
///
/// Used for Alpaca bar-range query params and [raw.slot_start] / [raw.t] in
/// [market_data_snapshots] so fetch, persist, and gap checks always agree.
static String wireUtc(DateTime value) {
final DateTime u = value.toUtc();
String two(int n) => n.toString().padLeft(2, '0');
return '${u.year.toString().padLeft(4, '0')}-'
'${two(u.month)}-${two(u.day)}T'
'${two(u.hour)}:${two(u.minute)}:${two(u.second)}Z';
}
/// Wire form for the left edge of the 4-hour slot containing [slotStart].
static String slotStartWire(DateTime slotStart) =>
wireUtc(slotStartContaining(slotStart));
/// Parses a [wireUtc] / Alpaca RFC3339 timestamp, or `null` when invalid.
static DateTime? parseWire(String? wire) {
if (wire == null || wire.isEmpty) {
return null;
}
return DateTime.tryParse(wire)?.toUtc();
}
}

View File

@ -0,0 +1,138 @@
import 'dart:math';
import 'package:postgres/postgres.dart';
import 'market_data_db.dart' show MarketDataDb;
import 'market_history_bar_placeholder.dart';
import 'market_history_config.dart';
import 'tradable_assets_db.dart';
/// One symbol's rolling-window 4-hour bar summary for the guessing game.
class WeeklyMover {
WeeklyMover({
required this.symbol,
required this.openClose,
required this.currentClose,
required this.days,
});
final String symbol;
final num openClose;
final num currentClose;
/// Number of 4-hour bars in the window ( [MarketHistoryConfig.minBarsForGuess]).
final int days;
}
/// Read-side queries over 4-hour history bars for question rules.
class MarketHistoryQuery {
MarketHistoryQuery({
required Connection connection,
TradableAssetsDb? tradableAssetsDb,
}) : _connection = connection,
_tradableAssetsDb = tradableAssetsDb ?? TradableAssetsDb(connection);
final Connection _connection;
final TradableAssetsDb _tradableAssetsDb;
/// Symbols with [minBars] 4-hour closes in [`asOf` window, `asOf`), whose
/// newest bar is not older than [maxStalenessDays] before [asOf].
///
/// When [random] is set, eligible rows are sorted by symbol then shuffled for
/// a stable pick order across runs with the same seed.
Future<List<WeeklyMover>> weeklyMovers({
required DateTime asOf,
int minBars = MarketHistoryConfig.minBarsForGuess,
int windowDays = MarketHistoryConfig.windowDays,
int maxStalenessDays = 2,
Random? random,
}) async {
final DateTime until = asOf.toUtc();
final DateTime since =
until.subtract(Duration(days: windowDays));
final DateTime freshSince =
until.subtract(Duration(days: maxStalenessDays));
final List<String> active =
await _tradableAssetsDb.listActiveTradableSymbols();
if (active.isEmpty) {
return <WeeklyMover>[];
}
final Result rows = await _connection.execute(
Sql.named(
'''
WITH bars AS (
SELECT symbol, as_of, price
FROM market_data_snapshots
WHERE metric = 'bar'
AND timeframe = @timeframe
AND as_of >= @since
AND as_of < @until
AND symbol = ANY(@symbols)
AND ${MarketHistoryBarPlaceholder.sqlExcludePlaceholders}
AND price IS NOT NULL
),
agg AS (
SELECT
symbol,
COUNT(*)::int AS bar_count,
MIN(as_of) AS oldest_as_of,
MAX(as_of) AS newest_as_of
FROM bars
GROUP BY symbol
HAVING COUNT(*) >= @min_bars
AND MAX(as_of) >= @fresh_since
)
SELECT
a.symbol,
a.bar_count,
(
SELECT b.price
FROM bars b
WHERE b.symbol = a.symbol AND b.as_of = a.oldest_as_of
LIMIT 1
) AS open_close,
(
SELECT b.price
FROM bars b
WHERE b.symbol = a.symbol AND b.as_of = a.newest_as_of
LIMIT 1
) AS current_close
FROM agg a
ORDER BY a.symbol ASC
''',
),
parameters: <String, dynamic>{
'since': since,
'until': until,
'symbols': active,
'timeframe': MarketHistoryConfig.barTimeframe,
'min_bars': minBars,
'fresh_since': freshSince,
},
);
final List<WeeklyMover> movers = <WeeklyMover>[];
for (final ResultRow row in rows) {
final num? openClose = MarketDataDb.readOptionalNumeric(row[2]);
final num? currentClose = MarketDataDb.readOptionalNumeric(row[3]);
if (openClose == null || currentClose == null) {
continue;
}
movers.add(
WeeklyMover(
symbol: row[0]! as String,
openClose: openClose,
currentClose: currentClose,
days: (row[1]! as num).toInt(),
),
);
}
if (random != null && movers.length > 1) {
movers.shuffle(random);
}
return movers;
}
}

View File

@ -0,0 +1,457 @@
import 'dart:convert';
import 'package:postgres/postgres.dart';
import 'market_data_db.dart' show MarketDataDb;
import 'market_history_bar_placeholder.dart';
import 'market_history_config.dart';
import 'market_history_four_hour_slot.dart';
import 'tradable_assets_db.dart';
/// One 4-hour bar snapshot used in question audit comparisons.
class QuestionAuditBarSlot {
QuestionAuditBarSlot({
required this.asOf,
required this.avgPrice,
required this.volume,
this.open,
this.high,
this.low,
this.close,
});
final DateTime asOf;
final num? open;
final num? high;
final num? low;
final num? close;
final num avgPrice;
final num volume;
Map<String, dynamic> toJson() => <String, dynamic>{
'asOf': asOf.toIso8601String(),
if (open != null) 'open': open,
if (high != null) 'high': high,
if (low != null) 'low': low,
if (close != null) 'close': close,
'avgPrice': avgPrice,
'volume': volume,
};
}
/// One tradable symbol's last-two 4-hour bar deltas for question auditing.
class QuestionAuditAsset {
QuestionAuditAsset({
required this.symbol,
required this.priceDelta,
required this.volumeDelta,
required this.olderSlot,
required this.newerSlot,
});
final String symbol;
final num priceDelta;
final num volumeDelta;
final QuestionAuditBarSlot olderSlot;
final QuestionAuditBarSlot newerSlot;
Map<String, dynamic> toJson() => <String, dynamic>{
'symbol': symbol,
'priceDelta': priceDelta,
'volumeDelta': volumeDelta,
'olderSlot': olderSlot.toJson(),
'newerSlot': newerSlot.toJson(),
};
}
/// OHLC average for a bar; falls back to [closePrice] when [raw] lacks legs.
num averageBarPrice({required num? closePrice, Map<String, dynamic>? raw}) {
if (raw != null) {
final num? open = MarketDataDb.readOptionalNumeric(raw['o']);
final num? high = MarketDataDb.readOptionalNumeric(raw['h']);
final num? low = MarketDataDb.readOptionalNumeric(raw['l']);
final num? close = MarketDataDb.readOptionalNumeric(raw['c']);
if (open != null && high != null && low != null && close != null) {
return (open + high + low + close) / 4;
}
}
if (closePrice != null) {
return closePrice;
}
throw ArgumentError('Bar has no OHLC or close price');
}
QuestionAuditBarSlot barRowToSlot(_BarRow row) {
final Map<String, dynamic>? raw = row.raw;
final num? open = raw == null ? null : MarketDataDb.readOptionalNumeric(raw['o']);
final num? high = raw == null ? null : MarketDataDb.readOptionalNumeric(raw['h']);
final num? low = raw == null ? null : MarketDataDb.readOptionalNumeric(raw['l']);
final num? close =
raw == null ? row.closePrice : MarketDataDb.readOptionalNumeric(raw['c']);
return QuestionAuditBarSlot(
asOf: row.asOf,
open: open,
high: high,
low: low,
close: close ?? row.closePrice,
avgPrice: averageBarPrice(closePrice: row.closePrice, raw: raw),
volume: row.volume!,
);
}
/// Paginated question-audit payload for one [compareUntil] exclusive bound.
class QuestionAuditPage {
QuestionAuditPage({
required this.compareUntil,
required this.newerSlotStart,
required this.olderSlotStart,
required this.windowDays,
required this.assets,
required this.canStepOlder,
required this.canStepNewer,
this.stepOlderCompareUntil,
this.stepNewerCompareUntil,
});
/// Exclusive upper bound for this page (`endExclusive(newerSlotStart)`).
final DateTime compareUntil;
final DateTime newerSlotStart;
final DateTime olderSlotStart;
final int windowDays;
final List<QuestionAuditAsset> assets;
final bool canStepOlder;
final bool canStepNewer;
final DateTime? stepOlderCompareUntil;
final DateTime? stepNewerCompareUntil;
Map<String, dynamic> toJson() => <String, dynamic>{
'compareUntil': compareUntil.toIso8601String(),
'newerSlotStart': newerSlotStart.toIso8601String(),
'olderSlotStart': olderSlotStart.toIso8601String(),
'windowDays': windowDays,
'canStepOlder': canStepOlder,
'canStepNewer': canStepNewer,
if (stepOlderCompareUntil != null)
'stepOlderCompareUntil': stepOlderCompareUntil!.toIso8601String(),
if (stepNewerCompareUntil != null)
'stepNewerCompareUntil': stepNewerCompareUntil!.toIso8601String(),
'assets': assets.map((QuestionAuditAsset a) => a.toJson()).toList(),
};
}
/// Default view: last two **completed** 4-hour slots (newer = last completed).
DateTime questionAuditDefaultCompareUntil(DateTime now) {
final DateTime last =
MarketHistoryFourHourSlot.lastCompletedSlotStart(now.toUtc());
return MarketHistoryFourHourSlot.endExclusive(last);
}
/// Earliest [compareUntil] that still pairs two slots in the rolling window.
DateTime questionAuditMinCompareUntil(DateTime now, int windowDays) {
final DateTime first = MarketHistoryFourHourSlot.windowFirstSlotStart(
now.toUtc(),
windowDays,
);
final DateTime minNewerSlot = first.add(
const Duration(hours: MarketHistoryFourHourSlot.slotHours),
);
return MarketHistoryFourHourSlot.endExclusive(minNewerSlot);
}
/// The newer bar's slot start for a page keyed by [compareUntil].
///
/// [compareUntil] is always `endExclusive(newerSlotStart)`.
DateTime questionAuditNewerSlotStart(DateTime compareUntil) {
return MarketHistoryFourHourSlot.slotStartContaining(
compareUntil.toUtc().subtract(
const Duration(hours: MarketHistoryFourHourSlot.slotHours),
),
);
}
/// Snaps [requested] to a slot-aligned bound within [minUntil, maxUntil].
DateTime snapQuestionAuditCompareUntil({
required DateTime requested,
required DateTime minUntil,
required DateTime maxUntil,
}) {
final DateTime r = requested.toUtc();
if (r.isAfter(maxUntil)) {
return maxUntil;
}
final DateTime snapped = MarketHistoryFourHourSlot.endExclusive(
questionAuditNewerSlotStart(r),
);
if (snapped.isBefore(minUntil)) {
return minUntil;
}
if (snapped.isAfter(maxUntil)) {
return maxUntil;
}
return snapped;
}
/// Pair of slot starts for the page keyed by [compareUntil].
(DateTime newer, DateTime older) questionAuditSlotPair(DateTime compareUntil) {
final DateTime newer = questionAuditNewerSlotStart(compareUntil);
final DateTime older = newer.subtract(
const Duration(hours: MarketHistoryFourHourSlot.slotHours),
);
return (newer, older);
}
/// Steps back: newer becomes previous older (e.g. #1 vs #2 #2 vs #3).
DateTime questionAuditStepOlderCompareUntil({
required DateTime compareUntil,
required DateTime now,
}) {
final DateTime newerSlot = questionAuditNewerSlotStart(compareUntil);
final DateTime priorNewerSlot = newerSlot.subtract(
const Duration(hours: MarketHistoryFourHourSlot.slotHours),
);
return MarketHistoryFourHourSlot.endExclusive(priorNewerSlot);
}
/// Steps forward one completed slot, capped at [maxUntil].
DateTime questionAuditStepNewerCompareUntil({
required DateTime compareUntil,
required DateTime maxUntil,
}) {
final DateTime newerSlot = questionAuditNewerSlotStart(compareUntil);
final DateTime nextNewerSlot = newerSlot.add(
const Duration(hours: MarketHistoryFourHourSlot.slotHours),
);
final DateTime candidate =
MarketHistoryFourHourSlot.endExclusive(nextNewerSlot);
return candidate.isAfter(maxUntil) ? maxUntil : candidate;
}
/// Read-side audit of the two newest 4-hour bars per active tradable symbol.
class MarketHistoryQuestionAudit {
MarketHistoryQuestionAudit({
required Connection connection,
TradableAssetsDb? tradableAssetsDb,
}) : _connection = connection,
_tradableAssetsDb = tradableAssetsDb ?? TradableAssetsDb(connection);
final Connection _connection;
final TradableAssetsDb _tradableAssetsDb;
Future<QuestionAuditPage> page({
required DateTime now,
DateTime? compareUntil,
int windowDays = MarketHistoryConfig.windowDays,
String timeframe = MarketHistoryConfig.barTimeframe,
}) async {
final DateTime n = now.toUtc();
final DateTime calendarMax = questionAuditDefaultCompareUntil(n);
final DateTime minUntil = questionAuditMinCompareUntil(n, windowDays);
final DateTime? latestBarSlot = await _latestBarSlotStart(timeframe: timeframe);
final DateTime maxUntil = _effectiveMaxCompareUntil(
calendarMax: calendarMax,
latestBarSlot: latestBarSlot,
);
final DateTime until = compareUntil == null
? maxUntil
: snapQuestionAuditCompareUntil(
requested: compareUntil,
minUntil: minUntil,
maxUntil: maxUntil,
);
final (DateTime newerSlotStart, DateTime olderSlotStart) =
questionAuditSlotPair(until);
final bool canStepOlder = until.isAfter(minUntil);
final bool canStepNewer = until.isBefore(maxUntil);
final List<QuestionAuditAsset> assets = await assetsForSlotPair(
newerSlotStart: newerSlotStart,
olderSlotStart: olderSlotStart,
timeframe: timeframe,
);
return QuestionAuditPage(
compareUntil: until,
newerSlotStart: newerSlotStart,
olderSlotStart: olderSlotStart,
windowDays: windowDays,
assets: assets,
canStepOlder: canStepOlder,
canStepNewer: canStepNewer,
stepOlderCompareUntil: canStepOlder
? questionAuditStepOlderCompareUntil(
compareUntil: until,
now: n,
)
: null,
stepNewerCompareUntil: canStepNewer
? questionAuditStepNewerCompareUntil(
compareUntil: until,
maxUntil: maxUntil,
)
: null,
);
}
DateTime _effectiveMaxCompareUntil({
required DateTime calendarMax,
required DateTime? latestBarSlot,
}) {
if (latestBarSlot == null) {
return calendarMax;
}
final DateTime dataMax =
MarketHistoryFourHourSlot.endExclusive(latestBarSlot);
return dataMax.isBefore(calendarMax) ? dataMax : calendarMax;
}
/// Latest synced 4-hour bar slot, or `null` when history is empty.
Future<DateTime?> _latestBarSlotStart({required String timeframe}) async {
final Result result = await _connection.execute(
Sql.named(
'''
SELECT MAX(as_of) AS latest
FROM market_data_snapshots
WHERE metric = 'bar'
AND timeframe = @timeframe
AND price IS NOT NULL
AND ${MarketHistoryBarPlaceholder.sqlExcludePlaceholders}
''',
),
parameters: <String, dynamic>{'timeframe': timeframe},
);
if (result.isEmpty || result.first[0] == null) {
return null;
}
return MarketHistoryFourHourSlot.slotStartContaining(
(result.first[0]! as DateTime).toUtc(),
);
}
/// Loads the bar row for each symbol at [newerSlotStart] and [olderSlotStart].
Future<List<QuestionAuditAsset>> assetsForSlotPair({
required DateTime newerSlotStart,
required DateTime olderSlotStart,
String timeframe = MarketHistoryConfig.barTimeframe,
}) async {
final DateTime newer = MarketHistoryFourHourSlot.slotStartContaining(
newerSlotStart.toUtc(),
);
final DateTime older = MarketHistoryFourHourSlot.slotStartContaining(
olderSlotStart.toUtc(),
);
final String newerWire = MarketHistoryFourHourSlot.slotStartWire(newer);
final String olderWire = MarketHistoryFourHourSlot.slotStartWire(older);
final List<String> active =
await _tradableAssetsDb.listActiveTradableSymbols();
if (active.isEmpty) {
return <QuestionAuditAsset>[];
}
final Result rows = await _connection.execute(
Sql.named(
'''
SELECT symbol, as_of, price, volume, raw
FROM market_data_snapshots
WHERE metric = 'bar'
AND timeframe = @timeframe
AND symbol = ANY(@symbols)
AND price IS NOT NULL
AND ${MarketHistoryBarPlaceholder.sqlExcludePlaceholders}
AND (
as_of = @newer_slot
OR as_of = @older_slot
OR raw->>'slot_start' = @newer_wire
OR raw->>'slot_start' = @older_wire
)
ORDER BY symbol ASC, as_of DESC
''',
),
parameters: <String, dynamic>{
'symbols': active,
'timeframe': timeframe,
'newer_slot': newer,
'older_slot': older,
'newer_wire': newerWire,
'older_wire': olderWire,
},
);
final Map<String, Map<DateTime, _BarRow>> bySymbol =
<String, Map<DateTime, _BarRow>>{};
for (final ResultRow row in rows) {
final String symbol = row[0]! as String;
final DateTime asOf = MarketHistoryFourHourSlot.slotStartContaining(
(row[1]! as DateTime).toUtc(),
);
final Map<String, dynamic>? raw = _decodeRaw(row[4]);
bySymbol.putIfAbsent(symbol, () => <DateTime, _BarRow>{})[asOf] = _BarRow(
asOf: asOf,
closePrice: MarketDataDb.readOptionalNumeric(row[2]),
volume: MarketDataDb.readOptionalNumeric(row[3]),
raw: raw,
);
}
final List<QuestionAuditAsset> assets = <QuestionAuditAsset>[];
for (final MapEntry<String, Map<DateTime, _BarRow>> entry
in bySymbol.entries) {
final _BarRow? newerRow = entry.value[newer];
final _BarRow? olderRow = entry.value[older];
if (newerRow == null ||
olderRow == null ||
newerRow.volume == null ||
olderRow.volume == null) {
continue;
}
try {
final QuestionAuditBarSlot newerBar = barRowToSlot(newerRow);
final QuestionAuditBarSlot olderBar = barRowToSlot(olderRow);
assets.add(
QuestionAuditAsset(
symbol: entry.key,
priceDelta: newerBar.avgPrice - olderBar.avgPrice,
volumeDelta: newerBar.volume - olderBar.volume,
olderSlot: olderBar,
newerSlot: newerBar,
),
);
} on ArgumentError {
continue;
}
}
assets.sort(
(QuestionAuditAsset a, QuestionAuditAsset b) =>
a.symbol.compareTo(b.symbol),
);
return assets;
}
Map<String, dynamic>? _decodeRaw(Object? rawValue) {
if (rawValue == null) {
return null;
}
if (rawValue is Map<String, dynamic>) {
return rawValue;
}
return jsonDecode(rawValue.toString()) as Map<String, dynamic>;
}
}
class _BarRow {
const _BarRow({
required this.asOf,
required this.closePrice,
required this.volume,
required this.raw,
});
final DateTime asOf;
final num? closePrice;
final num? volume;
final Map<String, dynamic>? raw;
}

View File

@ -0,0 +1,50 @@
/// US equity regular-session calendar checks for 4-hour history slots (UTC).
abstract final class MarketHistoryTradingCalendar {
/// Saturday or Sunday (UTC calendar date of [instant]).
static bool isWeekendDayUtc(DateTime instant) {
final DateTime day = DateTime.utc(instant.year, instant.month, instant.day);
return day.weekday == DateTime.saturday || day.weekday == DateTime.sunday;
}
/// NYSE full-day closures (UTC date). Extend as needed.
static bool isNyseHolidayUtc(DateTime instant) {
return _nyseHolidays.contains(_dateKey(instant));
}
/// No regular US cash session on this UTC calendar day (weekend or NYSE holiday).
static bool isLikelyNoRegularSession(DateTime slotStart) {
final DateTime utc = slotStart.toUtc();
return isWeekendDayUtc(utc) || isNyseHolidayUtc(utc);
}
static String _dateKey(DateTime instant) {
final DateTime d = instant.toUtc();
String two(int n) => n.toString().padLeft(2, '0');
return '${d.year}-${two(d.month)}-${two(d.day)}';
}
/// NYSE observed full closures (UTC dates, YYYY-MM-DD).
static const Set<String> _nyseHolidays = <String>{
'2025-01-01',
'2025-01-20',
'2025-02-17',
'2025-04-18',
'2025-05-26',
'2025-06-19',
'2025-07-04',
'2025-09-01',
'2025-11-27',
'2025-12-25',
'2026-01-01',
'2026-01-19',
'2026-02-16',
'2026-04-03',
'2026-05-25',
'2026-06-19',
'2026-07-03',
'2026-09-07',
'2026-11-26',
'2026-12-25',
'2027-01-01',
};
}

View File

@ -0,0 +1,282 @@
import 'dart:convert';
import 'package:postgres/postgres.dart';
import 'market_history_config.dart';
import 'market_history_four_hour_slot.dart';
import 'tradable_assets_db.dart';
/// One UTC 4-hour slot within the rolling window.
class MarketHistorySlotCoverage {
const MarketHistorySlotCoverage({
required this.slotStart,
required this.completed,
required this.fullySynced,
required this.syncedSymbolCount,
required this.expectedSymbolCount,
});
final DateTime slotStart;
final bool completed;
final bool fullySynced;
final int syncedSymbolCount;
final int expectedSymbolCount;
Map<String, dynamic> toJson() => <String, dynamic>{
'slotStart': slotStart.toIso8601String(),
'completed': completed,
'fullySynced': fullySynced,
'syncedSymbolCount': syncedSymbolCount,
'expectedSymbolCount': expectedSymbolCount,
};
}
/// Slot rollup for one UTC calendar day.
class MarketHistoryDayCoverage {
const MarketHistoryDayCoverage({
required this.date,
required this.slots,
required this.completedSlots,
required this.fullySyncedSlots,
});
final DateTime date;
final List<MarketHistorySlotCoverage> slots;
final int completedSlots;
final int fullySyncedSlots;
Map<String, dynamic> toJson() => <String, dynamic>{
'date': dateWire(date),
'slotsPerDay': MarketHistoryFourHourSlot.slotsPerDay,
'completedSlots': completedSlots,
'fullySyncedSlots': fullySyncedSlots,
'slots': slots.map((MarketHistorySlotCoverage s) => s.toJson()).toList(),
};
}
/// Rolling-window slot consistency for the admin calendar.
class MarketHistoryWeekCoverageReport {
const MarketHistoryWeekCoverageReport({
required this.asOf,
required this.windowDays,
required this.slotsPerDay,
required this.symbolCount,
required this.days,
required this.isConsistent,
});
final DateTime asOf;
final int windowDays;
final int slotsPerDay;
final int symbolCount;
final List<MarketHistoryDayCoverage> days;
final bool isConsistent;
Map<String, dynamic> toJson() => <String, dynamic>{
'asOf': asOf.toIso8601String(),
'windowDays': windowDays,
'slotsPerDay': slotsPerDay,
'symbolCount': symbolCount,
'isConsistent': isConsistent,
'days': days.map((MarketHistoryDayCoverage d) => d.toJson()).toList(),
};
}
/// Validates 4-hour bar coverage per UTC day for the admin week view.
class MarketHistoryWeekCoverage {
MarketHistoryWeekCoverage({
required Connection connection,
TradableAssetsDb? tradableAssetsDb,
this.windowDays = MarketHistoryConfig.windowDays,
this.timeframe = MarketHistoryConfig.barTimeframe,
this.maxSymbols = MarketHistoryConfig.historySyncMaxSymbols,
}) : _tradableAssetsDb = tradableAssetsDb ?? TradableAssetsDb(connection),
_connection = connection;
final Connection _connection;
final TradableAssetsDb _tradableAssetsDb;
final int windowDays;
final String timeframe;
final int maxSymbols;
Future<MarketHistoryWeekCoverageReport> compute({DateTime? now}) async {
final DateTime tick = (now ?? DateTime.now()).toUtc();
final List<String> symbols = await _activeSymbols();
final int symbolCount = symbols.length;
final List<DateTime> calendarDays = _calendarDaysEndingToday(tick, windowDays);
final Map<String, Set<String>> symbolsBySlot =
await _loadSyncedSymbolsBySlot(tick, symbols);
final List<MarketHistoryDayCoverage> days = <MarketHistoryDayCoverage>[];
var isConsistent = symbolCount > 0;
for (final DateTime day in calendarDays) {
final List<MarketHistorySlotCoverage> slots = <MarketHistorySlotCoverage>[];
var completedSlots = 0;
var fullySyncedSlots = 0;
for (int hour = 0; hour < 24; hour += MarketHistoryFourHourSlot.slotHours) {
final DateTime slotStart = DateTime.utc(day.year, day.month, day.day, hour);
final bool completed = MarketHistoryFourHourSlot.hasEnded(slotStart, tick);
final Set<String> synced =
symbolsBySlot[_slotKey(slotStart)] ?? <String>{};
final int syncedCount = _countSyncedSymbols(synced, symbols);
final bool fullySynced =
symbolCount > 0 && completed && syncedCount >= symbolCount;
if (completed) {
completedSlots++;
if (fullySynced) {
fullySyncedSlots++;
} else {
isConsistent = false;
}
}
slots.add(
MarketHistorySlotCoverage(
slotStart: slotStart,
completed: completed,
fullySynced: fullySynced,
syncedSymbolCount: syncedCount,
expectedSymbolCount: symbolCount,
),
);
}
days.add(
MarketHistoryDayCoverage(
date: day,
slots: slots,
completedSlots: completedSlots,
fullySyncedSlots: fullySyncedSlots,
),
);
}
if (symbolCount == 0) {
isConsistent = false;
}
return MarketHistoryWeekCoverageReport(
asOf: tick,
windowDays: windowDays,
slotsPerDay: MarketHistoryFourHourSlot.slotsPerDay,
symbolCount: symbolCount,
days: days,
isConsistent: isConsistent,
);
}
Future<List<String>> _activeSymbols() async {
List<String> symbols = await _tradableAssetsDb.listActiveTradableSymbols();
if (symbols.length > maxSymbols) {
symbols = symbols.sublist(0, maxSymbols);
}
return symbols;
}
Future<Map<String, Set<String>>> _loadSyncedSymbolsBySlot(
DateTime now,
List<String> symbols,
) async {
if (symbols.isEmpty) {
return <String, Set<String>>{};
}
final DateTime firstDay =
_calendarDaysEndingToday(now, windowDays).first;
final DateTime since = DateTime.utc(firstDay.year, firstDay.month, firstDay.day);
final DateTime until =
MarketHistoryFourHourSlot.endExclusive(MarketHistoryFourHourSlot.slotStartContaining(now));
final Result rows = await _connection.execute(
Sql.named(
'''
SELECT symbol, as_of, raw
FROM market_data_snapshots
WHERE metric = 'bar'
AND timeframe = @timeframe
AND as_of >= @since
AND as_of < @until
AND symbol = ANY(@symbols)
''',
),
parameters: <String, dynamic>{
'timeframe': timeframe,
'since': since,
'until': until,
'symbols': symbols,
},
);
final Map<String, Set<String>> bySlot = <String, Set<String>>{};
for (final ResultRow row in rows) {
final String symbol = row[0]! as String;
final DateTime slotStart = _slotStartFromRow((row[1]! as DateTime).toUtc(), row[2]);
final String key = _slotKey(slotStart);
bySlot.putIfAbsent(key, () => <String>{}).add(symbol);
}
return bySlot;
}
static DateTime _slotStartFromRow(DateTime asOf, Object? rawValue) {
if (rawValue is Map<String, dynamic>) {
final String? slotStartRaw = rawValue['slot_start'] as String?;
if (slotStartRaw != null) {
final DateTime? parsed = DateTime.tryParse(slotStartRaw);
if (parsed != null) {
return parsed.toUtc();
}
}
} else if (rawValue != null) {
try {
final Map<String, dynamic> raw =
jsonDecode(rawValue.toString()) as Map<String, dynamic>;
final String? slotStartRaw = raw['slot_start'] as String?;
if (slotStartRaw != null) {
final DateTime? parsed = DateTime.tryParse(slotStartRaw);
if (parsed != null) {
return parsed.toUtc();
}
}
} on Object {
// Fall back to as_of bucketing below.
}
}
return MarketHistoryFourHourSlot.slotStartContaining(asOf);
}
static List<DateTime> calendarDaysEndingToday(DateTime now, int windowDays) {
final DateTime today = DateTime.utc(now.year, now.month, now.day);
return List<DateTime>.generate(
windowDays,
(int index) => today.subtract(Duration(days: windowDays - 1 - index)),
);
}
static List<DateTime> _calendarDaysEndingToday(DateTime now, int windowDays) =>
calendarDaysEndingToday(now, windowDays);
static String _slotKey(DateTime slotStart) =>
MarketHistoryFourHourSlot.slotStartWire(slotStart);
static int _countSyncedSymbols(Set<String> synced, List<String> expected) {
if (expected.isEmpty) {
return 0;
}
var count = 0;
for (final String symbol in expected) {
if (synced.contains(symbol)) {
count++;
}
}
return count;
}
}
String dateWire(DateTime date) =>
'${date.year.toString().padLeft(4, '0')}-'
'${date.month.toString().padLeft(2, '0')}-'
'${date.day.toString().padLeft(2, '0')}';

View File

@ -1,4 +1,5 @@
import 'market_data_db.dart'; import 'market_data_db.dart';
import 'market_history_query.dart';
import 'trading_config.dart'; import 'trading_config.dart';
/// Why a rule did not fire. `null` means the rule fired. /// Why a rule did not fire. `null` means the rule fired.
@ -9,6 +10,7 @@ enum RuleSkipReason {
aboveThreshold, aboveThreshold,
cooldown, cooldown,
zeroReferencePrice, zeroReferencePrice,
insufficientBars,
} }
/// Result of evaluating a single [TradingRuleConfig] against snapshots. /// Result of evaluating a single [TradingRuleConfig] against snapshots.
@ -22,6 +24,10 @@ class RuleEvaluation {
this.observedPrice, this.observedPrice,
this.questionText, this.questionText,
this.asOf, this.asOf,
this.symbolToken,
this.guessSymbol,
this.correctAnswer,
this.refDaysAgo,
}); });
final TradingRuleConfig rule; final TradingRuleConfig rule;
@ -38,6 +44,18 @@ class RuleEvaluation {
/// Most recent `as_of` across the snapshots used. /// Most recent `as_of` across the snapshots used.
final DateTime? asOf; final DateTime? asOf;
/// Obfuscated ticker token for `guess_weekly_move` (e.g. `ASSET_A`).
final String? symbolToken;
/// Real symbol stored server-side only (not in [questionText]).
final String? guessSymbol;
/// Expected swipe answer: `10` (up) or `-10` (down) for guess rules.
final num? correctAnswer;
/// Days ago label for the reference close in guess templates.
final int? refDaysAgo;
} }
/// Pure evaluation of trading rules over [MarketDataSnapshot] inputs. /// Pure evaluation of trading rules over [MarketDataSnapshot] inputs.
@ -59,9 +77,21 @@ class RuleEngine {
required Map<String, MarketDataSnapshot> snapshots, required Map<String, MarketDataSnapshot> snapshots,
DateTime? lastFiredAt, DateTime? lastFiredAt,
DateTime? now, DateTime? now,
WeeklyMover? weeklyMover,
String? symbolToken,
}) { }) {
final DateTime evaluatedAt = (now ?? _clock()).toUtc(); final DateTime evaluatedAt = (now ?? _clock()).toUtc();
if (rule.type == 'guess_weekly_move') {
return evaluateGuessWeeklyMove(
rule: rule,
mover: weeklyMover,
symbolToken: symbolToken,
lastFiredAt: lastFiredAt,
now: now,
);
}
if (rule.type != 'price_below_pct_of_ref') { if (rule.type != 'price_below_pct_of_ref') {
return RuleEvaluation( return RuleEvaluation(
rule: rule, rule: rule,
@ -145,6 +175,66 @@ class RuleEngine {
); );
} }
/// Evaluates [guess_weekly_move] using a pre-selected [mover].
RuleEvaluation evaluateGuessWeeklyMove({
required TradingRuleConfig rule,
required WeeklyMover? mover,
required String? symbolToken,
DateTime? lastFiredAt,
DateTime? now,
}) {
final DateTime evaluatedAt = (now ?? _clock()).toUtc();
if (rule.type != 'guess_weekly_move') {
return RuleEvaluation(
rule: rule,
fired: false,
skipReason: RuleSkipReason.unknownType,
);
}
if (mover == null || symbolToken == null) {
return RuleEvaluation(
rule: rule,
fired: false,
skipReason: RuleSkipReason.insufficientBars,
);
}
final num refPrice = mover.openClose;
final num currentClose = mover.currentClose;
if (refPrice == 0) {
return RuleEvaluation(
rule: rule,
fired: false,
skipReason: RuleSkipReason.zeroReferencePrice,
);
}
final num correctAnswer =
currentClose > refPrice ? 10 : (currentClose < refPrice ? -10 : 10);
final int refDaysAgo = mover.days;
final String questionText = _renderGuessTemplate(
rule.questionTemplate,
token: symbolToken,
refPrice: refPrice,
refDaysAgo: refDaysAgo,
);
return RuleEvaluation(
rule: rule,
fired: true,
refPrice: refPrice,
observedPrice: currentClose,
questionText: questionText,
symbolToken: symbolToken,
guessSymbol: mover.symbol,
correctAnswer: correctAnswer,
refDaysAgo: refDaysAgo,
);
}
bool _isCooldown(DateTime? lastFiredAt, DateTime now) { bool _isCooldown(DateTime? lastFiredAt, DateTime now) {
if (lastFiredAt == null) { if (lastFiredAt == null) {
return false; return false;
@ -175,4 +265,16 @@ class RuleEngine {
final num abs = value.abs(); final num abs = value.abs();
return abs.toStringAsFixed(abs < 10 ? 2 : 1); return abs.toStringAsFixed(abs < 10 ? 2 : 1);
} }
String _renderGuessTemplate(
String template, {
required String token,
required num refPrice,
required int refDaysAgo,
}) {
return template
.replaceAll('{{token}}', token)
.replaceAll('{{ref_price}}', _formatPrice(refPrice))
.replaceAll('{{ref_days_ago}}', refDaysAgo.toString());
}
} }

View File

@ -0,0 +1,23 @@
/// Maps real ticker symbols to opaque tokens shown in question text.
class SymbolObfuscator {
SymbolObfuscator({this.prefix = 'ASSET_'});
final String prefix;
/// Stable token for [symbol] among [universe] sorted lexicographically.
String tokenFor(String symbol, List<String> universe) {
final List<String> sorted = List<String>.from(universe)..sort();
final int index = sorted.indexOf(symbol);
if (index < 0) {
throw ArgumentError.value(symbol, 'symbol', 'not in universe');
}
return '$prefix${_letterForIndex(index)}';
}
static String _letterForIndex(int index) {
if (index < 26) {
return String.fromCharCode(65 + index);
}
return '${_letterForIndex(index ~/ 26 - 1)}${_letterForIndex(index % 26)}';
}
}

View File

@ -0,0 +1,229 @@
import 'dart:convert';
import 'package:postgres/postgres.dart';
import 'backfill_sync_item.dart';
/// Outcome of one [SyncRunRecorder.record] body.
///
/// The recorder ALWAYS emits one of these regardless of whether the body
/// threw, so callers can convert to a domain-specific result type without
/// re-implementing the start/finish bookkeeping.
class SyncRunOutcome {
SyncRunOutcome({
required this.id,
required this.kind,
required this.startedAt,
required this.finishedAt,
required this.rowsWritten,
required this.rowsRemoved,
this.slotsSynced,
this.error,
});
final int id;
final String kind;
final DateTime startedAt;
final DateTime finishedAt;
final int rowsWritten;
final int rowsRemoved;
final int? slotsSynced;
final String? error;
bool get succeeded => error == null;
}
/// Counts the body of a sync run reports back to the recorder.
class SyncRunCounts {
const SyncRunCounts({
this.rowsWritten = 0,
this.rowsRemoved = 0,
this.slotsSynced,
this.backfillItems,
this.error,
});
final int rowsWritten;
final int rowsRemoved;
/// Completed 4-hour slots written (market-history backfill only).
final int? slotsSynced;
/// Slot starts and symbol lists requested from Alpaca (backfill only).
final List<BackfillSyncItem>? backfillItems;
/// Non-fatal partial failure (e.g. one Alpaca batch 500 while others
/// succeeded). Recorded on the sync run without discarding [rowsWritten].
final String? error;
}
/// Wraps a closure with a `market_data_sync_runs` audit row.
///
/// On entry, INSERTs a row with `kind`, `started_at` and returns its id.
/// After the body completes (success or thrown exception), UPDATEs the
/// row with `finished_at`, `rows_written`, `rows_removed`, and `error`.
///
/// The body's exception is **swallowed** — the recorder records it and
/// returns a [SyncRunOutcome] with `error` set instead. This is the
/// "scheduler-friendly" contract every sync stage in §2/§3/§4 needs.
///
/// **Do not change that contract for §3/§4:** `MarketHistoryScheduler`
/// (§5) runs universe backfill cleanup in sequence and expects a
/// thrown Alpaca 500 in backfill to be recorded here without aborting
/// cleanup. Partial batch failures in `MarketDataHistorySync` rely on
/// the same behaviour.
class SyncRunRecorder {
SyncRunRecorder(this._connection);
final Connection _connection;
static const String abortSupersededMessage =
'aborted: superseded by new worker sync';
/// Closes every run still marked in-progress (crashed or superseded worker).
Future<int> abortAllInProgressRuns({
DateTime? now,
String? message,
}) async {
return _abortInProgress(
now: now,
olderThan: null,
message: message ?? abortSupersededMessage,
);
}
/// Closes in-progress runs older than [olderThan] (hung without finishing).
Future<int> abortStaleInProgressRuns({
DateTime? now,
Duration olderThan = const Duration(minutes: 30),
String? message,
}) async {
return _abortInProgress(
now: now,
olderThan: olderThan,
message: message ?? 'aborted: stale in-progress sync run',
);
}
Future<int> _abortInProgress({
required DateTime? now,
required Duration? olderThan,
required String message,
}) async {
final DateTime tick = (now ?? DateTime.now()).toUtc();
final Result rows;
if (olderThan == null) {
rows = await _connection.execute(
Sql.named(
'''
UPDATE market_data_sync_runs
SET finished_at = @finished_at,
error = COALESCE(error, @message)
WHERE finished_at IS NULL
''',
),
parameters: <String, dynamic>{
'finished_at': tick,
'message': message,
},
);
} else {
rows = await _connection.execute(
Sql.named(
'''
UPDATE market_data_sync_runs
SET finished_at = @finished_at,
error = COALESCE(error, @message)
WHERE finished_at IS NULL
AND started_at < @cutoff
''',
),
parameters: <String, dynamic>{
'finished_at': tick,
'message': message,
'cutoff': tick.subtract(olderThan),
},
);
}
return rows.affectedRows;
}
Future<SyncRunOutcome> record(
String kind,
Future<SyncRunCounts> Function() body, {
DateTime? now,
}) async {
final DateTime startedAt = (now ?? DateTime.now()).toUtc();
final Result inserted = await _connection.execute(
Sql.named(
'''
INSERT INTO market_data_sync_runs (kind, started_at)
VALUES (@kind, @started_at)
RETURNING id
''',
),
parameters: <String, dynamic>{
'kind': kind,
'started_at': startedAt,
},
);
final int id = (inserted.first[0]! as num).toInt();
int rowsWritten = 0;
int rowsRemoved = 0;
int? slotsSynced;
List<BackfillSyncItem>? backfillItems;
String? error;
try {
final SyncRunCounts counts = await body();
rowsWritten = counts.rowsWritten;
rowsRemoved = counts.rowsRemoved;
slotsSynced = counts.slotsSynced;
backfillItems = counts.backfillItems;
error = counts.error;
} on Object catch (e) {
// Always recorded, never rethrown the scheduler in §5 expects
// each stage to fail in isolation, not to bubble up.
error = e.toString();
}
final DateTime finishedAt = (now ?? DateTime.now()).toUtc();
await _connection.execute(
Sql.named(
'''
UPDATE market_data_sync_runs
SET finished_at = @finished_at,
rows_written = @rows_written,
rows_removed = @rows_removed,
slots_synced = @slots_synced,
backfill_items = @backfill_items::jsonb,
error = @error
WHERE id = @id
''',
),
parameters: <String, dynamic>{
'id': id,
'finished_at': finishedAt,
'rows_written': rowsWritten,
'rows_removed': rowsRemoved,
'slots_synced': slotsSynced ?? 0,
'backfill_items': backfillItems == null || backfillItems.isEmpty
? null
: jsonEncode(BackfillSyncItem.encodeList(backfillItems)),
'error': error,
},
);
return SyncRunOutcome(
id: id,
kind: kind,
startedAt: startedAt,
finishedAt: finishedAt,
rowsWritten: rowsWritten,
rowsRemoved: rowsRemoved,
slotsSynced: slotsSynced,
error: error,
);
}
}

View File

@ -0,0 +1,163 @@
import 'dart:convert';
import 'package:postgres/postgres.dart';
import '../alpaca/alpaca_models.dart';
/// Read-side row for [tradable_assets].
class TradableAssetRow {
TradableAssetRow({
required this.symbol,
required this.assetClass,
required this.tradable,
required this.fractionable,
required this.status,
required this.refreshedAt,
this.exchange,
this.name,
this.raw,
});
final String symbol;
final String assetClass;
final String? exchange;
final String? name;
final bool tradable;
final bool fractionable;
final String status;
final Map<String, dynamic>? raw;
final DateTime refreshedAt;
}
/// Postgres access for [tradable_assets] (the daily Alpaca asset universe).
///
/// Writers: only [TradableAssetsSync] (§2.2). Readers: the historical
/// backfill in §3 plus operator/query tooling.
class TradableAssetsDb {
TradableAssetsDb(this._connection);
final Connection _connection;
/// Upserts every entry in [assets] (PK on `symbol`) using [now] as the
/// shared `refreshed_at`, then marks any rows whose `refreshed_at` is
/// older than [now] as `status='inactive', tradable=false` preserving
/// the historical row but flagging it as no longer in the live universe.
///
/// The two phases run inside a single transaction so a partial failure
/// can't leave the universe half-updated.
Future<void> upsertAll(
List<AlpacaAsset> assets, {
DateTime? now,
}) async {
final DateTime ts = (now ?? DateTime.now()).toUtc();
await _connection.runTx((TxSession tx) async {
for (final AlpacaAsset asset in assets) {
await tx.execute(
Sql.named(
'''
INSERT INTO tradable_assets (
symbol, asset_class, exchange, name, tradable,
fractionable, status, raw, refreshed_at
) VALUES (
@symbol, @asset_class, @exchange, @name, @tradable,
@fractionable, @status, @raw::jsonb, @refreshed_at
)
ON CONFLICT (symbol) DO UPDATE SET
asset_class = EXCLUDED.asset_class,
exchange = EXCLUDED.exchange,
name = EXCLUDED.name,
tradable = EXCLUDED.tradable,
fractionable = EXCLUDED.fractionable,
status = EXCLUDED.status,
raw = EXCLUDED.raw,
refreshed_at = EXCLUDED.refreshed_at
''',
),
parameters: <String, dynamic>{
'symbol': asset.symbol,
'asset_class': asset.assetClass,
'exchange': asset.exchange,
'name': asset.name,
'tradable': asset.tradable,
'fractionable': asset.fractionable,
'status': asset.status,
'raw': asset.raw == null ? null : jsonEncode(asset.raw),
'refreshed_at': ts,
},
);
}
// Anything not seen in this batch hasn't been refreshed at [ts] yet.
// Flip it to inactive without bumping refreshed_at, so the audit
// trail still records "last seen as part of the live universe."
await tx.execute(
Sql.named(
'''
UPDATE tradable_assets
SET status = 'inactive', tradable = false
WHERE refreshed_at < @now
AND (status <> 'inactive' OR tradable = true)
''',
),
parameters: <String, dynamic>{'now': ts},
);
});
}
/// Symbols currently tradable on the active universe.
Future<List<String>> listActiveTradableSymbols() async {
final Result result = await _connection.execute(
'''
SELECT symbol
FROM tradable_assets
WHERE status = 'active' AND tradable = true
ORDER BY symbol ASC
''',
);
return result.map((ResultRow r) => r[0]! as String).toList(growable: false);
}
/// Single-row lookup, primarily for tests and admin tooling.
Future<TradableAssetRow?> getBySymbol(String symbol) async {
final Result result = await _connection.execute(
Sql.named(
'''
SELECT symbol, asset_class, exchange, name, tradable,
fractionable, status, raw, refreshed_at
FROM tradable_assets
WHERE symbol = @symbol
''',
),
parameters: <String, dynamic>{'symbol': symbol},
);
if (result.isEmpty) {
return null;
}
return _rowToModel(result.first);
}
TradableAssetRow _rowToModel(ResultRow row) {
final Object? rawValue = row[7];
Map<String, dynamic>? raw;
if (rawValue is Map<String, dynamic>) {
raw = rawValue;
} else if (rawValue is Map) {
raw = Map<String, dynamic>.from(rawValue);
} else if (rawValue != null) {
raw = jsonDecode(rawValue.toString()) as Map<String, dynamic>;
}
return TradableAssetRow(
symbol: row[0]! as String,
assetClass: row[1]! as String,
exchange: row[2] as String?,
name: row[3] as String?,
tradable: row[4]! as bool,
fractionable: row[5]! as bool,
status: row[6]! as String,
raw: raw,
refreshedAt: (row[8]! as DateTime).toUtc(),
);
}
}

View File

@ -0,0 +1,77 @@
import 'package:postgres/postgres.dart';
import '../alpaca/alpaca_assets_client.dart';
import '../alpaca/alpaca_models.dart';
import 'sync_run_recorder.dart';
import 'tradable_assets_db.dart';
/// Outcome of a single [TradableAssetsSync.runOnce] invocation.
///
/// On the happy path [error] is `null` and [rowsWritten] is the number of
/// asset rows upserted. On a failure path [error] holds the upstream
/// message and [rowsWritten] is `0` the exception itself is never
/// rethrown to callers (the scheduler in §5 must keep going).
class TradableAssetsSyncResult {
TradableAssetsSyncResult({
required this.rowsWritten,
required this.startedAt,
required this.finishedAt,
this.error,
});
final int rowsWritten;
final DateTime startedAt;
final DateTime finishedAt;
final String? error;
bool get succeeded => error == null;
}
/// Daily refresh of the `tradable_assets` cache from Alpaca.
///
/// One pass:
/// 1. [SyncRunRecorder] inserts a `market_data_sync_runs` row with
/// `kind='universe'`.
/// 2. Pull `/v2/assets?status=active&asset_class=us_equity`.
/// 3. Upsert into `tradable_assets`, marking missing symbols inactive.
/// 4. Recorder closes the row with `finished_at` + `rows_written` (or
/// `error` on failure).
///
/// The scheduler in §5 runs this once per `MARKET_UNIVERSE_REFRESH_HOURS`.
class TradableAssetsSync {
TradableAssetsSync({
required AlpacaAssetsClient assetsClient,
required TradableAssetsDb assetsDb,
required Connection connection,
}) : _assetsClient = assetsClient,
_assetsDb = assetsDb,
_recorder = SyncRunRecorder(connection);
final AlpacaAssetsClient _assetsClient;
final TradableAssetsDb _assetsDb;
final SyncRunRecorder _recorder;
static const String kind = 'universe';
Future<TradableAssetsSyncResult> runOnce({DateTime? now}) async {
final DateTime started = (now ?? DateTime.now()).toUtc();
final SyncRunOutcome outcome = await _recorder.record(
kind,
() async {
final List<AlpacaAsset> assets =
await _assetsClient.listActiveTradable();
await _assetsDb.upsertAll(assets, now: started);
return SyncRunCounts(rowsWritten: assets.length);
},
now: started,
);
return TradableAssetsSyncResult(
rowsWritten: outcome.rowsWritten,
startedAt: outcome.startedAt,
finishedAt: outcome.finishedAt,
error: outcome.error,
);
}
}

View File

@ -170,7 +170,7 @@ class TradingRuleConfig {
return TradingRuleConfig( return TradingRuleConfig(
id: json['id']! as String, id: json['id']! as String,
type: json['type']! as String, type: json['type']! as String,
symbol: json['symbol']! as String, symbol: json['symbol'] as String? ?? '*',
refMetric: json['ref_metric'] as String? ?? 'prev_close', refMetric: json['ref_metric'] as String? ?? 'prev_close',
thresholdPct: json['threshold_pct'] as num? ?? 0, thresholdPct: json['threshold_pct'] as num? ?? 0,
questionTemplate: json['question_template'] as String? ?? '', questionTemplate: json['question_template'] as String? ?? '',

View File

@ -7,7 +7,10 @@ import '../question_service.dart';
import '../questions_db.dart'; import '../questions_db.dart';
import 'guardrails.dart'; import 'guardrails.dart';
import 'market_data_db.dart'; import 'market_data_db.dart';
import '../market_history_env.dart';
import 'market_history_query.dart';
import 'rule_engine.dart'; import 'rule_engine.dart';
import 'symbol_obfuscator.dart';
import 'trading_config.dart'; import 'trading_config.dart';
import 'trading_config_db.dart'; import 'trading_config_db.dart';
import 'user_trading_state_db.dart'; import 'user_trading_state_db.dart';
@ -36,6 +39,9 @@ class TradingPipeline {
required TradingConfigDb tradingConfigDb, required TradingConfigDb tradingConfigDb,
required UserTradingStateDb tradingStateDb, required UserTradingStateDb tradingStateDb,
RuleEngine? ruleEngine, RuleEngine? ruleEngine,
MarketHistoryQuery? marketHistoryQuery,
MarketHistoryEnv? marketHistoryEnv,
SymbolObfuscator? symbolObfuscator,
Guardrails? guardrails, Guardrails? guardrails,
int maxQueuedQuestions = 3, int maxQueuedQuestions = 3,
DateTime Function()? clock, DateTime Function()? clock,
@ -44,7 +50,10 @@ class TradingPipeline {
_marketDataDb = marketDataDb, _marketDataDb = marketDataDb,
_tradingConfigDb = tradingConfigDb, _tradingConfigDb = tradingConfigDb,
_tradingStateDb = tradingStateDb, _tradingStateDb = tradingStateDb,
_ruleEngine = ruleEngine ?? RuleEngine(), _ruleEngine = ruleEngine ?? RuleEngine(clock: clock),
_marketHistoryQuery = marketHistoryQuery,
_marketHistoryEnv = marketHistoryEnv ?? MarketHistoryEnv.fromMap(<String, String>{}),
_symbolObfuscator = symbolObfuscator ?? SymbolObfuscator(),
_guardrails = guardrails ?? Guardrails(), _guardrails = guardrails ?? Guardrails(),
_maxQueuedQuestions = maxQueuedQuestions, _maxQueuedQuestions = maxQueuedQuestions,
_clock = clock ?? DateTime.now; _clock = clock ?? DateTime.now;
@ -55,6 +64,9 @@ class TradingPipeline {
final TradingConfigDb _tradingConfigDb; final TradingConfigDb _tradingConfigDb;
final UserTradingStateDb _tradingStateDb; final UserTradingStateDb _tradingStateDb;
final RuleEngine _ruleEngine; final RuleEngine _ruleEngine;
final MarketHistoryQuery? _marketHistoryQuery;
final MarketHistoryEnv _marketHistoryEnv;
final SymbolObfuscator _symbolObfuscator;
final Guardrails _guardrails; final Guardrails _guardrails;
final int _maxQueuedQuestions; final int _maxQueuedQuestions;
final DateTime Function() _clock; final DateTime Function() _clock;
@ -99,46 +111,74 @@ class TradingPipeline {
continue; continue;
} }
final Map<String, MarketDataSnapshot> snapshots =
await _loadSnapshotsForRule(rule);
final DateTime? lastFiredAt = final DateTime? lastFiredAt =
await _tradingStateDb.getRuleLastFiredAt(firebaseUid, rule.id); await _tradingStateDb.getRuleLastFiredAt(firebaseUid, rule.id);
final RuleEvaluation result = _ruleEngine.evaluate( final RuleEvaluation result;
if (rule.type == 'guess_weekly_move') {
result = await _evaluateGuessRule(
firebaseUid: firebaseUid,
rule: rule,
lastFiredAt: lastFiredAt,
now: now,
);
} else {
final Map<String, MarketDataSnapshot> snapshots =
await _loadSnapshotsForRule(rule);
result = _ruleEngine.evaluate(
rule: rule, rule: rule,
snapshots: snapshots, snapshots: snapshots,
lastFiredAt: lastFiredAt, lastFiredAt: lastFiredAt,
now: now, now: now,
); );
}
if (!result.fired) { if (!result.fired) {
skipped.add('${rule.id}(${result.skipReason?.name ?? 'no_fire'})'); skipped.add('${rule.id}(${result.skipReason?.name ?? 'no_fire'})');
continue; continue;
} }
final String phase = rule.type == 'guess_weekly_move'
? TradingPhases.awaitAnswer
: TradingPhases.awaitConfirm;
final num correctAnswer =
result.correctAnswer ?? (rule.type == 'guess_weekly_move' ? 10 : 10);
final Map<String, dynamic> question = final Map<String, dynamic> question =
await _questionService.createAndDeliverQuestion( await _questionService.createAndDeliverQuestion(
assignedUserId: firebaseUid, assignedUserId: firebaseUid,
questionText: result.questionText!, questionText: result.questionText!,
correctAnswer: 10, correctAnswer: correctAnswer,
sourceTag: 'trading:rule:${rule.id}', sourceTag: 'trading:rule:${rule.id}',
pipelineKey: PipelineKeys.trading, pipelineKey: PipelineKeys.trading,
pipelineStep: '${rule.id}:${TradingPhases.awaitConfirm}', pipelineStep: '${rule.id}:$phase',
metadata: result.guessSymbol == null
? null
: <String, dynamic>{'guess_symbol': result.guessSymbol},
); );
questionsCreated++; questionsCreated++;
fired.add(rule.id); fired.add(rule.id);
if (rule.type == 'guess_weekly_move' && result.guessSymbol != null) {
await _tradingStateDb.recordGuessSymbolPicked(
firebaseUid: firebaseUid,
symbol: result.guessSymbol!,
at: now,
);
}
await _tradingStateDb.setRuleState( await _tradingStateDb.setRuleState(
firebaseUid: firebaseUid, firebaseUid: firebaseUid,
ruleId: rule.id, ruleId: rule.id,
state: <String, dynamic>{ state: <String, dynamic>{
'phase': TradingPhases.awaitConfirm, 'phase': phase,
'last_fired_at': now.toIso8601String(), 'last_fired_at': now.toIso8601String(),
'question_id': question['id'], 'question_id': question['id'],
'symbol': rule.symbol, 'symbol': result.guessSymbol ?? rule.symbol,
'observed_price': result.observedPrice, 'observed_price': result.observedPrice,
'ref_price': result.refPrice, 'ref_price': result.refPrice,
'pct': result.pricePct, 'pct': result.pricePct,
if (result.symbolToken != null) 'symbol_token': result.symbolToken,
}, },
); );
} catch (e, st) { } catch (e, st) {
@ -171,7 +211,12 @@ class TradingPipeline {
return; return;
} }
final List<String> parts = pipelineStep.split(':'); final List<String> parts = pipelineStep.split(':');
if (parts.length < 2 || parts[1] != TradingPhases.awaitConfirm) { if (parts.length < 2) {
return;
}
final String phase = parts[1];
if (phase != TradingPhases.awaitConfirm &&
phase != TradingPhases.awaitAnswer) {
return; return;
} }
final String ruleId = parts.first; final String ruleId = parts.first;
@ -195,13 +240,26 @@ class TradingPipeline {
return; return;
} }
final Map<String, dynamic>? priorState =
await _tradingStateDb.getRuleState(firebaseUid, ruleId);
if (rule.type == 'guess_weekly_move') {
await _handleGuessAnswer(
firebaseUid: firebaseUid,
rule: rule,
questionId: questionId,
userResponse: userResponse,
correctAnswer: correctAnswer,
priorState: priorState,
now: now,
);
return;
}
final BranchOutcome outcome = BranchDecision.yesNo( final BranchOutcome outcome = BranchDecision.yesNo(
userResponse: userResponse, userResponse: userResponse,
correctAnswer: correctAnswer, correctAnswer: correctAnswer,
); );
final Map<String, dynamic>? priorState =
await _tradingStateDb.getRuleState(firebaseUid, ruleId);
final Map<String, dynamic> baseState = <String, dynamic>{ final Map<String, dynamic> baseState = <String, dynamic>{
...?priorState, ...?priorState,
}; };
@ -259,6 +317,96 @@ class TradingPipeline {
} }
} }
Future<RuleEvaluation> _evaluateGuessRule({
required String firebaseUid,
required TradingRuleConfig rule,
required DateTime? lastFiredAt,
required DateTime now,
}) async {
if (_marketHistoryQuery == null) {
return RuleEvaluation(
rule: rule,
fired: false,
skipReason: RuleSkipReason.insufficientBars,
);
}
final List<WeeklyMover> movers = await _marketHistoryQuery.weeklyMovers(
asOf: now,
minBars: _marketHistoryEnv.minBarsForGuess,
windowDays: _marketHistoryEnv.windowDays,
);
if (movers.isEmpty) {
return RuleEvaluation(
rule: rule,
fired: false,
skipReason: RuleSkipReason.insufficientBars,
);
}
final List<String> universe =
movers.map((WeeklyMover m) => m.symbol).toList();
WeeklyMover? picked;
String? token;
for (final WeeklyMover mover in movers) {
final bool onCooldown = await _tradingStateDb.isGuessSymbolOnCooldown(
firebaseUid: firebaseUid,
symbol: mover.symbol,
now: now,
cooldownHours: _marketHistoryEnv.guessCooldownHours,
);
if (onCooldown) {
continue;
}
picked = mover;
token = _symbolObfuscator.tokenFor(mover.symbol, universe);
break;
}
return _ruleEngine.evaluateGuessWeeklyMove(
rule: rule,
mover: picked,
symbolToken: token,
lastFiredAt: lastFiredAt,
now: now,
);
}
Future<void> _handleGuessAnswer({
required String firebaseUid,
required TradingRuleConfig rule,
required String questionId,
required num userResponse,
required num correctAnswer,
required Map<String, dynamic>? priorState,
required DateTime now,
}) async {
final int scoreDelta = userResponse == correctAnswer ? 1 : -1;
final String symbol =
(priorState?['symbol'] as String?) ?? rule.symbol;
await _tradingStateDb.recordGuessScore(
firebaseUid: firebaseUid,
scoreDelta: scoreDelta,
symbol: symbol,
at: now,
);
final Map<String, dynamic> baseState = <String, dynamic>{
...?priorState,
'phase': TradingPhases.done,
'question_id': questionId,
'answer': userResponse == correctAnswer ? 'match' : 'miss',
'score_delta': scoreDelta,
'answered_at': now.toIso8601String(),
};
await _tradingStateDb.setRuleState(
firebaseUid: firebaseUid,
ruleId: rule.id,
state: baseState,
);
}
Future<Map<String, MarketDataSnapshot>> _loadSnapshotsForRule( Future<Map<String, MarketDataSnapshot>> _loadSnapshotsForRule(
TradingRuleConfig rule, TradingRuleConfig rule,
) async { ) async {

View File

@ -12,6 +12,8 @@ class UserTradingStateDb {
static const String rulesContextKey = 'rules'; static const String rulesContextKey = 'rules';
static const String pendingOrdersContextKey = 'pending_orders'; static const String pendingOrdersContextKey = 'pending_orders';
static const String skippedContextKey = 'skipped'; static const String skippedContextKey = 'skipped';
static const String guessScoreContextKey = 'guess_score';
static const String guessSymbolCooldownContextKey = 'guess_symbol_cooldown';
Future<void> ensureExists(String firebaseUid) async { Future<void> ensureExists(String firebaseUid) async {
await _connection.execute( await _connection.execute(
@ -177,6 +179,82 @@ class UserTradingStateDb {
await _writeContext(firebaseUid, context, touchEvalAt: false); await _writeContext(firebaseUid, context, touchEvalAt: false);
} }
Future<Map<String, dynamic>?> getGuessScore(String firebaseUid) async {
final Map<String, dynamic> context = await getContext(firebaseUid);
final Object? raw = context[guessScoreContextKey];
if (raw is Map) {
return Map<String, dynamic>.from(raw);
}
return null;
}
Future<void> recordGuessScore({
required String firebaseUid,
required int scoreDelta,
required String symbol,
required DateTime at,
}) async {
await ensureExists(firebaseUid);
final Map<String, dynamic> context = await getContext(firebaseUid);
final Map<String, dynamic> prior = Map<String, dynamic>.from(
context[guessScoreContextKey] as Map? ?? <String, dynamic>{},
);
final int total = ((prior['total'] as num?)?.toInt() ?? 0) + scoreDelta;
context[guessScoreContextKey] = <String, dynamic>{
'total': total,
'last': <String, dynamic>{
'score_delta': scoreDelta,
'symbol': symbol,
'at': at.toUtc().toIso8601String(),
},
};
await _writeContext(firebaseUid, context, touchEvalAt: true);
}
Future<DateTime?> getGuessSymbolLastPickedAt(
String firebaseUid,
String symbol,
) async {
final Map<String, dynamic> context = await getContext(firebaseUid);
final Map<String, dynamic> cooldown = Map<String, dynamic>.from(
context[guessSymbolCooldownContextKey] as Map? ?? <String, dynamic>{},
);
final String? raw = cooldown[symbol] as String?;
if (raw == null || raw.isEmpty) {
return null;
}
return DateTime.parse(raw).toUtc();
}
Future<bool> isGuessSymbolOnCooldown({
required String firebaseUid,
required String symbol,
required DateTime now,
int cooldownHours = 24,
}) async {
final DateTime? last =
await getGuessSymbolLastPickedAt(firebaseUid, symbol);
if (last == null) {
return false;
}
return now.difference(last) < Duration(hours: cooldownHours);
}
Future<void> recordGuessSymbolPicked({
required String firebaseUid,
required String symbol,
required DateTime at,
}) async {
await ensureExists(firebaseUid);
final Map<String, dynamic> context = await getContext(firebaseUid);
final Map<String, dynamic> cooldown = Map<String, dynamic>.from(
context[guessSymbolCooldownContextKey] as Map? ?? <String, dynamic>{},
);
cooldown[symbol] = at.toUtc().toIso8601String();
context[guessSymbolCooldownContextKey] = cooldown;
await _writeContext(firebaseUid, context, touchEvalAt: true);
}
Future<void> recordSkip({ Future<void> recordSkip({
required String firebaseUid, required String firebaseUid,
required String ruleId, required String ruleId,

View File

@ -0,0 +1,193 @@
import 'dart:io';
import 'package:postgres/postgres.dart';
import '../trading/market_data_history.dart';
import '../trading/market_data_retention.dart';
import '../trading/sync_run_recorder.dart';
import '../trading/tradable_assets_sync.dart';
import 'market_history_scheduler_config.dart';
/// Which scheduler stages ran during [MarketHistoryScheduler.runIfDue].
class MarketHistorySchedulerReport {
MarketHistorySchedulerReport({required this.ranStages});
final List<String> ranStages;
}
/// Market-history pipeline: universe backfill cleanup.
///
/// Before each run, stale or orphaned in-progress `market_data_sync_runs` rows
/// are aborted so a hung prior sync cannot block the worker.
class MarketHistoryScheduler {
MarketHistoryScheduler({
required Connection connection,
this.config = const MarketHistorySchedulerConfig(),
Future<void> Function(DateTime now)? runUniverse,
Future<void> Function(DateTime now)? runBackfill,
Future<void> Function(DateTime now)? runCleanup,
Future<bool> Function(DateTime now)? backfillIsDue,
}) : _connection = connection,
_recorder = SyncRunRecorder(connection),
_runUniverse = runUniverse,
_runBackfill = runBackfill,
_runCleanup = runCleanup,
_backfillIsDue = backfillIsDue;
final Connection _connection;
final SyncRunRecorder _recorder;
final MarketHistorySchedulerConfig config;
final Future<void> Function(DateTime now)? _runUniverse;
final Future<void> Function(DateTime now)? _runBackfill;
final Future<void> Function(DateTime now)? _runCleanup;
final Future<bool> Function(DateTime now)? _backfillIsDue;
bool _pipelineActive = false;
Future<MarketHistorySchedulerReport> runIfDue(DateTime now) async {
final DateTime tick = now.toUtc();
await _prepareForPipeline(tick);
if (_pipelineActive) {
return MarketHistorySchedulerReport(ranStages: <String>[]);
}
_pipelineActive = true;
try {
final List<String> ran = <String>[];
await _maybeRunStage(
tick: tick,
kind: TradableAssetsSync.kind,
cadenceHours: config.universeRefreshHours,
runner: _runUniverse,
ran: ran,
);
await _maybeRunStage(
tick: tick,
kind: MarketDataHistorySync.kind,
cadenceHours: config.historySyncHours,
runner: _runBackfill,
isDue: _backfillIsDue == null
? null
: (DateTime tick) => _isDue(
tick,
MarketDataHistorySync.kind,
config.historySyncHours,
slotGate: _backfillIsDue,
),
ran: ran,
);
await _maybeRunStage(
tick: tick,
kind: MarketDataRetention.kind,
cadenceHours: config.cleanupHours,
runner: _runCleanup,
ran: ran,
);
return MarketHistorySchedulerReport(ranStages: ran);
} finally {
_pipelineActive = false;
}
}
Future<void> _prepareForPipeline(DateTime tick) async {
final int staleAborted = await _recorder.abortStaleInProgressRuns(
now: tick,
olderThan: Duration(minutes: config.staleSyncRunMinutes),
);
if (_pipelineActive) {
if (staleAborted == 0) {
stderr.writeln(
'Market history sync still running; skipped overlapping scheduler tick',
);
return;
}
_pipelineActive = false;
stderr.writeln(
'Aborted $staleAborted stale market history sync run(s); starting new pipeline',
);
}
final int aborted = await _recorder.abortAllInProgressRuns(now: tick);
if (aborted > 0) {
stderr.writeln(
'Aborted $aborted orphaned market history sync run(s) before new pipeline',
);
}
}
Future<void> _maybeRunStage({
required DateTime tick,
required String kind,
required int cadenceHours,
required Future<void> Function(DateTime now)? runner,
Future<bool> Function(DateTime now)? isDue,
required List<String> ran,
}) async {
if (runner == null) {
return;
}
if (!await _isDue(tick, kind, cadenceHours, slotGate: isDue)) {
return;
}
await runner(tick);
ran.add(kind);
}
Future<bool> _isDue(
DateTime now,
String kind,
int cadenceHours, {
Future<bool> Function(DateTime now)? slotGate,
}) async {
final DateTime? last = await _lastFinishedAt(kind);
if (config.syncHourUtc != null) {
if (now.hour < config.syncHourUtc!) {
return false;
}
if (slotGate == null && last != null && _sameUtcDate(last, now)) {
return false;
}
}
if (slotGate != null) {
return slotGate(now);
}
if (last == null) {
return true;
}
return now.difference(last) >= Duration(hours: cadenceHours);
}
Future<DateTime?> _lastFinishedAt(String kind) async {
final Result result = await _connection.execute(
Sql.named(
'''
SELECT finished_at
FROM market_data_sync_runs
WHERE kind = @kind
AND finished_at IS NOT NULL
AND (error IS NULL OR error NOT LIKE 'aborted:%')
ORDER BY finished_at DESC
LIMIT 1
''',
),
parameters: <String, dynamic>{'kind': kind},
);
if (result.isEmpty) {
return null;
}
return (result.first[0]! as DateTime).toUtc();
}
static bool _sameUtcDate(DateTime a, DateTime b) {
final DateTime au = a.toUtc();
final DateTime bu = b.toUtc();
return au.year == bu.year && au.month == bu.month && au.day == bu.day;
}
}

View File

@ -0,0 +1,21 @@
/// Cadence for [MarketHistoryScheduler]. Backfill uses slot pending, not [historySyncHours].
class MarketHistorySchedulerConfig {
const MarketHistorySchedulerConfig({
this.universeRefreshHours = 24,
this.historySyncHours = 24,
this.cleanupHours = 24,
this.syncHourUtc,
this.staleSyncRunMinutes = 30,
});
final int universeRefreshHours;
final int historySyncHours;
final int cleanupHours;
/// In-progress sync rows older than this are aborted before a new pipeline run.
final int staleSyncRunMinutes;
/// When set, stages only run when `now.hour >= syncHourUtc` (UTC) and
/// no successful run has already finished on the same UTC calendar day.
final int? syncHourUtc;
}

View File

@ -1,23 +1,38 @@
import 'dart:async'; import 'dart:async';
import 'dart:io'; import 'dart:io';
import 'package:meta/meta.dart';
import '../pipeline/question_pipeline.dart'; import '../pipeline/question_pipeline.dart';
import '../trading/trading_orchestrator.dart'; import '../trading/trading_orchestrator.dart';
import 'market_history_scheduler.dart';
/// Runs [QuestionPipeline.runMaintenanceCycle] on a fixed interval, and /// Runs [QuestionPipeline.runMaintenanceCycle] on a fixed interval, and
/// optionally [TradingOrchestrator.runMaintenanceCycle] right after when /// optionally [TradingOrchestrator.runMaintenanceCycle] right after when
/// trading is enabled. /// trading is enabled.
///
/// When [marketHistoryScheduler] is set, [MarketHistoryScheduler.runIfDue]
/// runs at the start of each tick (before the question pipeline).
class QuestionBackgroundWorker { class QuestionBackgroundWorker {
QuestionBackgroundWorker({ QuestionBackgroundWorker({
required QuestionPipeline pipeline, required QuestionPipeline pipeline,
required Duration interval, required Duration interval,
TradingOrchestrator? tradingOrchestrator, TradingOrchestrator? tradingOrchestrator,
MarketHistoryScheduler? marketHistoryScheduler,
Future<void> Function()? tradingMaintenanceRunner,
DateTime Function()? clock,
}) : _pipeline = pipeline, }) : _pipeline = pipeline,
_interval = interval, _interval = interval,
_tradingOrchestrator = tradingOrchestrator; _tradingOrchestrator = tradingOrchestrator,
_marketHistoryScheduler = marketHistoryScheduler,
_tradingMaintenanceRunner = tradingMaintenanceRunner,
_clock = clock ?? DateTime.now;
final QuestionPipeline _pipeline; final QuestionPipeline _pipeline;
final TradingOrchestrator? _tradingOrchestrator; final TradingOrchestrator? _tradingOrchestrator;
final MarketHistoryScheduler? _marketHistoryScheduler;
final Future<void> Function()? _tradingMaintenanceRunner;
final DateTime Function() _clock;
final Duration _interval; final Duration _interval;
Timer? _timer; Timer? _timer;
bool _running = false; bool _running = false;
@ -28,7 +43,8 @@ class QuestionBackgroundWorker {
} }
stdout.writeln( stdout.writeln(
'Question background worker started (interval ${_interval.inSeconds}s, ' 'Question background worker started (interval ${_interval.inSeconds}s, '
'trading=${_tradingOrchestrator != null})', 'trading=${_tradingOrchestrator != null || _tradingMaintenanceRunner != null}, '
'marketHistory=${_marketHistoryScheduler != null})',
); );
_timer = Timer.periodic(_interval, (_) => _tick()); _timer = Timer.periodic(_interval, (_) => _tick());
unawaited(_tick()); unawaited(_tick());
@ -40,17 +56,33 @@ class QuestionBackgroundWorker {
_pipeline.close(); _pipeline.close();
} }
@visibleForTesting
Future<void> runTickForTest() => _tick();
Future<void> _tick() async { Future<void> _tick() async {
if (_running) { if (_running) {
return; return;
} }
_running = true; _running = true;
if (_marketHistoryScheduler != null) {
try {
await _marketHistoryScheduler.runIfDue(_clock());
} catch (e, st) {
stderr.writeln('Market history scheduler tick failed: $e\n$st');
}
}
try { try {
await _pipeline.runMaintenanceCycle(); await _pipeline.runMaintenanceCycle();
} catch (e, st) { } catch (e, st) {
stderr.writeln('Question background worker tick failed: $e\n$st'); stderr.writeln('Question background worker tick failed: $e\n$st');
} }
if (_tradingOrchestrator != null) { if (_tradingMaintenanceRunner != null) {
try {
await _tradingMaintenanceRunner();
} catch (e, st) {
stderr.writeln('Trading orchestrator tick failed: $e\n$st');
}
} else if (_tradingOrchestrator != null) {
try { try {
await _tradingOrchestrator.runMaintenanceCycle(); await _tradingOrchestrator.runMaintenanceCycle();
} catch (e, st) { } catch (e, st) {

View File

@ -0,0 +1,53 @@
-- 005_market_history.sql
--
-- Adds the rolling 7-day market-history substrate:
-- * timeframe column + idempotent unique observation key on
-- market_data_snapshots
-- * tradable_assets cache for the daily Alpaca asset universe
-- * market_data_sync_runs audit table
-- See TODO.md §1.
ALTER TABLE market_data_snapshots
ADD COLUMN IF NOT EXISTS timeframe TEXT NOT NULL DEFAULT 'tick';
ALTER TABLE market_data_snapshots
DROP CONSTRAINT IF EXISTS market_data_snapshots_unique_obs;
ALTER TABLE market_data_snapshots
ADD CONSTRAINT market_data_snapshots_unique_obs
UNIQUE (symbol, metric, timeframe, as_of);
CREATE INDEX IF NOT EXISTS market_data_snapshots_asof_idx
ON market_data_snapshots (as_of DESC);
CREATE TABLE IF NOT EXISTS tradable_assets (
symbol TEXT PRIMARY KEY,
asset_class TEXT NOT NULL DEFAULT 'us_equity',
exchange TEXT,
name TEXT,
tradable BOOLEAN NOT NULL DEFAULT true,
fractionable BOOLEAN NOT NULL DEFAULT false,
status TEXT NOT NULL DEFAULT 'active',
raw JSONB,
refreshed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS tradable_assets_status_idx
ON tradable_assets (status, tradable);
CREATE TABLE IF NOT EXISTS market_data_sync_runs (
id BIGSERIAL PRIMARY KEY,
kind TEXT NOT NULL,
started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
finished_at TIMESTAMPTZ,
rows_written INTEGER NOT NULL DEFAULT 0,
rows_removed INTEGER NOT NULL DEFAULT 0,
error TEXT
);
CREATE INDEX IF NOT EXISTS market_data_sync_runs_kind_started_idx
ON market_data_sync_runs (kind, started_at DESC);
-- The market_data_archive table is intentionally deferred to TODO.md
-- section 4.2 (Phase 2 archive mode) and will be added when
-- MarketDataRetention.runArchiveAndCleanup is implemented.

View File

@ -0,0 +1,21 @@
-- 006_market_data_archive.sql — Phase 2 archive-before-delete (TODO §4.2).
CREATE TABLE IF NOT EXISTS market_data_archive (
id BIGSERIAL PRIMARY KEY,
symbol TEXT NOT NULL,
asset_class TEXT NOT NULL DEFAULT 'us_equity',
feed TEXT NOT NULL DEFAULT 'iex',
metric TEXT NOT NULL,
timeframe TEXT NOT NULL DEFAULT 'tick',
price NUMERIC,
volume NUMERIC,
as_of TIMESTAMPTZ NOT NULL,
raw JSONB,
archived_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS market_data_archive_asof_idx
ON market_data_archive (as_of DESC);
CREATE INDEX IF NOT EXISTS market_data_archive_symbol_asof_idx
ON market_data_archive (symbol, as_of DESC);

View File

@ -0,0 +1,3 @@
-- Server-side question metadata (e.g. guess_symbol for obfuscated trading questions).
ALTER TABLE questions
ADD COLUMN IF NOT EXISTS metadata JSONB NOT NULL DEFAULT '{}';

View File

@ -0,0 +1,25 @@
-- 008_market_history_four_hour.sql
--
-- Market history uses Alpaca 4Hour bars (six UTC slots per day).
-- Drops legacy 1Day history rows, constrains timeframes, adds query index,
-- and records slots_synced on backfill audit rows.
DELETE FROM market_data_snapshots
WHERE metric = 'bar' AND timeframe = '1Day';
DELETE FROM market_data_archive
WHERE metric = 'bar' AND timeframe = '1Day';
ALTER TABLE market_data_snapshots
DROP CONSTRAINT IF EXISTS market_data_snapshots_timeframe_check;
ALTER TABLE market_data_snapshots
ADD CONSTRAINT market_data_snapshots_timeframe_check
CHECK (timeframe IN ('tick', '1Min', '1Hour', '4Hour', '1Day'));
CREATE INDEX IF NOT EXISTS market_data_snapshots_bar_4h_idx
ON market_data_snapshots (symbol, as_of DESC)
WHERE metric = 'bar' AND timeframe = '4Hour';
ALTER TABLE market_data_sync_runs
ADD COLUMN IF NOT EXISTS slots_synced INTEGER NOT NULL DEFAULT 0;

View File

@ -0,0 +1,4 @@
-- Per-slot backfill audit: UTC slot start + symbols requested from Alpaca.
ALTER TABLE market_data_sync_runs
ADD COLUMN IF NOT EXISTS backfill_items JSONB;

View File

@ -0,0 +1,154 @@
import 'dart:convert';
import 'package:cyberhybridhub_server/alpaca/alpaca_assets_client.dart';
import 'package:cyberhybridhub_server/alpaca/alpaca_env.dart';
import 'package:cyberhybridhub_server/alpaca/alpaca_models.dart';
import 'package:http/http.dart' as http;
import 'package:test/test.dart';
import '../helpers/fixture_loader.dart';
import '../helpers/mock_http_client.dart';
void main() {
late FixtureLoader fixtures;
late AlpacaEnv env;
setUp(() {
fixtures = FixtureLoader();
env = AlpacaEnv.fromMap(<String, String>{
'ALPACA_API_KEY_ID': 'test-key',
'ALPACA_API_SECRET_KEY': 'test-secret',
'ALPACA_TRADING_BASE_URL': 'https://paper-api.alpaca.markets',
'ALPACA_DATA_BASE_URL': 'https://data.alpaca.markets',
});
});
test('listActiveTradable issues GET with auth headers and query', () async {
final String body = await fixtures.loadString('alpaca_assets_active.json');
final MockHttpClient mock = MockHttpClient()
..whenGet(
'/v2/assets',
http.Response(body, 200, headers: <String, String>{
'content-type': 'application/json',
}),
);
final AlpacaAssetsClient client =
AlpacaAssetsClient(env: env, httpClient: mock);
await client.listActiveTradable();
expect(mock.requests, hasLength(1));
final http.BaseRequest req = mock.requests.single;
expect(req.method, 'GET');
expect(
req.url.toString(),
startsWith('https://paper-api.alpaca.markets/v2/assets'),
);
expect(req.url.queryParameters['status'], 'active');
expect(req.url.queryParameters['asset_class'], 'us_equity');
expect(req.headers['APCA-API-KEY-ID'], 'test-key');
expect(req.headers['APCA-API-SECRET-KEY'], 'test-secret');
});
test('listActiveTradable parses the fixture into AlpacaAsset rows',
() async {
final String body = await fixtures.loadString('alpaca_assets_active.json');
final MockHttpClient mock = MockHttpClient()
..whenGet(
'/v2/assets',
http.Response(body, 200, headers: <String, String>{
'content-type': 'application/json',
}),
);
final AlpacaAssetsClient client =
AlpacaAssetsClient(env: env, httpClient: mock);
final List<AlpacaAsset> assets = await client.listActiveTradable();
expect(assets, hasLength(5));
final AlpacaAsset aapl =
assets.firstWhere((AlpacaAsset a) => a.symbol == 'AAPL');
expect(aapl.assetClass, 'us_equity');
expect(aapl.exchange, 'NASDAQ');
expect(aapl.name, 'Apple Inc. Common Stock');
expect(aapl.status, 'active');
expect(aapl.tradable, isTrue);
expect(aapl.fractionable, isTrue);
final AlpacaAsset pinkSheet =
assets.firstWhere((AlpacaAsset a) => a.symbol == 'PNKZZ');
expect(pinkSheet.tradable, isFalse);
expect(pinkSheet.fractionable, isFalse);
final AlpacaAsset brk =
assets.firstWhere((AlpacaAsset a) => a.symbol == 'BRK.B');
expect(brk.tradable, isTrue);
expect(brk.fractionable, isFalse);
});
test('401 unauthorized throws AlpacaAssetsException with code + body',
() async {
final MockHttpClient mock = MockHttpClient()
..whenGet(
'/v2/assets',
http.Response(
jsonEncode(<String, dynamic>{'message': 'forbidden.'}),
401,
headers: <String, String>{'content-type': 'application/json'},
),
);
final AlpacaAssetsClient client =
AlpacaAssetsClient(env: env, httpClient: mock);
await expectLater(
client.listActiveTradable(),
throwsA(
isA<AlpacaAssetsException>()
.having((AlpacaAssetsException e) => e.message, 'message',
contains('401'))
.having((AlpacaAssetsException e) => e.message, 'body',
contains('forbidden')),
),
);
});
test('500 server error throws AlpacaAssetsException with code + body',
() async {
final MockHttpClient mock = MockHttpClient()
..whenGet(
'/v2/assets',
http.Response('upstream exploded', 500),
);
final AlpacaAssetsClient client =
AlpacaAssetsClient(env: env, httpClient: mock);
await expectLater(
client.listActiveTradable(),
throwsA(
isA<AlpacaAssetsException>()
.having((AlpacaAssetsException e) => e.message, 'message',
contains('500'))
.having((AlpacaAssetsException e) => e.message, 'body',
contains('upstream exploded')),
),
);
});
test('empty array response returns [] and does not throw', () async {
final MockHttpClient mock = MockHttpClient()
..whenGet(
'/v2/assets',
http.Response('[]', 200, headers: <String, String>{
'content-type': 'application/json',
}),
);
final AlpacaAssetsClient client =
AlpacaAssetsClient(env: env, httpClient: mock);
final List<AlpacaAsset> assets = await client.listActiveTradable();
expect(assets, isEmpty);
});
}

View File

@ -0,0 +1,64 @@
@Tags(['alpaca'])
library;
import 'package:cyberhybridhub_server/alpaca/alpaca_assets_client.dart';
import 'package:cyberhybridhub_server/alpaca/alpaca_env.dart';
import 'package:cyberhybridhub_server/alpaca/alpaca_models.dart';
import 'package:dotenv/dotenv.dart';
import 'package:test/test.dart';
void main() {
late AlpacaEnv env;
AlpacaAssetsClient? client;
setUpAll(() {
final DotEnv dotenv = DotEnv(includePlatformEnvironment: true)
..load(['.env']);
const List<String> alpacaKeys = <String>[
'ALPACA_API_KEY_ID',
'ALPACA_API_SECRET_KEY',
'ALPACA_TRADING_BASE_URL',
'ALPACA_DATA_BASE_URL',
'ALPACA_DATA_FEED',
'ALPACA_ALLOW_LIVE',
];
final Map<String, String> envMap = <String, String>{
for (final String key in alpacaKeys)
if (dotenv[key] != null && dotenv[key]!.isNotEmpty) key: dotenv[key]!,
};
env = AlpacaEnv.fromMap(envMap);
if (env.hasCredentials) {
client = AlpacaAssetsClient(env: env);
}
});
tearDownAll(() {
client?.close();
});
test('live listActiveTradable returns >100 us_equity assets', () async {
if (!env.hasCredentials) {
markTestSkipped(
'Set ALPACA_API_KEY_ID and ALPACA_API_SECRET_KEY in server/.env',
);
return;
}
final List<AlpacaAsset> assets = await client!.listActiveTradable();
// Alpaca's live US-equity catalog has thousands of entries; >100 is a
// safe lower bound that catches "we got an empty/auth-error payload"
// without being brittle.
expect(assets.length, greaterThan(100));
expect(
assets.every((AlpacaAsset a) => a.assetClass == 'us_equity'),
isTrue,
reason: 'asset_class=us_equity filter should be enforced server-side',
);
expect(
assets.every((AlpacaAsset a) => a.status == 'active'),
isTrue,
reason: 'status=active filter should be enforced server-side',
);
});
}

View File

@ -1,5 +1,7 @@
import 'package:cyberhybridhub_server/alpaca/alpaca_env.dart'; import 'package:cyberhybridhub_server/alpaca/alpaca_env.dart';
import 'package:cyberhybridhub_server/alpaca/alpaca_market_data_client.dart'; import 'package:cyberhybridhub_server/alpaca/alpaca_market_data_client.dart';
import 'package:cyberhybridhub_server/alpaca/alpaca_models.dart';
import 'package:http/http.dart' as http;
import 'package:test/test.dart'; import 'package:test/test.dart';
import '../helpers/fixture_loader.dart'; import '../helpers/fixture_loader.dart';
@ -53,4 +55,105 @@ void main() {
expect(mock.requests.single.url.queryParameters['symbols'], 'SPY'); expect(mock.requests.single.url.queryParameters['symbols'], 'SPY');
expect(mock.requests.single.url.queryParameters['timeframe'], '1Day'); expect(mock.requests.single.url.queryParameters['timeframe'], '1Day');
}); });
group('getBarsRange', () {
final DateTime start = DateTime.utc(2026, 5, 20);
final DateTime end = DateTime.utc(2026, 5, 27);
test('builds query string with start, end, timeframe, feed, symbols, limit',
() async {
final Map<String, dynamic> barsJson =
await fixtures.loadJson('alpaca_daily_bars.json');
final MockHttpClient mock = MockHttpClient()
..whenGetJson('/bars', barsJson);
final AlpacaMarketDataClient client =
AlpacaMarketDataClient(env: env, httpClient: mock);
await client.getBarsRange(
symbols: <String>['SPY', 'AAPL'],
timeframe: '1Day',
start: start,
end: end,
);
expect(mock.requests, hasLength(1));
final Uri url = mock.requests.single.url;
expect(url.queryParameters['symbols'], 'SPY,AAPL');
expect(url.queryParameters['timeframe'], '1Day');
expect(url.queryParameters['feed'], 'iex');
expect(url.queryParameters['limit'], '10000');
expect(url.queryParameters['start'], '2026-05-20T00:00:00Z');
expect(url.queryParameters['end'], '2026-05-27T00:00:00Z');
});
test('follows pagination and merges bars per symbol', () async {
final Map<String, dynamic> page1 =
await fixtures.loadJson('alpaca_bars_7d_multi_page1.json');
final Map<String, dynamic> page2 =
await fixtures.loadJson('alpaca_bars_7d_multi_page2.json');
final MockHttpClient mock = MockHttpClient()
..whenGetQueuedJson('/bars', page1)
..whenGetQueuedJson('/bars', page2);
final AlpacaMarketDataClient client =
AlpacaMarketDataClient(env: env, httpClient: mock);
final AlpacaBarsResponse merged = await client.getBarsRange(
symbols: <String>['SPY', 'AAPL'],
timeframe: '1Day',
start: start,
end: end,
);
expect(mock.requests, hasLength(2));
expect(mock.requests[1].url.queryParameters['page_token'], 'abc');
expect(merged.barsBySymbol['SPY'], hasLength(3));
expect(merged.barsBySymbol['AAPL'], hasLength(3));
});
test('stops after maxPages even when next_page_token is present', () async {
final Map<String, dynamic> page1 =
await fixtures.loadJson('alpaca_bars_7d_multi_page1.json');
final MockHttpClient mock = MockHttpClient()
..whenGetQueuedJson('/bars', page1);
final AlpacaMarketDataClient client =
AlpacaMarketDataClient(env: env, httpClient: mock);
await client.getBarsRange(
symbols: <String>['SPY'],
timeframe: '1Day',
start: start,
end: end,
maxPages: 1,
);
expect(mock.requests, hasLength(1));
});
test('429 throws AlpacaMarketDataException containing rate', () async {
final MockHttpClient mock = MockHttpClient()
..whenGet(
'/bars',
http.Response('rate limit exceeded', 429),
);
final AlpacaMarketDataClient client =
AlpacaMarketDataClient(env: env, httpClient: mock);
await expectLater(
client.getBarsRange(
symbols: <String>['SPY'],
timeframe: '1Day',
start: start,
end: end,
),
throwsA(
isA<AlpacaMarketDataException>().having(
(AlpacaMarketDataException e) => e.message,
'message',
contains('rate'),
),
),
);
});
});
} }

View File

@ -0,0 +1,62 @@
@Tags(['alpaca'])
library;
import 'package:cyberhybridhub_server/alpaca/alpaca_env.dart';
import 'package:cyberhybridhub_server/alpaca/alpaca_market_data_client.dart';
import 'package:cyberhybridhub_server/alpaca/alpaca_models.dart';
import 'package:dotenv/dotenv.dart';
import 'package:test/test.dart';
void main() {
late AlpacaEnv env;
AlpacaMarketDataClient? client;
setUpAll(() {
final DotEnv dotenv = DotEnv(includePlatformEnvironment: true)
..load(['.env']);
const List<String> alpacaKeys = <String>[
'ALPACA_API_KEY_ID',
'ALPACA_API_SECRET_KEY',
'ALPACA_TRADING_BASE_URL',
'ALPACA_DATA_BASE_URL',
'ALPACA_DATA_FEED',
'ALPACA_ALLOW_LIVE',
];
final Map<String, String> envMap = <String, String>{
for (final String key in alpacaKeys)
if (dotenv[key] != null && dotenv[key]!.isNotEmpty) key: dotenv[key]!,
};
env = AlpacaEnv.fromMap(envMap);
if (env.hasCredentials) {
client = AlpacaMarketDataClient(env: env);
}
});
tearDownAll(() {
client?.close();
});
test('live getBarsRange for SPY returns at least 3 daily bars in 7d window',
() async {
if (!env.hasCredentials) {
markTestSkipped(
'Set ALPACA_API_KEY_ID and ALPACA_API_SECRET_KEY in server/.env',
);
return;
}
final DateTime end = DateTime.now().toUtc();
final DateTime start = end.subtract(const Duration(days: 7));
final AlpacaBarsResponse response = await client!.getBarsRange(
symbols: <String>['SPY'],
timeframe: '1Day',
start: start,
end: end,
);
final List<AlpacaBar>? bars = response.barsBySymbol['SPY'];
expect(bars, isNotNull);
expect(bars!.length, greaterThanOrEqualTo(3));
});
}

View File

@ -0,0 +1,77 @@
import 'package:cyberhybridhub_server/market_history_env.dart';
import 'package:test/test.dart';
void main() {
group('MarketHistoryEnv.fromMap', () {
test('defaults when env keys are empty', () {
final MarketHistoryEnv env = MarketHistoryEnv.fromMap(<String, String>{});
expect(env.syncEnabled, isFalse);
expect(env.windowDays, 7);
expect(env.retentionDays, 7);
expect(env.archiveEnabled, isFalse);
expect(env.universeRefreshHours, 24);
expect(env.historySyncHours, 24);
expect(env.cleanupHours, 24);
expect(env.syncHourUtc, isNull);
expect(env.historySyncBatchSize, 50);
expect(env.historySyncMaxSymbols, 2000);
expect(env.minBarsForGuess, 5);
expect(env.guessCooldownHours, 24);
expect(env.apiRequestsPerMinute, 200);
expect(env.staleSyncRunMinutes, 30);
});
test('MARKET_HISTORY_API_REQUESTS_PER_MINUTE overrides default', () {
final MarketHistoryEnv env = MarketHistoryEnv.fromMap(<String, String>{
'MARKET_HISTORY_API_REQUESTS_PER_MINUTE': '120',
});
expect(env.apiRequestsPerMinute, 120);
});
test('MARKET_HISTORY_SYNC_ENABLED=true without trading throws', () {
final MarketHistoryEnv env = MarketHistoryEnv.fromMap(<String, String>{
'MARKET_HISTORY_SYNC_ENABLED': 'true',
});
expect(
() => env.assertConsistent(tradingEnabled: false),
throwsStateError,
);
});
test('MARKET_HISTORY_WINDOW_DAYS=0 throws', () {
expect(
() => MarketHistoryEnv.fromMap(<String, String>{
'MARKET_HISTORY_WINDOW_DAYS': '0',
}),
throwsArgumentError,
);
});
test('MARKET_HISTORY_WINDOW_DAYS negative throws', () {
expect(
() => MarketHistoryEnv.fromMap(<String, String>{
'MARKET_HISTORY_WINDOW_DAYS': '-3',
}),
throwsArgumentError,
);
});
test('MARKET_HISTORY_SYNC_HOUR_UTC=24 throws', () {
expect(
() => MarketHistoryEnv.fromMap(<String, String>{
'MARKET_HISTORY_SYNC_HOUR_UTC': '24',
}),
throwsArgumentError,
);
});
test('MARKET_HISTORY_SYNC_HOUR_UTC=10 is accepted', () {
final MarketHistoryEnv env = MarketHistoryEnv.fromMap(<String, String>{
'MARKET_HISTORY_SYNC_HOUR_UTC': '10',
});
expect(env.syncHourUtc, 10);
});
});
}

View File

@ -0,0 +1,67 @@
[
{
"id": "b0b6dd9d-8b9b-48a9-ba46-b9d54906e415",
"class": "us_equity",
"exchange": "NASDAQ",
"symbol": "AAPL",
"name": "Apple Inc. Common Stock",
"status": "active",
"tradable": true,
"marginable": true,
"shortable": true,
"easy_to_borrow": true,
"fractionable": true
},
{
"id": "8ccae427-5dd0-45b3-b5fe-7ba5e422c766",
"class": "us_equity",
"exchange": "NASDAQ",
"symbol": "MSFT",
"name": "Microsoft Corporation Common Stock",
"status": "active",
"tradable": true,
"marginable": true,
"shortable": true,
"easy_to_borrow": true,
"fractionable": true
},
{
"id": "a8ab8ab8-7777-4444-aaaa-cccccccccccc",
"class": "us_equity",
"exchange": "ARCA",
"symbol": "SPY",
"name": "SPDR S&P 500 ETF Trust",
"status": "active",
"tradable": true,
"marginable": true,
"shortable": true,
"easy_to_borrow": true,
"fractionable": true
},
{
"id": "11111111-2222-3333-4444-555555555555",
"class": "us_equity",
"exchange": "OTC",
"symbol": "PNKZZ",
"name": "Pink Sheet Test Co.",
"status": "active",
"tradable": false,
"marginable": false,
"shortable": false,
"easy_to_borrow": false,
"fractionable": false
},
{
"id": "66666666-7777-8888-9999-000000000000",
"class": "us_equity",
"exchange": "NYSE",
"symbol": "BRK.B",
"name": "Berkshire Hathaway Inc. Class B",
"status": "active",
"tradable": true,
"marginable": true,
"shortable": true,
"easy_to_borrow": true,
"fractionable": false
}
]

View File

@ -0,0 +1,29 @@
{
"bars": {
"SPY": [
{ "t": "2026-05-25T12:00:00Z", "o": 490, "h": 492, "l": 488, "c": 491, "v": 4000000 },
{ "t": "2026-05-25T16:00:00Z", "o": 491, "h": 493, "l": 489, "c": 492, "v": 4100000 },
{ "t": "2026-05-25T20:00:00Z", "o": 492, "h": 494, "l": 490, "c": 493, "v": 4200000 },
{ "t": "2026-05-26T00:00:00Z", "o": 493, "h": 495, "l": 491, "c": 494, "v": 4300000 },
{ "t": "2026-05-26T04:00:00Z", "o": 494, "h": 496, "l": 492, "c": 495, "v": 4400000 },
{ "t": "2026-05-26T08:00:00Z", "o": 495, "h": 497, "l": 493, "c": 496, "v": 4500000 }
],
"AAPL": [
{ "t": "2026-05-25T12:00:00Z", "o": 180, "h": 182, "l": 179, "c": 181.5, "v": 5000000 },
{ "t": "2026-05-25T16:00:00Z", "o": 181.5, "h": 183, "l": 180, "c": 182, "v": 5100000 },
{ "t": "2026-05-25T20:00:00Z", "o": 182, "h": 184, "l": 181, "c": 183, "v": 5200000 },
{ "t": "2026-05-26T00:00:00Z", "o": 183, "h": 185, "l": 182, "c": 184, "v": 5300000 },
{ "t": "2026-05-26T04:00:00Z", "o": 184, "h": 186, "l": 183, "c": 185, "v": 5400000 },
{ "t": "2026-05-26T08:00:00Z", "o": 185, "h": 187, "l": 184, "c": 186, "v": 5500000 }
],
"MSFT": [
{ "t": "2026-05-25T12:00:00Z", "o": 410, "h": 412, "l": 408, "c": 411, "v": 3000000 },
{ "t": "2026-05-25T16:00:00Z", "o": 411, "h": 413, "l": 409, "c": 412, "v": 3100000 },
{ "t": "2026-05-25T20:00:00Z", "o": 412, "h": 414, "l": 410, "c": 413, "v": 3200000 },
{ "t": "2026-05-26T00:00:00Z", "o": 413, "h": 415, "l": 411, "c": 414, "v": 3300000 },
{ "t": "2026-05-26T04:00:00Z", "o": 414, "h": 416, "l": 412, "c": 415, "v": 3400000 },
{ "t": "2026-05-26T08:00:00Z", "o": 415, "h": 417, "l": 413, "c": 416, "v": 3500000 }
]
},
"next_page_token": null
}

View File

@ -0,0 +1,32 @@
{
"bars": {
"SPY": [
{ "t": "2026-05-20T04:00:00Z", "o": 490, "h": 492, "l": 488, "c": 491, "v": 40000000 },
{ "t": "2026-05-21T04:00:00Z", "o": 491, "h": 495, "l": 489, "c": 494, "v": 41000000 },
{ "t": "2026-05-22T04:00:00Z", "o": 494, "h": 498, "l": 492, "c": 497, "v": 42000000 },
{ "t": "2026-05-23T04:00:00Z", "o": 497, "h": 500, "l": 495, "c": 499, "v": 43000000 },
{ "t": "2026-05-24T04:00:00Z", "o": 499, "h": 501, "l": 496, "c": 500, "v": 44000000 },
{ "t": "2026-05-25T04:00:00Z", "o": 500, "h": 502, "l": 497, "c": 501, "v": 45000000 },
{ "t": "2026-05-26T04:00:00Z", "o": 501, "h": 504, "l": 499, "c": 503, "v": 46000000 }
],
"AAPL": [
{ "t": "2026-05-20T04:00:00Z", "o": 180, "h": 182, "l": 179, "c": 181.5, "v": 50000000 },
{ "t": "2026-05-21T04:00:00Z", "o": 181.5, "h": 184, "l": 180.5, "c": 183, "v": 48000000 },
{ "t": "2026-05-22T04:00:00Z", "o": 183, "h": 185, "l": 182, "c": 184.5, "v": 46000000 },
{ "t": "2026-05-23T04:00:00Z", "o": 184.5, "h": 186, "l": 183, "c": 185, "v": 47000000 },
{ "t": "2026-05-24T04:00:00Z", "o": 185, "h": 187, "l": 184, "c": 186, "v": 48000000 },
{ "t": "2026-05-25T04:00:00Z", "o": 186, "h": 188, "l": 185, "c": 187, "v": 49000000 },
{ "t": "2026-05-26T04:00:00Z", "o": 187, "h": 189, "l": 186, "c": 188, "v": 50000000 }
],
"MSFT": [
{ "t": "2026-05-20T04:00:00Z", "o": 410, "h": 412, "l": 408, "c": 411, "v": 30000000 },
{ "t": "2026-05-21T04:00:00Z", "o": 411, "h": 414, "l": 409, "c": 413, "v": 31000000 },
{ "t": "2026-05-22T04:00:00Z", "o": 413, "h": 416, "l": 411, "c": 415, "v": 32000000 },
{ "t": "2026-05-23T04:00:00Z", "o": 415, "h": 417, "l": 412, "c": 416, "v": 33000000 },
{ "t": "2026-05-24T04:00:00Z", "o": 416, "h": 418, "l": 413, "c": 417, "v": 34000000 },
{ "t": "2026-05-25T04:00:00Z", "o": 417, "h": 419, "l": 414, "c": 418, "v": 35000000 },
{ "t": "2026-05-26T04:00:00Z", "o": 418, "h": 421, "l": 416, "c": 420, "v": 36000000 }
]
},
"next_page_token": null
}

View File

@ -0,0 +1,33 @@
{
"bars": {
"SPY": [
{
"t": "2026-05-20T04:00:00Z",
"o": 490.0,
"h": 492.0,
"l": 488.0,
"c": 491.0,
"v": 40000000
},
{
"t": "2026-05-21T04:00:00Z",
"o": 491.0,
"h": 495.0,
"l": 489.0,
"c": 494.0,
"v": 41000000
}
],
"AAPL": [
{
"t": "2026-05-20T04:00:00Z",
"o": 180.0,
"h": 182.0,
"l": 179.0,
"c": 181.5,
"v": 50000000
}
]
},
"next_page_token": "abc"
}

View File

@ -0,0 +1,33 @@
{
"bars": {
"SPY": [
{
"t": "2026-05-22T04:00:00Z",
"o": 494.0,
"h": 498.0,
"l": 492.0,
"c": 497.0,
"v": 42000000
}
],
"AAPL": [
{
"t": "2026-05-21T04:00:00Z",
"o": 181.5,
"h": 184.0,
"l": 180.5,
"c": 183.0,
"v": 48000000
},
{
"t": "2026-05-22T04:00:00Z",
"o": 183.0,
"h": 185.0,
"l": 182.0,
"c": 184.5,
"v": 46000000
}
]
},
"next_page_token": null
}

View File

@ -0,0 +1,142 @@
import 'package:cyberhybridhub_server/trading/backfill_sync_item.dart';
import 'package:cyberhybridhub_server/trading/market_history_admin_logic.dart';
AdminSyncRunRecord adminSyncRun({
required int id,
required String kind,
required DateTime startedAt,
DateTime? finishedAt,
int rowsWritten = 0,
int rowsRemoved = 0,
int slotsSynced = 0,
List<BackfillSyncItem>? backfillItems,
String? error,
}) {
return AdminSyncRunRecord(
id: id,
kind: kind,
startedAt: startedAt,
finishedAt: finishedAt,
rowsWritten: rowsWritten,
rowsRemoved: rowsRemoved,
slotsSynced: slotsSynced,
backfillItems: backfillItems ?? const <BackfillSyncItem>[],
error: error,
);
}
List<AdminSyncRunRecord> fixtureAllSuccessRecentFirst(DateTime base) {
return <AdminSyncRunRecord>[
adminSyncRun(
id: 3,
kind: 'cleanup',
startedAt: base,
finishedAt: base.add(const Duration(minutes: 1)),
rowsRemoved: 100,
),
adminSyncRun(
id: 2,
kind: 'backfill',
startedAt: base.subtract(const Duration(hours: 1)),
finishedAt: base.subtract(const Duration(minutes: 59)),
rowsWritten: 500,
),
adminSyncRun(
id: 1,
kind: 'universe',
startedAt: base.subtract(const Duration(hours: 2)),
finishedAt: base.subtract(const Duration(hours: 1, minutes: 59)),
rowsWritten: 8000,
),
];
}
List<AdminSyncRunRecord> fixtureRateLimitUnresolved(DateTime base) {
return <AdminSyncRunRecord>[
adminSyncRun(
id: 10,
kind: 'backfill',
startedAt: base,
finishedAt: base.add(const Duration(minutes: 2)),
rowsWritten: 0,
error: 'AlpacaMarketDataException: rate limited: 429',
),
];
}
List<AdminSyncRunRecord> fixtureFailedThenSuccessSameKind(DateTime base) {
return <AdminSyncRunRecord>[
adminSyncRun(
id: 21,
kind: 'backfill',
startedAt: base.subtract(const Duration(hours: 2)),
finishedAt: base.subtract(const Duration(hours: 1, minutes: 58)),
error: '429',
),
adminSyncRun(
id: 22,
kind: 'backfill',
startedAt: base,
finishedAt: base.add(const Duration(minutes: 1)),
rowsWritten: 1200,
),
];
}
List<AdminSyncRunRecord> fixturePartialBackfillError(DateTime base) {
return <AdminSyncRunRecord>[
adminSyncRun(
id: 30,
kind: 'backfill',
startedAt: base,
finishedAt: base.add(const Duration(minutes: 5)),
rowsWritten: 12000,
error: 'batch MSFT,AAPL: server 500',
),
];
}
List<AdminSyncRunRecord> fixtureInProgressStale(DateTime base) {
return <AdminSyncRunRecord>[
adminSyncRun(
id: 40,
kind: 'cleanup',
startedAt: base.subtract(const Duration(hours: 2)),
finishedAt: null,
),
];
}
List<AdminSyncRunRecord> fixtureMixedKindsMixedOutcomes(DateTime base) {
return <AdminSyncRunRecord>[
adminSyncRun(
id: 50,
kind: 'cleanup',
startedAt: base,
finishedAt: base.add(const Duration(minutes: 1)),
rowsRemoved: 4200,
),
adminSyncRun(
id: 51,
kind: 'backfill',
startedAt: base.subtract(const Duration(hours: 1)),
finishedAt: base.subtract(const Duration(minutes: 55)),
rowsWritten: 8000,
error: 'partial batch failure',
),
adminSyncRun(
id: 52,
kind: 'universe',
startedAt: base.subtract(const Duration(hours: 3)),
finishedAt: base.subtract(const Duration(hours: 2, minutes: 58)),
rowsWritten: 8200,
),
adminSyncRun(
id: 53,
kind: 'backfill',
startedAt: base.subtract(const Duration(hours: 5)),
finishedAt: base.subtract(const Duration(hours: 4, minutes: 58)),
error: '429',
),
];
}

View File

@ -8,6 +8,7 @@ class MockHttpClient extends http.BaseClient {
: _responses = responses ?? <String, _MatchedResponse>{}; : _responses = responses ?? <String, _MatchedResponse>{};
final Map<String, _MatchedResponse> _responses; final Map<String, _MatchedResponse> _responses;
final List<_QueuedGet> _getQueue = <_QueuedGet>[];
final List<http.BaseRequest> requests = <http.BaseRequest>[]; final List<http.BaseRequest> requests = <http.BaseRequest>[];
/// Captured request bodies indexed by request order in [requests]. /// Captured request bodies indexed by request order in [requests].
@ -27,6 +28,53 @@ class MockHttpClient extends http.BaseClient {
); );
} }
/// Returns [response] for the next GET whose path ends with [pathSuffix].
///
/// Useful for paginated endpoints where the same path is hit multiple
/// times with different `page_token` query values.
void whenGetQueued(String pathSuffix, http.Response response) {
_getQueue.add(_QueuedGet(pathSuffix, response));
}
void whenGetQueuedJson(String pathSuffix, Map<String, dynamic> body,
{int statusCode = 200}) {
whenGetQueued(
pathSuffix,
http.Response(jsonEncode(body), statusCode, headers: <String, String>{
'content-type': 'application/json',
}),
);
}
/// Returns [response] when [predicate] matches the request URI.
void whenGetWhere(
String pathSuffix,
bool Function(Uri uri) predicate,
http.Response response,
) {
_responses['GET:$pathSuffix:${_responses.length}'] = _MatchedResponse(
method: 'GET',
response: response,
pathSuffix: pathSuffix,
uriPredicate: predicate,
);
}
void whenGetWhereJson(
String pathSuffix,
bool Function(Uri uri) predicate,
Map<String, dynamic> body, {
int statusCode = 200,
}) {
whenGetWhere(
pathSuffix,
predicate,
http.Response(jsonEncode(body), statusCode, headers: <String, String>{
'content-type': 'application/json',
}),
);
}
void whenPost(String pathSuffix, http.Response response) { void whenPost(String pathSuffix, http.Response response) {
_responses['POST:$pathSuffix'] = _responses['POST:$pathSuffix'] =
_MatchedResponse(method: 'POST', response: response); _MatchedResponse(method: 'POST', response: response);
@ -50,13 +98,35 @@ class MockHttpClient extends http.BaseClient {
final String path = request.url.path; final String path = request.url.path;
final String method = request.method.toUpperCase(); final String method = request.method.toUpperCase();
if (method == 'GET' && _getQueue.isNotEmpty) {
final int idx = _getQueue.indexWhere(
(_QueuedGet q) => path.endsWith(q.pathSuffix),
);
if (idx >= 0) {
final _QueuedGet queued = _getQueue.removeAt(idx);
return http.StreamedResponse(
Stream<List<int>>.value(utf8.encode(queued.response.body)),
queued.response.statusCode,
headers: queued.response.headers,
request: request,
);
}
}
for (final MapEntry<String, _MatchedResponse> entry in _responses.entries) { for (final MapEntry<String, _MatchedResponse> entry in _responses.entries) {
final _MatchedResponse match = entry.value; final _MatchedResponse match = entry.value;
if (match.method != method) continue; if (match.method != method) continue;
final String suffix = entry.key.startsWith('$method:') final String suffix = match.pathSuffix ??
? entry.key.substring(method.length + 1) (entry.key.startsWith('$method:')
: entry.key; ? entry.key.substring(method.length + 1).split(':').first
if (path.endsWith(suffix) || request.url.toString().contains(suffix)) { : entry.key);
final bool pathOk =
path.endsWith(suffix) || request.url.toString().contains(suffix);
if (!pathOk) continue;
if (match.uriPredicate != null && !match.uriPredicate!(request.url)) {
continue;
}
return http.StreamedResponse( return http.StreamedResponse(
Stream<List<int>>.value(utf8.encode(match.response.body)), Stream<List<int>>.value(utf8.encode(match.response.body)),
match.response.statusCode, match.response.statusCode,
@ -64,7 +134,6 @@ class MockHttpClient extends http.BaseClient {
request: request, request: request,
); );
} }
}
return http.StreamedResponse( return http.StreamedResponse(
Stream<List<int>>.value(utf8.encode('{}')), Stream<List<int>>.value(utf8.encode('{}')),
404, 404,
@ -84,9 +153,23 @@ class MockHttpClient extends http.BaseClient {
void close() {} void close() {}
} }
class _QueuedGet {
_QueuedGet(this.pathSuffix, this.response);
final String pathSuffix;
final http.Response response;
}
class _MatchedResponse { class _MatchedResponse {
_MatchedResponse({required this.method, required this.response}); _MatchedResponse({
required this.method,
required this.response,
this.pathSuffix,
this.uriPredicate,
});
final String method; final String method;
final http.Response response; final http.Response response;
final String? pathSuffix;
final bool Function(Uri uri)? uriPredicate;
} }

View File

@ -11,7 +11,7 @@ import 'package:cyberhybridhub_server/trading/user_trading_state_db.dart';
import 'package:dotenv/dotenv.dart'; import 'package:dotenv/dotenv.dart';
import 'package:postgres/postgres.dart'; import 'package:postgres/postgres.dart';
/// Integration test Postgres: [cyberhybridhub_test] with migrations 001004. /// Integration test Postgres: [cyberhybridhub_test] with migrations 001009.
class TestDb { class TestDb {
TestDb._(this.db, this._connection, this.databaseUrl); TestDb._(this.db, this._connection, this.databaseUrl);
@ -125,6 +125,8 @@ class TestDb {
TRUNCATE TABLE TRUNCATE TABLE
trade_orders, trade_orders,
market_data_snapshots, market_data_snapshots,
market_data_sync_runs,
tradable_assets,
user_trading_state, user_trading_state,
user_trading_config, user_trading_config,
questions, questions,

View File

@ -2,6 +2,7 @@
library; library;
import 'package:cyberhybridhub_server/trading/market_data_db.dart'; import 'package:cyberhybridhub_server/trading/market_data_db.dart';
import 'package:cyberhybridhub_server/trading/market_history_four_hour_slot.dart';
import 'package:test/test.dart'; import 'package:test/test.dart';
import '../helpers/test_db.dart'; import '../helpers/test_db.dart';
@ -56,4 +57,295 @@ void main() {
greaterThan(older.toUtc().millisecondsSinceEpoch), greaterThan(older.toUtc().millisecondsSinceEpoch),
); );
}); });
group('upsertSnapshot', () {
test('re-upsert updates price and raw without duplicating row', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final MarketDataDb db = testDb!.marketDataDb;
final DateTime asOf = DateTime.utc(2026, 5, 23, 13);
await db.upsertSnapshot(
symbol: 'SPY',
metric: 'bar',
timeframe: '1Day',
asOf: asOf,
price: 500,
volume: 1000,
raw: <String, dynamic>{'c': 500, 'v': 1000},
);
await db.upsertSnapshot(
symbol: 'SPY',
metric: 'bar',
timeframe: '1Day',
asOf: asOf,
price: 505,
volume: 1100,
raw: <String, dynamic>{'c': 505, 'v': 1100},
);
final List<MarketDataSnapshot> rows = await db.barsForSymbol(
symbol: 'SPY',
timeframe: '1Day',
since: asOf.subtract(const Duration(seconds: 1)),
until: asOf.add(const Duration(seconds: 1)),
);
expect(rows, hasLength(1));
expect(rows.single.price, 505);
expect(rows.single.volume, 1100);
expect(rows.single.raw?['c'], 505);
expect(rows.single.raw?['v'], 1100);
});
});
group('barsForSymbol', () {
test('returns rows ordered by as_of ASC within [since, until)', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final MarketDataDb db = testDb!.marketDataDb;
final DateTime t1 = DateTime.utc(2026, 5, 20);
final DateTime t2 = DateTime.utc(2026, 5, 21);
final DateTime t3 = DateTime.utc(2026, 5, 22);
for (final DateTime t in <DateTime>[t1, t2, t3]) {
await db.upsertSnapshot(
symbol: 'SPY',
metric: 'bar',
timeframe: '1Day',
asOf: t,
price: t.day.toDouble(),
);
}
final List<MarketDataSnapshot> rows = await db.barsForSymbol(
symbol: 'SPY',
timeframe: '1Day',
since: t1,
until: t3,
);
expect(rows, hasLength(2));
expect(rows.map((MarketDataSnapshot r) => r.asOf), <DateTime>[t1, t2]);
});
test('returns empty list when no rows match', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final List<MarketDataSnapshot> rows =
await testDb!.marketDataDb.barsForSymbol(
symbol: 'NOPE',
timeframe: '1Day',
since: DateTime.utc(2026, 1, 1),
until: DateTime.utc(2026, 1, 2),
);
expect(rows, isEmpty);
});
});
test('latestSyncedAsOf returns newest as_of or null', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final MarketDataDb db = testDb!.marketDataDb;
final DateTime older = DateTime.utc(2026, 5, 20);
final DateTime newer = DateTime.utc(2026, 5, 22);
expect(await db.latestSyncedAsOf('SPY', '1Day'), isNull);
await db.upsertSnapshot(
symbol: 'SPY',
metric: 'bar',
timeframe: '1Day',
asOf: older,
price: 490,
);
await db.upsertSnapshot(
symbol: 'SPY',
metric: 'bar',
timeframe: '1Day',
asOf: newer,
price: 500,
);
expect(await db.latestSyncedAsOf('SPY', '1Day'), newer);
});
test('symbolsWithBarForSlot matches canonical slot_start wire string', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final MarketDataDb db = testDb!.marketDataDb;
const String timeframe = '4Hour';
final DateTime slotStart = DateTime.utc(2026, 5, 26, 8);
final String slotWire = MarketHistoryFourHourSlot.slotStartWire(slotStart);
await db.upsertSnapshot(
symbol: 'AAPL',
metric: 'bar',
timeframe: timeframe,
asOf: slotStart,
price: 186,
raw: <String, dynamic>{
'slot_start': slotWire,
't': slotWire,
},
);
final Set<String> synced = await db.symbolsWithBarForSlot(
symbols: <String>['AAPL', 'MSFT'],
slotStart: slotStart,
timeframe: timeframe,
);
expect(synced, <String>{'AAPL'});
});
test('symbolsWithBarForSlot falls back to as_of slot bucket for legacy rows', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final MarketDataDb db = testDb!.marketDataDb;
const String timeframe = '4Hour';
final DateTime slotStart = DateTime.utc(2026, 5, 26, 8);
final DateTime barAt = slotStart.add(const Duration(hours: 1));
await db.upsertSnapshot(
symbol: 'AAPL',
metric: 'bar',
timeframe: timeframe,
asOf: barAt,
price: 186,
raw: <String, dynamic>{
// Different wire format than Dart's toIso8601String() — must still count.
'slot_start': '2026-05-26T08:00:00Z',
},
);
final Set<String> synced = await db.symbolsWithBarForSlot(
symbols: <String>['AAPL', 'MSFT'],
slotStart: slotStart,
timeframe: timeframe,
);
expect(synced, <String>{'AAPL'});
});
test('symbolsWithBarForSlot matches via slot_start bucket when wire differs', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final MarketDataDb db = testDb!.marketDataDb;
const String timeframe = '4Hour';
final DateTime slotStart = DateTime.utc(2026, 5, 26, 8);
await db.upsertSnapshot(
symbol: 'AAPL',
metric: 'bar',
timeframe: timeframe,
asOf: slotStart.add(const Duration(hours: 4)),
price: 186,
raw: <String, dynamic>{
'slot_start': '2026-05-26T08:00:00.000Z',
},
);
final Set<String> synced = await db.symbolsWithBarForSlot(
symbols: <String>['AAPL'],
slotStart: slotStart,
timeframe: timeframe,
);
expect(synced, <String>{'AAPL'});
});
test('symbolsWithBarForSlot does not match the next slot boundary as prior slot',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final MarketDataDb db = testDb!.marketDataDb;
const String timeframe = '4Hour';
final DateTime slotStart = DateTime.utc(2026, 5, 26, 8);
await db.upsertSnapshot(
symbol: 'AAPL',
metric: 'bar',
timeframe: timeframe,
asOf: DateTime.utc(2026, 5, 26, 12),
price: 186,
);
final Set<String> synced = await db.symbolsWithBarForSlot(
symbols: <String>['AAPL'],
slotStart: slotStart,
timeframe: timeframe,
);
expect(synced, isEmpty);
});
test('symbolsWithBarForSlot counts no_data placeholder as synced', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final MarketDataDb db = testDb!.marketDataDb;
const String timeframe = '4Hour';
final DateTime slotStart = DateTime.utc(2026, 5, 30, 20);
await db.upsertNoDataBarPlaceholder(
symbol: 'A',
slotStart: slotStart,
timeframe: timeframe,
checkedAt: DateTime.utc(2026, 5, 31),
);
final Set<String> synced = await db.symbolsWithBarForSlot(
symbols: <String>['A', 'B'],
slotStart: slotStart,
timeframe: timeframe,
);
expect(synced, <String>{'A'});
});
} }

View File

@ -0,0 +1,775 @@
@Tags(['integration', 'postgres'])
library;
import 'package:cyberhybridhub_server/alpaca/alpaca_env.dart';
import 'package:cyberhybridhub_server/alpaca/alpaca_market_data_client.dart';
import 'package:cyberhybridhub_server/alpaca/alpaca_models.dart';
import 'package:cyberhybridhub_server/trading/market_data_history.dart';
import 'package:cyberhybridhub_server/trading/market_history_config.dart';
import 'package:cyberhybridhub_server/trading/market_history_api_rate_limiter.dart';
import 'package:cyberhybridhub_server/trading/market_history_four_hour_slot.dart';
import 'package:cyberhybridhub_server/trading/tradable_assets_db.dart';
import 'package:http/http.dart' as http;
import 'package:postgres/postgres.dart';
import 'package:test/test.dart';
import '../helpers/fixture_loader.dart';
import '../helpers/mock_http_client.dart';
import '../helpers/test_db.dart';
Future<void> _seedTradables(
Connection connection,
List<String> symbols,
) async {
final TradableAssetsDb db = TradableAssetsDb(connection);
await db.upsertAll(
symbols
.map(
(String symbol) => AlpacaAsset(
symbol: symbol,
assetClass: 'us_equity',
status: 'active',
tradable: true,
fractionable: true,
),
)
.toList(),
now: DateTime.utc(2026, 5, 26, 10),
);
}
void main() {
TestDb? testDb;
late FixtureLoader fixtures;
late AlpacaEnv env;
setUpAll(() async {
testDb = await TestDb.open();
fixtures = FixtureLoader();
env = AlpacaEnv.fromMap(<String, String>{
'ALPACA_API_KEY_ID': 'test-key',
'ALPACA_API_SECRET_KEY': 'test-secret',
'ALPACA_DATA_BASE_URL': 'https://data.alpaca.markets',
'ALPACA_DATA_FEED': 'iex',
});
});
tearDown(() async {
if (testDb != null) {
await testDb!.truncateTradingTables();
}
});
tearDownAll(() async {
await testDb?.close();
});
MarketDataHistorySync makeSync({
required MockHttpClient mock,
int batchSize = 100,
int maxSymbols = 2000,
int windowDays = 1,
int apiRequestsPerMinute = 10000,
MarketHistoryApiRateLimiter? rateLimiter,
Duration rateLimitCooldown = Duration.zero,
List<Duration>? sleepLog,
}) {
return MarketDataHistorySync(
marketDataClient: AlpacaMarketDataClient(env: env, httpClient: mock),
tradableAssetsDb: TradableAssetsDb(testDb!.connection),
marketDataDb: testDb!.marketDataDb,
connection: testDb!.connection,
batchSize: batchSize,
maxSymbols: maxSymbols,
windowDays: windowDays,
apiRequestsPerMinute: apiRequestsPerMinute,
rateLimiter: rateLimiter,
rateLimitCooldown: rateLimitCooldown,
sleep: sleepLog == null
? null
: (Duration d) async {
sleepLog.add(d);
},
);
}
group('runOnce — 4-hour slots', () {
final DateTime now = DateTime.utc(2026, 5, 26, 12);
test('cold start upserts completed slots in window and uses 4Hour', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await _seedTradables(
testDb!.connection,
<String>['SPY', 'AAPL', 'MSFT'],
);
final Map<String, dynamic> barsJson =
await fixtures.loadJson('alpaca_bars_4h_window.json');
final MockHttpClient mock = MockHttpClient()
..whenGetJson('/bars', barsJson);
final MarketDataHistorySyncResult result =
await makeSync(mock: mock, windowDays: 1).runOnce(now: now);
expect(result.error, isNull);
expect(result.rowsWritten, 18);
expect(result.slotsSynced, 6);
final Result rows = await testDb!.connection.execute(
'''
SELECT metric, timeframe, COUNT(*)::int
FROM market_data_snapshots
GROUP BY metric, timeframe
''',
);
expect(rows.first[0], 'bar');
expect(rows.first[1], MarketHistoryConfig.barTimeframe);
expect((rows.first[2]! as num).toInt(), 18);
final Uri firstBarRequest = mock.requests
.firstWhere((http.BaseRequest r) => r.url.path.endsWith('/bars'))
.url;
expect(
firstBarRequest.queryParameters['timeframe'],
MarketHistoryFourHourSlot.alpacaTimeframe,
);
expect(
DateTime.parse(firstBarRequest.queryParameters['start']!).toUtc(),
DateTime.utc(2026, 5, 26, 8),
);
expect(
DateTime.parse(firstBarRequest.queryParameters['end']!).toUtc(),
MarketHistoryFourHourSlot.endInclusive(
DateTime.utc(2026, 5, 26, 8),
),
);
final Result runs = await testDb!.connection.execute(
'''
SELECT kind, rows_written, slots_synced, backfill_items, error
FROM market_data_sync_runs
''',
);
expect(runs.single[0], 'backfill');
expect((runs.single[1]! as num).toInt(), 18);
expect((runs.single[2]! as num).toInt(), 6);
final List<dynamic> items =
runs.single[3]! as List<dynamic>;
expect(items, isNotEmpty);
final Map<String, dynamic> firstItem =
items.first as Map<String, dynamic>;
expect(firstItem['slotStart'], isNotNull);
expect(firstItem['symbols'], isA<List<dynamic>>());
expect(runs.single[4], isNull);
});
test('stored slot_start wire matches Alpaca start query param', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await _seedTradables(
testDb!.connection,
<String>['SPY'],
);
final Map<String, dynamic> barsJson =
await fixtures.loadJson('alpaca_bars_4h_window.json');
final MockHttpClient mock = MockHttpClient()
..whenGetJson('/bars', barsJson);
await makeSync(mock: mock, windowDays: 1).runOnce(now: now);
final String alpacaStart = mock.requests
.firstWhere((http.BaseRequest r) => r.url.path.endsWith('/bars'))
.url
.queryParameters['start']!;
final Result rows = await testDb!.connection.execute(
Sql.named(
'''
SELECT raw->>'slot_start' AS slot_start
FROM market_data_snapshots
WHERE symbol = 'SPY'
AND metric = 'bar'
AND timeframe = @timeframe
AND raw->>'slot_start' = @slot_start
LIMIT 1
''',
),
parameters: <String, dynamic>{
'timeframe': MarketHistoryConfig.barTimeframe,
'slot_start': alpacaStart,
},
);
expect(rows.single[0], alpacaStart);
expect(alpacaStart, '2026-05-26T08:00:00Z');
});
test('re-run is idempotent with zero rows when fully synced', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await _seedTradables(testDb!.connection, <String>['SPY', 'AAPL', 'MSFT']);
final Map<String, dynamic> barsJson =
await fixtures.loadJson('alpaca_bars_4h_window.json');
final MockHttpClient mock = MockHttpClient()
..whenGetJson('/bars', barsJson);
final MarketDataHistorySync sync =
makeSync(mock: mock, windowDays: 1);
final MarketDataHistorySyncResult r1 = await sync.runOnce(now: now);
mock.requests.clear();
final MarketDataHistorySyncResult r2 = await sync.runOnce(now: now);
expect(r1.rowsWritten, 18);
expect(r2.rowsWritten, 0);
expect(
mock.requests.where((http.BaseRequest r) => r.url.path.endsWith('/bars')),
isEmpty,
);
});
test('re-run skips symbols when bar is stored with alternate slot_start wire format',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await _seedTradables(
testDb!.connection,
<String>['AAPL', 'MSFT', 'SPY'],
);
final List<DateTime> completed =
MarketHistoryFourHourSlot.completedSlotStartsInWindow(now, 1);
for (final DateTime slotStart in completed) {
for (final String symbol in <String>['AAPL', 'MSFT', 'SPY']) {
await testDb!.marketDataDb.upsertSnapshot(
symbol: symbol,
metric: 'bar',
timeframe: MarketHistoryConfig.barTimeframe,
asOf: slotStart.add(const Duration(minutes: 30)),
price: 100,
raw: <String, dynamic>{
'slot_start': slotStart.toUtc().toIso8601String().replaceAll(
'.000',
'',
),
},
);
}
}
final MockHttpClient mock = MockHttpClient();
final MarketDataHistorySyncResult result = await makeSync(
mock: mock,
windowDays: 1,
).runOnce(now: now);
expect(result.rowsWritten, 0);
expect(
mock.requests.where((http.BaseRequest r) => r.url.path.endsWith('/bars')),
isEmpty,
);
});
test('does not fetch while current slot is still open', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await _seedTradables(testDb!.connection, <String>['SPY']);
final Map<String, dynamic> barsJson =
await fixtures.loadJson('alpaca_bars_4h_window.json');
final MockHttpClient mock = MockHttpClient()
..whenGetJson('/bars', barsJson);
final MarketDataHistorySync sync =
makeSync(mock: mock, windowDays: 1);
final DateTime midSlot = DateTime.utc(2026, 5, 26, 10, 30);
await sync.runOnce(now: midSlot);
mock.requests.clear();
final MarketDataHistorySyncResult second =
await sync.runOnce(now: midSlot);
expect(second.rowsWritten, 0);
final Iterable<http.BaseRequest> barRequests = mock.requests.where(
(http.BaseRequest r) => r.url.path.endsWith('/bars'),
);
for (final http.BaseRequest request in barRequests) {
final String? start = request.url.queryParameters['start'];
expect(start, isNotNull);
expect(
DateTime.parse(start!).toUtc(),
isNot(DateTime.utc(2026, 5, 26, 12)),
reason: 'must not fetch the still-open slot',
);
}
});
test('fetches only the newly completed slot after prior sync', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await _seedTradables(testDb!.connection, <String>['SPY']);
final List<DateTime> completed =
MarketHistoryFourHourSlot.completedSlotStartsInWindow(now, 1);
final DateTime targetSlot = DateTime.utc(2026, 5, 26, 8);
for (final DateTime slotStart in completed) {
if (slotStart == targetSlot) {
continue;
}
await testDb!.marketDataDb.upsertSnapshot(
symbol: 'SPY',
metric: 'bar',
timeframe: MarketHistoryConfig.barTimeframe,
asOf: slotStart.add(const Duration(hours: 1)),
price: 495,
raw: <String, dynamic>{
'slot_start': slotStart.toIso8601String(),
},
);
}
final MockHttpClient mock = MockHttpClient()
..whenGetJson('/bars', <String, dynamic>{
'bars': <String, dynamic>{
'SPY': <Map<String, dynamic>>[
<String, dynamic>{
't': '2026-05-26T08:00:00Z',
'o': 495,
'h': 497,
'l': 493,
'c': 496,
'v': 4500000,
},
],
},
'next_page_token': null,
});
await makeSync(mock: mock, windowDays: 1).runOnce(now: now);
final String start = mock.requests.single.url.queryParameters['start']!;
expect(DateTime.parse(start).toUtc(), DateTime.utc(2026, 5, 26, 8));
expect(
mock.requests.single.url.queryParameters['timeframe'],
'4Hour',
);
});
test('partial outage persists successful batch and records error', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await _seedTradables(
testDb!.connection,
<String>['SPY', 'AAPL', 'MSFT'],
);
final Map<String, dynamic> okJson =
await fixtures.loadJson('alpaca_bars_4h_window.json');
final MockHttpClient mock = MockHttpClient()
..whenGetWhereJson(
'/bars',
(Uri uri) => uri.queryParameters['symbols'] == 'AAPL,MSFT',
okJson,
)
..whenGetWhere(
'/bars',
(Uri uri) => uri.queryParameters['symbols'] == 'SPY',
http.Response('upstream exploded', 500),
);
final MarketDataHistorySyncResult result = await makeSync(
mock: mock,
batchSize: 2,
windowDays: 1,
).runOnce(now: now);
expect(result.error, isNotNull);
expect(result.error, contains('SPY'));
expect(result.rowsWritten, greaterThan(0));
});
test('stores no_data placeholder when Alpaca returns empty bars for requested symbols',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await _seedTradables(
testDb!.connection,
<String>['A', 'AA'],
);
final MockHttpClient mock = MockHttpClient()
..whenGetJson('/bars', <String, dynamic>{
'bars': <String, dynamic>{},
'next_page_token': null,
});
final MarketDataHistorySyncResult result = await makeSync(
mock: mock,
batchSize: 10,
windowDays: 1,
).runOnce(now: now);
expect(result.error, isNull);
expect(result.rowsWritten, greaterThan(0));
final Result rows = await testDb!.connection.execute(
Sql.named(
'''
SELECT symbol, raw->>'no_data' AS no_data, raw->>'slot_start' AS slot_start
FROM market_data_snapshots
WHERE symbol IN ('A', 'AA')
AND metric = 'bar'
AND timeframe = @timeframe
''',
),
parameters: <String, dynamic>{
'timeframe': MarketHistoryConfig.barTimeframe,
},
);
expect(rows.length, greaterThanOrEqualTo(2));
for (final ResultRow row in rows) {
expect(row[1], 'true');
expect(row[2], isNotNull);
}
mock.requests.clear();
final MarketDataHistorySyncResult second = await makeSync(
mock: mock,
batchSize: 10,
windowDays: 1,
).runOnce(now: now);
expect(
mock.requests.where((http.BaseRequest r) => r.url.path.endsWith('/bars')),
isEmpty,
reason: 'placeholders should satisfy gap check for synced symbols',
);
expect(second.error, isNull);
});
test('stores no_data placeholder when Alpaca bars are outside the requested slot',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await _seedTradables(testDb!.connection, <String>['SPY']);
final MockHttpClient mock = MockHttpClient()
..whenGetJson('/bars', <String, dynamic>{
'bars': <String, dynamic>{
'SPY': <Map<String, dynamic>>[
<String, dynamic>{
't': '2026-05-26T12:00:00Z',
'o': 495,
'h': 497,
'l': 493,
'c': 496,
'v': 4500000,
},
],
},
'next_page_token': null,
});
final MarketDataHistorySyncResult result = await makeSync(
mock: mock,
windowDays: 1,
).runOnce(now: now);
expect(result.error, isNull);
expect(result.rowsWritten, greaterThan(0));
final Result rows = await testDb!.connection.execute(
Sql.named(
'''
SELECT raw->>'no_data' AS no_data, raw->>'slot_start' AS slot_start
FROM market_data_snapshots
WHERE symbol = 'SPY' AND metric = 'bar' AND timeframe = @timeframe
AND raw->>'slot_start' = '2026-05-26T08:00:00Z'
''',
),
parameters: <String, dynamic>{
'timeframe': MarketHistoryConfig.barTimeframe,
},
);
expect(rows.single[0], 'true');
expect(rows.single[1], '2026-05-26T08:00:00Z');
});
test('stores market_closed placeholder on weekend with no error', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await _seedTradables(testDb!.connection, <String>['A']);
final DateTime sundayAfterWeekend = DateTime.utc(2026, 5, 31, 0, 30);
final MockHttpClient mock = MockHttpClient()
..whenGetJson('/bars', <String, dynamic>{
'bars': <String, dynamic>{},
'next_page_token': null,
});
final MarketDataHistorySyncResult result = await makeSync(
mock: mock,
batchSize: 10,
windowDays: 1,
).runOnce(now: sundayAfterWeekend);
expect(result.error, isNull);
expect(result.rowsWritten, greaterThan(0));
final Result rows = await testDb!.connection.execute(
Sql.named(
'''
SELECT raw->>'source' AS source
FROM market_data_snapshots
WHERE symbol = 'A'
AND metric = 'bar'
AND timeframe = @timeframe
AND raw->>'slot_start' = '2026-05-30T20:00:00Z'
LIMIT 1
''',
),
parameters: <String, dynamic>{
'timeframe': MarketHistoryConfig.barTimeframe,
},
);
expect(rows, isNotEmpty);
expect(rows.first[0], 'market_closed');
});
test('batching issues one Alpaca call per slot per batch', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await _seedTradables(
testDb!.connection,
<String>['S1', 'S2', 'S3', 'S4', 'S5'],
);
final Map<String, dynamic> barsJson =
await fixtures.loadJson('alpaca_bars_4h_window.json');
final MockHttpClient mock = MockHttpClient()
..whenGetJson('/bars', barsJson);
await makeSync(mock: mock, batchSize: 2, windowDays: 1).runOnce(
now: now,
);
final int barRequests = mock.requests
.where((http.BaseRequest r) => r.url.path.endsWith('/bars'))
.length;
expect(barRequests, 6 * 3);
});
test('new symbol is fetched without re-requesting fully synced symbols', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await _seedTradables(
testDb!.connection,
<String>['SPY', 'AAPL', 'MSFT'],
);
final Map<String, dynamic> barsJson =
await fixtures.loadJson('alpaca_bars_4h_window.json');
final MockHttpClient mock = MockHttpClient()
..whenGetJson('/bars', barsJson);
final MarketDataHistorySync sync =
makeSync(mock: mock, windowDays: 1);
await sync.runOnce(now: now);
await _seedTradables(
testDb!.connection,
<String>['SPY', 'AAPL', 'MSFT', 'NVDA'],
);
mock.requests.clear();
await sync.runOnce(now: now);
final Iterable<http.BaseRequest> barRequests = mock.requests.where(
(http.BaseRequest r) => r.url.path.endsWith('/bars'),
);
expect(barRequests, isNotEmpty);
for (final http.BaseRequest request in barRequests) {
final String? symbols = request.url.queryParameters['symbols'];
expect(symbols, isNotNull);
expect(symbols!.split(','), everyElement('NVDA'));
}
});
});
group('rate limit', () {
final DateTime now = DateTime.utc(2026, 5, 26, 12);
test('429 waits one minute, retries, and saves partial progress', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await _seedTradables(testDb!.connection, <String>['SPY']);
final Map<String, dynamic> barsJson =
await fixtures.loadJson('alpaca_bars_4h_window.json');
final MockHttpClient mock = MockHttpClient()
..whenGetQueued('/bars', http.Response('rate limited', 429))
..whenGetQueuedJson('/bars', barsJson)
..whenGetJson('/bars', barsJson);
final List<Duration> sleeps = <Duration>[];
final MarketDataHistorySyncResult result = await makeSync(
mock: mock,
batchSize: 1,
windowDays: 1,
rateLimitCooldown: const Duration(minutes: 1),
sleepLog: sleeps,
).runOnce(now: now);
expect(sleeps, <Duration>[const Duration(minutes: 1)]);
expect(result.rowsWritten, greaterThan(0));
expect(result.error, isNull);
});
test('429 after cooldown stops run and keeps partial rows', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await _seedTradables(
testDb!.connection,
<String>['SPY', 'AAPL', 'MSFT'],
);
final Map<String, dynamic> okJson =
await fixtures.loadJson('alpaca_bars_4h_window.json');
final MockHttpClient mock = MockHttpClient()
..whenGetWhereJson(
'/bars',
(Uri uri) => uri.queryParameters['symbols'] == 'AAPL,MSFT',
okJson,
)
..whenGetWhere(
'/bars',
(Uri uri) => uri.queryParameters['symbols'] == 'SPY',
http.Response('rate limited', 429),
);
final List<Duration> sleeps = <Duration>[];
final MarketDataHistorySyncResult result = await makeSync(
mock: mock,
batchSize: 2,
windowDays: 1,
rateLimitCooldown: const Duration(minutes: 1),
sleepLog: sleeps,
).runOnce(now: now);
expect(sleeps, <Duration>[const Duration(minutes: 1)]);
expect(result.error, isNotNull);
expect(result.error, contains('rate limited'));
expect(result.error, contains('partial sync saved'));
expect(result.rowsWritten, greaterThan(0));
final Result okBatchCount = await testDb!.connection.execute(
Sql.named(
'''
SELECT COUNT(*)::int FROM market_data_snapshots
WHERE symbol IN ('AAPL', 'MSFT') AND metric = 'bar'
''',
),
);
expect((okBatchCount.first[0]! as num).toInt(), greaterThan(0));
final Result spyCount = await testDb!.connection.execute(
"SELECT COUNT(*)::int FROM market_data_snapshots WHERE symbol = 'SPY'",
);
expect((spyCount.first[0]! as num).toInt(), 0);
});
});
group('hasPendingSlots', () {
test('true on cold start and false when window is caught up', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await _seedTradables(testDb!.connection, <String>['SPY']);
final MarketDataHistorySync sync = makeSync(
mock: MockHttpClient(),
windowDays: 1,
);
final DateTime now = DateTime.utc(2026, 5, 26, 12);
expect(await sync.hasPendingSlots(now), isTrue);
final Map<String, dynamic> barsJson =
await fixtures.loadJson('alpaca_bars_4h_window.json');
final MockHttpClient mock = MockHttpClient()
..whenGetJson('/bars', barsJson);
await makeSync(mock: mock, windowDays: 1).runOnce(now: now);
expect(await sync.hasPendingSlots(now), isFalse);
});
});
}

View File

@ -0,0 +1,319 @@
@Tags(['integration', 'postgres'])
library;
import 'package:cyberhybridhub_server/trading/market_data_db.dart';
import 'package:cyberhybridhub_server/trading/market_data_retention.dart';
import 'package:postgres/postgres.dart';
import 'package:test/test.dart';
import '../helpers/test_db.dart';
void main() {
TestDb? testDb;
setUpAll(() async {
testDb = await TestDb.open();
});
tearDown(() async {
if (testDb != null) {
await testDb!.connection.execute('TRUNCATE TABLE market_data_archive');
await testDb!.truncateTradingTables();
}
});
tearDownAll(() async {
await testDb?.close();
});
MarketDataRetention retention({void Function(String sql)? onExecute}) {
return MarketDataRetention(
connection: testDb!.connection,
onExecute: onExecute,
);
}
group('runCleanup (hard delete)', () {
test('deletes rows older than window and keeps rows within window',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final MarketDataDb db = testDb!.marketDataDb;
final DateTime now = DateTime.utc(2026, 5, 26, 12);
final DateTime cutoff = now.subtract(const Duration(days: 7));
int removedCount = 0;
int keptCount = 0;
for (int day = 0; day < 10; day++) {
final DateTime asOf = now.subtract(Duration(days: 14 - day));
await db.upsertSnapshot(
symbol: 'SPY',
metric: 'bar',
timeframe: '1Day',
asOf: asOf,
price: 490 + day,
);
if (asOf.isBefore(cutoff)) {
removedCount++;
} else {
keptCount++;
}
}
final MarketDataRetentionResult result = await retention().runCleanup(
now: now,
);
expect(result.error, isNull);
expect(result.rowsRemoved, removedCount);
expect(keptCount, greaterThan(0));
final Result remaining = await testDb!.connection.execute(
'SELECT COUNT(*)::int FROM market_data_snapshots',
);
expect((remaining.first[0]! as num).toInt(), keptCount);
});
test('empty table returns rowsRemoved 0 without throwing', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final MarketDataRetentionResult result = await retention().runCleanup(
now: DateTime.utc(2026, 5, 26, 12),
);
expect(result.rowsRemoved, 0);
expect(result.error, isNull);
});
test('batchSize issues multiple DELETE statements for large backlog',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
const int rowCount = 5000;
const int batchSize = 1000;
final DateTime now = DateTime.utc(2026, 5, 26, 12);
final DateTime oldAsOf = now.subtract(const Duration(days: 14));
await testDb!.connection.execute(
Sql.named(
'''
INSERT INTO market_data_snapshots (symbol, metric, timeframe, as_of, price)
SELECT 'SYM' || g::text, 'bar', '1Day', @as_of, 100
FROM generate_series(1, @count) AS g
''',
),
parameters: <String, dynamic>{
'as_of': oldAsOf,
'count': rowCount,
},
);
int deleteStatements = 0;
final MarketDataRetentionResult result = await MarketDataRetention(
connection: testDb!.connection,
batchSize: batchSize,
onExecute: (String sql) {
if (sql.toUpperCase().contains('DELETE FROM MARKET_DATA_SNAPSHOTS')) {
deleteStatements++;
}
},
).runCleanup(now: now);
expect(result.rowsRemoved, rowCount);
expect(deleteStatements, greaterThanOrEqualTo(5));
});
test('writes market_data_sync_runs row with kind cleanup and rows_removed',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final DateTime now = DateTime.utc(2026, 5, 26, 12);
await testDb!.marketDataDb.upsertSnapshot(
symbol: 'SPY',
metric: 'bar',
timeframe: '1Day',
asOf: now.subtract(const Duration(days: 10)),
price: 480,
);
await retention().runCleanup(now: now);
final Result runs = await testDb!.connection.execute(
'''
SELECT kind, rows_removed, finished_at, error
FROM market_data_sync_runs
ORDER BY id DESC
LIMIT 1
''',
);
expect(runs, hasLength(1));
expect(runs.first[0], 'cleanup');
expect((runs.first[1]! as num).toInt(), 1);
expect(runs.first[2], isNotNull);
expect(runs.first[3], isNull);
});
test('rows within window are never deleted (by id)', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final MarketDataDb db = testDb!.marketDataDb;
final DateTime now = DateTime.utc(2026, 5, 26, 12);
final MarketDataSnapshot kept = await db.upsertSnapshot(
symbol: 'SPY',
metric: 'bar',
timeframe: '1Day',
asOf: now.subtract(const Duration(days: 2)),
price: 500,
);
await db.upsertSnapshot(
symbol: 'SPY',
metric: 'bar',
timeframe: '1Day',
asOf: now.subtract(const Duration(days: 10)),
price: 480,
);
await retention().runCleanup(now: now);
final Result row = await testDb!.connection.execute(
Sql.named('SELECT id FROM market_data_snapshots WHERE id = @id'),
parameters: <String, dynamic>{'id': kept.id},
);
expect(row, hasLength(1));
});
});
group('runArchiveAndCleanup', () {
test('archiveEnabled copies expired rows then deletes them', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final DateTime now = DateTime.utc(2026, 5, 26, 12);
await testDb!.marketDataDb.upsertSnapshot(
symbol: 'SPY',
metric: 'bar',
timeframe: '1Day',
asOf: now.subtract(const Duration(days: 10)),
price: 480,
volume: 1000,
raw: <String, dynamic>{'c': 480},
);
await testDb!.marketDataDb.upsertSnapshot(
symbol: 'SPY',
metric: 'bar',
timeframe: '1Day',
asOf: now.subtract(const Duration(days: 2)),
price: 500,
);
final MarketDataRetentionResult result =
await retention().runArchiveAndCleanup(now: now);
expect(result.error, isNull);
expect(result.rowsRemoved, 1);
final Result archiveCount = await testDb!.connection.execute(
'SELECT COUNT(*)::int FROM market_data_archive',
);
expect((archiveCount.first[0]! as num).toInt(), 1);
final Result liveCount = await testDb!.connection.execute(
'SELECT COUNT(*)::int FROM market_data_snapshots',
);
expect((liveCount.first[0]! as num).toInt(), 1);
});
test('archive failure rolls back delete and records error', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final DateTime now = DateTime.utc(2026, 5, 26, 12);
await testDb!.marketDataDb.upsertSnapshot(
symbol: 'SPY',
metric: 'bar',
timeframe: '1Day',
asOf: now.subtract(const Duration(days: 10)),
price: 480,
);
await testDb!.connection.execute(
'ALTER TABLE market_data_archive RENAME TO market_data_archive_hidden',
);
final MarketDataRetentionResult result =
await retention().runArchiveAndCleanup(now: now);
expect(result.error, isNotNull);
expect(result.rowsRemoved, 0);
final Result liveCount = await testDb!.connection.execute(
'SELECT COUNT(*)::int FROM market_data_snapshots',
);
expect((liveCount.first[0]! as num).toInt(), 1);
await testDb!.connection.execute(
'ALTER TABLE market_data_archive_hidden RENAME TO market_data_archive',
);
});
test('runCleanup does not write to archive table', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final DateTime now = DateTime.utc(2026, 5, 26, 12);
await testDb!.marketDataDb.upsertSnapshot(
symbol: 'SPY',
metric: 'bar',
timeframe: '1Day',
asOf: now.subtract(const Duration(days: 10)),
price: 480,
);
await retention().runCleanup(now: now);
final Result archiveCount = await testDb!.connection.execute(
'SELECT COUNT(*)::int FROM market_data_archive',
);
expect((archiveCount.first[0]! as num).toInt(), 0);
});
});
}

View File

@ -0,0 +1,821 @@
@Tags(<String>['integration', 'postgres'])
library;
import 'dart:convert';
import 'package:cyberhybridhub_server/firebase_auth.dart';
import 'package:cyberhybridhub_server/handlers/market_history_admin_handler.dart';
import 'package:cyberhybridhub_server/trading/market_data_history.dart';
import 'package:cyberhybridhub_server/trading/market_history_four_hour_slot.dart';
import 'package:cyberhybridhub_server/trading/market_data_retention.dart';
import 'package:cyberhybridhub_server/trading/market_history_admin_actions.dart';
import 'package:cyberhybridhub_server/trading/sync_run_recorder.dart';
import 'package:cyberhybridhub_server/trading/tradable_assets_sync.dart';
import 'package:postgres/postgres.dart';
import 'package:shelf/shelf.dart';
import 'package:test/test.dart';
import '../helpers/test_db.dart';
class _FakeAuthVerifier extends FirebaseAuthVerifier {
_FakeAuthVerifier() : super('test-key');
@override
Future<String?> verifyBearerToken(String? authorization) async {
if (authorization == null || !authorization.startsWith('Bearer ')) {
return null;
}
final String token = authorization.substring('Bearer '.length).trim();
return token.isEmpty ? null : token;
}
}
Future<Response> _get(
Handler handler, {
required String path,
String? bearer,
}) async {
return await Future<Response>.value(
handler(
Request(
'GET',
Uri.parse('http://localhost$path'),
headers: <String, String>{
if (bearer != null) 'Authorization': 'Bearer $bearer',
},
),
),
);
}
Future<Response> _post(
Handler handler, {
required String path,
String? bearer,
}) async {
return await Future<Response>.value(
handler(
Request(
'POST',
Uri.parse('http://localhost$path'),
headers: <String, String>{
if (bearer != null) 'Authorization': 'Bearer $bearer',
},
),
),
);
}
Future<void> _insertRun(
Connection connection, {
required String kind,
required DateTime startedAt,
DateTime? finishedAt,
int rowsWritten = 0,
int rowsRemoved = 0,
String? error,
}) async {
await connection.execute(
Sql.named(
'''
INSERT INTO market_data_sync_runs
(kind, started_at, finished_at, rows_written, rows_removed, error)
VALUES
(@kind, @started_at, @finished_at, @rows_written, @rows_removed, @error)
''',
),
parameters: <String, dynamic>{
'kind': kind,
'started_at': startedAt.toUtc(),
'finished_at': finishedAt?.toUtc(),
'rows_written': rowsWritten,
'rows_removed': rowsRemoved,
'error': error,
},
);
}
void main() {
TestDb? testDb;
setUpAll(() async {
testDb = await TestDb.open();
});
tearDown(() async {
if (testDb != null) {
await testDb!.truncateTradingTables();
}
});
tearDownAll(() async {
await testDb?.close();
});
test('returns 403 for authenticated non-admin user', () async {
if (testDb == null) {
markTestSkipped('Set DATABASE_URL or TEST_DATABASE_URL for integration tests');
return;
}
final Handler handler = marketHistoryAdminHandler(
auth: _FakeAuthVerifier(),
connection: testDb!.connection,
adminFirebaseUids: <String>{'admin-uid'},
);
final Response response = await _get(
handler,
path: '/v1/admin/market-history/sync-runs',
bearer: 'non-admin',
);
expect(response.statusCode, 403);
});
test('returns newest-first runs and pinned unresolved failures', () async {
if (testDb == null) {
markTestSkipped('Set DATABASE_URL or TEST_DATABASE_URL for integration tests');
return;
}
final DateTime t0 = DateTime.utc(2026, 5, 26, 10);
await _insertRun(
testDb!.connection,
kind: 'backfill',
startedAt: t0,
finishedAt: t0.add(const Duration(minutes: 1)),
rowsWritten: 10,
error: 'rate limited: 429',
);
await _insertRun(
testDb!.connection,
kind: 'cleanup',
startedAt: t0.add(const Duration(hours: 1)),
finishedAt: t0.add(const Duration(hours: 1, minutes: 1)),
rowsRemoved: 100,
);
final Handler handler = marketHistoryAdminHandler(
auth: _FakeAuthVerifier(),
connection: testDb!.connection,
adminFirebaseUids: <String>{'admin-uid'},
);
final Response response = await _get(
handler,
path: '/v1/admin/market-history/sync-runs?limit=50',
bearer: 'admin-uid',
);
expect(response.statusCode, 200);
final Map<String, dynamic> body =
jsonDecode(await response.readAsString()) as Map<String, dynamic>;
final List<dynamic> pinned = body['pinned'] as List<dynamic>;
final List<dynamic> runs = body['runs'] as List<dynamic>;
expect((pinned.first as Map<String, dynamic>)['kind'], 'backfill');
expect((pinned.first as Map<String, dynamic>)['severity'], 'rate_limit');
expect((runs.first as Map<String, dynamic>)['kind'], 'cleanup');
});
test('pinned clears after later success for same kind', () async {
if (testDb == null) {
markTestSkipped('Set DATABASE_URL or TEST_DATABASE_URL for integration tests');
return;
}
final DateTime t0 = DateTime.utc(2026, 5, 26, 10);
await _insertRun(
testDb!.connection,
kind: 'backfill',
startedAt: t0,
finishedAt: t0.add(const Duration(minutes: 1)),
rowsWritten: 10,
error: '429',
);
await _insertRun(
testDb!.connection,
kind: 'backfill',
startedAt: t0.add(const Duration(hours: 2)),
finishedAt: t0.add(const Duration(hours: 2, minutes: 1)),
rowsWritten: 40,
);
final Handler handler = marketHistoryAdminHandler(
auth: _FakeAuthVerifier(),
connection: testDb!.connection,
adminFirebaseUids: <String>{'admin-uid'},
);
final Response response = await _get(
handler,
path: '/v1/admin/market-history/sync-runs',
bearer: 'admin-uid',
);
final Map<String, dynamic> body =
jsonDecode(await response.readAsString()) as Map<String, dynamic>;
expect((body['pinned'] as List<dynamic>), isEmpty);
});
test('supports kind filter, limit, and before pagination', () async {
if (testDb == null) {
markTestSkipped('Set DATABASE_URL or TEST_DATABASE_URL for integration tests');
return;
}
final DateTime t0 = DateTime.utc(2026, 5, 26, 10);
await _insertRun(
testDb!.connection,
kind: 'cleanup',
startedAt: t0,
finishedAt: t0.add(const Duration(minutes: 1)),
rowsRemoved: 10,
);
await _insertRun(
testDb!.connection,
kind: 'cleanup',
startedAt: t0.add(const Duration(hours: 1)),
finishedAt: t0.add(const Duration(hours: 1, minutes: 1)),
rowsRemoved: 20,
);
await _insertRun(
testDb!.connection,
kind: 'backfill',
startedAt: t0.add(const Duration(hours: 2)),
finishedAt: t0.add(const Duration(hours: 2, minutes: 1)),
rowsWritten: 99,
);
final Handler handler = marketHistoryAdminHandler(
auth: _FakeAuthVerifier(),
connection: testDb!.connection,
adminFirebaseUids: <String>{'admin-uid'},
);
final Response filtered = await _get(
handler,
path: '/v1/admin/market-history/sync-runs?kind=cleanup&limit=1',
bearer: 'admin-uid',
);
final Map<String, dynamic> filteredBody =
jsonDecode(await filtered.readAsString()) as Map<String, dynamic>;
final List<dynamic> filteredRuns = filteredBody['runs'] as List<dynamic>;
expect(filteredRuns, hasLength(1));
expect((filteredRuns.first as Map<String, dynamic>)['kind'], 'cleanup');
final String before = filteredBody['nextBefore'] as String;
final Response nextPage = await _get(
handler,
path: '/v1/admin/market-history/sync-runs?kind=cleanup&before=$before&limit=5',
bearer: 'admin-uid',
);
final Map<String, dynamic> nextBody =
jsonDecode(await nextPage.readAsString()) as Map<String, dynamic>;
final List<dynamic> nextRuns = nextBody['runs'] as List<dynamic>;
expect(nextRuns, hasLength(1));
});
test('POST resync returns 202 and creates run rows', () async {
if (testDb == null) {
markTestSkipped('Set DATABASE_URL or TEST_DATABASE_URL for integration tests');
return;
}
Future<void> record(String kind, DateTime now) async {
final SyncRunRecorder recorder = SyncRunRecorder(testDb!.connection);
await recorder.record(
kind,
() async => const SyncRunCounts(rowsWritten: 1),
now: now,
);
}
final MarketHistoryAdminActions actions = MarketHistoryAdminActions(
connection: testDb!.connection,
runUniverse: (DateTime now) => record(TradableAssetsSync.kind, now),
runBackfill: (DateTime now) => record(MarketDataHistorySync.kind, now),
runCleanup: (DateTime now, bool archive, int windowDays) =>
record(MarketDataRetention.kind, now),
);
final Handler handler = marketHistoryAdminHandler(
auth: _FakeAuthVerifier(),
connection: testDb!.connection,
adminFirebaseUids: <String>{'admin-uid'},
actions: actions,
);
final Response response = await _post(
handler,
path: '/v1/admin/market-data/resync',
bearer: 'admin-uid',
);
expect(response.statusCode, 202);
final Map<String, dynamic> body =
jsonDecode(await response.readAsString()) as Map<String, dynamic>;
expect((body['runIds'] as List<dynamic>), hasLength(2));
});
test('POST cleanup routes archive flag for admin', () async {
if (testDb == null) {
markTestSkipped('Set DATABASE_URL or TEST_DATABASE_URL for integration tests');
return;
}
bool? seenArchive;
final MarketHistoryAdminActions actions = MarketHistoryAdminActions(
connection: testDb!.connection,
runUniverse: (_) async {},
runBackfill: (_) async {},
runCleanup: (DateTime now, bool archive, int windowDays) async {
seenArchive = archive;
final SyncRunRecorder recorder = SyncRunRecorder(testDb!.connection);
await recorder.record(
MarketDataRetention.kind,
() async => const SyncRunCounts(rowsRemoved: 5),
now: now,
);
},
);
final Handler handler = marketHistoryAdminHandler(
auth: _FakeAuthVerifier(),
connection: testDb!.connection,
adminFirebaseUids: <String>{'admin-uid'},
actions: actions,
);
final Response response = await _post(
handler,
path: '/v1/admin/market-data/cleanup?archive=true',
bearer: 'admin-uid',
);
expect(response.statusCode, 202);
expect(seenArchive, isTrue);
});
test('POST trigger returns 403 for non-admin', () async {
if (testDb == null) {
markTestSkipped('Set DATABASE_URL or TEST_DATABASE_URL for integration tests');
return;
}
final Handler handler = marketHistoryAdminHandler(
auth: _FakeAuthVerifier(),
connection: testDb!.connection,
adminFirebaseUids: <String>{'admin-uid'},
actions: MarketHistoryAdminActions(
connection: testDb!.connection,
runUniverse: (_) async {},
runBackfill: (_) async {},
runCleanup: (_, __, ___) async {},
),
);
final Response response = await _post(
handler,
path: '/v1/admin/market-data/resync',
bearer: 'not-admin',
);
expect(response.statusCode, 403);
});
test('POST resync aborts orphaned in-progress run and returns 202', () async {
if (testDb == null) {
markTestSkipped('Set DATABASE_URL or TEST_DATABASE_URL for integration tests');
return;
}
await testDb!.connection.execute(
Sql.named(
'''
INSERT INTO market_data_sync_runs (kind, started_at, finished_at)
VALUES ('backfill', @started_at, NULL)
''',
),
parameters: <String, dynamic>{
'started_at': DateTime.utc(2026, 5, 27, 12),
},
);
final MarketHistoryAdminActions actions = MarketHistoryAdminActions(
connection: testDb!.connection,
runUniverse: (_) async {},
runBackfill: (_) async {},
runCleanup: (_, __, ___) async {},
);
final Handler handler = marketHistoryAdminHandler(
auth: _FakeAuthVerifier(),
connection: testDb!.connection,
adminFirebaseUids: <String>{'admin-uid'},
actions: actions,
);
final Response response = await _post(
handler,
path: '/v1/admin/market-data/resync',
bearer: 'admin-uid',
);
expect(response.statusCode, 202);
});
test('returns portal config for admin sync-runs', () async {
if (testDb == null) {
markTestSkipped('Set DATABASE_URL or TEST_DATABASE_URL for integration tests');
return;
}
final Handler handler = marketHistoryAdminHandler(
auth: _FakeAuthVerifier(),
connection: testDb!.connection,
adminFirebaseUids: <String>{'admin-uid'},
portalConfig: const MarketHistoryAdminPortalConfig(
archiveEnabled: true,
windowDays: 7,
retentionDays: 14,
syncEnabled: true,
),
);
final Response response = await _get(
handler,
path: '/v1/admin/market-history/sync-runs',
bearer: 'admin-uid',
);
expect(response.statusCode, 200);
final Map<String, dynamic> body =
jsonDecode(await response.readAsString()) as Map<String, dynamic>;
final Map<String, dynamic> config =
body['config'] as Map<String, dynamic>;
expect(config['archiveEnabled'], isTrue);
expect(config['windowDays'], 7);
expect(config['retentionDays'], 14);
expect(config['syncEnabled'], isTrue);
});
test('sync-runs response excludes raw snapshot fields', () async {
if (testDb == null) {
markTestSkipped('Set DATABASE_URL or TEST_DATABASE_URL for integration tests');
return;
}
final DateTime t0 = DateTime.utc(2026, 5, 26, 10);
await _insertRun(
testDb!.connection,
kind: 'backfill',
startedAt: t0,
finishedAt: t0.add(const Duration(minutes: 1)),
rowsWritten: 100,
error: 'batch failed: upstream 500',
);
final Handler handler = marketHistoryAdminHandler(
auth: _FakeAuthVerifier(),
connection: testDb!.connection,
adminFirebaseUids: <String>{'admin-uid'},
);
final Response response = await _get(
handler,
path: '/v1/admin/market-history/sync-runs',
bearer: 'admin-uid',
);
final Map<String, dynamic> body =
jsonDecode(await response.readAsString()) as Map<String, dynamic>;
final List<dynamic> all = <dynamic>[
...(body['runs'] as List<dynamic>? ?? <dynamic>[]),
...(body['pinned'] as List<dynamic>? ?? <dynamic>[]),
];
for (final dynamic item in all) {
final Map<String, dynamic> run = item as Map<String, dynamic>;
expect(run.containsKey('raw'), isFalse);
expect(run.containsKey('bars'), isFalse);
expect(run.containsKey('snapshots'), isFalse);
}
});
test('question-audit returns last-two bar deltas for admin', () async {
if (testDb == null) {
markTestSkipped('Set DATABASE_URL or TEST_DATABASE_URL for integration tests');
return;
}
final DateTime now = DateTime.now().toUtc();
final DateTime newerSlot = MarketHistoryFourHourSlot.lastCompletedSlotStart(now);
final DateTime olderSlot = newerSlot.subtract(const Duration(hours: 4));
final DateTime oldestSlot = olderSlot.subtract(const Duration(hours: 4));
await testDb!.connection.execute(
Sql.named(
'''
INSERT INTO tradable_assets (symbol, asset_class, status, tradable, refreshed_at)
VALUES ('AAA', 'us_equity', 'active', true, @refreshed_at)
''',
),
parameters: <String, dynamic>{'refreshed_at': now},
);
Future<void> insertBar({
required DateTime asOf,
required num open,
required num high,
required num low,
required num close,
required num volume,
}) async {
await testDb!.connection.execute(
Sql.named(
'''
INSERT INTO market_data_snapshots (
symbol, asset_class, feed, metric, timeframe, price, volume, as_of, raw
) VALUES (
'AAA', 'us_equity', 'iex', 'bar', '4Hour', @close, @volume, @as_of, @raw::jsonb
)
''',
),
parameters: <String, dynamic>{
'as_of': asOf,
'close': close,
'volume': volume,
'raw': jsonEncode(<String, dynamic>{
'o': open,
'h': high,
'l': low,
'c': close,
'v': volume,
'slot_start': asOf.toIso8601String(),
}),
},
);
}
await insertBar(
asOf: oldestSlot,
open: 8,
high: 9,
low: 7,
close: 8,
volume: 80,
);
await insertBar(
asOf: olderSlot,
open: 10,
high: 12,
low: 8,
close: 10,
volume: 100,
);
await insertBar(
asOf: newerSlot,
open: 12,
high: 14,
low: 10,
close: 12,
volume: 150,
);
final Handler handler = marketHistoryAdminHandler(
auth: _FakeAuthVerifier(),
connection: testDb!.connection,
adminFirebaseUids: <String>{'admin-uid'},
portalConfig: const MarketHistoryAdminPortalConfig(
archiveEnabled: false,
windowDays: 7,
retentionDays: 7,
syncEnabled: false,
),
);
final Response response = await _get(
handler,
path: '/v1/admin/market-history/question-audit',
bearer: 'admin-uid',
);
expect(response.statusCode, 200);
final Map<String, dynamic> body =
jsonDecode(await response.readAsString()) as Map<String, dynamic>;
expect(body['windowDays'], 7);
expect(body['canStepNewer'], isFalse);
expect(body['canStepOlder'], isTrue);
expect(
DateTime.parse(body['compareUntil'] as String).toUtc(),
MarketHistoryFourHourSlot.endExclusive(newerSlot),
);
final List<dynamic> assets = body['assets'] as List<dynamic>;
expect(assets, hasLength(1));
final Map<String, dynamic> aaa = assets.first as Map<String, dynamic>;
expect(aaa['symbol'], 'AAA');
expect(aaa['priceDelta'], 2);
expect(aaa['volumeDelta'], 50);
expect(aaa.containsKey('raw'), isFalse);
final Map<String, dynamic> older =
aaa['olderSlot'] as Map<String, dynamic>;
final Map<String, dynamic> newer =
aaa['newerSlot'] as Map<String, dynamic>;
expect(older['avgPrice'], 10);
expect(older['volume'], 100);
expect(older['open'], 10);
expect(newer['avgPrice'], 12);
expect(newer['volume'], 150);
expect(newer['close'], 12);
expect(
DateTime.parse(body['newerSlotStart'] as String).toUtc(),
newerSlot,
);
expect(
DateTime.parse(body['olderSlotStart'] as String).toUtc(),
olderSlot,
);
final String olderBound = body['stepOlderCompareUntil'] as String;
final Response stepped = await _get(
handler,
path:
'/v1/admin/market-history/question-audit?asOf=${Uri.encodeQueryComponent(olderBound)}',
bearer: 'admin-uid',
);
expect(stepped.statusCode, 200);
final Map<String, dynamic> steppedBody =
jsonDecode(await stepped.readAsString()) as Map<String, dynamic>;
expect(steppedBody['canStepNewer'], isTrue);
final Map<String, dynamic> steppedAsset =
(steppedBody['assets'] as List<dynamic>).first as Map<String, dynamic>;
expect((steppedAsset['newerSlot'] as Map<String, dynamic>)['close'], 10);
expect(
DateTime.parse(steppedBody['newerSlotStart'] as String).toUtc(),
olderSlot,
);
});
test('question-audit omits symbols missing either slot', () async {
if (testDb == null) {
markTestSkipped('Set DATABASE_URL or TEST_DATABASE_URL for integration tests');
return;
}
final DateTime now = DateTime.now().toUtc();
final DateTime newerSlot = MarketHistoryFourHourSlot.lastCompletedSlotStart(now);
final DateTime olderSlot = newerSlot.subtract(const Duration(hours: 4));
await testDb!.connection.execute(
Sql.named(
'''
INSERT INTO tradable_assets (symbol, asset_class, status, tradable, refreshed_at)
VALUES
('AAA', 'us_equity', 'active', true, @refreshed_at),
('BBB', 'us_equity', 'active', true, @refreshed_at)
''',
),
parameters: <String, dynamic>{'refreshed_at': now},
);
Future<void> insertBar(String symbol, DateTime asOf, num close) async {
await testDb!.connection.execute(
Sql.named(
'''
INSERT INTO market_data_snapshots (
symbol, asset_class, feed, metric, timeframe, price, volume, as_of, raw
) VALUES (
@symbol, 'us_equity', 'iex', 'bar', '4Hour', @close, 100, @as_of,
@raw::jsonb
)
''',
),
parameters: <String, dynamic>{
'symbol': symbol,
'as_of': asOf,
'close': close,
'raw': jsonEncode(<String, dynamic>{
'o': close,
'h': close,
'l': close,
'c': close,
'v': 100,
'slot_start': asOf.toIso8601String(),
}),
},
);
}
await insertBar('AAA', olderSlot, 10);
await insertBar('AAA', newerSlot, 12);
await insertBar('BBB', newerSlot, 20);
final Handler handler = marketHistoryAdminHandler(
auth: _FakeAuthVerifier(),
connection: testDb!.connection,
adminFirebaseUids: <String>{'admin-uid'},
);
final Response response = await _get(
handler,
path: '/v1/admin/market-history/question-audit',
bearer: 'admin-uid',
);
expect(response.statusCode, 200);
final Map<String, dynamic> body =
jsonDecode(await response.readAsString()) as Map<String, dynamic>;
final List<dynamic> assets = body['assets'] as List<dynamic>;
expect(assets, hasLength(1));
expect((assets.first as Map<String, dynamic>)['symbol'], 'AAA');
});
test('week-coverage validates slot consistency for admin', () async {
if (testDb == null) {
markTestSkipped('Set DATABASE_URL or TEST_DATABASE_URL for integration tests');
return;
}
final DateTime now = DateTime.now().toUtc();
final DateTime slotStart =
MarketHistoryFourHourSlot.lastCompletedSlotStart(now);
await testDb!.connection.execute(
Sql.named(
'''
INSERT INTO tradable_assets (symbol, asset_class, status, tradable, refreshed_at)
VALUES ('AAA', 'us_equity', 'active', true, @refreshed_at)
''',
),
parameters: <String, dynamic>{'refreshed_at': now},
);
await testDb!.connection.execute(
Sql.named(
'''
INSERT INTO market_data_snapshots (
symbol, asset_class, feed, metric, timeframe, price, as_of, raw
) VALUES (
'AAA', 'us_equity', 'iex', 'bar', '4Hour', 100,
@as_of,
@raw::jsonb
)
''',
),
parameters: <String, dynamic>{
'as_of': slotStart.add(const Duration(hours: 1)),
'raw': jsonEncode(<String, dynamic>{
'slot_start': slotStart.toIso8601String(),
}),
},
);
final Handler handler = marketHistoryAdminHandler(
auth: _FakeAuthVerifier(),
connection: testDb!.connection,
adminFirebaseUids: <String>{'admin-uid'},
portalConfig: const MarketHistoryAdminPortalConfig(
archiveEnabled: false,
windowDays: 7,
retentionDays: 7,
syncEnabled: false,
),
);
final Response response = await _get(
handler,
path: '/v1/admin/market-history/week-coverage',
bearer: 'admin-uid',
);
expect(response.statusCode, 200);
final Map<String, dynamic> body =
jsonDecode(await response.readAsString()) as Map<String, dynamic>;
expect(body['windowDays'], 7);
expect(body['slotsPerDay'], 6);
expect(body['symbolCount'], 1);
expect(body['isConsistent'], isFalse);
final List<dynamic> days = body['days'] as List<dynamic>;
expect(days, hasLength(7));
final Map<String, dynamic> today =
days.last as Map<String, dynamic>;
expect(today['fullySyncedSlots'], 1);
expect(today['completedSlots'], greaterThan(0));
});
test('resync returns 503 when sync is disabled in portal config', () async {
if (testDb == null) {
markTestSkipped('Set DATABASE_URL or TEST_DATABASE_URL for integration tests');
return;
}
final Handler handler = marketHistoryAdminHandler(
auth: _FakeAuthVerifier(),
connection: testDb!.connection,
adminFirebaseUids: <String>{'admin-uid'},
actions: null,
portalConfig: const MarketHistoryAdminPortalConfig(
archiveEnabled: false,
windowDays: 7,
retentionDays: 7,
syncEnabled: false,
),
);
final Response response = await _post(
handler,
path: '/v1/admin/market-data/resync',
bearer: 'admin-uid',
);
expect(response.statusCode, 503);
final Map<String, dynamic> body =
jsonDecode(await response.readAsString()) as Map<String, dynamic>;
expect(body['error'], contains('MARKET_HISTORY_SYNC_ENABLED=false'));
});
}

View File

@ -0,0 +1,184 @@
@Tags(<String>['integration', 'postgres', 'risk'])
library;
import 'dart:convert';
import 'package:cyberhybridhub_server/firebase_auth.dart';
import 'package:cyberhybridhub_server/handlers/market_history_admin_handler.dart';
import 'package:cyberhybridhub_server/trading/market_history_admin_logic.dart';
import 'package:postgres/postgres.dart';
import 'package:shelf/shelf.dart';
import 'package:test/test.dart';
import '../helpers/admin_sync_run_fixtures.dart';
import '../helpers/test_db.dart';
class _FakeAuthVerifier extends FirebaseAuthVerifier {
_FakeAuthVerifier() : super('test-key');
@override
Future<String?> verifyBearerToken(String? authorization) async {
if (authorization == null || !authorization.startsWith('Bearer ')) {
return null;
}
final String token = authorization.substring('Bearer '.length).trim();
return token.isEmpty ? null : token;
}
}
Future<Response> _get(Handler handler, {required String bearer}) async {
return await Future<Response>.value(
handler(
Request(
'GET',
Uri.parse('http://localhost/v1/admin/market-history/sync-runs'),
headers: <String, String>{'Authorization': 'Bearer $bearer'},
),
),
);
}
Future<void> _insertRecord(
Connection connection,
AdminSyncRunRecord run,
) async {
await connection.execute(
Sql.named(
'''
INSERT INTO market_data_sync_runs
(kind, started_at, finished_at, rows_written, rows_removed, error)
VALUES
(@kind, @started_at, @finished_at, @rows_written, @rows_removed, @error)
''',
),
parameters: <String, dynamic>{
'kind': run.kind,
'started_at': run.startedAt.toUtc(),
'finished_at': run.finishedAt?.toUtc(),
'rows_written': run.rowsWritten,
'rows_removed': run.rowsRemoved,
'error': run.error,
},
);
}
void main() {
TestDb? testDb;
setUpAll(() async {
testDb = await TestDb.open();
});
tearDown(() async {
if (testDb != null) {
await testDb!.truncateTradingTables();
}
});
tearDownAll(() async {
await testDb?.close();
});
group('Section 8 risk checklist — server', () {
test('mixed kinds interleaving pins only unresolved backfill failures', () async {
if (testDb == null) {
markTestSkipped('Set DATABASE_URL or TEST_DATABASE_URL for integration tests');
return;
}
final DateTime base = DateTime.utc(2026, 5, 27, 12);
for (final AdminSyncRunRecord run in fixtureMixedKindsMixedOutcomes(base)) {
await _insertRecord(testDb!.connection, run);
}
final Handler handler = marketHistoryAdminHandler(
auth: _FakeAuthVerifier(),
connection: testDb!.connection,
adminFirebaseUids: <String>{'admin-uid'},
);
final Response response = await _get(handler, bearer: 'admin-uid');
expect(response.statusCode, 200);
final Map<String, dynamic> body =
jsonDecode(await response.readAsString()) as Map<String, dynamic>;
final List<dynamic> pinned = body['pinned'] as List<dynamic>;
expect(pinned, isNotEmpty);
expect(
pinned.every(
(dynamic item) => (item as Map<String, dynamic>)['kind'] == 'backfill',
),
isTrue,
);
expect(
pinned.any(
(dynamic item) =>
(item as Map<String, dynamic>)['severity'] == 'rate_limit',
),
isTrue,
);
});
test('empty database returns safe empty payload', () async {
if (testDb == null) {
markTestSkipped('Set DATABASE_URL or TEST_DATABASE_URL for integration tests');
return;
}
final Handler handler = marketHistoryAdminHandler(
auth: _FakeAuthVerifier(),
connection: testDb!.connection,
adminFirebaseUids: <String>{'admin-uid'},
);
final Response response = await _get(handler, bearer: 'admin-uid');
expect(response.statusCode, 200);
final Map<String, dynamic> body =
jsonDecode(await response.readAsString()) as Map<String, dynamic>;
expect(body['runs'], isEmpty);
expect(body['pinned'], isEmpty);
expect(body['nextBefore'], isNull);
});
test('rate-limit capitalization variants surface as rate_limit severity', () async {
if (testDb == null) {
markTestSkipped('Set DATABASE_URL or TEST_DATABASE_URL for integration tests');
return;
}
final DateTime t0 = DateTime.utc(2026, 5, 27, 10);
for (final String message in <String>[
'HTTP 429 Too Many Requests',
'Rate Limit exceeded',
'RATE LIMITED by upstream',
]) {
await _insertRecord(
testDb!.connection,
adminSyncRun(
id: 0,
kind: 'backfill',
startedAt: t0,
finishedAt: t0.add(const Duration(minutes: 1)),
error: message,
),
);
final Handler handler = marketHistoryAdminHandler(
auth: _FakeAuthVerifier(),
connection: testDb!.connection,
adminFirebaseUids: <String>{'admin-uid'},
);
final Response response = await _get(handler, bearer: 'admin-uid');
final Map<String, dynamic> body =
jsonDecode(await response.readAsString()) as Map<String, dynamic>;
final List<dynamic> pinned = body['pinned'] as List<dynamic>;
expect(
(pinned.first as Map<String, dynamic>)['severity'],
'rate_limit',
reason: message,
);
await testDb!.truncateTradingTables();
}
});
});
}

View File

@ -0,0 +1,169 @@
@Tags(['integration', 'postgres'])
library;
import 'dart:math';
import 'package:cyberhybridhub_server/alpaca/alpaca_models.dart';
import 'package:cyberhybridhub_server/trading/market_history_config.dart';
import 'package:cyberhybridhub_server/trading/market_history_query.dart';
import 'package:cyberhybridhub_server/trading/tradable_assets_db.dart';
import 'package:test/test.dart';
import '../helpers/test_db.dart';
void main() {
TestDb? testDb;
setUpAll(() async {
testDb = await TestDb.open();
});
tearDown(() async {
if (testDb != null) {
await testDb!.truncateTradingTables();
}
});
tearDownAll(() async {
await testDb?.close();
});
Future<void> seedBars({
required String symbol,
required List<num> closes,
required DateTime asOf,
}) async {
for (int i = 0; i < closes.length; i++) {
await testDb!.marketDataDb.upsertSnapshot(
symbol: symbol,
metric: 'bar',
timeframe: MarketHistoryConfig.barTimeframe,
asOf: asOf.subtract(Duration(hours: 4 * (closes.length - 1 - i))),
price: closes[i],
volume: 1000,
);
}
}
group('weeklyMovers', () {
test('returns symbols with minBars and open/current closes', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final DateTime asOf = DateTime.utc(2026, 5, 26, 12);
final DateTime windowStart = asOf.subtract(const Duration(days: 7));
await TradableAssetsDb(testDb!.connection).upsertAll(
<AlpacaAsset>[
AlpacaAsset(
symbol: 'SPY',
assetClass: 'us_equity',
tradable: true,
fractionable: true,
status: 'active',
),
AlpacaAsset(
symbol: 'STALE',
assetClass: 'us_equity',
tradable: true,
fractionable: true,
status: 'active',
),
],
now: asOf,
);
await seedBars(
symbol: 'SPY',
closes: <num>[500, 501, 502, 503, 504, 505, 510],
asOf: asOf,
);
await seedBars(
symbol: 'STALE',
closes: <num>[100, 101, 102, 103, 104],
asOf: asOf,
);
// Newest STALE bar is 9 days before asOf (> 2d stale).
await testDb!.marketDataDb.upsertSnapshot(
symbol: 'STALE',
metric: 'bar',
timeframe: MarketHistoryConfig.barTimeframe,
asOf: asOf.subtract(const Duration(days: 9)),
price: 104,
);
final MarketHistoryQuery query =
MarketHistoryQuery(connection: testDb!.connection);
final List<WeeklyMover> movers = await query.weeklyMovers(
asOf: asOf,
minBars: 5,
);
expect(movers, hasLength(1));
expect(movers.single.symbol, 'SPY');
expect(movers.single.openClose, 500);
expect(movers.single.currentClose, 505);
expect(movers.single.days, 6);
});
test('deterministic selection order with seeded random', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final DateTime asOf = DateTime.utc(2026, 5, 26, 12);
await TradableAssetsDb(testDb!.connection).upsertAll(
<AlpacaAsset>[
AlpacaAsset(
symbol: 'AAA',
assetClass: 'us_equity',
tradable: true,
fractionable: true,
status: 'active',
),
AlpacaAsset(
symbol: 'ZZZ',
assetClass: 'us_equity',
tradable: true,
fractionable: true,
status: 'active',
),
],
now: asOf,
);
for (final String symbol in <String>['AAA', 'ZZZ']) {
await seedBars(
symbol: symbol,
closes: <num>[10, 11, 12, 13, 14, 15, 16],
asOf: asOf,
);
}
final MarketHistoryQuery query =
MarketHistoryQuery(connection: testDb!.connection);
final List<WeeklyMover> a = await query.weeklyMovers(
asOf: asOf,
minBars: 5,
random: Random(42),
);
final List<WeeklyMover> b = await query.weeklyMovers(
asOf: asOf,
minBars: 5,
random: Random(42),
);
expect(a.map((WeeklyMover m) => m.symbol).toList(),
b.map((WeeklyMover m) => m.symbol).toList());
expect(a.map((WeeklyMover m) => m.symbol).toList(), isNotEmpty);
});
});
}

View File

@ -0,0 +1,361 @@
@Tags(['integration', 'postgres'])
library;
import 'package:cyberhybridhub_server/trading/market_data_history.dart';
import 'package:cyberhybridhub_server/trading/market_data_retention.dart';
import 'package:cyberhybridhub_server/trading/sync_run_recorder.dart';
import 'package:cyberhybridhub_server/trading/tradable_assets_sync.dart';
import 'package:cyberhybridhub_server/workers/market_history_scheduler.dart';
import 'package:cyberhybridhub_server/workers/market_history_scheduler_config.dart';
import 'package:postgres/postgres.dart';
import 'package:test/test.dart';
import '../helpers/test_db.dart';
void main() {
TestDb? testDb;
setUpAll(() async {
testDb = await TestDb.open();
});
tearDown(() async {
if (testDb != null) {
await testDb!.truncateTradingTables();
}
});
tearDownAll(() async {
await testDb?.close();
});
MarketHistoryScheduler scheduler({
MarketHistorySchedulerConfig? config,
Future<void> Function(DateTime now)? runUniverse,
Future<void> Function(DateTime now)? runBackfill,
Future<void> Function(DateTime now)? runCleanup,
Future<bool> Function(DateTime now)? backfillIsDue,
}) {
return MarketHistoryScheduler(
connection: testDb!.connection,
config: config ?? const MarketHistorySchedulerConfig(),
runUniverse: runUniverse,
runBackfill: runBackfill,
runCleanup: runCleanup,
backfillIsDue: backfillIsDue,
);
}
Future<void> recordStage(String kind, DateTime now) async {
final SyncRunRecorder recorder = SyncRunRecorder(testDb!.connection);
await recorder.record(
kind,
() async => const SyncRunCounts(rowsWritten: 1),
now: now,
);
}
group('runIfDue', () {
test('cold start runs universe, backfill, cleanup in order', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final List<String> order = <String>[];
final DateTime t0 = DateTime.utc(2026, 5, 26, 12);
await scheduler(
runUniverse: (DateTime now) async {
order.add('universe');
await recordStage(TradableAssetsSync.kind, now);
},
runBackfill: (DateTime now) async {
order.add('backfill');
await recordStage(MarketDataHistorySync.kind, now);
},
runCleanup: (DateTime now) async {
order.add('cleanup');
await recordStage(MarketDataRetention.kind, now);
},
).runIfDue(t0);
expect(order, <String>['universe', 'backfill', 'cleanup']);
final Result runs = await testDb!.connection.execute(
'''
SELECT kind FROM market_data_sync_runs
ORDER BY id ASC
''',
);
expect(
runs.map((ResultRow r) => r[0]).toList(),
<String>['universe', 'backfill', 'cleanup'],
);
});
test('same-day re-run does not execute stages again', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final DateTime t0 = DateTime.utc(2026, 5, 26, 12);
int calls = 0;
final MarketHistoryScheduler s = scheduler(
runUniverse: (DateTime now) async {
calls++;
await recordStage(TradableAssetsSync.kind, now);
},
runBackfill: (DateTime now) async {
calls++;
await recordStage(MarketDataHistorySync.kind, now);
},
runCleanup: (DateTime now) async {
calls++;
await recordStage(MarketDataRetention.kind, now);
},
);
await s.runIfDue(t0);
final int runsBefore =
(await testDb!.connection.execute('SELECT COUNT(*) FROM market_data_sync_runs'))
.first[0]! as int;
await s.runIfDue(t0.add(const Duration(hours: 1)));
expect(calls, 3);
final int runsAfter =
(await testDb!.connection.execute('SELECT COUNT(*) FROM market_data_sync_runs'))
.first[0]! as int;
expect(runsAfter, runsBefore);
});
test('next day runs all stages again', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final DateTime t0 = DateTime.utc(2026, 5, 26, 12);
int calls = 0;
final MarketHistoryScheduler s = scheduler(
runUniverse: (DateTime now) async {
calls++;
await recordStage(TradableAssetsSync.kind, now);
},
runBackfill: (DateTime now) async {
calls++;
await recordStage(MarketDataHistorySync.kind, now);
},
runCleanup: (DateTime now) async {
calls++;
await recordStage(MarketDataRetention.kind, now);
},
);
await s.runIfDue(t0);
await s.runIfDue(t0.add(const Duration(hours: 24)));
expect(calls, 6);
});
test('per-stage cadence: at T0+24h only backfill and cleanup run', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final DateTime t0 = DateTime.utc(2026, 5, 26, 12);
final List<String> ran = <String>[];
final MarketHistoryScheduler s = scheduler(
config: const MarketHistorySchedulerConfig(
universeRefreshHours: 48,
historySyncHours: 24,
cleanupHours: 24,
),
runUniverse: (DateTime now) async {
ran.add('universe');
await recordStage(TradableAssetsSync.kind, now);
},
runBackfill: (DateTime now) async {
ran.add('backfill');
await recordStage(MarketDataHistorySync.kind, now);
},
runCleanup: (DateTime now) async {
ran.add('cleanup');
await recordStage(MarketDataRetention.kind, now);
},
);
await s.runIfDue(t0);
await s.runIfDue(t0.add(const Duration(hours: 24)));
expect(ran, <String>['universe', 'backfill', 'cleanup', 'backfill', 'cleanup']);
});
test('failure isolation: backfill error still runs cleanup', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final DateTime t0 = DateTime.utc(2026, 5, 26, 12);
final Connection connection = testDb!.connection;
await scheduler(
runUniverse: (DateTime now) => recordStage(TradableAssetsSync.kind, now),
runBackfill: (DateTime now) async {
final SyncRunRecorder recorder = SyncRunRecorder(connection);
await recorder.record(
MarketDataHistorySync.kind,
() async {
throw Exception('backfill boom');
},
now: now,
);
},
runCleanup: (DateTime now) => recordStage(MarketDataRetention.kind, now),
).runIfDue(t0);
final Result runs = await connection.execute(
'''
SELECT kind, error IS NOT NULL AS has_error
FROM market_data_sync_runs
ORDER BY id ASC
''',
);
expect(runs, hasLength(3));
expect(runs[0][0], 'universe');
expect(runs[0][1], isFalse);
expect(runs[1][0], 'backfill');
expect(runs[1][1], isTrue);
expect(runs[2][0], 'cleanup');
expect(runs[2][1], isFalse);
});
test('slot-based backfill runs again same day when new slot is pending',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
bool pending = true;
int backfillCalls = 0;
final MarketHistoryScheduler s = scheduler(
backfillIsDue: (DateTime now) async => pending,
runUniverse: (DateTime now) => recordStage(TradableAssetsSync.kind, now),
runBackfill: (DateTime now) async {
backfillCalls++;
await recordStage(MarketDataHistorySync.kind, now);
pending = false;
},
runCleanup: (DateTime now) => recordStage(MarketDataRetention.kind, now),
);
final DateTime t0 = DateTime.utc(2026, 5, 26, 12);
await s.runIfDue(t0);
expect(backfillCalls, 1);
pending = true;
await s.runIfDue(t0.add(const Duration(hours: 1)));
expect(backfillCalls, 2);
});
test('aborts orphaned in-progress run before starting pipeline', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await testDb!.connection.execute(
Sql.named(
'''
INSERT INTO market_data_sync_runs (kind, started_at, finished_at)
VALUES ('backfill', @started_at, NULL)
''',
),
parameters: <String, dynamic>{
'started_at': DateTime.utc(2026, 5, 26, 10),
},
);
final DateTime t0 = DateTime.utc(2026, 5, 26, 12);
await scheduler(
runUniverse: (DateTime now) => recordStage(TradableAssetsSync.kind, now),
runBackfill: (DateTime now) => recordStage(MarketDataHistorySync.kind, now),
runCleanup: (DateTime now) => recordStage(MarketDataRetention.kind, now),
).runIfDue(t0);
final Result rows = await testDb!.connection.execute(
'''
SELECT kind, finished_at IS NULL AS open, error
FROM market_data_sync_runs
ORDER BY id ASC
''',
);
expect(rows.first[0], 'backfill');
expect(rows.first[1], false);
expect(rows.first[2], contains('aborted'));
expect(rows, hasLength(4));
});
test('syncHourUtc blocks before hour and same UTC day', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final MarketHistoryScheduler s = scheduler(
config: const MarketHistorySchedulerConfig(syncHourUtc: 10),
runUniverse: (DateTime now) async {
await recordStage(TradableAssetsSync.kind, now);
},
runBackfill: (DateTime now) async {
await recordStage(MarketDataHistorySync.kind, now);
},
runCleanup: (DateTime now) async {
await recordStage(MarketDataRetention.kind, now);
},
);
final MarketHistorySchedulerReport beforeHour =
await s.runIfDue(DateTime.utc(2026, 5, 26, 9));
expect(beforeHour.ranStages, isEmpty);
final DateTime tRun = DateTime.utc(2026, 5, 26, 11);
final MarketHistorySchedulerReport first =
await s.runIfDue(tRun);
expect(first.ranStages, hasLength(3));
final MarketHistorySchedulerReport sameDay =
await s.runIfDue(DateTime.utc(2026, 5, 26, 12));
expect(sameDay.ranStages, isEmpty);
});
});
}

View File

@ -0,0 +1,411 @@
@Tags(['integration', 'postgres'])
library;
import 'package:cyberhybridhub_server/trading/market_data_history.dart';
import 'package:cyberhybridhub_server/trading/sync_run_recorder.dart';
import 'package:postgres/postgres.dart';
import 'package:test/test.dart';
import '../helpers/test_db.dart';
void main() {
TestDb? testDb;
setUpAll(() async {
testDb = await TestDb.open();
});
tearDown(() async {
if (testDb != null) {
await testDb!.truncateTradingTables();
}
});
tearDownAll(() async {
await testDb?.close();
});
group('market_data_snapshots — timeframe + unique observation', () {
test(
'upsert on (symbol, metric, timeframe, as_of) keeps a single row '
'and updates price', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final Connection connection = testDb!.connection;
final DateTime asOf = DateTime.utc(2026, 5, 23, 13);
// First insert.
await connection.execute(
Sql.named(
'''
INSERT INTO market_data_snapshots (
symbol, metric, timeframe, price, as_of
) VALUES (
'SPY', 'bar', '1Day', 500, @as_of
)
ON CONFLICT (symbol, metric, timeframe, as_of) DO UPDATE
SET price = EXCLUDED.price
''',
),
parameters: <String, dynamic>{'as_of': asOf},
);
// Re-upsert the same observation with a new price.
await connection.execute(
Sql.named(
'''
INSERT INTO market_data_snapshots (
symbol, metric, timeframe, price, as_of
) VALUES (
'SPY', 'bar', '1Day', 505, @as_of
)
ON CONFLICT (symbol, metric, timeframe, as_of) DO UPDATE
SET price = EXCLUDED.price
''',
),
parameters: <String, dynamic>{'as_of': asOf},
);
final Result rows = await connection.execute(
Sql.named(
'''
SELECT price::float
FROM market_data_snapshots
WHERE symbol = 'SPY'
AND metric = 'bar'
AND timeframe = '1Day'
AND as_of = @as_of
''',
),
parameters: <String, dynamic>{'as_of': asOf},
);
expect(rows, hasLength(1));
expect(rows.first[0], 505.0);
});
test('timeframe defaults to tick and accepts 1Min/1Hour/1Day', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final Connection connection = testDb!.connection;
final DateTime asOfBase = DateTime.utc(2026, 5, 23, 14);
// Insert without naming timeframe must default to 'tick'.
await connection.execute(
Sql.named(
'''
INSERT INTO market_data_snapshots (symbol, metric, price, as_of)
VALUES ('SPY', 'last_trade', 491, @as_of)
''',
),
parameters: <String, dynamic>{'as_of': asOfBase},
);
final Result defaulted = await connection.execute(
Sql.named(
'''
SELECT timeframe
FROM market_data_snapshots
WHERE symbol = 'SPY' AND as_of = @as_of
''',
),
parameters: <String, dynamic>{'as_of': asOfBase},
);
expect(defaulted, hasLength(1));
expect(defaulted.first[0], 'tick');
// Each accepted timeframe value can be inserted.
const List<String> timeframes = <String>['1Min', '1Hour', '4Hour', '1Day'];
for (int i = 0; i < timeframes.length; i++) {
final String tf = timeframes[i];
final DateTime asOf = asOfBase.add(Duration(minutes: i + 1));
await connection.execute(
Sql.named(
'''
INSERT INTO market_data_snapshots (
symbol, metric, timeframe, price, as_of
) VALUES (
'SPY', 'bar', @timeframe, 500, @as_of
)
''',
),
parameters: <String, dynamic>{'timeframe': tf, 'as_of': asOf},
);
}
final Result accepted = await connection.execute(
Sql.named(
'''
SELECT timeframe
FROM market_data_snapshots
WHERE symbol = 'SPY' AND metric = 'bar'
ORDER BY timeframe
''',
),
);
expect(
accepted.map((ResultRow r) => r[0]).toList(),
<String>['1Day', '1Hour', '1Min', '4Hour'],
);
});
});
group('tradable_assets', () {
test('rejects duplicate symbol via primary key', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final Connection connection = testDb!.connection;
await connection.execute(
'''
INSERT INTO tradable_assets (symbol, asset_class, status, tradable)
VALUES ('SPY', 'us_equity', 'active', true)
''',
);
await expectLater(
connection.execute(
'''
INSERT INTO tradable_assets (symbol, asset_class, status, tradable)
VALUES ('SPY', 'us_equity', 'active', true)
''',
),
throwsA(isA<ServerException>()),
);
});
test(
'status/tradable lookup uses the tradable_assets_status_idx index',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final Connection connection = testDb!.connection;
// Seed enough rows that the planner has a real choice.
for (int i = 0; i < 60; i++) {
await connection.execute(
Sql.named(
'''
INSERT INTO tradable_assets (symbol, asset_class, status, tradable)
VALUES (@symbol, 'us_equity', @status, @tradable)
''',
),
parameters: <String, dynamic>{
'symbol': 'SYM$i',
'status': i.isEven ? 'active' : 'inactive',
'tradable': i.isEven,
},
);
}
await connection.execute('ANALYZE tradable_assets');
// Force the planner to consider the index even on a small dataset.
// SET LOCAL only persists inside an explicit transaction; use plain
// SET on this short-lived test session and restore afterwards.
await connection.execute('SET enable_seqscan = off');
final Result plan;
try {
plan = await connection.execute(
'''
EXPLAIN (FORMAT TEXT)
SELECT symbol FROM tradable_assets
WHERE status = 'active' AND tradable = true
''',
);
} finally {
await connection.execute('SET enable_seqscan = on');
}
final String planText =
plan.map((ResultRow r) => r[0]?.toString() ?? '').join('\n');
// Either an Index Scan or a Bitmap Index Scan referencing the index
// is acceptable; both prove the index is in use.
expect(
planText.contains('tradable_assets_status_idx'),
isTrue,
reason: 'expected plan to use tradable_assets_status_idx, '
'got:\n$planText',
);
});
});
group('market_data_sync_runs', () {
test(
'records kind/started_at/finished_at/rows_written/rows_removed/error',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final Connection connection = testDb!.connection;
final DateTime started = DateTime.utc(2026, 5, 26, 10);
final DateTime finished = DateTime.utc(2026, 5, 26, 10, 0, 5);
await connection.execute(
Sql.named(
'''
INSERT INTO market_data_sync_runs (
kind, started_at, finished_at, rows_written, rows_removed, error
) VALUES (
'universe', @started, @finished, 12, 0, NULL
)
''',
),
parameters: <String, dynamic>{
'started': started,
'finished': finished,
},
);
await connection.execute(
Sql.named(
'''
INSERT INTO market_data_sync_runs (
kind, started_at, finished_at, rows_written, rows_removed, error
) VALUES (
'cleanup', @started, @finished, 0, 42, 'partial outage: MSFT'
)
''',
),
parameters: <String, dynamic>{
'started': started,
'finished': finished,
},
);
// kind must reject NULL.
await expectLater(
connection.execute(
'''
INSERT INTO market_data_sync_runs (started_at)
VALUES (now())
''',
),
throwsA(isA<ServerException>()),
);
final Result runs = await connection.execute(
'''
SELECT kind, rows_written, rows_removed, error
FROM market_data_sync_runs
ORDER BY kind
''',
);
expect(
runs.map((ResultRow r) => r[0]).toList(),
<String>['cleanup', 'universe'],
);
expect((runs.elementAt(0)[1]! as num).toInt(), 0);
expect((runs.elementAt(0)[2]! as num).toInt(), 42);
expect(runs.elementAt(0)[3], 'partial outage: MSFT');
expect((runs.elementAt(1)[1]! as num).toInt(), 12);
expect((runs.elementAt(1)[2]! as num).toInt(), 0);
expect(runs.elementAt(1)[3], isNull);
});
test('slots_synced defaults to 0 and is stored on backfill runs', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final SyncRunRecorder recorder = SyncRunRecorder(testDb!.connection);
await recorder.record(
MarketDataHistorySync.kind,
() async => const SyncRunCounts(rowsWritten: 9, slotsSynced: 3),
now: DateTime.utc(2026, 5, 26, 12),
);
final Result rows = await testDb!.connection.execute(
'''
SELECT slots_synced, rows_written
FROM market_data_sync_runs
WHERE kind = 'backfill'
''',
);
expect((rows.single[0]! as num).toInt(), 3);
expect((rows.single[1]! as num).toInt(), 9);
});
});
group('008 four-hour history constraints', () {
test('rejects invalid bar timeframe', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await expectLater(
testDb!.connection.execute(
Sql.named(
'''
INSERT INTO market_data_snapshots (
symbol, metric, timeframe, price, as_of
) VALUES (
'SPY', 'bar', '2Hour', 100, @as_of
)
''',
),
parameters: <String, dynamic>{
'as_of': DateTime.utc(2026, 5, 26, 8),
},
),
throwsA(isA<ServerException>()),
);
});
test('accepts 4Hour bar rows and partial index exists', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
await testDb!.marketDataDb.upsertSnapshot(
symbol: 'SPY',
metric: 'bar',
timeframe: '4Hour',
asOf: DateTime.utc(2026, 5, 26, 8),
price: 500,
);
final Result indexes = await testDb!.connection.execute(
'''
SELECT indexname
FROM pg_indexes
WHERE tablename = 'market_data_snapshots'
AND indexname = 'market_data_snapshots_bar_4h_idx'
''',
);
expect(indexes, hasLength(1));
});
});
}

View File

@ -0,0 +1,83 @@
@Tags(['integration', 'postgres'])
library;
import 'package:cyberhybridhub_server/trading/market_history_config.dart';
import 'package:cyberhybridhub_server/trading/market_history_four_hour_slot.dart';
import 'package:cyberhybridhub_server/trading/market_history_week_coverage.dart';
import 'package:cyberhybridhub_server/trading/tradable_assets_db.dart';
import 'package:cyberhybridhub_server/alpaca/alpaca_models.dart';
import 'package:test/test.dart';
import '../helpers/test_db.dart';
void main() {
TestDb? testDb;
setUpAll(() async {
testDb = await TestDb.open();
});
tearDown(() async {
if (testDb != null) {
await testDb!.truncateTradingTables();
}
});
tearDownAll(() async {
await testDb?.close();
});
test('no_data placeholder counts toward fullySynced slot', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final DateTime now = DateTime.utc(2026, 5, 31, 0, 30);
final DateTime slotStart = DateTime.utc(2026, 5, 30, 20);
await TradableAssetsDb(testDb!.connection).upsertAll(
<AlpacaAsset>[
AlpacaAsset(
symbol: 'AAA',
assetClass: 'us_equity',
status: 'active',
tradable: true,
fractionable: true,
),
],
now: now,
);
await testDb!.marketDataDb.upsertNoDataBarPlaceholder(
symbol: 'AAA',
slotStart: slotStart,
timeframe: MarketHistoryConfig.barTimeframe,
checkedAt: now,
source: 'market_closed',
);
final MarketHistoryWeekCoverageReport report =
await MarketHistoryWeekCoverage(
connection: testDb!.connection,
windowDays: 2,
maxSymbols: 2000,
).compute(now: now);
expect(report.symbolCount, 1);
final MarketHistoryDayCoverage saturday = report.days.singleWhere(
(MarketHistoryDayCoverage day) => day.date == DateTime.utc(2026, 5, 30),
);
final MarketHistorySlotCoverage slot = saturday.slots.singleWhere(
(MarketHistorySlotCoverage s) =>
s.slotStart == DateTime.utc(2026, 5, 30, 20),
);
expect(slot.completed, isTrue);
expect(slot.fullySynced, isTrue);
expect(slot.syncedSymbolCount, 1);
});
}

View File

@ -0,0 +1,112 @@
@Tags(['integration', 'postgres'])
library;
import 'package:cyberhybridhub_server/pipeline/question_pipeline.dart';
import 'package:cyberhybridhub_server/question_service.dart';
import 'package:cyberhybridhub_server/questions_db.dart';
import 'package:cyberhybridhub_server/signalr/questions_hub_connections.dart';
import 'package:cyberhybridhub_server/workers/market_history_scheduler.dart';
import 'package:cyberhybridhub_server/workers/question_background_worker.dart';
import 'package:test/test.dart';
import '../helpers/test_db.dart';
void main() {
TestDb? testDb;
setUpAll(() async {
testDb = await TestDb.open();
});
tearDownAll(() async {
await testDb?.close();
});
group('QuestionBackgroundWorker wireup', () {
test('scheduler runs before trading maintenance', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final List<String> callOrder = <String>[];
final QuestionsDb questionsDb = QuestionsDb(testDb!.connection);
final QuestionPipeline pipeline = QuestionPipeline(
questionsDb: questionsDb,
questionService: QuestionService(
questionsDb: questionsDb,
hubConnections: QuestionsHubConnections(),
),
testMode: true,
);
final MarketHistoryScheduler scheduler = MarketHistoryScheduler(
connection: testDb!.connection,
runUniverse: (_) async {
callOrder.add('scheduler');
},
runBackfill: (_) async {},
runCleanup: (_) async {},
);
final QuestionBackgroundWorker worker = QuestionBackgroundWorker(
pipeline: pipeline,
interval: const Duration(hours: 1),
marketHistoryScheduler: scheduler,
tradingMaintenanceRunner: () async {
callOrder.add('trading');
},
);
await worker.runTickForTest();
expect(callOrder.indexOf('scheduler'), lessThan(callOrder.indexOf('trading')));
expect(callOrder, contains('scheduler'));
expect(callOrder, contains('trading'));
});
test('scheduler exception does not block trading', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final QuestionsDb questionsDb = QuestionsDb(testDb!.connection);
final QuestionPipeline pipeline = QuestionPipeline(
questionsDb: questionsDb,
questionService: QuestionService(
questionsDb: questionsDb,
hubConnections: QuestionsHubConnections(),
),
testMode: true,
);
final MarketHistoryScheduler scheduler = MarketHistoryScheduler(
connection: testDb!.connection,
runUniverse: (_) async {
throw Exception('scheduler boom');
},
runBackfill: (_) async {},
runCleanup: (_) async {},
);
var tradingRan = false;
final QuestionBackgroundWorker worker = QuestionBackgroundWorker(
pipeline: pipeline,
interval: const Duration(hours: 1),
marketHistoryScheduler: scheduler,
tradingMaintenanceRunner: () async {
tradingRan = true;
},
);
await worker.runTickForTest();
expect(tradingRan, isTrue);
});
});
}

View File

@ -0,0 +1,174 @@
@Tags(['integration', 'postgres'])
library;
import 'package:cyberhybridhub_server/alpaca/alpaca_models.dart';
import 'package:cyberhybridhub_server/trading/tradable_assets_db.dart';
import 'package:test/test.dart';
import '../helpers/test_db.dart';
AlpacaAsset _asset(
String symbol, {
bool tradable = true,
bool fractionable = true,
String status = 'active',
String? exchange = 'NASDAQ',
String? name,
}) {
return AlpacaAsset(
symbol: symbol,
assetClass: 'us_equity',
exchange: exchange,
name: name ?? '$symbol Inc.',
status: status,
tradable: tradable,
fractionable: fractionable,
raw: <String, dynamic>{
'symbol': symbol,
'class': 'us_equity',
'exchange': exchange,
'name': name ?? '$symbol Inc.',
'status': status,
'tradable': tradable,
'fractionable': fractionable,
},
);
}
void main() {
TestDb? testDb;
setUpAll(() async {
testDb = await TestDb.open();
});
tearDown(() async {
if (testDb != null) {
await testDb!.truncateTradingTables();
}
});
tearDownAll(() async {
await testDb?.close();
});
test('upsertAll inserts new symbols with the supplied refreshed_at',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final TradableAssetsDb db = TradableAssetsDb(testDb!.connection);
final DateTime t0 = DateTime.utc(2026, 5, 26, 10);
await db.upsertAll(<AlpacaAsset>[_asset('A'), _asset('B'), _asset('C')],
now: t0);
final TradableAssetRow? a = await db.getBySymbol('A');
final TradableAssetRow? b = await db.getBySymbol('B');
final TradableAssetRow? c = await db.getBySymbol('C');
expect(a, isNotNull);
expect(b, isNotNull);
expect(c, isNotNull);
expect(a!.tradable, isTrue);
expect(a.status, 'active');
expect(a.refreshedAt, t0);
expect(b!.refreshedAt, t0);
expect(c!.refreshedAt, t0);
});
test(
'second upsertAll updates B*, leaves C content unchanged but bumps '
'refreshed_at, inserts D, and marks A inactive without deleting it',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final TradableAssetsDb db = TradableAssetsDb(testDb!.connection);
final DateTime t0 = DateTime.utc(2026, 5, 26, 10);
await db.upsertAll(
<AlpacaAsset>[
_asset('A'),
_asset('B', name: 'B Original Inc.', exchange: 'NYSE'),
_asset('C', exchange: 'NASDAQ'),
],
now: t0,
);
final DateTime t1 = DateTime.utc(2026, 5, 27, 10);
await db.upsertAll(
<AlpacaAsset>[
// B with new content (name + exchange).
_asset('B', name: 'B Renamed Corp.', exchange: 'NASDAQ'),
// C with identical content as t0 should bump refreshed_at only.
_asset('C', exchange: 'NASDAQ'),
// D is new.
_asset('D'),
],
now: t1,
);
final TradableAssetRow? a = await db.getBySymbol('A');
final TradableAssetRow? b = await db.getBySymbol('B');
final TradableAssetRow? c = await db.getBySymbol('C');
final TradableAssetRow? d = await db.getBySymbol('D');
// A is preserved as a historical record but flipped to inactive.
expect(a, isNotNull, reason: 'A must NOT be deleted — history preserved');
expect(a!.status, 'inactive');
expect(a.tradable, isFalse);
// refreshed_at on the inactivated row stays at the original t0 so
// operators can see "last seen as active" in the audit trail.
expect(a.refreshedAt, t0);
// B was updated in place: same row, new content, new refreshed_at.
expect(b, isNotNull);
expect(b!.name, 'B Renamed Corp.');
expect(b.exchange, 'NASDAQ');
expect(b.refreshedAt, t1);
// C content unchanged but refreshed_at bumped to t1.
expect(c, isNotNull);
expect(c!.exchange, 'NASDAQ');
expect(c.refreshedAt, t1);
// D inserted.
expect(d, isNotNull);
expect(d!.refreshedAt, t1);
});
test('listActiveTradableSymbols filters to active AND tradable', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final TradableAssetsDb db = TradableAssetsDb(testDb!.connection);
final DateTime t0 = DateTime.utc(2026, 5, 26, 10);
await db.upsertAll(
<AlpacaAsset>[
_asset('AAA'), // active + tradable
_asset('BBB'), // active + tradable
_asset('CCC', tradable: false), // active but not tradable
_asset('DDD', status: 'inactive'), // inactive
],
now: t0,
);
final List<String> symbols = await db.listActiveTradableSymbols();
expect(symbols.toSet(), <String>{'AAA', 'BBB'});
});
}

View File

@ -0,0 +1,177 @@
@Tags(['integration', 'postgres'])
library;
import 'package:cyberhybridhub_server/alpaca/alpaca_assets_client.dart';
import 'package:cyberhybridhub_server/alpaca/alpaca_env.dart';
import 'package:cyberhybridhub_server/trading/tradable_assets_db.dart';
import 'package:cyberhybridhub_server/trading/tradable_assets_sync.dart';
import 'package:http/http.dart' as http;
import 'package:postgres/postgres.dart';
import 'package:test/test.dart';
import '../helpers/fixture_loader.dart';
import '../helpers/mock_http_client.dart';
import '../helpers/test_db.dart';
void main() {
TestDb? testDb;
late FixtureLoader fixtures;
late AlpacaEnv env;
setUpAll(() async {
testDb = await TestDb.open();
fixtures = FixtureLoader();
env = AlpacaEnv.fromMap(<String, String>{
'ALPACA_API_KEY_ID': 'test-key',
'ALPACA_API_SECRET_KEY': 'test-secret',
'ALPACA_TRADING_BASE_URL': 'https://paper-api.alpaca.markets',
});
});
tearDown(() async {
if (testDb != null) {
await testDb!.truncateTradingTables();
}
});
tearDownAll(() async {
await testDb?.close();
});
TradableAssetsSync makeSync({required MockHttpClient mock}) {
return TradableAssetsSync(
assetsClient: AlpacaAssetsClient(env: env, httpClient: mock),
assetsDb: TradableAssetsDb(testDb!.connection),
connection: testDb!.connection,
);
}
test(
'runOnce with the fixture upserts 5 assets and writes a universe '
'sync_runs row with finished_at',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final String body = await fixtures.loadString('alpaca_assets_active.json');
final MockHttpClient mock = MockHttpClient()
..whenGet(
'/v2/assets',
http.Response(body, 200, headers: <String, String>{
'content-type': 'application/json',
}),
);
final TradableAssetsSync sync = makeSync(mock: mock);
final TradableAssetsSyncResult result = await sync.runOnce();
expect(result.error, isNull);
expect(result.rowsWritten, 5);
final List<String> active = await TradableAssetsDb(testDb!.connection)
.listActiveTradableSymbols();
expect(
active.toSet(),
<String>{'AAPL', 'MSFT', 'SPY', 'BRK.B'},
reason: 'PNKZZ has tradable=false and must be excluded',
);
final Result runs = await testDb!.connection.execute(
'''
SELECT kind, rows_written, rows_removed, error,
started_at, finished_at
FROM market_data_sync_runs
ORDER BY id ASC
''',
);
expect(runs, hasLength(1));
final ResultRow row = runs.first;
expect(row[0], 'universe');
expect((row[1]! as num).toInt(), 5);
expect((row[2]! as num).toInt(), 0);
expect(row[3], isNull, reason: 'no error on happy path');
expect(row[4], isNotNull);
expect(row[5], isNotNull);
});
test(
'runOnce records the error when the Alpaca client throws and does '
'not propagate the exception',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final MockHttpClient mock = MockHttpClient()
..whenGet(
'/v2/assets',
http.Response('upstream exploded', 500),
);
final TradableAssetsSync sync = makeSync(mock: mock);
// Must not throw failures get recorded, not raised.
final TradableAssetsSyncResult result = await sync.runOnce();
expect(result.rowsWritten, 0);
expect(result.error, isNotNull);
expect(result.error, contains('500'));
final Result runs = await testDb!.connection.execute(
'''
SELECT kind, rows_written, error, finished_at
FROM market_data_sync_runs
ORDER BY id ASC
''',
);
expect(runs, hasLength(1));
expect(runs.first[0], 'universe');
expect((runs.first[1]! as num).toInt(), 0);
expect(runs.first[2]?.toString(), contains('500'));
expect(runs.first[3], isNotNull,
reason: 'finished_at must always be set, even on failure');
});
test('two consecutive runs produce identical row counts (idempotent)',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
final String body = await fixtures.loadString('alpaca_assets_active.json');
final MockHttpClient mock = MockHttpClient()
..whenGet(
'/v2/assets',
http.Response(body, 200, headers: <String, String>{
'content-type': 'application/json',
}),
);
final TradableAssetsSync sync = makeSync(mock: mock);
final TradableAssetsSyncResult r1 = await sync.runOnce();
final TradableAssetsSyncResult r2 = await sync.runOnce();
expect(r1.rowsWritten, 5);
expect(r2.rowsWritten, 5);
expect(r1.error, isNull);
expect(r2.error, isNull);
final Result count = await testDb!.connection
.execute("SELECT COUNT(*) FROM tradable_assets WHERE status = 'active'");
expect((count.first[0]! as num).toInt(), 5);
final Result runs = await testDb!.connection
.execute('SELECT COUNT(*) FROM market_data_sync_runs');
expect((runs.first[0]! as num).toInt(), 2);
});
}

View File

@ -0,0 +1,292 @@
@Tags(['integration', 'postgres'])
library;
import 'package:cyberhybridhub_server/alpaca/alpaca_models.dart';
import 'package:cyberhybridhub_server/pipeline/question_pipeline.dart';
import 'package:cyberhybridhub_server/trading/market_history_config.dart';
import 'package:cyberhybridhub_server/trading/market_history_query.dart';
import 'package:cyberhybridhub_server/trading/tradable_assets_db.dart';
import 'package:cyberhybridhub_server/trading/trading_pipeline.dart';
import 'package:cyberhybridhub_server/trading/user_trading_state_db.dart';
import 'package:postgres/postgres.dart';
import 'package:test/test.dart';
import '../helpers/test_db.dart';
void main() {
TestDb? testDb;
setUpAll(() async {
testDb = await TestDb.open();
});
tearDown(() async {
if (testDb != null) {
await testDb!.truncateTradingTables();
}
});
tearDownAll(() async {
await testDb?.close();
});
final DateTime _testNow = DateTime.utc(2026, 5, 26, 12);
Future<void> _seedGuessUniverse() async {
await TradableAssetsDb(testDb!.connection).upsertAll(
<AlpacaAsset>[
AlpacaAsset(
symbol: 'SPY',
assetClass: 'us_equity',
tradable: true,
fractionable: true,
status: 'active',
),
],
now: _testNow,
);
final List<num> closes = <num>[500, 501, 502, 503, 504, 505, 510];
for (int i = 0; i < closes.length; i++) {
await testDb!.marketDataDb.upsertSnapshot(
symbol: 'SPY',
metric: 'bar',
timeframe: MarketHistoryConfig.barTimeframe,
asOf: _testNow.subtract(Duration(hours: 4 * (closes.length - 1 - i))),
price: closes[i],
);
}
}
Future<TradingPipeline> _guessPipeline() async {
return TradingPipeline(
questionsDb: testDb!.questionsDb,
questionService: testDb!.questionService(),
marketDataDb: testDb!.marketDataDb,
tradingConfigDb: testDb!.tradingConfigDb,
tradingStateDb: testDb!.userTradingStateDb,
marketHistoryQuery: MarketHistoryQuery(connection: testDb!.connection),
clock: () => _testNow,
);
}
Future<void> _enableGuessRule(String uid) async {
await testDb!.seedUser(uid);
await testDb!.tradingConfigDb.upsertUserConfig(
firebaseUid: uid,
templateName: 'default_paper_watchlist',
enabled: true,
config: <String, dynamic>{
'rules': <Map<String, dynamic>>[
<String, dynamic>{
'id': 'guess_weekly_move',
'type': 'guess_weekly_move',
'question_template':
'{{token}} was {{ref_price}} {{ref_days_ago}} days ago. Swipe +10 if up, -10 if down.',
},
<String, dynamic>{
'id': 'dip_confirm',
'threshold_pct': 0,
},
],
},
);
}
group('guess_weekly_move pipeline', () {
test('evaluate creates obfuscated question with metadata.guess_symbol',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
const String uid = 'guess-eval-uid';
await _enableGuessRule(uid);
await _seedGuessUniverse();
final TradingPipeline pipeline = await _guessPipeline();
final TradingEvaluationResult result = await pipeline.evaluate(uid);
expect(result.questionsCreated, 1);
expect(result.rulesFired, <String>['guess_weekly_move']);
final Result rows = await testDb!.connection.execute(
Sql.named(
'''
SELECT pipeline_key, pipeline_step, question_text, metadata
FROM questions
WHERE assigned_user_id = @uid
''',
),
parameters: <String, dynamic>{'uid': uid},
);
expect(rows, hasLength(1));
expect(rows.first[0], PipelineKeys.trading);
expect(rows.first[1], 'guess_weekly_move:${TradingPhases.awaitAnswer}');
final String text = rows.first[2]! as String;
expect(text, contains('ASSET_A'));
expect(text, isNot(contains('SPY')));
final Map<String, dynamic> metadata =
rows.first[3]! as Map<String, dynamic>;
expect(metadata['guess_symbol'], 'SPY');
});
test('matching direction records score_delta +1', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
const String uid = 'guess-score-match-uid';
await _enableGuessRule(uid);
await _seedGuessUniverse();
final TradingPipeline pipeline = await _guessPipeline();
await pipeline.evaluate(uid);
final List<Map<String, dynamic>> open =
await testDb!.questionsDb.listUnansweredQuestions(uid);
final Map<String, dynamic>? updated =
await testDb!.questionsDb.submitAnswer(
questionId: open.single['id'] as String,
assignedUserId: uid,
userResponse: 10,
);
await pipeline.handleAnswer(
firebaseUid: uid,
answeredQuestion: updated!,
userResponse: 10,
);
final Map<String, dynamic>? score =
await testDb!.userTradingStateDb.getGuessScore(uid);
expect(score, isNotNull);
expect(score!['total'], 1);
expect((score['last'] as Map)['score_delta'], 1);
});
test('non-matching direction records score_delta -1', () async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
const String uid = 'guess-score-miss-uid';
await _enableGuessRule(uid);
await _seedGuessUniverse();
final TradingPipeline pipeline = await _guessPipeline();
await pipeline.evaluate(uid);
final List<Map<String, dynamic>> open =
await testDb!.questionsDb.listUnansweredQuestions(uid);
final Map<String, dynamic>? updated =
await testDb!.questionsDb.submitAnswer(
questionId: open.single['id'] as String,
assignedUserId: uid,
userResponse: -10,
);
await pipeline.handleAnswer(
firebaseUid: uid,
answeredQuestion: updated!,
userResponse: -10,
);
final Map<String, dynamic>? score =
await testDb!.userTradingStateDb.getGuessScore(uid);
expect(score!['total'], -1);
expect((score['last'] as Map)['score_delta'], -1);
});
test('handleAnswer never stages pending orders; actuator not invoked',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
const String uid = 'guess-no-actuator-uid';
await _enableGuessRule(uid);
await _seedGuessUniverse();
final TradingPipeline pipeline = await _guessPipeline();
await pipeline.evaluate(uid);
final List<Map<String, dynamic>> open =
await testDb!.questionsDb.listUnansweredQuestions(uid);
final Map<String, dynamic>? updated =
await testDb!.questionsDb.submitAnswer(
questionId: open.single['id'] as String,
assignedUserId: uid,
userResponse: 10,
);
await pipeline.handleAnswer(
firebaseUid: uid,
answeredQuestion: updated!,
userResponse: 10,
);
final List<Map<String, dynamic>> pending =
await testDb!.userTradingStateDb.listPendingOrders(uid);
expect(pending, isEmpty);
final Result orders = await testDb!.connection.execute(
Sql.named(
'SELECT COUNT(*)::int FROM trade_orders WHERE firebase_uid = @uid',
),
parameters: <String, dynamic>{'uid': uid},
);
expect(orders.first[0], 0);
});
test('symbol cooldown prevents re-pick within GUESS_COOLDOWN_HOURS',
() async {
if (testDb == null) {
markTestSkipped(
'Set DATABASE_URL or TEST_DATABASE_URL for integration tests',
);
return;
}
const String uid = 'guess-cooldown-uid';
await _enableGuessRule(uid);
await _seedGuessUniverse();
final TradingPipeline pipeline = await _guessPipeline();
await pipeline.evaluate(uid);
final List<Map<String, dynamic>> open =
await testDb!.questionsDb.listUnansweredQuestions(uid);
final Map<String, dynamic>? updated =
await testDb!.questionsDb.submitAnswer(
questionId: open.single['id'] as String,
assignedUserId: uid,
userResponse: 10,
);
await pipeline.handleAnswer(
firebaseUid: uid,
answeredQuestion: updated!,
userResponse: 10,
);
final TradingEvaluationResult again = await pipeline.evaluate(uid);
expect(again.questionsCreated, 0);
expect(
again.rulesSkipped,
contains('guess_weekly_move(insufficientBars)'),
);
});
});
}

Some files were not shown because too many files have changed in this diff Show More