462 lines
16 KiB
Markdown
462 lines
16 KiB
Markdown
# Flutter Admin Portal — Development Plan
|
||
|
||
**Status:** Planning
|
||
**Related:** `TODO.md` §8 (operational CLIs), §8.5 (admin API), rolling market-history pipeline (§1–§7)
|
||
**Audience:** Mobile/web Flutter developers, API maintainers, operators
|
||
|
||
---
|
||
|
||
## 1. Purpose
|
||
|
||
Build an **admin-only** area in the Flutter app that shows a **live audit log** of periodic market-history operations:
|
||
|
||
- **Universe sync** (`kind=universe`) — tradable asset refresh
|
||
- **Historical backfill** (`kind=backfill`) — ended **4-hour UTC slots** (`4Hour` bars) into `market_data_snapshots`; open slot not fetched yet
|
||
- **Retention / cleanup** (`kind=cleanup`) — prune (or archive-then-delete) expired rows
|
||
|
||
Operators use it to confirm the rolling 7-day window is healthy, see when cleanup ran, and **notice Alpaca/API failures or rate limits immediately** without reading server logs or SQL.
|
||
|
||
This plan **replaces the need for §8 shell CLIs** for day-to-day product use. CLIs remain optional for CI/SRE (see §12).
|
||
|
||
---
|
||
|
||
## 2. Goals and non-goals
|
||
|
||
### Goals
|
||
|
||
| # | Goal |
|
||
|---|------|
|
||
| G1 | Chronological log: **newest events at the top**; scroll down for older history |
|
||
| G2 | **Pin unresolved failures** (API errors, rate limits, partial batch failures) at the top until a **successful retry** for the same `kind` (and optionally same error “thread”) |
|
||
| G3 | **Expandable list rows** — collapsed summary; expanded shows structured detail **without** raw JSON payloads (no bar blobs, no `tradable_assets.raw`) |
|
||
| G4 | For API failures, expanded view shows **full operator-relevant error text** (HTTP status, Alpaca message body summary, batch scope) |
|
||
| G5 | Optional: **on-demand** “Run sync” / “Run cleanup” from the portal (API-triggered), with new rows appearing in the log |
|
||
| G6 | Reuse existing auth (Firebase ID token → API) and app theme/navigation patterns |
|
||
|
||
### Non-goals (v1)
|
||
|
||
- End-user (non-admin) access to sync logs
|
||
- Displaying raw `market_data_snapshots` or per-symbol bar tables
|
||
- Full log export / SIEM integration (future)
|
||
- Replacing the background worker; scheduler still runs automatically when `MARKET_HISTORY_SYNC_ENABLED=true`
|
||
- Building §8 `bin/sync_market_history.dart` / `bin/cleanup_market_history.dart` (defer unless ops requests them)
|
||
|
||
---
|
||
|
||
## 3. Source of truth (already in Postgres)
|
||
|
||
All events are recorded in **`market_data_sync_runs`** (migration `005_market_history.sql`):
|
||
|
||
| Column | Meaning |
|
||
|--------|---------|
|
||
| `id` | Stable row id |
|
||
| `kind` | `universe` \| `backfill` \| `cleanup` |
|
||
| `started_at` | UTC start |
|
||
| `finished_at` | UTC end (null if crash mid-run — treat as in-progress) |
|
||
| `rows_written` | Upserted rows (universe/backfill) |
|
||
| `rows_removed` | Deleted rows (cleanup) |
|
||
| `error` | `NULL` = success; non-null = failure or **partial** failure message |
|
||
|
||
**Kinds map to UI labels:**
|
||
|
||
| `kind` | UI title | Typical success summary |
|
||
|--------|----------|-------------------------|
|
||
| `universe` | Asset universe sync | N assets refreshed |
|
||
| `backfill` | 4-hour slot backfill (ended slots only) | N bar rows written |
|
||
| `cleanup` | Data retention cleanup | N rows removed |
|
||
|
||
**Error shapes today (no schema change required for v1):**
|
||
|
||
- Thrown exception → `error` is `e.toString()` (e.g. `AlpacaMarketDataException: getBarsRange rate limited: 429 …`)
|
||
- Partial backfill → `error` may list batch failures while `rows_written > 0` (`SyncRunCounts.error` in `market_data_history.dart`)
|
||
|
||
**v1 recommendation:** Parse `error` text client-side for keywords (`429`, `rate`, `Alpaca`, `batch`) to set severity and pin rules. **v2 (optional):** add `error_code`, `http_status`, `severity` columns or a JSON `detail` object on insert (server change).
|
||
|
||
---
|
||
|
||
## 4. List ordering and “pin until resolved”
|
||
|
||
### 4.1 Default sort
|
||
|
||
Primary: **`started_at` DESC** (most recent at top).
|
||
|
||
### 4.2 Pin rule (failure priority)
|
||
|
||
An event is **pinned** when:
|
||
|
||
1. `error IS NOT NULL` **or** `finished_at IS NULL` (stuck in-progress > N minutes → show as warning), **and**
|
||
2. There is **no later successful run** of the **same `kind`** with `error IS NULL` and `finished_at IS NOT NULL` **after** this row’s `started_at`.
|
||
|
||
**Pinned block** appears **above** the chronological list (fixed section or visually distinct header: “Needs attention”).
|
||
|
||
When a new successful run for that `kind` completes, the older failed row **unpins** and appears only in normal history order.
|
||
|
||
### 4.3 Rate limits and API errors
|
||
|
||
Treat as **high severity** when `error` matches (case-insensitive):
|
||
|
||
- `429`, `rate limit`, `rate limited`
|
||
- `AlpacaMarketDataException`, `AlpacaAssetsException`, `AlpacaTradingException`
|
||
- HTTP `5xx` patterns in message
|
||
|
||
Pinned rate-limit rows use a **warning/error** color and icon until the pin rule clears.
|
||
|
||
### 4.4 Partial success (backfill)
|
||
|
||
If `rows_written > 0` **and** `error != null`:
|
||
|
||
- Collapsed: **“Partial success”** badge + counts
|
||
- Expanded: show `rows_written`, full `error` text (batch list), **do not** show raw bar JSON
|
||
- Pin until next **fully successful** backfill (`error` null) **or** product decision: pin until error null even if partial (stricter — **recommended for ops**)
|
||
|
||
---
|
||
|
||
## 5. Backend API (prerequisite)
|
||
|
||
No admin market-history endpoints exist today. Implement **before** Flutter UI (or in parallel with mock API).
|
||
|
||
### 5.1 Auth model
|
||
|
||
| Approach | Recommendation |
|
||
|----------|----------------|
|
||
| Firebase + allowlist | `ADMIN_FIREBASE_UIDS=uid1,uid2` in server env; middleware rejects others with `403` |
|
||
| Custom claim | `admin: true` on Firebase token; server verifies claim |
|
||
| Separate admin API key | Avoid for mobile; poor UX |
|
||
|
||
Match existing pattern: `Authorization: Bearer <Firebase ID token>` like `QuestionsApiService`.
|
||
|
||
**Do not** expose these under `/v1/me/...` without admin checks — use `/v1/admin/...` prefix.
|
||
|
||
### 5.2 Endpoints (v1)
|
||
|
||
#### `GET /v1/admin/market-history/sync-runs`
|
||
|
||
Query params:
|
||
|
||
| Param | Default | Description |
|
||
|-------|---------|-------------|
|
||
| `limit` | `50` | Page size |
|
||
| `before` | — | Cursor: `id` or `started_at` of oldest item in current page (for infinite scroll) |
|
||
| `kind` | — | Optional filter: `universe`, `backfill`, `cleanup` |
|
||
|
||
Response:
|
||
|
||
```json
|
||
{
|
||
"runs": [
|
||
{
|
||
"id": 123,
|
||
"kind": "cleanup",
|
||
"startedAt": "2026-05-26T10:00:00Z",
|
||
"finishedAt": "2026-05-26T10:00:05Z",
|
||
"rowsWritten": 0,
|
||
"rowsRemoved": 4200,
|
||
"error": null,
|
||
"severity": "ok",
|
||
"status": "success",
|
||
"durationMs": 5000,
|
||
"summary": "Removed 4,200 expired snapshots"
|
||
}
|
||
],
|
||
"pinned": [
|
||
{
|
||
"id": 120,
|
||
"kind": "backfill",
|
||
"error": "getBarsRange rate limited: 429 ...",
|
||
"severity": "rate_limit",
|
||
"status": "failed",
|
||
"summary": "Backfill partial: 12,000 rows; rate limited"
|
||
}
|
||
],
|
||
"nextBefore": "2026-05-25T08:00:00Z"
|
||
}
|
||
```
|
||
|
||
Server computes `pinned[]` with the rule in §4.2.
|
||
Server computes `severity`: `ok` \| `warning` \| `error` \| `rate_limit` from `error` text + `finished_at`.
|
||
**Never** include `raw` JSON from snapshots or assets in this API.
|
||
|
||
#### `GET /v1/admin/market-history/sync-runs/{id}`
|
||
|
||
Single run for expanded row refresh (optional if list payload is enough).
|
||
|
||
#### `POST /v1/admin/market-data/resync` (optional v1.1)
|
||
|
||
Runs universe + slot backfill (ended `4Hour` slots in window, not the open slot). Returns `{ "runIds": […] }` (202). `windowDays` on cleanup only today.
|
||
|
||
#### `POST /v1/admin/market-data/cleanup` (optional v1.1)
|
||
|
||
Query: `windowDays`, `archive=true|false`.
|
||
Runs `MarketDataRetention.run(archive: …)` once.
|
||
|
||
### 5.3 Implementation notes (server)
|
||
|
||
- New handler: `server/lib/handlers/market_history_admin_handler.dart`
|
||
- Mount only when `ADMIN_PORTAL_ENABLED=true` (or reuse `TRADING_DEV_ENDPOINTS_ENABLED` — **prefer dedicated flag**)
|
||
- Reuse `TradableAssetsSync`, `MarketDataHistorySync`, `MarketDataRetention` — same as `bin/server.dart` scheduler wiring
|
||
- List query: indexed `ORDER BY started_at DESC` (index exists: `market_data_sync_runs_kind_started_idx`)
|
||
|
||
### 5.4 Error detail for API failures (v2 enhancement)
|
||
|
||
When recording sync failures, optionally persist structured fields:
|
||
|
||
```json
|
||
{
|
||
"httpStatus": 429,
|
||
"endpoint": "GET /v2/stocks/bars",
|
||
"message": "...",
|
||
"batchSymbols": ["AAPL", "MSFT"]
|
||
}
|
||
```
|
||
|
||
Store in new `detail JSONB` column or append to `error` as delimited text for v1.
|
||
|
||
---
|
||
|
||
## 6. Flutter architecture
|
||
|
||
### 6.1 Entry and routing
|
||
|
||
| Item | Proposal |
|
||
|------|----------|
|
||
| Route | `/admin/market-history` or dedicated `AdminPortalScreen` pushed from a hidden entry |
|
||
| Gate | `AdminGate` widget: calls `GET /v1/admin/market-history/sync-runs?limit=1` — `403` → “Not authorized” |
|
||
| Discovery | Long-press app logo, `kDebugMode` menu item, or profile flag `isAdmin` from `GET /v1/me/profile` extension |
|
||
|
||
Do **not** show admin nav to all signed-in users.
|
||
|
||
### 6.2 Layering (match existing app)
|
||
|
||
```
|
||
lib/
|
||
admin/
|
||
models/
|
||
sync_run_event.dart # DTO from API
|
||
services/
|
||
market_history_admin_api.dart # HTTP + auth headers (mirror QuestionsApiService)
|
||
repositories/
|
||
sync_run_log_repository.dart # paging, pin merge, refresh
|
||
screens/
|
||
market_history_log_screen.dart
|
||
widgets/
|
||
sync_run_expansion_tile.dart
|
||
pinned_alerts_section.dart
|
||
sync_run_status_chip.dart
|
||
```
|
||
|
||
- **API service:** `apiBaseUrl` from `lib/config/api_config.dart`, token via `AuthService.instance.getIdToken()`
|
||
- **State:** `Listenable`/`ChangeNotifier` or `flutter_riverpod` if project adopts it later; v1 can use `StatefulWidget` + repository like profile sync
|
||
|
||
### 6.3 `SyncRunEvent` model (client)
|
||
|
||
Fields from API plus derived:
|
||
|
||
- `isPinned` (from `pinned` array or client recompute)
|
||
- `Severity` enum: `ok`, `warning`, `error`, `rateLimit`
|
||
- `displayTitle` from `kind`
|
||
- `collapsedSubtitle` — one line: time ago + summary
|
||
- `expandedSections` — `List<DetailSection>` title/body pairs (no raw JSON)
|
||
|
||
### 6.4 Expanded row content (templates)
|
||
|
||
**Success — cleanup**
|
||
|
||
- Started / finished (local timezone)
|
||
- Duration
|
||
- Rows removed
|
||
- Archive mode on/off (if API adds flag later)
|
||
|
||
**Success — backfill**
|
||
|
||
- Rows written
|
||
- Window days (from env or API meta)
|
||
- “No errors”
|
||
|
||
**Failure / rate limit**
|
||
|
||
- Full `error` string (selectable text)
|
||
- Parsed HTTP status if regex finds it
|
||
- Kind + run id (for support tickets)
|
||
- Hint: “Will retry on next scheduled run” or button **Retry now** if POST endpoint exists
|
||
|
||
**Partial backfill**
|
||
|
||
- Rows written (success count)
|
||
- Error block (batch failures)
|
||
- Do **not** render `raw` from bars
|
||
|
||
**In progress**
|
||
|
||
- Started at; spinner if `finished_at == null` and started < 30m ago
|
||
- Stale run warning if > 30m without `finished_at`
|
||
|
||
---
|
||
|
||
## 7. UI specification
|
||
|
||
### 7.1 Screen layout
|
||
|
||
```
|
||
┌─────────────────────────────────────┐
|
||
│ ← Market history log [↻] │ AppBar: refresh
|
||
├─────────────────────────────────────┤
|
||
│ NEEDS ATTENTION (0–n) │ Pinned section (hidden if empty)
|
||
│ ⚠ Backfill — Rate limited 2h ago │ ExpansionTile, error styling
|
||
│ ✗ Cleanup — failed 1d ago │
|
||
├─────────────────────────────────────┤
|
||
│ HISTORY │ Section header
|
||
│ ✓ Cleanup — 4.2k removed 10:00 │ Newest first
|
||
│ ✓ Backfill — 98k written 09:00 │
|
||
│ ✓ Universe — 8.2k assets 08:00 │
|
||
│ ... │ Infinite scroll
|
||
└─────────────────────────────────────┘
|
||
```
|
||
|
||
### 7.2 Collapsed row
|
||
|
||
- Leading icon by `severity` / `kind`
|
||
- Title: `{Kind label} — {short status}`
|
||
- Trailing: relative time (`2h ago`)
|
||
- Optional chips: `rows_written`, `rows_removed`
|
||
|
||
### 7.3 Expanded row
|
||
|
||
- `ExpansionTile` children: labeled rows (not monospace dumps)
|
||
- **Selectable** `SelectableText` for full error message
|
||
- Copy-to-clipboard button for error block
|
||
|
||
### 7.4 Refresh and pagination
|
||
|
||
| Action | Behavior |
|
||
|--------|----------|
|
||
| Pull-to-refresh | Reload first page + recompute pins |
|
||
| Scroll end | `before=` cursor load next page, **append** to history (not pinned) |
|
||
| Auto-refresh | Optional 60s timer while screen visible |
|
||
|
||
### 7.5 On-demand actions (v1.1)
|
||
|
||
AppBar overflow menu:
|
||
|
||
- **Run backfill now** → `POST resync` → show in-progress row → refresh
|
||
- **Run cleanup now** → `POST cleanup` → confirm dialog (destructive) → refresh
|
||
|
||
Disable buttons while any `kind` has in-progress run (`finished_at == null`).
|
||
|
||
---
|
||
|
||
## 8. Phased delivery
|
||
|
||
### Phase A — Read-only log (MVP)
|
||
|
||
| Track | Work |
|
||
|-------|------|
|
||
| Server | `GET sync-runs` + admin auth + `pinned` computation |
|
||
| Flutter | Models, API service, log screen, expansion tiles, pull-to-refresh |
|
||
| QA | Seed `market_data_sync_runs` in test DB; verify pin/unpin when new success inserted |
|
||
|
||
**Exit:** Operator can view retention/cleanup/backfill history on device/emulator.
|
||
|
||
### Phase B — Failure UX polish
|
||
|
||
- Severity icons/colors, rate-limit copy, partial-success template
|
||
- Copy error, relative timestamps, empty/loading/error states
|
||
- Widget tests for sort/pin merge logic
|
||
|
||
### Phase C — On-demand triggers
|
||
|
||
- `POST resync`, `POST cleanup`
|
||
- Flutter buttons + confirmation + in-progress state
|
||
|
||
### Phase D — Structured errors (optional)
|
||
|
||
- Server writes `detail JSONB`; Flutter renders HTTP status, endpoint, batch symbols without parsing free text
|
||
|
||
---
|
||
|
||
## 9. Testing
|
||
|
||
### Server
|
||
|
||
| Test | Description |
|
||
|------|-------------|
|
||
| Integration | `test/integration/market_history_admin_handler_test.dart` — 403 non-admin, 200 list shape, pinned logic |
|
||
| Fixture rows | Success cleanup, failed backfill 429, partial backfill |
|
||
|
||
### Flutter
|
||
|
||
| Test | Description |
|
||
|------|-------------|
|
||
| Unit | `sync_run_log_repository_test.dart` — merge pinned + chronological, pin clearance |
|
||
| Unit | `sync_run_event_test.dart` — severity from error string |
|
||
| Widget | Expansion tile shows/hides detail; pinned section visibility |
|
||
| Integration | Mock HTTP → screen goldens or pump tests |
|
||
|
||
### Manual
|
||
|
||
1. Enable worker sync; wait for 3 kinds in log
|
||
2. Simulate 429 (mock Alpaca or test env throttle)
|
||
3. Confirm pin → run successful backfill → pin clears
|
||
|
||
---
|
||
|
||
## 10. Security and compliance
|
||
|
||
- Admin routes **only** with verified admin identity
|
||
- Logs may contain Alpaca error bodies — **no PII**, but treat as internal ops data
|
||
- Do not log API keys; ensure `error` column never stores secrets (review `SyncRunRecorder` callers)
|
||
- CORS: same as existing API (`apiCorsHeaders`)
|
||
- Production: `ADMIN_PORTAL_ENABLED=false` by default
|
||
|
||
---
|
||
|
||
## 11. Configuration (align with §7 env)
|
||
|
||
Document in `server/README.md` when implemented:
|
||
|
||
| Env var | Purpose |
|
||
|---------|---------|
|
||
| `ADMIN_PORTAL_ENABLED` | Mount `/v1/admin/market-history/*` |
|
||
| `ADMIN_FIREBASE_UIDS` | Comma-separated allowlist |
|
||
| `MARKET_HISTORY_*` | Window, sync cadence (worker); shown in expanded row as read-only meta |
|
||
|
||
---
|
||
|
||
## 12. Relationship to §8 CLIs
|
||
|
||
| Capability | Admin portal | §8 CLI |
|
||
|------------|----------------|--------|
|
||
| View sync/cleanup log | Yes (primary) | No |
|
||
| On-demand sync/cleanup | Yes (Phase C API) | Yes (planned) |
|
||
| CI / headless / no UI | No | Yes |
|
||
| Ops SSH without app | No | Yes |
|
||
|
||
**Recommendation:** Implement **portal + admin API** first; defer §8 CLIs unless SRE requests them.
|
||
|
||
---
|
||
|
||
## 13. Open decisions
|
||
|
||
| # | Question | Default proposal |
|
||
|---|----------|------------------|
|
||
| 1 | Admin discovery in app? | Debug-only entry + allowlist |
|
||
| 2 | Pin partial backfill with `rows_written > 0`? | Yes, until `error` null |
|
||
| 3 | Separate web-only admin build? | Single app, gated route (v1) |
|
||
| 4 | Real-time updates? | Pull-to-refresh only (v1); WebSocket later |
|
||
| 5 | `finished_at` null stale threshold? | 30 minutes → warning row |
|
||
|
||
---
|
||
|
||
## 14. Acceptance criteria
|
||
|
||
- [ ] Admin user sees **newest** history events at top of main list
|
||
- [ ] Failed/rate-limited runs appear in **Needs attention** until same `kind` succeeds later
|
||
- [ ] Expand row shows operational detail **without** raw market-data JSON
|
||
- [ ] API error expanded view shows **complete** `error` text from server
|
||
- [ ] Non-admin receives 403 and no data
|
||
- [ ] Pull-to-refresh and pagination load additional older events
|
||
- [ ] (Phase C) On-demand cleanup/sync creates visible new log entries
|
||
|
||
---
|
||
|
||
*Document version: 1.0 — Flutter admin portal for `market_data_sync_runs` audit log.*
|