2026-05-31 13:11:09 -05:00

16 KiB

Raw Blame History

Flutter Admin Portal — Development Plan

Status: Planning
Related: TODO.md §8 (operational CLIs), §8.5 (admin API), rolling market-history pipeline (§1–§7)
Audience: Mobile/web Flutter developers, API maintainers, operators

1. Purpose

Build an admin-only area in the Flutter app that shows a live audit log of periodic market-history operations:

Universe sync (kind=universe) — tradable asset refresh
Historical backfill (kind=backfill) — ended 4-hour UTC slots (4Hour bars) into market_data_snapshots; open slot not fetched yet
Retention / cleanup (kind=cleanup) — prune (or archive-then-delete) expired rows

Operators use it to confirm the rolling 5-trading-day window is healthy, see when cleanup ran, and notice Alpaca/API failures or rate limits immediately without reading server logs or SQL.

This plan replaces the need for §8 shell CLIs for day-to-day product use. CLIs remain optional for CI/SRE (see §12).

2. Goals and non-goals

Goals

#	Goal
G1	Chronological log: newest events at the top; scroll down for older history
G2	Pin unresolved failures (API errors, rate limits, partial batch failures) at the top until a successful retry for the same `kind` (and optionally same error “thread”)
G3	Expandable list rows — collapsed summary; expanded shows structured detail without raw JSON payloads (no bar blobs, no `tradable_assets.raw`)
G4	For API failures, expanded view shows full operator-relevant error text (HTTP status, Alpaca message body summary, batch scope)
G5	Optional: on-demand “Run sync” / “Run cleanup” from the portal (API-triggered), with new rows appearing in the log
G6	Reuse existing auth (Firebase ID token → API) and app theme/navigation patterns

Non-goals (v1)

End-user (non-admin) access to sync logs
Displaying raw market_data_snapshots or per-symbol bar tables
Full log export / SIEM integration (future)
Replacing the background worker; scheduler still runs automatically when MARKET_HISTORY_SYNC_ENABLED=true
Building §8 bin/sync_market_history.dart / bin/cleanup_market_history.dart (defer unless ops requests them)

3. Source of truth (already in Postgres)

All events are recorded in market_data_sync_runs (migration 005_market_history.sql):

Column	Meaning
`id`	Stable row id
`kind`	`universe` \| `backfill` \| `cleanup`
`started_at`	UTC start
`finished_at`	UTC end (null if crash mid-run — treat as in-progress)
`rows_written`	Upserted rows (universe/backfill)
`rows_removed`	Deleted rows (cleanup)
`error`	`NULL` = success; non-null = failure or partial failure message

Kinds map to UI labels:

`kind`	UI title	Typical success summary
`universe`	Asset universe sync	N assets refreshed
`backfill`	4-hour slot backfill (ended slots only)	N bar rows written
`cleanup`	Data retention cleanup	N rows removed

Error shapes today (no schema change required for v1):

Thrown exception → error is e.toString() (e.g. AlpacaMarketDataException: getBarsRange rate limited: 429 …)
Partial backfill → error may list batch failures while rows_written > 0 (SyncRunCounts.error in market_data_history.dart)

v1 recommendation: Parse error text client-side for keywords (429, rate, Alpaca, batch) to set severity and pin rules. v2 (optional): add error_code, http_status, severity columns or a JSON detail object on insert (server change).

4. List ordering and “pin until resolved”

4.1 Default sort

Primary: started_at DESC (most recent at top).

4.2 Pin rule (failure priority)

An event is pinned when:

error IS NOT NULL or finished_at IS NULL (stuck in-progress > N minutes → show as warning), and
There is no later successful run of the same kind with error IS NULL and finished_at IS NOT NULL after this row’s started_at.

Pinned block appears above the chronological list (fixed section or visually distinct header: “Needs attention”).

When a new successful run for that kind completes, the older failed row unpins and appears only in normal history order.

4.3 Rate limits and API errors

Treat as high severity when error matches (case-insensitive):

429, rate limit, rate limited
AlpacaMarketDataException, AlpacaAssetsException, AlpacaTradingException
HTTP 5xx patterns in message

Pinned rate-limit rows use a warning/error color and icon until the pin rule clears.

4.4 Partial success (backfill)

If rows_written > 0 and error != null:

Collapsed: “Partial success” badge + counts
Expanded: show rows_written, full error text (batch list), do not show raw bar JSON
Pin until next fully successful backfill (error null) or product decision: pin until error null even if partial (stricter — recommended for ops)

5. Backend API (prerequisite)

No admin market-history endpoints exist today. Implement before Flutter UI (or in parallel with mock API).

5.1 Auth model

Approach	Recommendation
Firebase + allowlist	`ADMIN_FIREBASE_UIDS=uid1,uid2` in server env; middleware rejects others with `403`
Custom claim	`admin: true` on Firebase token; server verifies claim
Separate admin API key	Avoid for mobile; poor UX

Match existing pattern: Authorization: Bearer <Firebase ID token> like QuestionsApiService.

Do not expose these under /v1/me/... without admin checks — use /v1/admin/... prefix.

5.2 Endpoints (v1)

`GET /v1/admin/market-history/sync-runs`

Query params:

Param	Default	Description
`limit`	`50`	Page size
`before`	—	Cursor: `id` or `started_at` of oldest item in current page (for infinite scroll)
`kind`	—	Optional filter: `universe`, `backfill`, `cleanup`

Response:

{
  "runs": [
    {
      "id": 123,
      "kind": "cleanup",
      "startedAt": "2026-05-26T10:00:00Z",
      "finishedAt": "2026-05-26T10:00:05Z",
      "rowsWritten": 0,
      "rowsRemoved": 4200,
      "error": null,
      "severity": "ok",
      "status": "success",
      "durationMs": 5000,
      "summary": "Removed 4,200 expired snapshots"
    }
  ],
  "pinned": [
    {
      "id": 120,
      "kind": "backfill",
      "error": "getBarsRange rate limited: 429 ...",
      "severity": "rate_limit",
      "status": "failed",
      "summary": "Backfill partial: 12,000 rows; rate limited"
    }
  ],
  "nextBefore": "2026-05-25T08:00:00Z"
}

Server computes pinned[] with the rule in §4.2.
Server computes severity: ok | warning | error | rate_limit from error text + finished_at.
Never include raw JSON from snapshots or assets in this API.

`GET /v1/admin/market-history/sync-runs/{id}`

Single run for expanded row refresh (optional if list payload is enough).

`POST /v1/admin/market-data/resync` (optional v1.1)

Runs universe + slot backfill (ended 4Hour slots in window, not the open slot). Returns { "runIds": […] } (202). windowDays on cleanup only today.

`POST /v1/admin/market-data/cleanup` (optional v1.1)

Query: windowDays, archive=true|false.
Runs MarketDataRetention.run(archive: …) once.

5.3 Implementation notes (server)

New handler: server/lib/handlers/market_history_admin_handler.dart
Mount only when ADMIN_PORTAL_ENABLED=true (or reuse TRADING_DEV_ENDPOINTS_ENABLED — prefer dedicated flag)
Reuse TradableAssetsSync, MarketDataHistorySync, MarketDataRetention — same as bin/server.dart scheduler wiring
List query: indexed ORDER BY started_at DESC (index exists: market_data_sync_runs_kind_started_idx)

5.4 Error detail for API failures (v2 enhancement)

When recording sync failures, optionally persist structured fields:

{
  "httpStatus": 429,
  "endpoint": "GET /v2/stocks/bars",
  "message": "...",
  "batchSymbols": ["AAPL", "MSFT"]
}

Store in new detail JSONB column or append to error as delimited text for v1.

6. Flutter architecture

6.1 Entry and routing

Item	Proposal
Route	`/admin/market-history` or dedicated `AdminPortalScreen` pushed from a hidden entry
Gate	`AdminGate` widget: calls `GET /v1/admin/market-history/sync-runs?limit=1` — `403` → “Not authorized”
Discovery	Long-press app logo, `kDebugMode` menu item, or profile flag `isAdmin` from `GET /v1/me/profile` extension

Do not show admin nav to all signed-in users.

6.2 Layering (match existing app)

lib/
  admin/
    models/
      sync_run_event.dart          # DTO from API
    services/
      market_history_admin_api.dart  # HTTP + auth headers (mirror QuestionsApiService)
    repositories/
      sync_run_log_repository.dart   # paging, pin merge, refresh
    screens/
      market_history_log_screen.dart
    widgets/
      sync_run_expansion_tile.dart
      pinned_alerts_section.dart
      sync_run_status_chip.dart

API service: apiBaseUrl from lib/config/api_config.dart, token via AuthService.instance.getIdToken()
State: Listenable/ChangeNotifier or flutter_riverpod if project adopts it later; v1 can use StatefulWidget + repository like profile sync

6.3 `SyncRunEvent` model (client)

Fields from API plus derived:

isPinned (from pinned array or client recompute)
Severity enum: ok, warning, error, rateLimit
displayTitle from kind
collapsedSubtitle — one line: time ago + summary
expandedSections — List<DetailSection> title/body pairs (no raw JSON)

6.4 Expanded row content (templates)

Success — cleanup

Started / finished (local timezone)
Duration
Rows removed
Archive mode on/off (if API adds flag later)

Success — backfill

Rows written
Window days (from env or API meta)
“No errors”

Failure / rate limit

Full error string (selectable text)
Parsed HTTP status if regex finds it
Kind + run id (for support tickets)
Hint: “Will retry on next scheduled run” or button Retry now if POST endpoint exists

Partial backfill

Rows written (success count)
Error block (batch failures)
Do not render raw from bars

In progress

Started at; spinner if finished_at == null and started < 30m ago
Stale run warning if > 30m without finished_at

7. UI specification

7.1 Screen layout

┌─────────────────────────────────────┐
│ ←  Market history log        [↻]   │  AppBar: refresh
├─────────────────────────────────────┤
│ NEEDS ATTENTION (0–n)               │  Pinned section (hidden if empty)
│  ⚠ Backfill — Rate limited  2h ago │  ExpansionTile, error styling
│  ✗ Cleanup — failed         1d ago   │
├─────────────────────────────────────┤
│ HISTORY                             │  Section header
│  ✓ Cleanup — 4.2k removed   10:00  │  Newest first
│  ✓ Backfill — 98k written   09:00  │
│  ✓ Universe — 8.2k assets   08:00  │
│  ...                                │  Infinite scroll
└─────────────────────────────────────┘

7.2 Collapsed row

Leading icon by severity / kind
Title: {Kind label} — {short status}
Trailing: relative time (2h ago)
Optional chips: rows_written, rows_removed

7.3 Expanded row

ExpansionTile children: labeled rows (not monospace dumps)
Selectable SelectableText for full error message
Copy-to-clipboard button for error block

7.4 Refresh and pagination

Action	Behavior
Pull-to-refresh	Reload first page + recompute pins
Scroll end	`before=` cursor load next page, append to history (not pinned)
Auto-refresh	Optional 60s timer while screen visible

7.5 On-demand actions (v1.1)

AppBar overflow menu:

Run backfill now → POST resync → show in-progress row → refresh
Run cleanup now → POST cleanup → confirm dialog (destructive) → refresh

Disable buttons while any kind has in-progress run (finished_at == null).

8. Phased delivery

Phase A — Read-only log (MVP)

Track	Work
Server	`GET sync-runs` + admin auth + `pinned` computation
Flutter	Models, API service, log screen, expansion tiles, pull-to-refresh
QA	Seed `market_data_sync_runs` in test DB; verify pin/unpin when new success inserted

Exit: Operator can view retention/cleanup/backfill history on device/emulator.

Phase B — Failure UX polish

Severity icons/colors, rate-limit copy, partial-success template
Copy error, relative timestamps, empty/loading/error states
Widget tests for sort/pin merge logic

Phase C — On-demand triggers

POST resync, POST cleanup
Flutter buttons + confirmation + in-progress state

Phase D — Structured errors (optional)

Server writes detail JSONB; Flutter renders HTTP status, endpoint, batch symbols without parsing free text

9. Testing

Server

Test	Description
Integration	`test/integration/market_history_admin_handler_test.dart` — 403 non-admin, 200 list shape, pinned logic
Fixture rows	Success cleanup, failed backfill 429, partial backfill

Flutter

Test	Description
Unit	`sync_run_log_repository_test.dart` — merge pinned + chronological, pin clearance
Unit	`sync_run_event_test.dart` — severity from error string
Widget	Expansion tile shows/hides detail; pinned section visibility
Integration	Mock HTTP → screen goldens or pump tests

Manual

Enable worker sync; wait for 3 kinds in log
Simulate 429 (mock Alpaca or test env throttle)
Confirm pin → run successful backfill → pin clears

10. Security and compliance

Admin routes only with verified admin identity
Logs may contain Alpaca error bodies — no PII, but treat as internal ops data
Do not log API keys; ensure error column never stores secrets (review SyncRunRecorder callers)
CORS: same as existing API (apiCorsHeaders)
Production: ADMIN_PORTAL_ENABLED=false by default

11. Configuration (align with §7 env)

Document in server/README.md when implemented:

Env var	Purpose
`ADMIN_PORTAL_ENABLED`	Mount `/v1/admin/market-history/*`
`ADMIN_FIREBASE_UIDS`	Comma-separated allowlist
`MARKET_HISTORY_*`	Window, sync cadence (worker); shown in expanded row as read-only meta

12. Relationship to §8 CLIs

Capability	Admin portal	§8 CLI
View sync/cleanup log	Yes (primary)	No
On-demand sync/cleanup	Yes (Phase C API)	Yes (planned)
CI / headless / no UI	No	Yes
Ops SSH without app	No	Yes

Recommendation: Implement portal + admin API first; defer §8 CLIs unless SRE requests them.

13. Open decisions

#	Question	Default proposal
1	Admin discovery in app?	Debug-only entry + allowlist
2	Pin partial backfill with `rows_written > 0`?	Yes, until `error` null
3	Separate web-only admin build?	Single app, gated route (v1)
4	Real-time updates?	Pull-to-refresh only (v1); WebSocket later
5	`finished_at` null stale threshold?	30 minutes → warning row

14. Acceptance criteria

Admin user sees newest history events at top of main list
Failed/rate-limited runs appear in Needs attention until same kind succeeds later
Expand row shows operational detail without raw market-data JSON
API error expanded view shows complete error text from server
Non-admin receives 403 and no data
Pull-to-refresh and pagination load additional older events
(Phase C) On-demand cleanup/sync creates visible new log entries

Document version: 1.0 — Flutter admin portal for market_data_sync_runs audit log.

16 KiB Raw Blame History Unescape Escape