# Flutter Admin Portal — Development Plan

**Status:** Planning  
**Related:** `TODO.md` §8 (operational CLIs), §8.5 (admin API), rolling market-history pipeline (§1–§7)  
**Audience:** Mobile/web Flutter developers, API maintainers, operators  

---

## 1. Purpose

Build an **admin-only** area in the Flutter app that shows a **live audit log** of periodic market-history operations:

- **Universe sync** (`kind=universe`) — tradable asset refresh  
- **Historical backfill** (`kind=backfill`) — ended **4-hour UTC slots** (`4Hour` bars) into `market_data_snapshots`; open slot not fetched yet  
- **Retention / cleanup** (`kind=cleanup`) — prune (or archive-then-delete) expired rows  

Operators use it to confirm the rolling 7-day window is healthy, see when cleanup ran, and **notice Alpaca/API failures or rate limits immediately** without reading server logs or SQL.

This plan **replaces the need for §8 shell CLIs** for day-to-day product use. CLIs remain optional for CI/SRE (see §12).

---

## 2. Goals and non-goals

### Goals

| # | Goal |
|---|------|
| G1 | Chronological log: **newest events at the top**; scroll down for older history |
| G2 | **Pin unresolved failures** (API errors, rate limits, partial batch failures) at the top until a **successful retry** for the same `kind` (and optionally same error “thread”) |
| G3 | **Expandable list rows** — collapsed summary; expanded shows structured detail **without** raw JSON payloads (no bar blobs, no `tradable_assets.raw`) |
| G4 | For API failures, expanded view shows **full operator-relevant error text** (HTTP status, Alpaca message body summary, batch scope) |
| G5 | Optional: **on-demand** “Run sync” / “Run cleanup” from the portal (API-triggered), with new rows appearing in the log |
| G6 | Reuse existing auth (Firebase ID token → API) and app theme/navigation patterns |

### Non-goals (v1)

- End-user (non-admin) access to sync logs  
- Displaying raw `market_data_snapshots` or per-symbol bar tables  
- Full log export / SIEM integration (future)  
- Replacing the background worker; scheduler still runs automatically when `MARKET_HISTORY_SYNC_ENABLED=true`  
- Building §8 `bin/sync_market_history.dart` / `bin/cleanup_market_history.dart` (defer unless ops requests them)

---

## 3. Source of truth (already in Postgres)

All events are recorded in **`market_data_sync_runs`** (migration `005_market_history.sql`):

| Column | Meaning |
|--------|---------|
| `id` | Stable row id |
| `kind` | `universe` \| `backfill` \| `cleanup` |
| `started_at` | UTC start |
| `finished_at` | UTC end (null if crash mid-run — treat as in-progress) |
| `rows_written` | Upserted rows (universe/backfill) |
| `rows_removed` | Deleted rows (cleanup) |
| `error` | `NULL` = success; non-null = failure or **partial** failure message |

**Kinds map to UI labels:**

| `kind` | UI title | Typical success summary |
|--------|----------|-------------------------|
| `universe` | Asset universe sync | N assets refreshed |
| `backfill` | 4-hour slot backfill (ended slots only) | N bar rows written |
| `cleanup` | Data retention cleanup | N rows removed |

**Error shapes today (no schema change required for v1):**

- Thrown exception → `error` is `e.toString()` (e.g. `AlpacaMarketDataException: getBarsRange rate limited: 429 …`)  
- Partial backfill → `error` may list batch failures while `rows_written > 0` (`SyncRunCounts.error` in `market_data_history.dart`)  

**v1 recommendation:** Parse `error` text client-side for keywords (`429`, `rate`, `Alpaca`, `batch`) to set severity and pin rules. **v2 (optional):** add `error_code`, `http_status`, `severity` columns or a JSON `detail` object on insert (server change).

---

## 4. List ordering and “pin until resolved”

### 4.1 Default sort

Primary: **`started_at` DESC** (most recent at top).

### 4.2 Pin rule (failure priority)

An event is **pinned** when:

1. `error IS NOT NULL` **or** `finished_at IS NULL` (stuck in-progress > N minutes → show as warning), **and**  
2. There is **no later successful run** of the **same `kind`** with `error IS NULL` and `finished_at IS NOT NULL` **after** this row’s `started_at`.

**Pinned block** appears **above** the chronological list (fixed section or visually distinct header: “Needs attention”).

When a new successful run for that `kind` completes, the older failed row **unpins** and appears only in normal history order.

### 4.3 Rate limits and API errors

Treat as **high severity** when `error` matches (case-insensitive):

- `429`, `rate limit`, `rate limited`  
- `AlpacaMarketDataException`, `AlpacaAssetsException`, `AlpacaTradingException`  
- HTTP `5xx` patterns in message  

Pinned rate-limit rows use a **warning/error** color and icon until the pin rule clears.

### 4.4 Partial success (backfill)

If `rows_written > 0` **and** `error != null`:

- Collapsed: **“Partial success”** badge + counts  
- Expanded: show `rows_written`, full `error` text (batch list), **do not** show raw bar JSON  
- Pin until next **fully successful** backfill (`error` null) **or** product decision: pin until error null even if partial (stricter — **recommended for ops**)

---

## 5. Backend API (prerequisite)

No admin market-history endpoints exist today. Implement **before** Flutter UI (or in parallel with mock API).

### 5.1 Auth model

| Approach | Recommendation |
|----------|----------------|
| Firebase + allowlist | `ADMIN_FIREBASE_UIDS=uid1,uid2` in server env; middleware rejects others with `403` |
| Custom claim | `admin: true` on Firebase token; server verifies claim |
| Separate admin API key | Avoid for mobile; poor UX |

Match existing pattern: `Authorization: Bearer <Firebase ID token>` like `QuestionsApiService`.

**Do not** expose these under `/v1/me/...` without admin checks — use `/v1/admin/...` prefix.

### 5.2 Endpoints (v1)

#### `GET /v1/admin/market-history/sync-runs`

Query params:

| Param | Default | Description |
|-------|---------|-------------|
| `limit` | `50` | Page size |
| `before` | — | Cursor: `id` or `started_at` of oldest item in current page (for infinite scroll) |
| `kind` | — | Optional filter: `universe`, `backfill`, `cleanup` |

Response:

```json
{
  "runs": [
    {
      "id": 123,
      "kind": "cleanup",
      "startedAt": "2026-05-26T10:00:00Z",
      "finishedAt": "2026-05-26T10:00:05Z",
      "rowsWritten": 0,
      "rowsRemoved": 4200,
      "error": null,
      "severity": "ok",
      "status": "success",
      "durationMs": 5000,
      "summary": "Removed 4,200 expired snapshots"
    }
  ],
  "pinned": [
    {
      "id": 120,
      "kind": "backfill",
      "error": "getBarsRange rate limited: 429 ...",
      "severity": "rate_limit",
      "status": "failed",
      "summary": "Backfill partial: 12,000 rows; rate limited"
    }
  ],
  "nextBefore": "2026-05-25T08:00:00Z"
}
```

Server computes `pinned[]` with the rule in §4.2.  
Server computes `severity`: `ok` \| `warning` \| `error` \| `rate_limit` from `error` text + `finished_at`.  
**Never** include `raw` JSON from snapshots or assets in this API.

#### `GET /v1/admin/market-history/sync-runs/{id}`

Single run for expanded row refresh (optional if list payload is enough).

#### `POST /v1/admin/market-data/resync` (optional v1.1)

Runs universe + slot backfill (ended `4Hour` slots in window, not the open slot). Returns `{ "runIds": […] }` (202). `windowDays` on cleanup only today.

#### `POST /v1/admin/market-data/cleanup` (optional v1.1)

Query: `windowDays`, `archive=true|false`.  
Runs `MarketDataRetention.run(archive: …)` once.

### 5.3 Implementation notes (server)

- New handler: `server/lib/handlers/market_history_admin_handler.dart`  
- Mount only when `ADMIN_PORTAL_ENABLED=true` (or reuse `TRADING_DEV_ENDPOINTS_ENABLED` — **prefer dedicated flag**)  
- Reuse `TradableAssetsSync`, `MarketDataHistorySync`, `MarketDataRetention` — same as `bin/server.dart` scheduler wiring  
- List query: indexed `ORDER BY started_at DESC` (index exists: `market_data_sync_runs_kind_started_idx`)

### 5.4 Error detail for API failures (v2 enhancement)

When recording sync failures, optionally persist structured fields:

```json
{
  "httpStatus": 429,
  "endpoint": "GET /v2/stocks/bars",
  "message": "...",
  "batchSymbols": ["AAPL", "MSFT"]
}
```

Store in new `detail JSONB` column or append to `error` as delimited text for v1.

---

## 6. Flutter architecture

### 6.1 Entry and routing

| Item | Proposal |
|------|----------|
| Route | `/admin/market-history` or dedicated `AdminPortalScreen` pushed from a hidden entry |
| Gate | `AdminGate` widget: calls `GET /v1/admin/market-history/sync-runs?limit=1` — `403` → “Not authorized” |
| Discovery | Long-press app logo, `kDebugMode` menu item, or profile flag `isAdmin` from `GET /v1/me/profile` extension |

Do **not** show admin nav to all signed-in users.

### 6.2 Layering (match existing app)

```
lib/
  admin/
    models/
      sync_run_event.dart          # DTO from API
    services/
      market_history_admin_api.dart  # HTTP + auth headers (mirror QuestionsApiService)
    repositories/
      sync_run_log_repository.dart   # paging, pin merge, refresh
    screens/
      market_history_log_screen.dart
    widgets/
      sync_run_expansion_tile.dart
      pinned_alerts_section.dart
      sync_run_status_chip.dart
```

- **API service:** `apiBaseUrl` from `lib/config/api_config.dart`, token via `AuthService.instance.getIdToken()`  
- **State:** `Listenable`/`ChangeNotifier` or `flutter_riverpod` if project adopts it later; v1 can use `StatefulWidget` + repository like profile sync  

### 6.3 `SyncRunEvent` model (client)

Fields from API plus derived:

- `isPinned` (from `pinned` array or client recompute)  
- `Severity` enum: `ok`, `warning`, `error`, `rateLimit`  
- `displayTitle` from `kind`  
- `collapsedSubtitle` — one line: time ago + summary  
- `expandedSections` — `List<DetailSection>` title/body pairs (no raw JSON)

### 6.4 Expanded row content (templates)

**Success — cleanup**

- Started / finished (local timezone)  
- Duration  
- Rows removed  
- Archive mode on/off (if API adds flag later)

**Success — backfill**

- Rows written  
- Window days (from env or API meta)  
- “No errors”

**Failure / rate limit**

- Full `error` string (selectable text)  
- Parsed HTTP status if regex finds it  
- Kind + run id (for support tickets)  
- Hint: “Will retry on next scheduled run” or button **Retry now** if POST endpoint exists  

**Partial backfill**

- Rows written (success count)  
- Error block (batch failures)  
- Do **not** render `raw` from bars  

**In progress**

- Started at; spinner if `finished_at == null` and started &lt; 30m ago  
- Stale run warning if &gt; 30m without `finished_at`

---

## 7. UI specification

### 7.1 Screen layout

```
┌─────────────────────────────────────┐
│ ←  Market history log        [↻]   │  AppBar: refresh
├─────────────────────────────────────┤
│ NEEDS ATTENTION (0–n)               │  Pinned section (hidden if empty)
│  ⚠ Backfill — Rate limited  2h ago │  ExpansionTile, error styling
│  ✗ Cleanup — failed         1d ago   │
├─────────────────────────────────────┤
│ HISTORY                             │  Section header
│  ✓ Cleanup — 4.2k removed   10:00  │  Newest first
│  ✓ Backfill — 98k written   09:00  │
│  ✓ Universe — 8.2k assets   08:00  │
│  ...                                │  Infinite scroll
└─────────────────────────────────────┘
```

### 7.2 Collapsed row

- Leading icon by `severity` / `kind`  
- Title: `{Kind label} — {short status}`  
- Trailing: relative time (`2h ago`)  
- Optional chips: `rows_written`, `rows_removed`

### 7.3 Expanded row

- `ExpansionTile` children: labeled rows (not monospace dumps)  
- **Selectable** `SelectableText` for full error message  
- Copy-to-clipboard button for error block  

### 7.4 Refresh and pagination

| Action | Behavior |
|--------|----------|
| Pull-to-refresh | Reload first page + recompute pins |
| Scroll end | `before=` cursor load next page, **append** to history (not pinned) |
| Auto-refresh | Optional 60s timer while screen visible |

### 7.5 On-demand actions (v1.1)

AppBar overflow menu:

- **Run backfill now** → `POST resync` → show in-progress row → refresh  
- **Run cleanup now** → `POST cleanup` → confirm dialog (destructive) → refresh  

Disable buttons while any `kind` has in-progress run (`finished_at == null`).

---

## 8. Phased delivery

### Phase A — Read-only log (MVP)

| Track | Work |
|-------|------|
| Server | `GET sync-runs` + admin auth + `pinned` computation |
| Flutter | Models, API service, log screen, expansion tiles, pull-to-refresh |
| QA | Seed `market_data_sync_runs` in test DB; verify pin/unpin when new success inserted |

**Exit:** Operator can view retention/cleanup/backfill history on device/emulator.

### Phase B — Failure UX polish

- Severity icons/colors, rate-limit copy, partial-success template  
- Copy error, relative timestamps, empty/loading/error states  
- Widget tests for sort/pin merge logic  

### Phase C — On-demand triggers

- `POST resync`, `POST cleanup`  
- Flutter buttons + confirmation + in-progress state  

### Phase D — Structured errors (optional)

- Server writes `detail JSONB`; Flutter renders HTTP status, endpoint, batch symbols without parsing free text  

---

## 9. Testing

### Server

| Test | Description |
|------|-------------|
| Integration | `test/integration/market_history_admin_handler_test.dart` — 403 non-admin, 200 list shape, pinned logic |
| Fixture rows | Success cleanup, failed backfill 429, partial backfill |

### Flutter

| Test | Description |
|------|-------------|
| Unit | `sync_run_log_repository_test.dart` — merge pinned + chronological, pin clearance |
| Unit | `sync_run_event_test.dart` — severity from error string |
| Widget | Expansion tile shows/hides detail; pinned section visibility |
| Integration | Mock HTTP → screen goldens or pump tests |

### Manual

1. Enable worker sync; wait for 3 kinds in log  
2. Simulate 429 (mock Alpaca or test env throttle)  
3. Confirm pin → run successful backfill → pin clears  

---

## 10. Security and compliance

- Admin routes **only** with verified admin identity  
- Logs may contain Alpaca error bodies — **no PII**, but treat as internal ops data  
- Do not log API keys; ensure `error` column never stores secrets (review `SyncRunRecorder` callers)  
- CORS: same as existing API (`apiCorsHeaders`)  
- Production: `ADMIN_PORTAL_ENABLED=false` by default  

---

## 11. Configuration (align with §7 env)

Document in `server/README.md` when implemented:

| Env var | Purpose |
|---------|---------|
| `ADMIN_PORTAL_ENABLED` | Mount `/v1/admin/market-history/*` |
| `ADMIN_FIREBASE_UIDS` | Comma-separated allowlist |
| `MARKET_HISTORY_*` | Window, sync cadence (worker); shown in expanded row as read-only meta |

---

## 12. Relationship to §8 CLIs

| Capability | Admin portal | §8 CLI |
|------------|----------------|--------|
| View sync/cleanup log | Yes (primary) | No |
| On-demand sync/cleanup | Yes (Phase C API) | Yes (planned) |
| CI / headless / no UI | No | Yes |
| Ops SSH without app | No | Yes |

**Recommendation:** Implement **portal + admin API** first; defer §8 CLIs unless SRE requests them.

---

## 13. Open decisions

| # | Question | Default proposal |
|---|----------|------------------|
| 1 | Admin discovery in app? | Debug-only entry + allowlist |
| 2 | Pin partial backfill with `rows_written > 0`? | Yes, until `error` null |
| 3 | Separate web-only admin build? | Single app, gated route (v1) |
| 4 | Real-time updates? | Pull-to-refresh only (v1); WebSocket later |
| 5 | `finished_at` null stale threshold? | 30 minutes → warning row |

---

## 14. Acceptance criteria

- [ ] Admin user sees **newest** history events at top of main list  
- [ ] Failed/rate-limited runs appear in **Needs attention** until same `kind` succeeds later  
- [ ] Expand row shows operational detail **without** raw market-data JSON  
- [ ] API error expanded view shows **complete** `error` text from server  
- [ ] Non-admin receives 403 and no data  
- [ ] Pull-to-refresh and pagination load additional older events  
- [ ] (Phase C) On-demand cleanup/sync creates visible new log entries  

---

*Document version: 1.0 — Flutter admin portal for `market_data_sync_runs` audit log.*