218 lines
8.4 KiB
Markdown
218 lines
8.4 KiB
Markdown
# Cyber Hybrid Hub API
|
||
|
||
Postgres-backed profile API for the Flutter app.
|
||
|
||
## Setup
|
||
|
||
1. Create the database (once):
|
||
|
||
```bash
|
||
createdb cyberhybridhub
|
||
```
|
||
|
||
2. Copy and edit environment variables:
|
||
|
||
```bash
|
||
cp .env.example .env
|
||
```
|
||
|
||
3. Install dependencies and run (from this `server/` directory):
|
||
|
||
```bash
|
||
dart pub get
|
||
dart run bin/server.dart
|
||
```
|
||
|
||
The API listens on `http://localhost:3000` by default (`PORT` in `.env`).
|
||
|
||
## Tests
|
||
|
||
From the repo root (loads `server/.env` automatically):
|
||
|
||
```bash
|
||
./scripts/test-server.sh # unit + DB integration (no live Alpaca)
|
||
./scripts/test-server-alpaca.sh # live SPY quote — requires keys in server/.env
|
||
```
|
||
|
||
Or from `server/`:
|
||
|
||
```bash
|
||
# Uses DATABASE_URL from the environment or server/.env
|
||
export DATABASE_URL=postgresql://postgres:PASSWORD@localhost:5432/cyberhybridhub
|
||
dart pub get
|
||
dart test
|
||
```
|
||
|
||
Integration tests apply migrations `001`–`009` on `cyberhybridhub_test` and truncate
|
||
trading tables between cases. Optional override: `TEST_DATABASE_URL`.
|
||
|
||
## Endpoints
|
||
|
||
| Method | Path | Auth |
|
||
|--------|------|------|
|
||
| `GET` | `/v1/me/profile` | `Authorization: Bearer <Firebase ID token>` |
|
||
| `PUT` | `/v1/me/profile` | same |
|
||
| `POST` | `/v1/me/incoming-question` | same — pushes a question to the client via SignalR |
|
||
| `POST` | `/v1/me/questions/bootstrap` | ensure starter question at login |
|
||
| `GET` | `/v1/me/questions` | list unanswered questions (queue order) |
|
||
| `POST` | `/v1/me/questions/{id}/answer` | submit answer (`{"answer": 0}` default) |
|
||
| `POST` | `/v1/me/questions/{id}/defer` | move question to end of queue |
|
||
|
||
## SignalR — incoming questions
|
||
|
||
Hub URL: `http://localhost:3000/hubs/questions`
|
||
|
||
The Flutter app calls `POST /v1/me/questions/bootstrap` once at login to ensure a
|
||
starter question exists (random correct answer from -10 to 10) when the user has none.
|
||
After sign-in it connects to SignalR and listens for `ReceiveQuestion`. On each new
|
||
WebSocket connection the API only delivers an existing unanswered question — it does
|
||
not create new rows.
|
||
|
||
Client payload (correct answer is not sent):
|
||
|
||
```json
|
||
{
|
||
"id": "uuid",
|
||
"assignedUserId": "firebase-uid",
|
||
"text": "...",
|
||
"sentAt": "...",
|
||
"unansweredCount": 2
|
||
}
|
||
```
|
||
|
||
`unansweredCount` is the number of unanswered rows for that user (shown in the app when greater than 1).
|
||
|
||
`questions` table: `id` (UUID), `assigned_user_id`, `question_text`, `user_response`
|
||
(nullable), `correct_answer`, `created_at`, `modified_at`.
|
||
|
||
## Background question pipeline
|
||
|
||
A background worker runs inside the API process (enabled by default). On each
|
||
interval it walks registered users, fetches data from public web APIs, and
|
||
enqueues pipeline questions when the user's queue is not full.
|
||
|
||
| Env var | Default | Purpose |
|
||
|---------|---------|---------|
|
||
| `QUESTION_WORKER_ENABLED` | `true` | Set to `false` to disable the worker |
|
||
| `QUESTION_WORKER_INTERVAL_SECONDS` | `60` | Seconds between maintenance cycles |
|
||
| `QUESTION_PIPELINE_TEST_MODE` | `false` | Use random -10..10 starter-style questions instead of API copy |
|
||
|
||
**External APIs used**
|
||
|
||
- [REST Countries](https://restcountries.com/) — population and capital facts
|
||
- [Open-Meteo](https://open-meteo.com/) — current temperature by city
|
||
|
||
**Branching flow**
|
||
|
||
1. **Track choice** — user swipes toward +10 (weather) or -10 (geography).
|
||
2. **Geography** — yes/no population threshold, then capital confirmation;
|
||
wrong population guess triggers a recovery question.
|
||
3. **Weather** — yes/no warm/cool for a random European city, then a follow-up
|
||
to continue weather or switch to geography.
|
||
|
||
When a user submits an answer (`POST .../answer`), the pipeline evaluates the
|
||
response and may immediately create the next branched question and push it over
|
||
SignalR.
|
||
|
||
Pipeline state is stored in `user_pipeline_state`; questions may include
|
||
`source_tag`, `pipeline_key`, and `pipeline_step` columns (migration
|
||
`003_question_pipeline.sql`).
|
||
|
||
Test from the shell (replace `ID_TOKEN`):
|
||
|
||
```bash
|
||
curl -s -X POST http://localhost:3000/v1/me/incoming-question \
|
||
-H "Authorization: Bearer ID_TOKEN" \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"text":"What is your preferred contact method?"}'
|
||
```
|
||
|
||
## Market history
|
||
|
||
Alpaca **`1Min`** bars aggregated into two **US regular-session** half-days per trading day
|
||
(morning **9:30–12:45 ET**, afternoon **12:45–16:00 ET**, ~195 minutes each).
|
||
Stored as `metric=bar`, `timeframe=sessionHalf`. Rolling window: `MARKET_HISTORY_WINDOW_DAYS` (default 7).
|
||
|
||
**Backfill** (`kind=backfill`): fetches each **ended** slot still missing in DB; skips the open slot.
|
||
Throttled to `MARKET_HISTORY_API_REQUESTS_PER_MINUTE` (default 200/min). On HTTP 429: wait 1 minute, retry once; if still limited, save partial rows and resume next tick.
|
||
Runs when `hasPendingSlots` is true (worker tick). One bars request per slot × symbol batch.
|
||
Before each worker or admin pipeline, orphaned `market_data_sync_runs` rows with
|
||
`finished_at IS NULL` are closed (stale rows first, then any remaining orphans) so a
|
||
crashed prior sync cannot block new work.
|
||
**Universe** / **cleanup** keep hour-based cadence. `guess_weekly_move` reads these bars.
|
||
|
||
Requires `TRADING_ENABLED=true` when `MARKET_HISTORY_SYNC_ENABLED=true`.
|
||
|
||
**Migration `008`:** legacy `1Day` / `4Hour` history cleanup and `slots_synced` on sync runs.
|
||
|
||
**Migration `010`:** deletes `4Hour` bars, adds `sessionHalf` timeframe + partial index.
|
||
|
||
**Migration `009`:** adds `market_data_sync_runs.backfill_items` JSONB — per-slot UTC start + symbol list for each backfill run.
|
||
|
||
| Env var | Default | Purpose |
|
||
|---------|---------|---------|
|
||
| `MARKET_HISTORY_SYNC_ENABLED` | `false` | Worker: universe → backfill → cleanup |
|
||
| `MARKET_HISTORY_WINDOW_DAYS` | `7` | Oldest slot start to retain / backfill |
|
||
| `MARKET_HISTORY_RETENTION_DAYS` | `7` | Delete `as_of` older than this |
|
||
| `MARKET_HISTORY_ARCHIVE_ENABLED` | `false` | Archive before delete |
|
||
| `MARKET_UNIVERSE_REFRESH_HOURS` | `24` | Min hours between universe syncs |
|
||
| `MARKET_HISTORY_SYNC_HOURS` | `24` | Unused for backfill (slot-gated); fallback if no slot gate |
|
||
| `MARKET_HISTORY_CLEANUP_HOURS` | `24` | Min hours between cleanups |
|
||
| `MARKET_HISTORY_SYNC_HOUR_UTC` | *(unset)* | UTC hour floor; same-day cap applies to universe/cleanup only |
|
||
| `HISTORY_SYNC_BATCH_SIZE` | `50` | Symbols per bars request |
|
||
| `HISTORY_SYNC_MAX_SYMBOLS` | `2000` | Max symbols per backfill run |
|
||
| `MARKET_HISTORY_API_REQUESTS_PER_MINUTE` | `200` | Max Alpaca HTTP calls per rolling minute during backfill |
|
||
| `MARKET_HISTORY_SYNC_STALE_MINUTES` | `30` | Abort in-progress sync rows older than this before a new pipeline |
|
||
| `MIN_BARS_FOR_GUESS` | `5` | Min 4-hour bars for guess eligibility |
|
||
| `GUESS_COOLDOWN_HOURS` | `24` | Per-symbol guess cooldown |
|
||
|
||
## Admin portal (market history log)
|
||
|
||
Read-only audit log and on-demand triggers for universe sync, backfill, and
|
||
retention cleanup. Mounted when `ADMIN_PORTAL_ENABLED=true`.
|
||
|
||
| Env var | Default | Purpose |
|
||
|---------|---------|---------|
|
||
| `ADMIN_PORTAL_ENABLED` | `false` | Mount `/v1/admin/market-history/*` and `/v1/admin/market-data/*` |
|
||
| `ADMIN_FIREBASE_UIDS` | *(empty)* | Comma-separated Firebase UIDs allowed to call admin routes |
|
||
|
||
Requires `TRADING_ENABLED=true`, Alpaca credentials, and
|
||
`QUESTION_PIPELINE_TEST_MODE=false` for on-demand resync/cleanup when
|
||
`MARKET_HISTORY_SYNC_ENABLED=true`. When sync is disabled, the admin log and
|
||
week-coverage calendar remain available read-only; resync/cleanup return `503`.
|
||
Scheduled worker sync also requires `MARKET_HISTORY_SYNC_ENABLED=true`.
|
||
|
||
| Method | Path | Auth |
|
||
|--------|------|------|
|
||
| `GET` | `/v1/admin/market-history/sync-runs` | Admin Firebase UID |
|
||
| `POST` | `/v1/admin/market-data/resync` | Admin Firebase UID |
|
||
| `POST` | `/v1/admin/market-data/cleanup?archive=true\|false` | Admin Firebase UID |
|
||
|
||
`sync-runs` includes optional `config` when the server has market-history env loaded:
|
||
|
||
```json
|
||
{ "archiveEnabled": true, "windowDays": 7, "retentionDays": 7, "syncEnabled": true }
|
||
```
|
||
|
||
Flutter uses `config.archiveEnabled` to show the archive checkbox in the cleanup
|
||
confirm dialog.
|
||
|
||
### Admin portal tests
|
||
|
||
From the repo root:
|
||
|
||
```bash
|
||
./scripts/test-admin-portal.sh # server + Flutter admin suites
|
||
./scripts/check-admin-portal-coverage.sh # coverage + Section 2 thresholds
|
||
```
|
||
|
||
CI: `.github/workflows/admin-portal.yml` runs both on admin-related changes.
|
||
|
||
## Flutter client
|
||
|
||
Run the app with the API URL (defaults to `http://localhost:3000`):
|
||
|
||
```bash
|
||
flutter run --dart-define=API_BASE_URL=http://localhost:3000
|
||
```
|