updates on the ui and updates agents

docs/agent-platform-phase7-plan.md

@@ -0,0 +1,777 @@
+# Doormile Agent Platform — Phase 7: Decision Memory, Deployment, Executors
+
+Status: **mostly implemented 2026-10-08 (uncommitted, not deployed, no migration run)**
+Written 2026-10-08 · Scope: `krow_talent_app`, `doormile_backend`, `AI_engine`, `kubernetes`
+
+---
+
+## Status (2026-10-08)
+
+Verified: backend `go build`/`go vet` clean, 20 packages pass · engine 148 tests
+(3 pre-existing import errors) · console 1397 pass (13 pre-existing
+`agentStudio.test.jsx` failures) · `kubectl kustomize manifests/doormile` renders
+clean. **Nothing committed. No migration has run. No SQL in this work has ever
+executed against a database.**
+
+| Item | State |
+|---|---|
+| A1 config/secrets manifest, `envFrom`, probes, resources | **built** — placeholders unreplaced |
+| A2 secrets out of git | **not done** — documented in `manifests/doormile/SECRETS.md`, including the kustomize-clobbers-hand-created-secrets trap. Five more placeholder keys were added. |
+| A3 `anthropic>=0.49.0` · A4 `claude-opus-5-5` | **done** |
+| B0.1 `CREATE EXTENSION vector` · B0.2 `/similar` → POST | **done** |
+| B1 provider (OpenAI `text-embedding-3-small`, 1536) | **done** — my call, flagged |
+| B2 `core/embeddings.py` + decision wiring | **done** |
+| B3 outcome sweeper + rules + 14 tests | **done** |
+| B4 `core/memory.py` + both agents | **done** |
+| B5 retention/prune | **done** — ivfflat `lists` re-tune still needs a row count |
+| B6 `tenantid` scoping | **done** |
+| B2.5 historical backfill | **done** — `internal/ai/outcomes/backfill.go` |
+| B7 persist console findings | **done** — `aiskillfindings`, 4 routes, `findingReport.js`, 15 tests |
+| C1 health surface + `ai-engine.yaml` + Dockerfile/compose port | **done** — image not built or pushed |
+| C2 kustomization + `deploy-doormile.sh` | **done** |
+| C3 probes/resources/PDB | **done** — StatefulSet→Deployment not done |
+| C4 ingress | **built** — apply order matters, see the file header |
+| C5 NATS port note | **done** — `docs/ARCHITECTURE.md` |
+| C6 engine replica safety | **see correction 1 — was already safe for a different reason** |
+| D1 admin batch-assign + console executor | **done** |
+| D2 `alert_low_battery_rider` | **done** |
+| D3 five review-only tools | **1 of 5 done** — `trigger_auto_dispatch` turned out to BE batch-assign under another name. The other four need endpoints that do not exist. |
+| Agents-page snapshot → live endpoint | **backend done** — engine serves `GET /agents/status`; the console still reads the snapshot, so the swap is one fetch away |
+
+## Two things this plan got wrong
+
+**Correction 1 — C6. The engine's stall dedup was never per-pod.**
+This plan claimed `ExceptionAgent` kept its dedup in an in-process TTL set and
+that moving it to Redis was the prerequisite for scaling out. Wrong:
+`_claim_stall` / `_claim_stall_handled` (`agents/exception_agent.py:336`) have
+been Redis `SET NX` with a TTL all along — cross-pod, self-expiring, durable
+across restarts. The comment there saying "replaces the old unbounded in-memory
+set" describes what it REPLACED; I read it as current state.
+
+The real blocker is different and still real: `core/message_bus.py:284` and
+`:312` bind **durable push consumers with fixed names**, and a durable push
+consumer admits one active subscriber unless created with a deliver group. A
+second replica does not duplicate work — it fails to bind. Scaling out means
+giving those subscriptions a deliver group (or moving to pull consumers).
+`replicas: 1` stands, for the corrected reason.
+
+**Correction 2 — the registry was already honest about the simulated agents.**
+This plan said the three fictional-data agents were "presented as live" and
+should be marked. They already were: `seed.go` has `StatusSimulation` for
+`HUB_AGENT`, `FLEET_AGENT` and `ROUTE_OPTIMIZER`, with purposes reading "an
+in-memory fleet of 19 fake vehicles" and "8 hard-coded fictional hubs". The
+console snapshot (`src/lib/agentNetwork.js`) was also honest about activity
+("Zero tasks received", `publishes: 'Nothing on the bus.'`, `api: 'No.'`) but
+silent on the data being invented — so the three entries now say so, and the two
+surfaces agree.
+
+I overstated that surface as "telling anyone something untrue". It was
+incomplete, not false.
+
+---
+
+## Track A — Unblock (do this first; nothing else matters until it lands)
+
+### A1. `INTERNAL_API_KEY` is missing from the cluster — **everything engine↔backend is 401**
+
+`middlewares/internal_auth.go:13` reads `INTERNAL_API_KEY` and **fails closed when
+unset**:
+
+```go
+expected := os.Getenv("INTERNAL_API_KEY")
+if expected == "" || c.Get("X-Internal-Key") != expected {
+```
+
+`kubernetes/manifests/doormile/miletruth.yaml` never sets it. The engine's
+`docker-compose.yml` *does* pass it. So the engine sends a key the backend
+rejects, and in-cluster **every** `/api/v1/internal/*` call 401s:
+
+- `GET /internal/ai/registry` — the registry poll (this is why "engine reads
+  registry in prod" was never confirmable)
+- `POST /internal/agent-decisions` — the decision log, so Insights sees nothing
+- `GET /internal/express/riders`, `/express/bookings`, `POST /express/assign`
+- `POST /internal/bookings/:id/reassign`, `POST /internal/notify`
+
+**Step 1 — find out whether git matches reality.** The `kubernetes` repo's log is
+full of `Restore…` / `Recreate…` commits, so the manifest may already be fiction:
+
+```bash
+kubectl -n doormile get statefulset doormile -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}{"\n"}{end}' | sort
+```
+
+`config/config.go` reads 29 env vars. If that list is much shorter, the live
+cluster has hand-applied drift and **the next `kubectl apply -f` wipes it.**
+
+**Step 2 — close the gap in git, not by hand.** Add to `doormile-secrets`
+(`stringData`) and reference from the StatefulSet. Absent from the manifest today
+and read by `config.go`:
+
+`INTERNAL_API_KEY`, `JWT_SECRET_KEY`, `APP_PORT`, `ENV`, `TRUSTED_PROXIES`,
+`AI_LAYER_BASE_URL`, `ROUTE_OPTIMIZER_URL`, `GEOCODER_URL`, `GEOCODER_EMAIL`,
+`SMTP_HOST`, `SMTP_PORT`, `SMTP_USER`, `SMTP_PASSWORD`, `SMTP_FROM`,
+`CLIENT_ONBOARDING_OWNERS`, `PLAYGROUND_LLM_API_KEY`, `PLAYGROUND_LLM_BASE_URL`,
+`PLAYGROUND_LLM_MODEL`, `REDIS_USER`.
+
+Secrets vs ConfigMap: `INTERNAL_API_KEY`, `JWT_SECRET_KEY`, `SMTP_PASSWORD`,
+`PLAYGROUND_LLM_API_KEY` are secrets. The rest belong in a `doormile-config`
+ConfigMap so a URL change is not a secret edit.
+
+**Files**
+| File | Change |
+|---|---|
+| `kubernetes/manifests/doormile/miletruth.yaml` | modify — add the 19 env vars to the StatefulSet + `doormile-secrets` |
+| `kubernetes/manifests/doormile/doormile-config.yaml` | **new** — ConfigMap for the non-secret vars |
+| `AI_engine/.env` (server-local, untracked content) | modify — same `INTERNAL_API_KEY` value |
+
+Read-only, no change: `doormile_backend/middlewares/internal_auth.go:13`,
+`doormile_backend/config/config.go:60-102` (the contract being satisfied).
+
+> `INTERNAL_API_KEY` must be **byte-identical** in the backend Secret and the
+> engine's env. Generate once (`openssl rand -hex 32`), set both.
+
+**Acceptance:** from inside the cluster,
+`curl -H "X-Internal-Key: $KEY" http://doormile-service.doormile:8081/api/v1/internal/ai/registry`
+returns 200 with a body and an `ETag`; a second call with `If-None-Match` returns
+304. Backend logs show `ai playground: enabled`. `GET /admin/ai/status` reports
+the engine as following the registry (`aiRegistryController.go:208` counts the
+poll).
+
+### A2. Plaintext secrets are committed
+
+`miletruth.yaml:19-21` has `DB_PASSWORD`, `REDIS_PASSWORD`, `NATS_PASSWORD` as
+literal in git (the same password reused for all three). Same class as the committed Firebase keys from the
+CX audit. Pick one and apply it before A1 adds *more* secrets to that file:
+Sealed Secrets, SOPS, or an out-of-git `kubectl create secret` with the manifest
+holding only a reference.
+
+Do **not** delete or untrack anything here without per-file approval — see the
+standing rule. Rotating that shared password is a separate decision; this item is only
+about stopping new secrets entering git.
+
+**Files**
+| File | Change |
+|---|---|
+| `kubernetes/manifests/doormile/miletruth.yaml` | modify — `stringData` block becomes a reference |
+| `kubernetes/manifests/doormile/sealed-secrets.yaml` | **new** — if Sealed Secrets is the chosen route |
+| `kubernetes/docs/DEPLOY.md` | modify — document how secrets are supplied now |
+
+Same pattern already in git at `kubernetes/manifests/core/core-secrets.yaml` and
+`manifests/nearle/nearle-secrets.yaml` — whatever is chosen should cover those too,
+but that is outside this phase.
+
+### A3. `anthropic>=0.21.0` floor is wrong
+
+`AI_engine/requirements.txt` pins `anthropic>=0.21.0`, but `core/llm.py:117`
+sends `thinking: {"type": "adaptive"}` and `output_config.effort` — parameters
+that old SDK does not know. There is no lockfile, so a fresh `pip install` in a
+rebuilt image can resolve to something that 400s on every LLM call.
+
+Bump to a current floor and pin the image build. `openai>=1.0.0` is already
+present (unused today — Track B uses it).
+
+**Files**
+| File | Change |
+|---|---|
+| `AI_engine/requirements.txt` | modify — raise the `anthropic` floor |
+| `AI_engine/Dockerfile` | modify — optional: `pip install` from a lockfile instead |
+| `AI_engine/requirements-dev.txt` | check — may pin the same packages |
+
+### A4. `LLM_MODEL` default is a generation behind
+
+`core/llm.py:22` defaults to `claude-opus-4-8` ($5/$25 per MTok). `claude-opus-5-5`
+is both cheaper ($4/$20) and more capable. One-line change; the registry's
+per-agent `model` pin already overrides it (`registry.model(agent_id)`), so this
+only moves the floor.
+
+**Files**
+| File | Change |
+|---|---|
+| `AI_engine/core/llm.py:22` | modify — default `LLM_MODEL` |
+| `AI_engine/core/llm.py:11` | modify — the docstring naming the default |
+| `AI_engine/docker-compose.yml:45` | modify — `LLM_MODEL` fallback |
+| `AI_engine/tests/test_llm.py` | check — may assert the old default |
+
+Note `core/llm.py:109` special-cases Haiku 4.5 (rejects adaptive thinking and
+`effort`). Leave that branch alone; it is correct.
+
+---
+
+## Track B — RAG decision memory
+
+The skeleton exists and is wired to nothing. Four pieces are already built:
+
+| Piece | Where |
+|---|---|
+| `context_embedding vector(1536)` on `agent_decisions` | `migrations/migrate.go:92` |
+| ivfflat cosine index, `lists = 100` | `migrations/migrate.go:98` |
+| `POST /internal/agent-decisions` accepts `context_embedding` | `controllers/agentDecisionController.go:23` |
+| cosine kNN query + `PATCH /:id/outcome` | `agentDecisionController.go:74,120` |
+
+Nothing produces an embedding, nothing calls `/similar`, nothing records an
+outcome. There are exactly **two** decision types to cover —
+`assignment_failure` (`dispatch_agent.py:335`) and `stall_response`
+(`exception_agent.py:456`).
+
+### B0. Three defects to fix before writing any new code
+
+**B0.1 — `CREATE EXTENSION vector` appears nowhere in the repo.**
+`migrate.go:92` runs `ALTER TABLE agent_decisions ADD COLUMN … vector(1536)` and
+logs failure *non-fatally*. If the extension is not installed on `logistics`,
+both the column and the index silently fail and every retrieval 500s.
+
+```sql
+SELECT extname, extversion FROM pg_extension WHERE extname = 'vector';
+```
+
+If absent, add **before** line 92 in `Migrate()`:
+
+```go
+if res := db.Exec(`CREATE EXTENSION IF NOT EXISTS vector`); res.Error != nil {
+    utils.Error("❌ pgvector extension unavailable — decision memory disabled", "error", res.Error)
+}
+```
+
+Needs the `pgvector` extension available on the server and a role with rights to
+create it. On a managed Postgres this may be an admin action, not a migration —
+check before assuming.
+
+**B0.2 — `/similar` is a `GET` that requires a JSON body.**
+`routes.go:594` registers it as `GET`, and `agentDecisionController.go:84` calls
+`BodyParser` demanding an `embedding` array. nginx and most HTTP clients drop GET
+bodies — and doormile is served *through* host nginx (see C4). Change to:
+
+```go
+internal.Post("/agent-decisions/similar", controllers.FindSimilarDecisions)
+```
+
+No caller exists yet, so this breaks nothing.
+
+**B0.3 — `/similar` filters `WHERE outcome IS NOT NULL`, and nothing writes
+outcomes.** Even with embeddings flowing it returns zero rows forever. **B2 is
+not optional** — it is the half that makes retrieval worth anything.
+
+**Files**
+| File | Change |
+|---|---|
+| `doormile_backend/migrations/migrate.go:92` | modify — add `CREATE EXTENSION` above the `ALTER TABLE` |
+| `doormile_backend/routes/routes.go:594` | modify — `internal.Get` → `internal.Post` for `/agent-decisions/similar` |
+| `doormile_backend/controllers/agentDecisionController.go:74` | modify — comment the method change; body parsing already correct |
+| `doormile_backend/routes/routes_ai_registry_pg_test.go` | modify — add coverage for the POST shape |
+
+### B1. Embedding provider — decision required
+
+Anthropic has no embeddings endpoint, so this needs a second provider.
+
+| Option | Dims | Column change | Notes |
+|---|---|---|---|
+| **OpenAI `text-embedding-3-small`** | 1536 | **none** | ~$0.02/MTok. `openai>=1.0.0` already a dependency. Column was sized for it. |
+| Voyage `voyage-3` | 1024 | yes | Anthropic-recommended; new key, new vendor. |
+| Local `bge-small` / `MiniLM` | 384 | yes | Free, no egress, no key. +~400MB RAM, model in image, slower cold start. |
+
+**Recommendation: OpenAI `text-embedding-3-small`.** The schema already matches,
+the dependency is already there, and the second-provider line is already crossed
+(the playground runs on Groq). Revisit if data egress is a constraint — then take
+the local model and migrate the column to `vector(384)`.
+
+### B2. Write path — `AI_engine/core/embeddings.py`
+
+One function, modelled on `core/decisions.py`'s fire-and-forget discipline:
+
+```python
+async def embed(text: str) -> Optional[List[float]]:
+    """None on any failure. An embedding must never delay an agent's reaction."""
+```
+
+Requirements:
+
+- **Embed the same dict `build_payload` already stores.** Serialise the `facts`
+  dict deterministically (sorted keys) so the embedded text and the stored
+  `context` cannot drift apart. Add a `_context_text(facts)` helper and test it
+  pure, the way `build_payload` is tested.
+- Fail open — return `None`, log once, never raise into the agent path.
+- Small LRU cache: repeated stalls on one booking produce near-identical facts.
+- Gate on `EMBEDDINGS_ENABLED` **and** a registry skill flag, so it can be
+  switched off from Agent Studio without a redeploy
+  (`registry.skill_enabled(...)` already exists).
+
+Then extend `build_payload`/`record_decision` (`core/decisions.py:33,53`) to carry
+`context_embedding`. Both call sites (`dispatch_agent.py:335`,
+`exception_agent.py:456`) keep their signatures.
+
+**Acceptance:** after one stall,
+`SELECT count(*) FROM agent_decisions WHERE context_embedding IS NOT NULL` > 0.
+
+**Files**
+| File | Change |
+|---|---|
+| `AI_engine/core/embeddings.py` | **new** — `embed()`, `_context_text()`, LRU cache, fail-open |
+| `AI_engine/core/decisions.py:33` | modify — `build_payload` carries `context_embedding` |
+| `AI_engine/core/decisions.py:53` | modify — `record_decision` awaits the embed before posting |
+| `AI_engine/config/system_config.py` | modify — `EMBEDDINGS_ENABLED`, provider key, model name |
+| `AI_engine/docker-compose.yml` | modify — pass the embedding env through |
+| `AI_engine/tests/test_embeddings.py` | **new** — `_context_text` determinism, fail-open returns `None` |
+| `AI_engine/tests/test_registry_phase5.py:194` | modify — asserts the `record_decision` payload shape |
+
+Not touched: `agents/dispatch_agent.py:335` and `agents/exception_agent.py:456`
+keep their call signatures — the embedding is added inside `decisions.py`, so
+neither agent changes.
+
+### B3. Outcome loop — the part that makes retrieval useful
+
+`PATCH /internal/agent-decisions/:id/outcome` exists with no caller. Define, per
+decision type, what "it worked" means. Starting proposal:
+
+| Type | Outcome = `success` when | `failure` when |
+|---|---|---|
+| `stall_response` | booking reaches `Delivered` within its SLA window after the decision | SLA breached, or cancelled |
+| `assignment_failure` | booking gets an assignment within N minutes of the decision | still unassigned after N, or cancelled |
+
+Implement as a backend sweeper following the **established pattern** in
+`internal/assignment/sweeper.go:88` — ticker, `recover()` per tick, and the Redis
+lock that keeps one replica sweeping (`sweeper.go:104`). Wire it in `main.go`
+beside `go assignment.StartPendingSweeper()` (`main.go:244`).
+
+Two gotchas from the existing code:
+- Use `Receivedat`-style true instants, not `utils.DBNow` — `models/ai_runs.go:26`
+  documents that `DBNow` returns IST digits labelled UTC and is 5h30m off for
+  `timestamptz`. The same trap applies to `outcome_recorded_at`.
+- Leave `outcome` NULL while undecided. `/similar` already treats NULL as
+  "no evidence yet", which is correct.
+
+**Acceptance:** rows acquire non-NULL `outcome` within one sweep interval of
+their window closing, and `/insights` decision-outcome counts stop being all
+`pending`.
+
+**Files**
+| File | Change |
+|---|---|
+| `doormile_backend/internal/ai/outcomes/sweeper.go` | **new** — ticker + Redis lock, modelled on `internal/assignment/sweeper.go:88` |
+| `doormile_backend/internal/ai/outcomes/rules.go` | **new** — the per-decision-type success/failure predicates |
+| `doormile_backend/internal/ai/outcomes/sweeper_test.go` | **new** — interval, window, and both predicates |
+| `doormile_backend/main.go:244` | modify — `go outcomes.StartOutcomeSweeper()` beside the pending sweeper |
+| `doormile_backend/models/agentdecision.go` | modify — only if B6 adds `Tenantid` |
+
+Reference, not modified: `internal/assignment/sweeper.go:88-110` (the ticker +
+`recover()` + Redis single-replica lock pattern to copy),
+`models/ai_runs.go:26` (the `utils.DBNow` timezone trap to avoid).
+
+### B4. Read path — precedent in the prompt
+
+Before the LLM call in `exception_agent` / `dispatch_agent`, fetch the top-5
+similar **resolved** decisions and include them as precedent — "the last 5
+comparable situations and whether the action worked."
+
+- Feature-flag it on a registry skill so it is switchable from Agent Studio.
+- Hard timeout (~300ms) with fail-open to today's prompt. The current behaviour
+  is the floor; this can only raise it. (`core/registry.py` already follows this
+  rule; `ragRouter.js:9-14` documents the same discipline on the console side.)
+- Keep precedent **out** of the structured-output schema. It informs the prompt;
+  it must not become a field the model can invent.
+
+**Acceptance:** an eval run shows the decision quality moving. `AI_engine/evals/`
+already has the harness and cases (`stall_cases.jsonl`,
+`assignment_cases.jsonl`) — extend those rather than judging by eye.
+
+**Files**
+| File | Change |
+|---|---|
+| `AI_engine/core/memory.py` | **new** — `recall(decision_type, embedding, k)` → POST `/internal/agent-decisions/similar`, timeout + fail-open |
+| `AI_engine/core/llm.py:157` | modify — `decide_stall_response` accepts optional precedent |
+| `AI_engine/core/llm.py:227` | modify — `decide_assignment_failure` accepts optional precedent |
+| `AI_engine/agents/exception_agent.py:456` | modify — recall before the decide call |
+| `AI_engine/agents/dispatch_agent.py:335` | modify — recall before the decide call |
+| `AI_engine/tests/test_memory.py` | **new** — timeout fails open, empty recall changes nothing |
+| `AI_engine/evals/stall_eval.py` | modify — run with and without precedent |
+| `AI_engine/evals/assignment_eval.py` | modify — same |
+| `AI_engine/evals/stall_cases.jsonl` | modify — cases where precedent should change the answer |
+| `AI_engine/evals/assignment_cases.jsonl` | modify — same |
+| `doormile_backend/internal/ai/registry/seed.go:114` | modify — seed a `recall_similar_decisions` read tool + the skill flag that gates B4 |
+
+The registry seed row matters: without it the feature cannot be switched off from
+Agent Studio, which is the whole point of gating it on `registry.skill_enabled`.
+
+### B5. Hygiene
+
+- **`agent_decisions` has no retention.** `aiagentruns` purges at 30 days
+  (`telemetry/recorder.go:28,128`); decisions grow forever, and this is the table
+  retrieval scans. Decide a window — longer than 30 days, since old precedent is
+  the point. Suggest 180 days, or keep resolved rows and purge unresolved ones.
+- **Re-tune `lists = 100`.** That is right for roughly 100k–1M rows. Below ~10k
+  it over-partitions and recall drops. Check `count(*)` once embeddings flow;
+  consider HNSW instead if the pgvector version supports it.
+
+**Files**
+| File | Change |
+|---|---|
+| `doormile_backend/internal/ai/outcomes/sweeper.go` | modify — fold the decision purge into the same tick |
+| `doormile_backend/migrations/migrate.go:98` | modify — index tuning, once row count is known |
+
+### B6. Tenant isolation — decide before B4 ships, not after
+
+`agent_decisions` has **no tenant column**. If retrieved precedent crosses
+tenants, one client's operational history shapes decisions made for another.
+Given that console logins are already unscoped on `tenantid` NULL
+(`doormile-console-logins-unscoped`), this needs deciding up front.
+
+Recommendation: add `tenantid` to `agent_decisions`, have the engine populate it,
+and filter in the `/similar` query. Cheap now, expensive after the table fills.
+
+**Files**
+| File | Change |
+|---|---|
+| `doormile_backend/models/agentdecision.go` | modify — add `Tenantid *uint64` with an index |
+| `doormile_backend/controllers/agentDecisionController.go:18` | modify — accept `tenant_id` on create |
+| `doormile_backend/controllers/agentDecisionController.go:104` | modify — add `AND tenantid = ?` to the kNN query |
+| `AI_engine/core/decisions.py:33` | modify — `build_payload` carries the tenant |
+| `AI_engine/agents/exception_agent.py` · `dispatch_agent.py` | modify — source the tenant from the booking facts |
+| `doormile_backend/routes/routes_ai_registry_pg_test.go` | modify — a cross-tenant recall must return nothing |
+
+Nullable, because the engine will not always know the tenant. Decide whether a
+NULL tenant row is recallable by everyone or by no one — given
+`doormile-console-logins-unscoped`, **by no one** is the safer default.
+
+---
+
+## Track C — Kubernetes
+
+### C1. `AI_engine` is not in Kubernetes at all
+
+No manifest, no kustomization, no deploy script, and `docker-compose.yml` uses
+`build: .` with no registry push. It runs on a VM by compose.
+
+**Prerequisite: the engine has no HTTP server in production mode.**
+`main.py --production` starts no listener, so there is no liveness/readiness
+target and no `/metrics`. Without it, Kubernetes can only restart on process
+exit — a NATS-disconnected engine looks healthy forever.
+
+`fastapi` and `uvicorn` are already in `requirements.txt`. Add a small surface in
+`production_mode()`:
+
+- `GET /healthz` — process alive (event loop responsive)
+- `GET /readyz` — NATS connected **and** `registry.loaded` is true
+- `GET /metrics` — optional; decisions recorded, tool calls, LLM failures
+
+Readiness must include `registry.loaded`, otherwise a pod that cannot reach the
+backend serves traffic on env defaults while reporting healthy — exactly the A1
+failure mode, invisible again.
+
+Then: push the image to a registry (compose builds locally), and write
+`manifests/doormile/ai-engine.yaml` as a `Deployment` (it is stateless;
+`replicas: 1` to start — the agents are not yet idempotent across replicas, see
+C6).
+
+**Files**
+| File | Change |
+|---|---|
+| `AI_engine/core/health.py` | **new** — the aiohttp/FastAPI surface (`/healthz`, `/readyz`, `/metrics`) |
+| `AI_engine/main.py:130` | modify — start the health server inside `production_mode()` beside `registry.run(...)` |
+| `AI_engine/main.py` (`print_help`) | modify — document the health port |
+| `AI_engine/Dockerfile` | modify — `EXPOSE` the health port |
+| `AI_engine/docker-compose.yml` | modify — publish the port so compose and k8s behave alike |
+| `AI_engine/tests/test_health.py` | **new** — `/readyz` is red while `registry.loaded` is false |
+| `kubernetes/manifests/doormile/ai-engine.yaml` | **new** — Deployment + Service + probes + resources |
+
+`fastapi` and `uvicorn` are already in `requirements.txt` — no new dependency.
+Readiness must check `registry.loaded` (`core/registry.py:45`), not just the
+process, or A1's failure mode becomes invisible again.
+
+### C2. `doormile/` has no `kustomization.yaml`
+
+`alaska/`, `core/` and `nearle/` all have one. `doormile/` does not, and there is
+no `deploy-doormile.sh` alongside `deploy-core-stack.sh` / `deploy-nearle-stack.sh`.
+That is *why* it drifts. Add both.
+
+**Files**
+| File | Change |
+|---|---|
+| `kubernetes/manifests/doormile/kustomization.yaml` | **new** — list miletruth, config, ai-engine, pdb |
+| `kubernetes/deploy-doormile.sh` | **new** — copy the shape of `deploy-nearle-stack.sh` |
+| `kubernetes/scripts/sync_manifests.py` | check — may need the new namespace registering |
+| `kubernetes/docs/DEPLOY_CHECKLIST.md` | modify — add the doormile stack |
+
+Pattern to copy: `manifests/nearle/kustomization.yaml` + `deploy-nearle-stack.sh`.
+
+### C3. The `doormile` StatefulSet has no probes, resources or PDB
+
+`replicas: 3` with **no resource requests** means the scheduler can stack all
+three on one node, and **no readinessProbe** means a pod receives traffic before
+Postgres/Redis/NATS are connected. `core/` has `worker-pdb.yaml`; doormile has
+nothing.
+
+Add `resources.requests`/`limits`, a readinessProbe and livenessProbe against the
+backend's health route, and a PodDisruptionBudget (`minAvailable: 2`).
+
+Also: a `StatefulSet` for a stateless Go API is the wrong kind — it gives serial
+rollouts and no benefit. Switching to `Deployment` is low-risk and makes deploys
+faster. Not urgent; flagging because it is why rollouts feel slow.
+
+**Files**
+| File | Change |
+|---|---|
+| `kubernetes/manifests/doormile/miletruth.yaml:23` | modify — `resources`, `readinessProbe`, `livenessProbe` |
+| `kubernetes/manifests/doormile/doormile-pdb.yaml` | **new** — `minAvailable: 2`, copy `manifests/core/worker-pdb.yaml` |
+
+**No backend change needed** — the probe targets already exist and are correct:
+`GET /api/v1/health` (`routes/routes.go:46`, unauthenticated, always 200) for
+liveness, and `GET /api/v1/ready` (`routes/routes.go:50`) for readiness, which
+already returns **503** when Postgres or Redis is unreachable
+(`routes.go:68-70`). Point the probes at those; do not write new ones.
+
+Note `/ready` reports Redis GEO status without gating on it — deliberate, per the
+comment at `routes.go:73`. A readinessProbe on `/ready` therefore will not pull a
+pod out of service for a broken rider search, which is the intended behaviour.
+
+### C4. No ingress for `doormile`
+
+`nearle` and `alaska` are on `manifests/core/ingress-unified.yaml`. `doormile` is
+NodePort 30830 plus host nginx (`conf/nginx-doormile.conf`) — half-migrated. This
+also makes B0.2 (GET-with-body) a certainty rather than a risk.
+
+**Files**
+| File | Change |
+|---|---|
+| `kubernetes/manifests/core/ingress-unified.yaml` | modify — add a `doormile` rule (needs a ReferenceGrant if it stays cross-namespace, cf. `manifests/nearle/nearle-reference-grant.yaml`) |
+| `kubernetes/manifests/doormile/miletruth.yaml:91` | modify — `NodePort` → `ClusterIP` once the ingress serves it |
+| `kubernetes/conf/nginx-doormile.conf` | modify — retire or repoint, **only after** the ingress is verified |
+
+Do these in that order. Flipping the Service type before the ingress works takes
+the API offline.
+
+### C5. NATS is outside the cluster on two ports
+
+Backend uses `nats://66.116.226.161:4223`; `core-config.yaml:10` uses `:4222`.
+Worth a line in `docs/ARCHITECTURE.md` on which port is which and why, before the
+engine joins and needs to pick one.
+
+### C6. Decide replica safety before scaling the engine
+
+The telemetry recorder is already replica-safe (NATS queue group +
+`uq_aiagentruns_agent_task`). The **agents** are not obviously so: `exception_agent`
+has an in-process TTL dedup set (`exception_agent.py:339`), which is per-pod. Two
+engine replicas would each decide on the same stall. Keep `replicas: 1` until
+dedup moves to Redis.
+
+**Files** (only if scaling past 1 replica)
+| File | Change |
+|---|---|
+| `AI_engine/agents/exception_agent.py:339` | modify — TTL set → Redis `SET NX EX` |
+| `AI_engine/tests/test_stall_dedup.py` | modify — covers the current in-process behaviour |
+| `kubernetes/manifests/doormile/ai-engine.yaml` | modify — raise `replicas` |
+
+C5 (the NATS port note) is documentation only: `kubernetes/docs/ARCHITECTURE.md`.
+
+---
+
+## Track D — Executor backlog
+
+9 of 22 seeded tools are marked `REVIEW ONLY … No executor` in
+`internal/ai/registry/seed.go`. The registry is honest about it and the console
+renders them disabled. This is the feature list, in value order.
+
+### D1. `assign_riders` — needs an admin auto-assign route (not just auth)
+
+**Correcting an earlier assumption:** this is *not* a free auth fix.
+`actions.js:96-107` already explains why — `POST /admin/bookings/:id/assign-miler`
+(`routes.go:413` → `adminController.go:2965`) requires a **chosen rider per
+booking** (`{mileruserid}`), and a finding does not pick one. The hub route that
+*does* pick (`POST /hub/bookings/:id/auto-assign`, `routes.go:520`) is behind
+`HubStaffAuth` and 403s for every console login.
+
+**The cleanest route is an admin batch-assign**, better than the per-booking
+auto-assign first considered. `HubBatchAssign` (`controllers/hubController.go:1963`)
+already does exactly what a finding needs: it takes `bookingids[]`, picks riders
+via Redis GEO + scoring, and commits server-side in one call. Its only hub-specific
+parts — `c.Locals("hubid")` and `hubPincodePrefix(hubID)` (`hubController.go:1964-1966`)
+— are used **solely as a fallback when `bookingids` is empty** (`hubController.go:1981-1985`).
+The console always passes explicit ids, so that branch never runs.
+
+So: extract the body into a shared helper taking `(bookingIDs, capPerRider, actorID, scopeFn)`
+and have both the hub route and a new admin route call it. `scopeBookingsToOwnTenant`
+(`hubController.go:1986`) already works for admin logins.
+
+The console side is then nearly free — `batchAssignBookings` already exists at
+`src/api/doormile/endpoints.js:554` and already sends `{bookingids, max_per_rider}`.
+It just points at the hub URL that 403s. One URL change.
+
+Option (b), having the skill pick a rider via `nearby_milers` and call the
+existing `assign-miler`, is worse: more console work and it puts solver logic in
+the browser.
+
+**Files**
+| File | Change |
+|---|---|
+| `doormile_backend/controllers/hubController.go:1963` | modify — extract the shared assign helper out of `HubBatchAssign` |
+| `doormile_backend/controllers/adminController.go` | **add** `AdminBatchAssign` calling that helper |
+| `doormile_backend/routes/routes.go:413` | modify — register `adminAuth.Post("/bookings/batch-assign", …)` |
+| `krow_talent_app/src/api/doormile/endpoints.js:554` | modify — `/hub/bookings/batch-assign` → `/admin/bookings/batch-assign` |
+| `krow_talent_app/src/lib/assistant/agent/actions.js:122` | modify — add the `assignMiler` executor |
+| `krow_talent_app/src/lib/assistant/agent/actions.js:96-107` | modify — delete the "deliberately NOT an executor" note |
+| `krow_talent_app/tests/lib/agentActions.test.js` | modify — asserts the current executor set |
+| `doormile_backend/internal/ai/registry/seed.go` | modify — `assign_riders` description stops saying REVIEW ONLY |
+
+Check before starting: `endpoints.js:547-553` warns that Doormile-native batch
+assign commits with no preview/reconcile step and leaves multi-stop riders
+unsequenced. An agent-proposed assignment firing straight to commit is a
+behaviour decision, not just a wiring one — confirm that is wanted.
+
+This takes the console from 1 working verb to 2 and makes the highest-severity
+SLA finding actionable.
+
+### D2. `alert_low_battery_rider` — nearly free
+
+`seed.go` already targets `POST /admin/milers/:id/notify`, which is the endpoint
+`notify_riders` already uses successfully. This is a message-text change and an
+`EXECUTORS` entry, not a new capability.
+
+**Files**
+| File | Change |
+|---|---|
+| `krow_talent_app/src/lib/assistant/agent/actions.js:122` | modify — add the `alertLowBatteryRider` executor |
+| `krow_talent_app/src/lib/assistant/skills/definitions/RiderBatterySafetySkill.js` | check — confirm the proposal carries `milerId` |
+| `krow_talent_app/tests/lib/agentActions.test.js` | modify — same assertion as D1 |
+| `doormile_backend/internal/ai/registry/seed.go` | modify — drop REVIEW ONLY from the description |
+
+No backend change. `notifyMiler` already exists at
+`src/api/doormile/endpoints.js:389` → `POST /admin/milers/:id/notify`, which is
+the endpoint `seed.go` already names as the target.
+
+### D3. The rest need endpoints that do not exist
+
+`enforce_otp_verification`, `dispatch_hub_idle_parcels`, `trigger_auto_dispatch`,
+`enforce_cash_handoff`, `rebalance_riders` — all marked `Target: "none yet"`.
+Each is a product decision first. Not in this phase.
+
+---
+
+## Two more loose ends
+
+- **`src/lib/agentNetwork.js` is a hand-maintained snapshot** dated 16–20 Sep, and
+  the file says so honestly. Once C1 gives the engine an HTTP surface, add
+  `GET /agents/status` and swap the source. The file is deliberately shaped like
+  that response, so it is a change of source, not a rewrite.
+- **`src/lib/assistant/ragRouter.js` points at a `services/ai` sidecar that does
+  not exist in any repo.** `VITE_AI_URL` appears nowhere, so `isRagEnabled()` is
+  permanently false and the module is dead code. **Do not conflate this with
+  Track B** — ragRouter is semantic *intent routing* for the console assistant,
+  not decision memory. Lower value. Leave it dormant (it is correctly fail-open)
+  or decide to build the sidecar as its own piece of work.
+
+---
+
+## Consolidated file manifest
+
+**14 new files, 46 modified, across 4 repos.** Per-item detail is in the tracks above.
+
+### `doormile_backend` — 4 new, 17 modified
+
+| File | New? | Items |
+|---|---|---|
+| `internal/ai/outcomes/sweeper.go` | **new** | B3, B5 |
+| `internal/ai/outcomes/rules.go` | **new** | B3 |
+| `internal/ai/outcomes/sweeper_test.go` | **new** | B3 |
+| `migrations/migrate.go` | | B0.1 (:92), B5 (:98) |
+| `routes/routes.go` | | B0.2 (:594), D1 (:413) |
+| `controllers/agentDecisionController.go` | | B0.2 (:74), B6 (:18, :104) |
+| `controllers/hubController.go` | | D1 (:1963 — extract helper) |
+| `controllers/adminController.go` | | D1 (**add** `AdminBatchAssign`) |
+| `models/agentdecision.go` | | B6 |
+| `internal/ai/registry/seed.go` | | B4 (:114), D1, D2 |
+| `main.go` | | B3 (:244) |
+| `routes/routes_ai_registry_pg_test.go` | | B0.2, B6 |
+
+### `AI_engine` — 5 new, 20 modified
+
+| File | New? | Items |
+|---|---|---|
+| `core/embeddings.py` | **new** | B2 |
+| `core/memory.py` | **new** | B4 |
+| `core/health.py` | **new** | C1 |
+| `tests/test_embeddings.py` | **new** | B2 |
+| `tests/test_memory.py` · `tests/test_health.py` | **new** | B2, C1 |
+| `core/decisions.py` | | B2 (:33, :53), B6 |
+| `core/llm.py` | | A4 (:11, :22), B4 (:157, :227) |
+| `core/registry.py` | | — read-only (`loaded` consumed by C1) |
+| `agents/exception_agent.py` | | B4 (:456), B6, C6 (:339) |
+| `agents/dispatch_agent.py` | | B4 (:335), B6 |
+| `main.py` | | C1 (:130, `print_help`) |
+| `config/system_config.py` | | B2 |
+| `requirements.txt` · `Dockerfile` · `docker-compose.yml` | | A3, A4, C1 |
+| `evals/stall_eval.py` · `assignment_eval.py` · both `.jsonl` | | B4 |
+| `tests/test_registry_phase5.py` · `test_llm.py` · `test_stall_dedup.py` | | B2, A4, C6 |
+
+### `kubernetes` — 5 new, 7 modified
+
+| File | New? | Items |
+|---|---|---|
+| `manifests/doormile/doormile-config.yaml` | **new** | A1 |
+| `manifests/doormile/ai-engine.yaml` | **new** | C1, C6 |
+| `manifests/doormile/kustomization.yaml` | **new** | C2 |
+| `manifests/doormile/doormile-pdb.yaml` | **new** | C3 |
+| `deploy-doormile.sh` | **new** | C2 |
+| `manifests/doormile/miletruth.yaml` | | A1, A2, C3 (:23), C4 (:91) |
+| `manifests/core/ingress-unified.yaml` | | C4 |
+| `conf/nginx-doormile.conf` | | C4 (last) |
+| `scripts/sync_manifests.py` | | C2 |
+| `docs/DEPLOY.md` · `DEPLOY_CHECKLIST.md` · `ARCHITECTURE.md` | | A2, C2, C5 |
+
+### `krow_talent_app` — 0 new, 4 modified
+
+The console barely changes. Everything it needs already exists.
+
+| File | Items |
+|---|---|
+| `src/api/doormile/endpoints.js` | D1 (:554 — one URL) |
+| `src/lib/assistant/agent/actions.js` | D1 (:96-107, :122), D2 (:122) |
+| `tests/lib/agentActions.test.js` | D1, D2 |
+| `src/lib/assistant/skills/definitions/RiderBatterySafetySkill.js` | D2 — check only |
+
+Untouched on purpose: `src/lib/agentNetwork.js` (until C1 ships an endpoint) and
+`src/lib/assistant/ragRouter.js` (dormant, out of scope — see loose ends).
+
+### Files deliberately not touched
+
+- `src/lib` / `src/utils` duplicate trees — both have live importers, do not consolidate.
+- `AI_engine/core/tool_registry.py` — the 2-vs-22 tool gap is a separate decision, not this phase.
+- `AI_engine/customer_portal/`, `dashboard/` — not on any path this phase touches.
+
+## Decisions needed before coding
+
+1. **Embedding provider** — OpenAI 1536 (no column change), Voyage 1024, or local
+   384? (B1)
+2. **Is `pgvector` installed on `logistics`?** If creating extensions needs an
+   admin, that is a prerequisite, not a migration. (B0.1)
+3. **Outcome definitions** — are the two in B3 right, and what is N for
+   `assignment_failure`?
+4. **`tenantid` on `agent_decisions`** — add it now, or accept cross-tenant
+   precedent? (B6)
+5. **`assign_riders`** — route (a) admin auto-assign, or (b) console picks the
+   rider? (D1)
+6. **Decision retention window** — 180 days, or keep-resolved-purge-unresolved? (B5)
+
+## Sequencing
+
+```
+A1 ──> A3, A4 ──┬──> B0 ──> B2 ──> B3 ──> B4 ──> B5, B6
+                │
+                └──> C1(health) ──> C1(manifest) ──> C2 ──> C3 ──> C4
+A2 (independent, before A1 adds more secrets to git)
+D1, D2 (independent of everything above)
+```
+
+**A1 first and alone.** Until the internal key is set, the engine is not talking
+to the backend, so every Track B acceptance check would fail for the wrong
+reason. Run the `kubectl` check in A1 before writing any code — if git does not
+match the cluster, that changes the shape of Track C.
+
+Cheapest real step up a level, once A1 is in: **D1 + D2.** Two executors, no new
+LLM work, and it moves autonomy off the floor.
+
+---
+
+## Standing constraints
+
+- Nothing in this plan is committed, pushed, or deployed without being asked.
+- No migrations run against a real database without being asked.
+- No files deleted or untracked without per-file approval.
+- `src/lib` and `src/utils` in the console **both** have live importers — neither
+  tree is dead, do not consolidate them as part of this work.