# Deploying `9d3192a` to krow-2 The API on krow-2 is **currently down**, and stayed down on purpose. It refuses to start with: ``` ERROR fatal error="ANTHROPIC_API_KEY is set but is no longer read, and MODEL_API_KEY is empty: the Anthropic path was removed..." ``` That is a startup guard doing its job, not a crash. `34fa58a` removed the Anthropic path, and the deployment environment still describes the old one. The fix is two environment variables; everything else here is the rollout around it. **Prepared and verified locally. Not executed** — this machine has no working SSH to krow-2, and a production rollout is not something to do without the operator watching. --- ## 1. What is being deployed `9d3192a`, which is `HEAD` and already `origin/main`. Nothing needs pushing. Four commits since the last deploy point: | Commit | What | | --- | --- | | `34fa58a` | Anthropic path removed; the gateway speaks one wire protocol | | `bd9a8f9` | The shipped example envs can actually start (see §5) | | `7d83c16` | Compose comments: a container has no keyless option | | `9d3192a` | Model ids Groq actually serves; I7 measured | **No migrations.** `migrations/` is unchanged since `34fa58a`, so `migrate` will report nothing to apply and exit 0. This is a configuration rollout. ## 2. What was verified before writing this The whole path, locally, in the production image against a TLS Postgres and the real Groq API: | Step | Result | | --- | --- | | `docker build -f infrastructure/Dockerfile.api` | builds | | 11 migrations against Postgres 18 over TLS | all apply | | Boot with `APP_ENV=production`, `sslmode=require` | connects, **62 endpoints** | | `seed` + `importagents --org=krow-dev` | 245 records, 9 agents, 24 skills | | `POST /api/v1/auth/login` | session cookie issued | | `POST /api/v1/agents/{id}/runs` → Groq | `Completed`, 2 model calls, 1707 tokens | | Same with `Accept: text/event-stream` | `data: {"delta":"…"}` streams | | `make eval-live` ×2 | all three cases pass, I7 included | 62 endpoints is the number to expect. Fewer by two means the agent routes did not register, which means the credential is missing — see §6. ## 3. The environment change On krow-2, in the `.env` that `docker compose` reads (beside `docker-compose.yml`): ```bash MODEL_PROVIDER=openai MODEL_BASE_URL=https://api.groq.com/openai/v1 MODEL_API_KEY= MODEL_FAST=openai/gpt-oss-20b MODEL_BALANCED=openai/gpt-oss-120b MODEL_DEEP=openai/gpt-oss-120b ``` And **delete the `ANTHROPIC_API_KEY` line entirely.** Commenting it out is enough; leaving it set with an empty `MODEL_API_KEY` reproduces the failure. Two things that look like details and are not: - **Replace the value, do not just rename the variable.** An `sk-ant-…` key under the name `MODEL_API_KEY` passes every startup check — the process cannot tell one opaque string from another — and then fails every run with 401. Startup validation catches the shape of a stale configuration, never a wrong secret. - **The model ids are not interchangeable.** Groq no longer serves the `llama-3.1-8b-instant` / `llama-3.3-70b-versatile` pair that shipped in `34fa58a`; both were wrong the day they shipped and would have 400'd on every run. The ids above were checked against the live account. While in the file, confirm `HTTP_WRITE_TIMEOUT` is above `2m` — `180s` is the new default. Below that the container refuses to start. krow-2 only got past this before because someone had already overridden it. ## 4. Rollout ```bash cd git fetch origin && git checkout main && git pull --ff-only origin main git log --oneline -1 # expect 9d3192a cd infrastructure # edit .env per §3, then: docker compose up -d --build ``` `migrate` runs to completion before `api` starts and `api` will not start if it fails. Expect `migrate` to exit 0 having applied nothing. ## 5. Smoke test ```bash docker compose ps # api: Up (healthy) docker compose logs api | tail -20 # no fatal; "listening" with endpoints=62 curl -s http://127.0.0.1:8080/health # {"status":"ok"} ``` `"status":"degraded"` means the schema is missing or dirty, not a model problem. Then one real agent run, which is the only step that proves the model path — the earlier outage was invisible until run time: ```bash # from the host, against the published port curl -s -c /tmp/k.jar -X POST http://127.0.0.1:8080/api/v1/auth/login \ -H 'Content-Type: application/json' \ -d '{"email":"","password":""}' >/dev/null AGENT=$(docker compose exec -T api sh -c 'true' >/dev/null 2>&1; \ psql "$DATABASE_URL" -tAc \ "select id from agent_definitions where definition_id='activity-agent' and status='published' limit 1;") curl -s -b /tmp/k.jar -X POST "http://127.0.0.1:8080/api/v1/agents/$AGENT/runs" \ -H 'Content-Type: application/json' \ -d '{"input":"How many events happened in the last 7 days?"}' ``` Expect `"termination":"Completed"` and a non-zero `usage.totalTokens`. | Symptom | Cause | | --- | --- | | `404` on the runs route | no `MODEL_API_KEY`; the agent routes were never registered | | `the model credentials were refused` | the key is wrong — an Anthropic key renamed, or a bad Groq key | | `the model rejected the request` naming a model id | that id is not served; re-check §3 against `GET /v1/models` | | `502` from the proxy on slow runs | `HTTP_WRITE_TIMEOUT` below `2m` | ## 6. Rollback Nothing in the schema changed, so rollback is the previous image and the previous `.env`: ```bash cd infrastructure && git checkout && docker compose up -d --build ``` Restoring `ANTHROPIC_API_KEY` will **not** bring the old behaviour back at any commit from `34fa58a` onward — the provider is gone from the binary. Rolling back past it means rolling back the key too. ## 7. Not covered here - **The frontend needs no redeploy.** `krow-demo` is unchanged at `6249e00`; every change in this rollout is server-side. - **`EMBED_PROVIDER=ollama`** on krow-2 points at `host.docker.internal:11434`. Unrelated to this rollout, but if Ollama is not running on that host, dense retrieval degrades to keyword-only rather than failing loudly. - **The seed fixture generator** in `krow-demo` is a version behind the seeder and would delete the `employer@krow.app` account if run. Do not run `npm run seed:fixture` as part of a deploy.