Production answered 502 Bad Gateway on a non-streamed agent run. Nothing about
that was a gateway fault: krow-proxy already had proxy_read_timeout 3600s, and
the API pods were healthy with zero restarts throughout.
HTTP_WRITE_TIMEOUT was 30s. Every shipped agent runs at the `balanced` tier,
whose deadline is 60s, and the `deep` tier allows 120s. So the server aborted
the response on any run over half the time the runtime considered legal, the
proxy saw its upstream vanish mid-response, and it reported the only thing it
could. A gateway error for something no gateway did — which is why it looked
like infrastructure for as long as it did.
Delegation did not cause this; it made it routine. A parent that asks two
subagents takes longer than one answering alone, so a latent misconfiguration
became a reliable one. Verified: the exact request that returned 502 now
answers 200 in 18s.
Streaming is what hid it, and that is the part worth keeping in mind. The chat
panel uses SSE, so the product looked healthy while every non-streaming caller
got 502 on a slow question. A bug only reachable by the callers who do not yet
exist is one nobody reports.
So the value is now derived from the thing that constrains it — the default is
DeepestAgentDeadline plus headroom rather than a number typed once — and
validate() refuses anything below that deadline at startup. A slow,
intermittent, misattributed failure becomes a message on the first boot.
DeepestAgentDeadline is duplicated in internal/config rather than imported,
because internal/runtime already imports internal/config and a cycle to share
one number is a bad trade. TestConfigKnowsTheDeepestAgentDeadline asserts the
two agree, so drift is a build failure rather than a discovery. It also checks
that no tier exceeds it, or the name lies.
ORDERING, and it matters for the next deploy: the check refuses the old 30s, so
a pod carrying this image against an unpatched configmap will not boot.
Production's configmap is already 180s. The handover says so too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJvibeSc1JYXjatankqM1g
Both repositories carried GitHub Actions workflows on a Gitea remote and
nobody had confirmed a runner. There was not one: the 924 frontend checks,
the whole Go suite, the skip guard and the suite-shrank guard had never run
on a push, only when somebody remembered.
gitea/act_runner v0.6.1 is registered as krow-runner on the cluster host. This
commit is also the first push that can prove it picks up a job, which is the
failure mode worth catching — a runner that registers and never runs anything
looks identical to a healthy one in the Runners list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJvibeSc1JYXjatankqM1g
And correct the previous commit's closing claim, which was wrong.
c378bc0 said the Hiring activity chart "still renders empty ... the fault is
further down in that component, not in the data." That was a retraction of a
correct diagnosis, made from a screenshot taken before the rebase had reached
the browser. The chart was empty BECAUSE the data was stale, exactly as first
diagnosed, and rebasing fixed it. Checked properly this time: the area path
carries real values, and the rendered chart shows applications peaking at 16
around 8/18 with the screening and interview series drawn over it.
What was genuinely still missing was Hires. Staff was not rebased, so
hire_date stayed 35 days old with nothing inside the 30-day window and that
series drew nothing. It is rebased now, anchored on hire_date rather than
created_date, because the hire is the event the chart plots.
Leaving it behind had also introduced an inconsistency of my own making: once
applications moved, a candidate was hired last week according to their
application and five weeks ago according to their staff record. Rebasing them
together removes that.
All four series now render. Full suite green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJvibeSc1JYXjatankqM1g
Both documents are the first thing a new reader trusts, and several of their
claims were wrong — some wrong from the start, some overtaken by work this
week. A governing document that misdescribes the system is worse than none,
because it is believed.
CLAUDE.md §10 specified Python 3.11, FastAPI, SQLAlchemy and Alembic. The code
is Go and has never been anything else. That is corrected rather than quietly
deleted, so the next person understands the document drifted rather than
wondering which half to trust.
Also in CLAUDE.md: the tool count was 17 with one write and is 19 with two;
delegation now exists and §11's Orchestration row says what it guarantees; the
embedder deviation described a Voyage-or-stand-in choice that has since become
EMBED_PROVIDER with three options, of which production sets none.
The handover claimed three things that this session disproved by running them:
- "definition_versions is empty ... nothing has gone through it". It was not
empty in production; activity-agent had a v1 that the shipped file
contradicted, which is how a real drift was found. It now holds every
agent and skill.
- "Skills are still stored in user_preferences". They are rows in
skill_definitions, and are now versioned.
- "make eval-live ... has never been run". It has, it passes 3/3, and what
it established is recorded — including that the handbook corpus carries a
planted prompt injection which the agent refused and reported. That is I7
holding against a real model, which is worth more than the pass count.
The endpoint counts were one high throughout (55/57, not 56/58) — the delta of
two was always right, so the signal worked and the absolute numbers did not.
Added, because they cost time this week and would cost it again:
- the app reaches its database through pgbouncer, not PostgreSQL directly.
Enabling TLS on PostgreSQL does nothing for the application hop; pgbouncer
terminates 5432 and needs its own client_tls_sslmode.
- the seeded UserActivity is NOT anchored to today the way ShiftRecord is,
so it ages out of every window the activity tools offer. Twenty-three days
old as of writing: zero events in the last 7 days, 6 of 15 in the last 30.
The agent answers truthfully and the demo looks dead.
- the whole stack runs on Docker alone. Dockerfile.api builds every command
plus the migrate CLI, so a new machine needs neither Go nor psql — which
is how this one was set up, having no Homebrew.
- Ollama runs on the HOST, so a container reaches it at
host.docker.internal, not localhost. The old .env said localhost and would
have failed with nothing obviously wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJvibeSc1JYXjatankqM1g
Neither survived a machine change. CLAUDE.md sat in the directory ABOVE both
repositories, which is not a git repository at all, so the governing document
for the project existed on exactly one laptop. It is now in this repository;
place a copy at the parent level on a new machine, where it covers both.
docs/handover.md records what CLAUDE.md does not: what was decided and why,
what is deployed and how to verify it, and the conventions that produce
confident wrong numbers rather than errors — a score of 0 meaning "not rated",
screened_at being vestigial, shift data anchored to today.
Written because Claude Code's own memory is per-machine and keyed to the
absolute path of the checkout: it does not sync, and a different path on a new
machine reads a different folder. A file in the repository travels with the
code and is useful to a person besides.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186JgqQUCDS8ZwGmyw3ymWu
