Refuse an HTTP write timeout that would cut off a legal agent run
Some checks failed
CI / test (push) Failing after 5m32s
CI / fixture (push) Failing after 59s

Production answered 502 Bad Gateway on a non-streamed agent run. Nothing about
that was a gateway fault: krow-proxy already had proxy_read_timeout 3600s, and
the API pods were healthy with zero restarts throughout.

HTTP_WRITE_TIMEOUT was 30s. Every shipped agent runs at the `balanced` tier,
whose deadline is 60s, and the `deep` tier allows 120s. So the server aborted
the response on any run over half the time the runtime considered legal, the
proxy saw its upstream vanish mid-response, and it reported the only thing it
could. A gateway error for something no gateway did — which is why it looked
like infrastructure for as long as it did.

Delegation did not cause this; it made it routine. A parent that asks two
subagents takes longer than one answering alone, so a latent misconfiguration
became a reliable one. Verified: the exact request that returned 502 now
answers 200 in 18s.

Streaming is what hid it, and that is the part worth keeping in mind. The chat
panel uses SSE, so the product looked healthy while every non-streaming caller
got 502 on a slow question. A bug only reachable by the callers who do not yet
exist is one nobody reports.

So the value is now derived from the thing that constrains it — the default is
DeepestAgentDeadline plus headroom rather than a number typed once — and
validate() refuses anything below that deadline at startup. A slow,
intermittent, misattributed failure becomes a message on the first boot.

DeepestAgentDeadline is duplicated in internal/config rather than imported,
because internal/runtime already imports internal/config and a cycle to share
one number is a bad trade. TestConfigKnowsTheDeepestAgentDeadline asserts the
two agree, so drift is a build failure rather than a discovery. It also checks
that no tier exceeds it, or the name lies.

ORDERING, and it matters for the next deploy: the check refuses the old 30s, so
a pod carrying this image against an unpatched configmap will not boot.
Production's configmap is already 180s. The handover says so too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJvibeSc1JYXjatankqM1g
This commit is contained in:
2026-08-31 12:06:26 +05:30
parent fd1812e161
commit 57c2a52c1e
4 changed files with 160 additions and 1 deletions

View File

@@ -163,6 +163,25 @@ course, not a position in a window.
---
**HTTP_WRITE_TIMEOUT must exceed the deepest agent deadline.** It was 30s in
production while every shipped agent runs at the `balanced` tier, whose
deadline is 60s — so the server aborted the response on any run over half its
allowed time, and the proxy in front answered **502 Bad Gateway**. A gateway
error for something no gateway did, which is why it read as an infrastructure
fault: nginx was innocent and already had `proxy_read_timeout 3600s`.
Streaming hid it. The chat panel uses SSE and survives, so the product looked
healthy while any non-streaming caller — a webhook, a script, an integration —
got 502 on a slow question. Delegation made it routine rather than causing it:
a parent that asks two subagents takes longer than one answering alone.
Production is now 180s, and `config.validateWriteTimeout` refuses a value below
`DeepestAgentDeadline` at startup. NOTE THE ORDERING: that constant is 120s, so
a deployment still carrying the old 30s will now refuse to boot. Patch the
configmap before shipping an image that contains the check.
---
## Still outstanding
- `ANTHROPIC_API_KEY` was pasted into a chat transcript and is live in a