Refuse an HTTP write timeout that would cut off a legal agent run
Production answered 502 Bad Gateway on a non-streamed agent run. Nothing about that was a gateway fault: krow-proxy already had proxy_read_timeout 3600s, and the API pods were healthy with zero restarts throughout. HTTP_WRITE_TIMEOUT was 30s. Every shipped agent runs at the `balanced` tier, whose deadline is 60s, and the `deep` tier allows 120s. So the server aborted the response on any run over half the time the runtime considered legal, the proxy saw its upstream vanish mid-response, and it reported the only thing it could. A gateway error for something no gateway did — which is why it looked like infrastructure for as long as it did. Delegation did not cause this; it made it routine. A parent that asks two subagents takes longer than one answering alone, so a latent misconfiguration became a reliable one. Verified: the exact request that returned 502 now answers 200 in 18s. Streaming is what hid it, and that is the part worth keeping in mind. The chat panel uses SSE, so the product looked healthy while every non-streaming caller got 502 on a slow question. A bug only reachable by the callers who do not yet exist is one nobody reports. So the value is now derived from the thing that constrains it — the default is DeepestAgentDeadline plus headroom rather than a number typed once — and validate() refuses anything below that deadline at startup. A slow, intermittent, misattributed failure becomes a message on the first boot. DeepestAgentDeadline is duplicated in internal/config rather than imported, because internal/runtime already imports internal/config and a cycle to share one number is a bad trade. TestConfigKnowsTheDeepestAgentDeadline asserts the two agree, so drift is a build failure rather than a discovery. It also checks that no tier exceeds it, or the name lies. ORDERING, and it matters for the next deploy: the check refuses the old 30s, so a pod carrying this image against an unpatched configmap will not boot. Production's configmap is already 180s. The handover says so too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJvibeSc1JYXjatankqM1g
This commit is contained in:
@@ -163,6 +163,25 @@ course, not a position in a window.
|
||||
|
||||
---
|
||||
|
||||
**HTTP_WRITE_TIMEOUT must exceed the deepest agent deadline.** It was 30s in
|
||||
production while every shipped agent runs at the `balanced` tier, whose
|
||||
deadline is 60s — so the server aborted the response on any run over half its
|
||||
allowed time, and the proxy in front answered **502 Bad Gateway**. A gateway
|
||||
error for something no gateway did, which is why it read as an infrastructure
|
||||
fault: nginx was innocent and already had `proxy_read_timeout 3600s`.
|
||||
|
||||
Streaming hid it. The chat panel uses SSE and survives, so the product looked
|
||||
healthy while any non-streaming caller — a webhook, a script, an integration —
|
||||
got 502 on a slow question. Delegation made it routine rather than causing it:
|
||||
a parent that asks two subagents takes longer than one answering alone.
|
||||
|
||||
Production is now 180s, and `config.validateWriteTimeout` refuses a value below
|
||||
`DeepestAgentDeadline` at startup. NOTE THE ORDERING: that constant is 120s, so
|
||||
a deployment still carrying the old 30s will now refuse to boot. Patch the
|
||||
configmap before shipping an image that contains the check.
|
||||
|
||||
---
|
||||
|
||||
## Still outstanding
|
||||
|
||||
- `ANTHROPIC_API_KEY` was pasted into a chat transcript and is live in a
|
||||
|
||||
Reference in New Issue
Block a user