Suriyakumarvijayanayagam
57c2a52c1e
Production answered 502 Bad Gateway on a non-streamed agent run. Nothing about that was a gateway fault: krow-proxy already had proxy_read_timeout 3600s, and the API pods were healthy with zero restarts throughout. HTTP_WRITE_TIMEOUT was 30s. Every shipped agent runs at the `balanced` tier, whose deadline is 60s, and the `deep` tier allows 120s. So the server aborted the response on any run over half the time the runtime considered legal, the proxy saw its upstream vanish mid-response, and it reported the only thing it could. A gateway error for something no gateway did — which is why it looked like infrastructure for as long as it did. Delegation did not cause this; it made it routine. A parent that asks two subagents takes longer than one answering alone, so a latent misconfiguration became a reliable one. Verified: the exact request that returned 502 now answers 200 in 18s. Streaming is what hid it, and that is the part worth keeping in mind. The chat panel uses SSE, so the product looked healthy while every non-streaming caller got 502 on a slow question. A bug only reachable by the callers who do not yet exist is one nobody reports. So the value is now derived from the thing that constrains it — the default is DeepestAgentDeadline plus headroom rather than a number typed once — and validate() refuses anything below that deadline at startup. A slow, intermittent, misattributed failure becomes a message on the first boot. DeepestAgentDeadline is duplicated in internal/config rather than imported, because internal/runtime already imports internal/config and a cycle to share one number is a bad trade. TestConfigKnowsTheDeepestAgentDeadline asserts the two agree, so drift is a build failure rather than a discovery. It also checks that no tier exceeds it, or the name lies. ORDERING, and it matters for the next deploy: the check refuses the old 30s, so a pod carrying this image against an unpatched configmap will not boot. Production's configmap is already 180s. The handover says so too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJvibeSc1JYXjatankqM1g
63 lines
2.0 KiB
Go
63 lines
2.0 KiB
Go
package config
|
|
|
|
import (
|
|
"strings"
|
|
"testing"
|
|
"time"
|
|
)
|
|
|
|
// A write timeout below the deepest agent deadline is refused at startup.
|
|
//
|
|
// This is the misconfiguration that shipped: HTTP_WRITE_TIMEOUT=30s against a
|
|
// balanced deadline of 60s. The server aborted the response on any run over
|
|
// half its allowed time and the proxy in front answered 502, so it read as an
|
|
// infrastructure fault for months. Refusing it at startup turns a slow,
|
|
// intermittent, misattributed failure into a message on the first boot.
|
|
func TestValidateWriteTimeout(t *testing.T) {
|
|
withTimeout := func(d time.Duration) *Config {
|
|
c := &Config{}
|
|
c.HTTP.WriteTimeout = d
|
|
return c
|
|
}
|
|
|
|
for _, tc := range []struct {
|
|
name string
|
|
timeout time.Duration
|
|
wantErr bool
|
|
}{
|
|
{"the value that shipped", 30 * time.Second, true},
|
|
{"equal to the balanced deadline is still short of deep", 60 * time.Second, true},
|
|
{"one second under", DeepestAgentDeadline - time.Second, true},
|
|
{"exactly the deepest deadline", DeepestAgentDeadline, false},
|
|
{"comfortably above", DeepestAgentDeadline + 30*time.Second, false},
|
|
{"no deadline at all cuts nothing off", 0, false},
|
|
{"negative is treated as unset", -1, false},
|
|
} {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
err := withTimeout(tc.timeout).validateWriteTimeout()
|
|
if tc.wantErr && err == nil {
|
|
t.Fatalf("%s was accepted; it would abort a legal run", tc.timeout)
|
|
}
|
|
if !tc.wantErr && err != nil {
|
|
t.Fatalf("%s was refused: %v", tc.timeout, err)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
// The message has to name the fix. An operator reading it at 3am should not
|
|
// have to find the deep tier's deadline in another package.
|
|
func TestValidateWriteTimeoutSaysWhatToDo(t *testing.T) {
|
|
c := &Config{}
|
|
c.HTTP.WriteTimeout = 30 * time.Second
|
|
err := c.validateWriteTimeout()
|
|
if err == nil {
|
|
t.Fatal("expected a refusal")
|
|
}
|
|
for _, want := range []string{"HTTP_WRITE_TIMEOUT", "30s", "2m0s", "502"} {
|
|
if !strings.Contains(err.Error(), want) {
|
|
t.Errorf("the message does not mention %q:\n %v", want, err)
|
|
}
|
|
}
|
|
}
|