Krow/krow_backend
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove the Anthropic path; the gateway speaks one wire protocol
Some checks failed
CI / test (push) Failing after 4m38s
Details
CI / fixture (push) Failing after 7s
Details

Browse Source 
The platform now runs on Groq by default, through the OpenAI-compatible
chat-completions shape. That shape is not one vendor — Gemini, OpenRouter,
Together, vLLM and a local Ollama serve it too — so moving again stays
configuration rather than code.

Two things in the deleted file were not Anthropic's and would have gone
with it silently:

  withRetry / MaxAttempts / retryBackoff were defined in anthropic.go and
  CALLED BY openai.go. Deleting the file wholesale would have removed the
  retry policy of the provider that survived, and nothing in openai.go
  mentions it, so the loss would have been invisible until the next 429.
  The policy is a property of this platform's runs, not of a vendor's API;
  it now lives in retry.go where no provider can carry it off.

  StreamComplete had the same problem and moves to gateway.go, beside the
  Streamer interface whose comment already referenced it.

Three stale-configuration failures are now refused at startup instead of
being ignored. Each was verified firing through the real config.Load():

  MODEL_PROVIDER=anthropic — named separately from every other wrong value
  because it used to be correct. Ignoring it gives a stack that believes it
  is on Claude while every run goes to Groq and is billed there.

  ANTHROPIC_API_KEY set while MODEL_API_KEY is empty. Ignoring a key an
  operator did set is the worst version of this: they fail every run on a
  missing credential they are looking straight at.

  A leftover claude-* model id, naming the tier that carries it. This is
  the check the previous commit's error-detail work was diagnosing: such an
  id is accepted by this process, rejected by the provider, and 400s on
  EVERY run. "A model is wrong" does not say which of three lines to edit.

Defaults ship as a matched pair. defaultBaseURL and the three tier ids are
one decision, not four: an id is only meaningful against the service that
serves it, and a Groq id on an OpenAI base URL is the same failure from the
other side. The tiers also stop being one model — a tier whose cost does
not differ is a distinction that buys nothing.

Verified end to end against a stub of the wire, driving the real wiring
(config.Load in production mode, gateway.New, StreamComplete): streamed
deltas, tool-call decoding, the loopback credential exemption, and usage
totalling 150 rather than 190 — the cached-prefix subtraction still holds.

gofmt clean, go vet clean, 14/14 non-DB packages pass. httpserver still
needs a reachable database.

NOT verified: the I7 planted-injection eval. Removing this path removed the
only model whose refusal behaviour had been measured against it, so the new
default is unproven there until `make eval-live` runs with a real key. The
Groq model ids should also be confirmed against Groq's current lineup.
Flagged in CLAUDE.md §12 and docs/handover.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJvibeSc1JYXjatankqM1g
This commit is contained in:
Suriyakumarvijayanayagam
2026-09-05 11:52:21 +05:30
parent cf99866e12
commit 34fa58a6b9
18 changed files with 391 additions and 765 deletions
Download Patch File Download Diff File Expand all files Collapse all files

510
go-api/internal/gateway/anthropic.go
View File

@@ -1,510 +0,0 @@
package gateway
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"github.com/anthropics/anthropic-sdk-go"
"github.com/anthropics/anthropic-sdk-go/option"
)
// AnthropicGateway calls the Claude API.
type AnthropicGateway struct {
client anthropic.Client
cfg Config
}
// Compile-time proof that this satisfies the boundary.
var _ Gateway = (*AnthropicGateway)(nil)
// NewAnthropic builds a gateway over the Claude API.
//
// A missing key is not an error here. The service has to boot without model
// credentials — every endpoint that is not an agent run still works, and a
// developer running migrations should not need a key to do it. The failure
// surfaces at the first Complete, as a structured NotConfigured that the
// runtime can end a run with, rather than as a panic at startup.
func NewAnthropic(cfg Config) *AnthropicGateway {
opts := []option.RequestOption{}
if cfg.APIKey != "" {
opts = append(opts, option.WithAPIKey(cfg.APIKey))
}
return &AnthropicGateway{client: anthropic.NewClient(opts...), cfg: cfg}
}
// routing resolves a tier against this gateway's table.
func (g *AnthropicGateway) routing(t Tier) Routing { return g.cfg.routingFor(t) }
// sdkEffort maps the platform's effort vocabulary onto Anthropic's.
//
// A one-to-one mapping today, which is exactly why the neutral type is worth
// having: the platform's three levels are a statement about how much a turn is
// worth, and this function is where that statement meets one vendor's spelling
// of it. `max` is not reachable — see FromConfig.
func sdkEffort(e Effort) anthropic.OutputConfigEffort {
switch e {
case EffortLow:
return anthropic.OutputConfigEffortLow
case EffortXhigh:
return anthropic.OutputConfigEffortXhigh
default:
return anthropic.OutputConfigEffortHigh
}
}
// Complete sends one request and reports one result.
// MaxAttempts is how many times a transient failure is retried.
//
// Three total, not three retries. Past that the problem is not transient and a
// fourth call is just spending money on the same answer.
const MaxAttempts = 3
// retryBackoff is the pause before each retry.
//
// Short, and deliberately so: this sits inside a run that already has a
// wall-clock deadline, and a backoff long enough to be polite to the API is
// long enough to spend the caller's whole budget waiting. A run that cannot
// afford the wait dies on its deadline instead, which is the correct failure.
var retryBackoff = []time.Duration{400 * time.Millisecond, 1200 * time.Millisecond}
// Complete calls the model, retrying failures that are worth retrying.
//
// THE RETRY IS NOT DEFENSIVE POLISH. Error.Retryable() has existed since this
// package was written and had ZERO callers — the classification was built and
// never used, so a 529 "overloaded" killed a run that would have succeeded four
// hundred milliseconds later. Found by a real overload during live testing,
// where it presented as "the agent could not finish" with nothing to act on.
//
// Only genuinely transient failures qualify: rate limits, timeouts, and 5xx.
// A 400 is a malformed request and will be malformed again; a 401 is a bad
// credential and retrying it three times just gets refused three times.
//
// The run's context governs. A retry that would outlive the caller's deadline
// does not happen — the deadline belongs to the run, not to this function, and
// waiting past it would turn a bounded run into an unbounded one.
func (g *AnthropicGateway) Complete(ctx context.Context, req Request) (*Response, error) {
return withRetry(ctx, func() (*Response, error) { return g.complete(ctx, req) })
}
// withRetry runs one attempt until it succeeds, fails terminally, or runs out
// of attempts.
//
// SHARED BY EVERY PROVIDER, and it has to be. The retry policy is a property of
// this platform's runs — bounded attempts, short backoff, the caller's deadline
// winning — not of any one vendor's API. Left as a method, the second provider
// would have grown its own copy, and the two would have drifted the first time
// either was tuned.
func withRetry(ctx context.Context, once func() (*Response, error)) (*Response, error) {
var last error
for attempt := 0; attempt < MaxAttempts; attempt++ {
if attempt > 0 {
pause := retryBackoff[min(attempt-1, len(retryBackoff)-1)]
select {
case <-time.After(pause):
case <-ctx.Done():
// Out of time. The ORIGINAL failure is returned rather than the
// context error: "the model was overloaded" is what an operator
// needs to see, and "context deadline exceeded" would hide it.
return nil, last
}
}
resp, err := once()
if err == nil {
return resp, nil
}
last = err
var gwErr *Error
if !errors.As(err, &gwErr) || !gwErr.Retryable() {
return resp, err
}
}
return nil, last
}
// complete is one attempt.
func (g *AnthropicGateway) complete(ctx context.Context, req Request) (*Response, error) {
if g.cfg.APIKey == "" {
return nil, &Error{
Code: CodeNotConfigured,
Message: "no model credentials are configured for this deployment",
}
}
if err := req.Validate(); err != nil {
return nil, err
}
params, err := g.params(req)
if err != nil {
return nil, err
}
msg, err := g.client.Messages.New(ctx, params)
if err != nil {
return nil, translate(err)
}
return g.decode(msg, req)
}
// params builds the request both paths send.
//
// Extracted so the streaming and non-streaming calls cannot drift. They send
// the same model, the same thinking config, the same cache breakpoint and the
// same tools — an answer that differs depending on whether it was streamed
// would be the worst kind of bug to chase, because the transport is the last
// place anybody looks.
func (g *AnthropicGateway) params(req Request) (anthropic.MessageNewParams, error) {
route := g.routing(req.Tier)
maxTokens := req.MaxOutputTokens
if maxTokens <= 0 {
maxTokens = g.cfg.MaxOutputTokens
}
messages, err := encodeMessages(req.Messages)
if err != nil {
return anthropic.MessageNewParams{}, err
}
params := anthropic.MessageNewParams{
Model: anthropic.Model(route.Model),
MaxTokens: maxTokens,
Messages: messages,
// Adaptive thinking on every tier: the model decides how much to think,
// and effort sets the ceiling on that. A fixed token budget for
// reasoning is the deprecated shape and is rejected outright by the
// current models.
Thinking: anthropic.ThinkingConfigParamUnion{
OfAdaptive: &anthropic.ThinkingConfigAdaptiveParam{},
},
OutputConfig: anthropic.OutputConfigParam{Effort: sdkEffort(route.Effort)},
}
if len(req.Tools) > 0 {
params.Tools = encodeTools(req.Tools)
}
if s := strings.TrimSpace(req.System); s != "" {
// One cached block. The system prompt is the stable prefix of every
// turn in a run, and the render order is tools → system → messages, so
// a breakpoint here is the one that survives the conversation growing.
params.System = []anthropic.TextBlockParam{{
Text: s,
CacheControl: anthropic.NewCacheControlEphemeralParam(),
}}
}
return params, nil
}
// decode turns a finished message into a Response.
//
// Shared by both paths for the same reason params() is: a streamed message and
// a non-streamed one are the same object by the time they get here, and reading
// them differently would make streaming a second implementation of the answer.
func (g *AnthropicGateway) decode(msg *anthropic.Message, req Request) (*Response, error) {
route := g.routing(req.Tier)
usage := Usage{
InputTokens: msg.Usage.InputTokens,
OutputTokens: msg.Usage.OutputTokens,
CacheReadTokens: msg.Usage.CacheReadInputTokens,
CacheCreationTokens: msg.Usage.CacheCreationInputTokens,
}
// A refusal arrives as a successful HTTP response, so it has to be checked
// before the content is read. It is still billed, and the usage is carried
// on the error so the run's budget is charged for a turn that produced no
// text — a refusal that cost nothing on the ledger is a refusal the loop
// would happily repeat.
if msg.StopReason == anthropic.StopReasonRefusal {
return &Response{
StopReason: string(msg.StopReason),
Usage: usage,
Model: route.Model,
Tier: req.Tier,
}, &Error{
Code: CodeRefused,
Message: "the model declined this request",
Category: string(msg.StopDetails.Category),
}
}
var (
text strings.Builder
calls []ToolCall
)
for _, block := range msg.Content {
switch b := block.AsAny().(type) {
case anthropic.TextBlock:
text.WriteString(b.Text)
case anthropic.ToolUseBlock:
// The raw JSON, not a parsed value: current models vary their
// string escaping inside tool inputs, so this is handed to the
// handler's own decoder rather than matched on as a string here.
calls = append(calls, ToolCall{
ID: b.ID,
Name: b.Name,
Input: json.RawMessage(b.JSON.Input.Raw()),
})
}
}
return &Response{
Text: text.String(),
ToolCalls: calls,
StopReason: string(msg.StopReason),
Usage: usage,
Model: route.Model,
Tier: req.Tier,
}, nil
}
// encodeTools renders the tool definitions for the wire.
func encodeTools(defs []ToolDef) []anthropic.ToolUnionParam {
out := make([]anthropic.ToolUnionParam, 0, len(defs))
for _, d := range defs {
schema := anthropic.ToolInputSchemaParam{}
if props, ok := d.InputSchema["properties"].(map[string]any); ok {
schema.Properties = props
}
if req, ok := d.InputSchema["required"].([]string); ok {
schema.Required = req
}
tool := anthropic.ToolParam{
Name: d.Name,
Description: anthropic.String(d.Description),
InputSchema: schema,
}
out = append(out, anthropic.ToolUnionParam{OfTool: &tool})
}
return out
}
// encodeMessages renders a conversation for the wire.
//
// Tool results are variadic within ONE user message. Splitting them across
// several messages is accepted by the API and quietly teaches the model to stop
// making parallel calls — a performance regression with no error to trace it
// to, so the grouping is done here rather than left to callers.
func encodeMessages(msgs []Message) ([]anthropic.MessageParam, error) {
out := make([]anthropic.MessageParam, 0, len(msgs))
for i, m := range msgs {
var blocks []anthropic.ContentBlockParamUnion
if s := strings.TrimSpace(m.Text); s != "" {
blocks = append(blocks, anthropic.NewTextBlock(m.Text))
}
for _, c := range m.ToolCalls {
var input any
if len(c.Input) > 0 {
if err := json.Unmarshal(c.Input, &input); err != nil {
return nil, &Error{
Code: CodeInvalidRequest,
Message: fmt.Sprintf("messages[%d]: tool call %s carries invalid JSON", i, c.Name),
}
}
}
blocks = append(blocks, anthropic.NewToolUseBlock(c.ID, input, c.Name))
}
for _, r := range m.ToolResults {
blocks = append(blocks, anthropic.NewToolResultBlock(r.CallID, r.Content, r.IsError))
}
if len(blocks) == 0 {
continue
}
if m.Role == RoleAssistant {
out = append(out, anthropic.NewAssistantMessage(blocks...))
continue
}
out = append(out, anthropic.NewUserMessage(blocks...))
}
return out, nil
}
// translate turns an SDK error into one the runtime can branch on.
//
// A single broad class would lose the distinction the loop actually needs:
// whether sending the same request again could work. So the status is read and
// mapped, and anything unrecognised stays CodeUpstream with its status intact
// rather than being flattened into a generic failure.
func translate(err error) error {
if errors.Is(err, context.DeadlineExceeded) || errors.Is(err, context.Canceled) {
return &Error{Code: CodeTimeout, Message: "the model call did not complete in time", Cause: err}
}
var apierr *anthropic.Error
if !errors.As(err, &apierr) {
return &Error{Code: CodeUpstream, Message: "the model call failed", Cause: err}
}
// The upstream reason, carried through when there is one — the same rule the
// OpenAI path already follows, and for the same reason it gives: a 400 that
// says which model id or tool schema was rejected is worth more than "the
// model rejected the request", and the trajectory records only the message.
//
// This was not a hypothetical. Every run on a deployment failed with a bare
// `gateway.invalid_request`, and neither the run detail nor anything a
// client could read said why — the one fact needed to fix it was discarded
// here, three lines from where it arrived. `Cause` keeps the full error for
// a Go caller; nothing reads it by the time a run is persisted.
detail := anthropicErrorMessage(apierr.RawJSON())
withDetail := func(base string) string {
if detail == "" {
return base
}
return base + ": " + detail
}
switch apierr.StatusCode {
case 400:
return &Error{Code: CodeInvalidRequest, Message: withDetail("the model rejected the request"), Status: 400, Cause: err}
case 401, 403:
return &Error{Code: CodeUnauthorized, Message: withDetail("the model credentials were refused"), Status: apierr.StatusCode, Cause: err}
case 408:
return &Error{Code: CodeTimeout, Message: withDetail("the model call timed out"), Status: 408, Cause: err}
case 429:
return &Error{Code: CodeRateLimited, Message: withDetail("the model is rate limiting this deployment"), Status: 429, Cause: err}
case 529:
// Anthropic's "overloaded" — the service is up and temporarily out of
// capacity. Named separately from the 500s because it is the one that
// actually happens, and because a run dying on it is a run that would
// have succeeded a second later.
return &Error{Code: CodeUpstream, Message: "the model is temporarily overloaded",
Status: 529, Cause: err}
default:
// The status is IN the message, not only in the field. It cost an hour
// of debugging to learn that "the model call failed" was a 529 rather
// than a malformed tool schema, and the trajectory only records the
// message.
return &Error{
Code: CodeUpstream,
Message: withDetail(fmt.Sprintf("the model call failed (http %d)", apierr.StatusCode)),
Status: apierr.StatusCode, Cause: err,
}
}
}
// anthropicErrorMessage digs the human-readable reason out of an error body.
//
// Best-effort, exactly like its OpenAI counterpart: the envelope is documented
// and stable enough to be worth reading, and an unparseable body yields nothing
// rather than failing a failure.
func anthropicErrorMessage(raw string) string {
if raw == "" {
return ""
}
var envelope struct {
Error struct {
Message string `json:"message"`
Type string `json:"type"`
} `json:"error"`
}
if err := json.Unmarshal([]byte(raw), &envelope); err != nil {
return ""
}
if envelope.Error.Message != "" {
return envelope.Error.Message
}
return envelope.Error.Type
}
/* ── Streaming ──────────────────────────────────────────────────────────── */
// Stream is Complete, with the assistant's text delivered as it arrives.
//
// §6: "Stream partial assistant text as it arrives; buffer tool calls until
// complete." Both halves of that matter and they pull in opposite directions.
//
// TEXT IS STREAMED because a fifteen-second wait with nothing on screen reads
// as broken. The reader wants the first sentence while the rest is still being
// written, and that is the whole difference between a product and a spinner.
//
// TOOL CALLS ARE NOT. A tool call arrives as JSON assembled character by
// character across many events, and a half-built argument object is not a
// smaller version of the finished one — it is a different object, usually an
// invalid one. Dispatching on a partial call would run a tool with arguments
// the model had not finished choosing. So the accumulated message is decoded
// only once the stream closes, by exactly the same code the non-streaming path
// uses.
//
// onDelta is called from this goroutine, in order, and must not block for long
// — it is on the path between the model and the reader.
func (g *AnthropicGateway) Stream(ctx context.Context, req Request, onDelta func(string)) (*Response, error) {
if g.cfg.APIKey == "" {
return nil, &Error{
Code: CodeNotConfigured,
Message: "no model credentials are configured for this deployment",
}
}
if err := req.Validate(); err != nil {
return nil, err
}
params, err := g.params(req)
if err != nil {
return nil, err
}
stream := g.client.Messages.NewStreaming(ctx, params)
defer stream.Close()
var msg anthropic.Message
for stream.Next() {
event := stream.Current()
if err := msg.Accumulate(event); err != nil {
return nil, &Error{
Code: CodeUpstream,
Message: "the streamed response could not be assembled",
Cause: err,
}
}
// Text only. A thinking delta is the model's private reasoning and is
// not the answer; a tool-input delta is a fragment of JSON. Neither is
// something to put in front of a reader.
if event.Type == "content_block_delta" && event.Delta.Type == "text_delta" {
if d := event.Delta.Text; d != "" && onDelta != nil {
onDelta(d)
}
}
}
if err := stream.Err(); err != nil {
return nil, translate(err)
}
return g.decode(&msg, req)
}
// StreamComplete runs a request through whichever path the gateway supports.
//
// A gateway that cannot stream is not a broken gateway — every fake in the test
// suite is one, and so is any future provider without a streaming API. Falling
// back to Complete and delivering the finished text as a single delta keeps the
// caller's code identical either way, which is what stops streaming from
// becoming a second code path through the loop.
func StreamComplete(ctx context.Context, gw Gateway, req Request, onDelta func(string)) (*Response, error) {
// Normalised once, here, so no implementation has to guard it. A caller
// that does not want deltas passes nil — every eval and every test does —
// and an implementation that took that literally would panic on the first
// fragment. Making each Streamer remember the check is how one of them
// eventually forgets.
if onDelta == nil {
onDelta = func(string) {}
}
if s, ok := gw.(Streamer); ok {
return s.Stream(ctx, req, onDelta)
}
resp, err := gw.Complete(ctx, req)
if err == nil && resp != nil && resp.Text != "" && onDelta != nil {
onDelta(resp.Text)
}
return resp, err
}

46
go-api/internal/gateway/anthropic_error_test.go
View File

@@ -1,46 +0,0 @@
package gateway
import "testing"
// The reason a 400 gives is the whole diagnosis, and it used to be thrown away.
//
// A deployment answered every single run with "The agent could not finish —
// something it needed did not answer." Retrieval worked, the run was created,
// the provider was reached, and the trajectory recorded `gateway.invalid_request`
// and nothing else. The actual cause was one sentence long and Anthropic had
// sent it: the account was out of credit. Nothing a client, a log or a run
// detail could show said so, because `translate` dropped the body.
//
// The envelope below is the real one, copied from that response.
func TestAnthropicErrorMessage(t *testing.T) {
cases := []struct {
name string
raw string
want string
}{
{
name: "the outage this test exists for",
raw: `{"type":"error","error":{"type":"invalid_request_error",` +
`"message":"Your credit balance is too low to access the Anthropic API. ` +
`Please go to Plans & Billing to upgrade or purchase credits."},` +
`"request_id":"req_011CeiJqr8rTdpbZYhcL6VHG"}`,
want: "Your credit balance is too low to access the Anthropic API. " +
"Please go to Plans & Billing to upgrade or purchase credits.",
},
{
name: "a message-less envelope falls back to the type",
raw: `{"type":"error","error":{"type":"overloaded_error"}}`,
want: "overloaded_error",
},
// Best-effort by design: a failure to read a failure must not become a
// second failure.
{name: "unparseable body yields nothing", raw: `not json at all`, want: ""},
{name: "empty body yields nothing", raw: ``, want: ""},
}
for _, c := range cases {
if got := anthropicErrorMessage(c.raw); got != c.want {
t.Errorf("%s: anthropicErrorMessage() = %q, want %q", c.name, got, c.want)
}
}
}

26
go-api/internal/gateway/gateway.go
View File

@@ -292,3 +292,29 @@ func (r Request) Validate() error {
}
return nil
}
// StreamComplete runs a request through whichever path the gateway supports.
//
// A gateway that cannot stream is not a broken gateway — every fake in the test
// suite is one, and so is any future provider without a streaming API. Falling
// back to Complete and delivering the finished text as a single delta keeps the
// caller's code identical either way, which is what stops streaming from
// becoming a second code path through the loop.
func StreamComplete(ctx context.Context, gw Gateway, req Request, onDelta func(string)) (*Response, error) {
// Normalised once, here, so no implementation has to guard it. A caller
// that does not want deltas passes nil — every eval and every test does —
// and an implementation that took that literally would panic on the first
// fragment. Making each Streamer remember the check is how one of them
// eventually forgets.
if onDelta == nil {
onDelta = func(string) {}
}
if s, ok := gw.(Streamer); ok {
return s.Stream(ctx, req, onDelta)
}
resp, err := gw.Complete(ctx, req)
if err == nil && resp != nil && resp.Text != "" && onDelta != nil {
onDelta(resp.Text)
}
return resp, err
}

30
go-api/internal/gateway/gateway_test.go
View File

@@ -6,8 +6,6 @@ import (
"strings"
"testing"
"github.com/anthropics/anthropic-sdk-go"
"github.com/krow/krow-backend/go-api/internal/config"
)
@@ -71,7 +69,7 @@ func TestRequestValidate(t *testing.T) {
func TestCompleteWithoutCredentialsIsStructured(t *testing.T) {
// The service boots without a key on purpose. The failure has to arrive as
// something a run can terminate with, not as a panic or a bare string.
g := NewAnthropic(Config{})
g := NewOpenAI(Config{})
_, err := g.Complete(context.Background(), Request{
Messages: []Message{{Role: RoleUser, Text: "anything"}},
})
@@ -127,25 +125,29 @@ func TestFromConfigPinsEffortPerTier(t *testing.T) {
}
}
// The neutral effort vocabulary has to land on the vendor's own enum, and that
// mapping is the one thing FromConfig can no longer assert now that its result
// is provider-independent. Untested, a renamed SDK constant would silently
// The neutral effort vocabulary still has to land on a vendor's own spelling,
// and that mapping is the one thing FromConfig cannot assert now that its
// result is provider-independent. Untested, a renamed constant would silently
// route every tier to whatever the default arm returns.
func TestSDKEffortMapsToAnthropic(t *testing.T) {
cases := map[Effort]anthropic.OutputConfigEffort{
EffortLow: anthropic.OutputConfigEffortLow,
EffortHigh: anthropic.OutputConfigEffortHigh,
EffortXhigh: anthropic.OutputConfigEffortXhigh,
//
// This asserts POSITIONS, not words. OpenAI's scale runs minimal/low/medium/
// high against our low/high/xhigh, so `high` here is their "medium" — matching
// the spelling instead would collapse `fast` and `balanced` into neighbours.
func TestEffortMapsOntoTheProviderScale(t *testing.T) {
cases := map[Effort]string{
EffortLow: "low",
EffortHigh: "medium",
EffortXhigh: "high",
}
for neutral, want := range cases {
if got := sdkEffort(neutral); got != want {
t.Errorf("sdkEffort(%q) = %q, want %q", neutral, got, want)
if got := openAIEffort(neutral); got != want {
t.Errorf("openAIEffort(%q) = %q, want %q", neutral, got, want)
}
}
}
func TestRoutingSelectsPerTier(t *testing.T) {
g := NewAnthropic(Config{
g := NewOpenAI(Config{
Fast: Routing{Model: "m-fast"},
Balanced: Routing{Model: "m-balanced"},
Deep: Routing{Model: "m-deep"},

73
go-api/internal/gateway/retry.go Normal file
View File

@@ -0,0 +1,73 @@
package gateway
import (
"context"
"errors"
"time"
)
// MaxAttempts is how many times a transient failure is retried.
//
// Three total, not three retries. Past that the problem is not transient and a
// fourth call is just spending money on the same answer.
const MaxAttempts = 3
// retryBackoff is the pause before each retry.
//
// Short, and deliberately so: this sits inside a run that already has a
// wall-clock deadline, and a backoff long enough to be polite to the API is
// long enough to spend the caller's whole budget waiting. A run that cannot
// afford the wait dies on its deadline instead, which is the correct failure.
var retryBackoff = []time.Duration{400 * time.Millisecond, 1200 * time.Millisecond}
// withRetry runs one attempt until it succeeds, fails terminally, or runs out
// of attempts.
//
// THE RETRY IS NOT DEFENSIVE POLISH. Error.Retryable() has existed since this
// package was written and had ZERO callers — the classification was built and
// never used, so a 529 "overloaded" killed a run that would have succeeded four
// hundred milliseconds later. Found by a real overload during live testing,
// where it presented as "the agent could not finish" with nothing to act on.
//
// Only genuinely transient failures qualify: rate limits, timeouts, and 5xx.
// A 400 is a malformed request and will be malformed again; a 401 is a bad
// credential and retrying it three times just gets refused three times.
//
// The run's context governs. A retry that would outlive the caller's deadline
// does not happen — the deadline belongs to the run, not to this function, and
// waiting past it would turn a bounded run into an unbounded one.
//
// THIS FILE EXISTS BECAUSE THE POLICY OUTLIVED ITS FIRST PROVIDER. It was
// written inside the Anthropic implementation and used by both, so deleting
// that implementation would have deleted the retry policy of the one that
// remained — silently, because nothing about `openai.go` mentions it. The
// policy is a property of this platform's runs, not of any vendor's API, so it
// now lives somewhere no provider can take with it when it goes.
func withRetry(ctx context.Context, once func() (*Response, error)) (*Response, error) {
var last error
for attempt := 0; attempt < MaxAttempts; attempt++ {
if attempt > 0 {
pause := retryBackoff[min(attempt-1, len(retryBackoff)-1)]
select {
case <-time.After(pause):
case <-ctx.Done():
// Out of time. The ORIGINAL failure is returned rather than the
// context error: "the model was overloaded" is what an operator
// needs to see, and "context deadline exceeded" would hide it.
return nil, last
}
}
resp, err := once()
if err == nil {
return resp, nil
}
last = err
var gwErr *Error
if !errors.As(err, &gwErr) || !gwErr.Retryable() {
return resp, err
}
}
return nil, last
}

55
go-api/internal/gateway/routing.go
View File

@@ -4,26 +4,28 @@ import (
"github.com/krow/krow-backend/go-api/internal/config"
)
// Provider names the wire protocol a deployment talks.
// ProviderOpenAI names the only wire protocol this platform speaks.
//
// Two, not two hundred: "anthropic" is the Claude API, and "openai" is the
// chat-completions shape that Groq, Gemini, OpenRouter, Together, vLLM and
// Ollama all serve. That second one is the reason this constant exists at all
// — supporting those five providers is one implementation and five different
// base URLs, and pretending otherwise would grow a package per vendor.
const (
ProviderAnthropic = "anthropic"
ProviderOpenAI = "openai"
)
// One constant, not an enum, because there is one implementation. "openai" is
// the chat-completions shape — which is NOT only OpenAI. Groq, Gemini (through
// its compatible endpoint), OpenRouter, Together, vLLM and a local Ollama all
// serve it, and the difference between them is MODEL_BASE_URL and a model id,
// nothing more. Supporting six vendors is one implementation and six base URLs.
//
// The Anthropic path was removed deliberately, not lost. `MODEL_PROVIDER=anthropic`
// is now REFUSED at startup rather than ignored — see config.validateModel. A
// deployment carrying the old value must be told it moved, because the silent
// alternative is a stack that believes it is still on Claude while every run
// goes somewhere else.
const ProviderOpenAI = "openai"
// Effort is how hard a tier is allowed to think.
//
// PROVIDER-NEUTRAL ON PURPOSE. This was `anthropic.OutputConfigEffort` until a
// second provider existed, which meant the vendor's enum was baked into the
// routing table that every provider has to read. Nothing was wrong with it
// while there was one implementation; it became wrong the moment there were
// two, because the OpenAI path would have had to import the Anthropic SDK to
// learn how hard to think.
// PROVIDER-NEUTRAL ON PURPOSE, and the reason that mattered is now history
// worth keeping: this was a vendor SDK's own enum, baked into the routing table
// every provider has to read. Making it the platform's own vocabulary is what
// let that vendor be removed later without the routing table going with it —
// a one-line deletion instead of a re-typing of every tier.
//
// The three values are the platform's own vocabulary. Each implementation maps
// them onto whatever its API calls the same idea, and a provider with no such
@@ -55,8 +57,8 @@ type Routing struct {
// Built once at startup from the environment and passed in frozen, per §10.
// Nothing in this package reads the environment itself.
type Config struct {
// Provider selects the implementation. Empty means anthropic, so a
// deployment that predates the second provider keeps working untouched.
// Provider selects the implementation. Empty means openai, which is now
// the only one; config.validateModel refuses any other value.
Provider string
APIKey string
@@ -116,21 +118,18 @@ func FromConfig(c config.ModelConfig) Config {
// New builds the gateway a deployment's configuration asks for.
//
// The one place that maps a provider name to an implementation, so a caller
// wires a gateway without knowing which vendor answers. An unrecognised
// provider cannot reach here — config.validate rejects it at startup, where a
// typo is one loud failure instead of one per run.
// One provider, so this is a constructor rather than a choice. It survives the
// removal of the second implementation because the runtime wires itself through
// `gateway.New(gateway.FromConfig(...))` and should not learn a concrete type:
// the next provider is a change here and nowhere else.
func New(cfg Config) Gateway {
if cfg.Provider == ProviderOpenAI {
return NewOpenAI(cfg)
}
return NewAnthropic(cfg)
return NewOpenAI(cfg)
}
// routingFor resolves a tier against a table.
//
// Shared by both implementations: an unknown tier has already been normalised
// by ParseTier, so the default arm is reached only by a zero value.
// An unknown tier has already been normalised by ParseTier, so the default arm
// is reached only by a zero value.
func (c Config) routingFor(t Tier) Routing {
switch t {
case TierFast: