A free tier's ceiling is tokens per MINUTE, and one run can exceed a whole
minute's worth by itself: a three-call run measured 12,123 against a ceiling of
8,000. withRetry already fires three times and all three are refused, because
1.6 seconds of backoff does not buy back a minute's budget. The run ends
GatewayFailure and somebody reads "the model did not answer".
Retrying harder cannot fix a ceiling. Asking somebody else can — the ceilings
are per provider, so a second key is a second budget. Groq, Cerebras, Gemini,
Mistral and OpenRouter all serve the same chat-completions shape, which is why
this is a list of Configs and not a second implementation.
Configured as MODEL_FALLBACK_<n>_BASE_URL / _API_KEY / _FAST / _BALANCED /
_DEEP, numbered because five fields times three providers packed into one
delimited string is a parser nobody can read under pressure. Empty is the
ordinary case and returns the primary unwrapped, so a single-provider
deployment carries no wrapper and behaves exactly as before.
Failover is NOT unconditional, and the two guards are the design:
- Only a transient failure moves. Error.Retryable() already draws that line
for retries and it is the same line here. A 401 is this deployment's own
credential and a 400 is a malformed request; both fail identically at every
vendor, so trying three turns one visible fault into three invisible ones.
- Only an unpinned conversation moves. ToolCall.Extra carries provider
metadata echoed back verbatim — Gemini 3's thought signature — and a vendor
rejects a follow-up that drops its own. A conversation carrying any belongs
to whoever started it, so failover is available on the first model call,
which is where a rate limit usually lands anyway.
Streaming falls over only before the first fragment: once text is in the
reader's window, a second provider would continue that sentence in a different
voice.
What this does not do, since the gap is where the next bug lives: it does not
make a run cheaper, does not raise any one ceiling, and does not help when every
provider is exhausted at once. It turns one busy provider into a slower answer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Aravind
ace4db8db6