The platform could only talk to one vendor. Moving off Claude — for cost, or
because a client asks for Gemini — meant a rewrite behind an interface that
already had exactly the right shape and one implementation.
`openai` is not only OpenAI. Groq, Gemini's compatibility endpoint, OpenRouter,
Together, vLLM and a local Ollama all serve the chat-completions shape, so one
implementation reaches all of them and the difference between them is a base
URL and three model ids. That is why this is one file and not a package per
vendor.
`routing.go` had the vendor baked into the routing table every provider has to
read: effort was `anthropic.OutputConfigEffort`. Nothing was wrong with that
while there was one implementation; it became wrong the moment there were two,
because the OpenAI path would have had to import the Anthropic SDK to learn how
hard to think. Effort is now the platform's own three-value vocabulary and each
implementation maps it onto whatever its API calls the same idea.
THE ACCOUNTING DIFFERS BETWEEN THE TWO WIRES, and getting it wrong would have
been invisible. OpenAI reports prompt_tokens INCLUSIVE of the cached prefix;
Anthropic reports input tokens EXCLUSIVE of it and carries the cache
separately. Usage.Total() adds all four fields, so copying both numbers across
verbatim bills the cached prefix twice — worst on long conversations, which is
exactly where I3's budget matters most. The run would still answer; it would
just hit BudgetExceeded early, for no visible reason. normalise() subtracts,
and there is a test named after it.
Streamed tool calls are keyed by their wire index, not appended in arrival
order. Providers interleave the fragments of parallel calls, so appending
splices one call's arguments onto another's — and the result is usually two
calls that are each valid JSON and both wrong, which means the tools run with
inputs the model never chose and nothing errors. Mutation-checked: ignoring the
index produces `{"day"{"week":"friday"}:"next"}` and the test catches it.
Three configuration mistakes are refused at startup rather than at runtime:
- MODEL_BASE_URL without MODEL_PROVIDER=openai. The anthropic path has one
endpoint and ignores the field, so this is a deployment that believes it
switched providers and did not — every run still goes to Anthropic and is
still billed there, with nothing in the logs to say so. Cost is the whole
reason this change exists, and that is the one mistake that silently
defeats it.
- An unrecognised MODEL_PROVIDER, once at boot instead of once per run.
- A production deployment with no credential — except against localhost,
which needs none, and demanding one would make the free local path
impossible to configure.
reasoning_effort is opt-in via MODEL_REASONING_EFFORT. Reasoning models accept
it; most others reject the entire request with a 400 rather than ignoring an
unknown key, so every deployment would have had to opt out instead.
`make eval-live` now reads the same environment the service does and logs which
provider answered, because a suite that cannot say which model produced a
result is a suite whose result cannot be compared with another run's. That is
the point of this change: §12 leaves model hosting open, and this makes the
decision cheap to reverse and possible to settle on evidence. Weigh the I7 case
heaviest — a cheaper model that follows the planted injection is a security
regression, not a saving.
Default behaviour is unchanged: MODEL_PROVIDER unset means anthropic, and
ANTHROPIC_API_KEY still works, so no existing deployment needs an edit.
NOT verified against a live provider — no credential was available on this
machine. Tested against a fake endpoint covering both paths, and the three
guarantees above are mutation-checked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJvibeSc1JYXjatankqM1g
174 lines
7.8 KiB
YAML
174 lines
7.8 KiB
YAML
# ============================================================================
|
|
# Krow API — production stack
|
|
#
|
|
# cd infrastructure
|
|
# cp .env.docker.example .env
|
|
# docker compose up -d --build
|
|
#
|
|
# This file assumes a MANAGED PostgreSQL (RDS, Cloud SQL, Neon, a DBA's
|
|
# cluster) reachable from wherever these containers run. That is deliberate:
|
|
# a production database wants backups, point-in-time recovery, failover and a
|
|
# patch schedule, none of which a container in this file provides.
|
|
#
|
|
# To run PostgreSQL alongside the API — for staging, or to test the stack
|
|
# end-to-end on one host — layer the override on top:
|
|
#
|
|
# docker compose -f docker-compose.yml -f docker-compose.local-db.yml up -d
|
|
#
|
|
# ── ORDER OF OPERATIONS ─────────────────────────────────────────────────────
|
|
#
|
|
# `migrate` runs to completion BEFORE `api` starts, and `api` will not start if
|
|
# it fails. That is the expand/migrate/contract discipline from README.md
|
|
# expressed in the dependency graph rather than in a runbook: schema first,
|
|
# binary second, and every migration backwards-compatible with the version
|
|
# still running.
|
|
#
|
|
# ── TLS IS NOT OPTIONAL HERE ────────────────────────────────────────────────
|
|
#
|
|
# internal/config rejects DATABASE_SSLMODE=disable when APP_ENV=production, so
|
|
# this stack will refuse to start against a database that cannot do TLS. That
|
|
# is the intended behaviour and not something to work around by dropping to
|
|
# APP_ENV=staging — a staging flag on a production deployment is a lie the next
|
|
# person has to discover.
|
|
# ============================================================================
|
|
|
|
name: krow
|
|
|
|
# The database settings the API and the migration runner must agree on.
|
|
x-database-env: &database-env
|
|
DATABASE_HOST: ${DATABASE_HOST:?DATABASE_HOST is required}
|
|
DATABASE_PORT: ${DATABASE_PORT:-5432}
|
|
DATABASE_NAME: ${DATABASE_NAME:?DATABASE_NAME is required}
|
|
DATABASE_USER: ${DATABASE_USER:?DATABASE_USER is required}
|
|
DATABASE_PASSWORD: ${DATABASE_PASSWORD:?DATABASE_PASSWORD is required}
|
|
DATABASE_SCHEMA: ${DATABASE_SCHEMA:-public}
|
|
DATABASE_SSLMODE: ${DATABASE_SSLMODE:-require}
|
|
|
|
x-logging: &logging
|
|
driver: json-file
|
|
options:
|
|
max-size: "10m"
|
|
max-file: "5"
|
|
|
|
services:
|
|
# ── Schema ────────────────────────────────────────────────────────────────
|
|
# One-shot. Exits 0 when the schema is current, including when there was
|
|
# nothing to apply. `api` waits on that exit code.
|
|
#
|
|
# DATABASE_URL is a single pre-built string rather than assembled from the
|
|
# parts above because golang-migrate takes a URL, and a password containing
|
|
# @ : / ? or # must be percent-encoded in one. Compose cannot encode it, so
|
|
# the encoding is the operator's job — see .env.docker.example.
|
|
migrate:
|
|
image: migrate/migrate:v4.19.0
|
|
container_name: krow-migrate
|
|
volumes:
|
|
- ../migrations:/migrations:ro
|
|
command:
|
|
- -path=/migrations
|
|
- -database=${DATABASE_URL:?DATABASE_URL is required — see .env.docker.example}
|
|
- up
|
|
restart: "no"
|
|
logging: *logging
|
|
networks: [krow]
|
|
|
|
# ── API ───────────────────────────────────────────────────────────────────
|
|
api:
|
|
build:
|
|
context: ..
|
|
dockerfile: infrastructure/Dockerfile.api
|
|
args:
|
|
VERSION: ${VERSION:-dev}
|
|
image: ${IMAGE:-doormile/krowbackend:latest}
|
|
container_name: krow-api
|
|
depends_on:
|
|
migrate:
|
|
condition: service_completed_successfully
|
|
environment:
|
|
<<: *database-env
|
|
APP_ENV: ${APP_ENV:-production}
|
|
LOG_LEVEL: ${LOG_LEVEL:-info}
|
|
# The agent run routes are not registered without this, so a deployment
|
|
# without it answers 404 on /agents/{id}/runs and reports three fewer
|
|
# endpoints on /version. Empty by default: absent is a working API
|
|
# without Owliver, which is a legitimate way to run this.
|
|
# Provider selection. Empty MODEL_PROVIDER means anthropic, so a stack
|
|
# that predates the second provider comes up exactly as it did.
|
|
MODEL_PROVIDER: ${MODEL_PROVIDER:-}
|
|
MODEL_BASE_URL: ${MODEL_BASE_URL:-}
|
|
MODEL_API_KEY: ${MODEL_API_KEY:-}
|
|
MODEL_REASONING_EFFORT: ${MODEL_REASONING_EFFORT:-}
|
|
ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
|
|
MODEL_FAST: ${MODEL_FAST:-}
|
|
MODEL_BALANCED: ${MODEL_BALANCED:-}
|
|
MODEL_DEEP: ${MODEL_DEEP:-}
|
|
# voyage | ollama | lexical. Unset means ingest stores chunks that are
|
|
# keyword-searchable only — see the ingest output.
|
|
EMBED_PROVIDER: ${EMBED_PROVIDER:-}
|
|
EMBED_MODEL: ${EMBED_MODEL:-}
|
|
EMBED_DIMENSIONS: ${EMBED_DIMENSIONS:-}
|
|
# Ollama runs on the HOST, so "localhost" here would be this container.
|
|
# host.docker.internal is what reaches the host from inside it.
|
|
EMBED_BASE_URL: ${EMBED_BASE_URL:-}
|
|
VOYAGE_API_KEY: ${VOYAGE_API_KEY:-}
|
|
# 0.0.0.0 so the published port actually reaches the process. The binary
|
|
# defaults to 127.0.0.1, which inside a container means "nobody".
|
|
HTTP_HOST: 0.0.0.0
|
|
HTTP_PORT: "8080"
|
|
HTTP_READ_TIMEOUT: ${HTTP_READ_TIMEOUT:-15s}
|
|
HTTP_WRITE_TIMEOUT: ${HTTP_WRITE_TIMEOUT:-30s}
|
|
HTTP_IDLE_TIMEOUT: ${HTTP_IDLE_TIMEOUT:-60s}
|
|
HTTP_SHUTDOWN_TIMEOUT: ${HTTP_SHUTDOWN_TIMEOUT:-10s}
|
|
# Browser origins allowed to call this API. Empty means same-origin only,
|
|
# and the CORS middleware is then not installed at all. Cookies are the
|
|
# auth mechanism, so any origin listed here can hold a session.
|
|
#
|
|
# "*" is refused at startup: browsers reject Allow-Origin "*" together
|
|
# with credentials, so it would break authenticated calls rather than
|
|
# loosen anything.
|
|
HTTP_CORS_ORIGINS: ${HTTP_CORS_ORIGINS:-}
|
|
# lax | none | strict, or unset to let the server choose.
|
|
#
|
|
# "none" is required when the frontend is on a different registrable
|
|
# DOMAIN from the API — otherwise the browser withholds the cookie
|
|
# however correct the CORS headers are. A different ORIGIN on the same
|
|
# domain (platform.krowforce.com → mcp.krowforce.com) needs CORS but not
|
|
# this: a Lax cookie already travels between them, and "none" would give
|
|
# up the only CSRF protection this API has.
|
|
#
|
|
# Unset, the server derives it from HTTP_CORS_ORIGINS. The default below
|
|
# is deliberate: a compose deployment keeps Lax unless told otherwise.
|
|
HTTP_COOKIE_SAMESITE: ${HTTP_COOKIE_SAMESITE:-lax}
|
|
DATABASE_MAX_OPEN_CONNS: ${DATABASE_MAX_OPEN_CONNS:-25}
|
|
DATABASE_MIN_IDLE_CONNS: ${DATABASE_MIN_IDLE_CONNS:-2}
|
|
DATABASE_CONN_MAX_LIFETIME: ${DATABASE_CONN_MAX_LIFETIME:-30m}
|
|
DATABASE_CONNECT_TIMEOUT: ${DATABASE_CONNECT_TIMEOUT:-10s}
|
|
DATABASE_STATEMENT_TIMEOUT: ${DATABASE_STATEMENT_TIMEOUT:-10s}
|
|
ports:
|
|
# Bound to loopback by default. A reverse proxy in front of this is what
|
|
# terminates TLS; publishing 0.0.0.0:8080 would put a plain-HTTP service
|
|
# carrying session cookies straight onto the network.
|
|
- "${API_BIND:-127.0.0.1}:${API_PORT:-8080}:8080"
|
|
restart: unless-stopped
|
|
stop_grace_period: 30s
|
|
read_only: true
|
|
security_opt:
|
|
- no-new-privileges:true
|
|
cap_drop:
|
|
- ALL
|
|
tmpfs:
|
|
- /tmp:size=32m
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
cpus: "${API_CPU_LIMIT:-2}"
|
|
memory: ${API_MEMORY_LIMIT:-512M}
|
|
reservations:
|
|
memory: 128M
|
|
logging: *logging
|
|
networks: [krow]
|
|
|
|
networks:
|
|
krow:
|
|
driver: bridge
|