Production had no EMBED_PROVIDER, so every knowledge_chunk carried a null
embedding and a question only matched documents that shared its words. A person
asking about a family emergency got nothing from a document titled "shift cover
and cancellation".
Ollama rather than Voyage: internal/knowledge/embed.go calls it "the default
worth reaching for" — real semantics, no credential, no per-token cost, and no
tenant text leaving the cluster. Voyage needs an API key nobody has issued.
Bounded deliberately. The API pods share this node, so an unbounded model
server is a way to evict them; the memory limit means the kubelet kills the
embedder and nothing else. The 1Gi request is also what keeps it off the second
node, which has 1.2Gi allocatable and could not hold it.
Applied in three stages so nothing was pointed at an embedder that had not
been proven: deploy and pull the model, run reembed with the settings passed as
exec environment — 34 chunks in 11s, which proves connectivity without touching
live config — and only then patch krow-config and restart. Rolling back is
removing four keys and restarting.
Verified after: 55/55 on verify-deploy, and a question with no literal keyword
overlap with the corpus returned the relevant policy documents.
This file is the record of what was applied. It was applied by hand, which is
the same gap the README already admits for migrations — there is no deploy
pipeline, so a manifest in the repository is a description of the cluster
rather than the thing that produces it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJvibeSc1JYXjatankqM1g
The overlay had never been run against a fresh volume. Two faults, the first
hiding the second:
- postgres:16-alpine ships libssl but not the openssl CLI, so the first-boot
certificate generation exited 127 in a restart loop. It failed invisibly:
the 2>/dev/null on the openssl line swallowed sh's "not found" as well, so
`docker logs` was completely empty. openssl is now installed on the boot
that generates the certificate, inside the same guard, so a restart still
needs no network.
- the certificate was written into /var/lib/postgresql/data BEFORE initdb
ran, and initdb refuses to initialise a directory that is not empty. That
made a fresh volume unstartable regardless of the first fault. The
certificate now lives in its own volume, which keeps it persistent — the
reason it was put in the data directory — without touching the cluster's.
Separately, docker-compose.yml did not pass ANTHROPIC_API_KEY to the api
container, so a compose deployment could never register the agent run routes:
POST /agents/{id}/runs answered 404 and /version reported two endpoints fewer.
The model and embedder variables are now passed through, all defaulting to
empty so a deployment without them behaves exactly as it did.
Verified on a fresh volume: 56/56 verify-deploy checks against the resulting
stack, including a live agent run and 34 chunks embedded through Ollama.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJvibeSc1JYXjatankqM1g
CORS
cors.go never set Access-Control-Allow-Credentials, so the
cookie-authenticated API was unreadable from any cross-origin frontend:
the server answered correctly and the browser blocked the page from
reading it. Set for allowlisted origins on both the preflight and the
actual response. Three tests added.
HTTP_COOKIE_SAMESITE (lax|none|strict, default lax) is new. CORS is only
half of what a cross-origin browser call needs; SameSite is judged on
registrable domain, so a frontend on an unrelated domain gets perfect CORS
headers and still no cookie. "none" is the only value that survives that,
and validate() refuses it without the Secure flag.
The "*" rejection now explains itself: browsers refuse Allow-Origin "*"
together with credentials, so it would break every authenticated call
rather than loosen anything.
Transactional endpoints (api-contract.md 12.1)
POST /api/v1/job-applications/{id}/hire
POST /api/v1/job-postings/{id}/assignments
Replaces two client-side loops that wrote several records with no
transaction and no rollback. Each is now one endpoint and one transaction,
built over repo.Repo so org scoping, derived columns, type casts and error
translation are not re-derived. Authorization reuses the existing policy
table rather than adding a parallel one: a workflow is exactly as
privileged as the writes it performs. 13 tests, including both rollback
paths.
Bug fix in the repository layer
repo.bindValue handled int64/int/float64/string but not int32, which is
what pgx returns for a PostgreSQL `int` column. Nothing previously read a
record and wrote one of its fields elsewhere, so it never surfaced; the
hire flow does exactly that and failed with "ai_score must be a number".
Both KindInt and KindFloat now accept the widths pgx actually produces.
Deployment
infrastructure/Dockerfile.api multi-stage, cross-compiling (BUILDPLATFORM
+ GOARCH) so linux/amd64 builds from arm64 are compiled rather than
emulated. Alpine runtime, non-root uid 10001, 22.1 MB. Ships api, seed,
setpassword and migrate, plus the migrations, so a Kubernetes
initContainer can apply the schema from the same image and tag as the
API. HEALTHCHECK keys on status code, not body, so a "degraded" instance
is not pulled from rotation during a migration window.
infrastructure/docker-compose.yml migrations run to completion before the
API starts. Assumes a managed PostgreSQL; the local-db overlay adds one
with TLS enabled so APP_ENV=production is met rather than dodged.
scripts/drop_public_tables.go the one-off used to clear an unrelated
schema from krowdb on 2026-08-24, kept for the record. Build-tagged
ignore and gated on CONFIRM_DROP=yes.
Verified against PostgreSQL: 16/16 new tests pass, and the image was built,
run and exercised end to end (login, CORS preflight, authenticated reads,
transaction rollback).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CmQiGq73Uyfq7J4yR8Vxxw
