diff --git a/README.md b/README.md new file mode 100644 index 0000000..b5b1165 --- /dev/null +++ b/README.md @@ -0,0 +1,138 @@ +# Fiesta backend (`nearle`) + +The Go/Fiber API behind the Nearle Daily merchant console, the customer app, +the rider app and the in-store POS terminals. Postgres (`nearledb`) for +tenants, stores, products, stock and orders; a separate pgvector database for +the global product catalogue; Redis for POS presence; MQTT for the tills. + +This page is the map. Each section says what a thing is, how to use it, and +where the detail lives. + +## Run it + +```sh +export PATH="$PATH:$HOME/go/bin" # Go 1.24 lives there on the dev Macs +docker compose -f docker-compose.local.yml up -d # postgres :5433, pgvector :5434, redis :6379 +go run . # APP_ENV unset → .env.local, listens on :1122 +go test ./... +``` + +An empty database is not enough — startup runs migrations that assume the +live schema. `init/README.md` explains loading a schema dump first. + +Startup prints what it loaded and where it is pointed: + +``` +config: loaded .env.local +config: APP_ENV=local, listening on :1122, database nearle@localhost:5433/nearledb +scan: product search uses openai/all-minilm # or: EMBEDDING_PROVIDER not set, text-only +``` + +## Configuration + +Everything comes from environment variables, read once by `config.Load()`. + +| You want to… | Do this | +|---|---| +| Run locally | Nothing — `.env.local` is loaded by default | +| Run against production settings | `APP_ENV=production go run .` (⚠ every write is real) | +| See every variable and what it does | `.env.example` | +| Add a new setting | Add it to `config.Config` + `Load()` + `.env.example`, **and to the cluster** (`nearle-config` ConfigMap or `app-secrets` Secret in namespace `nearle`) — a variable in the file and not in the cluster is unset in production | +| Find out why it won't boot | Read the message — it lists *every* missing variable at once | + +Precedence is `real environment > .env. > .env`. The container gets +no `.env` file at all (`.dockerignore`); the `Dockerfile` sets +`APP_ENV=production` and the values come from Kubernetes. + +Full detail, including the committed-credentials situation: +**`docs/ENVIRONMENT.md`**. + +## Layout + +``` +main.go boot: config → databases → migrations → routes → MQTT → listen +config/ env-file loading, typed Config, validation +db/ Postgres (nearledb + catalogue), Redis, S3 image store +facade/ wires repositories → services → controllers (add new modules here) +routes/ one file per module; /live/api/v1/web/... (console) and /v1/mob/... (apps) +controllers/ HTTP in, HTTP out — parse, call the service, shape the envelope +services/ the rules; no SQL, no HTTP +repositories/ the SQL; nothing else +models/ request/response and table shapes +messaging/ MQTT ingest from tills, console live stream +utils/ small shared helpers (tokens, geo, embeddings, geocoding) +docs/ integration specs for the frontends and handoff notes +scratch/ one-off read/verify tools run with `go run ./scratch/` +init/ schema/seed for the local database +``` + +Every response uses the same envelope: +`{ "code": 200, "status": true, "message": "…", "details": … }`. +Business outcomes ("out of stock", "not registered") are 200s with a reason +in the body; HTTP errors mean the request could not be served at all. + +## Adding an endpoint + +1. **Model** the request/response in `models/`. +2. **Repository** method(s) in `repositories/` — SQL only, take a + `context.Context`, return `error`. +3. **Service** in `services/` — the rules, with sentinel errors + (`ErrXxxBadRequest`, `ErrXxxNotFound`) the controller can map to statuses. +4. **Controller** in `controllers/` — `BodyParser`/`Query`, call the service, + map sentinel errors to 400/404/503, everything else to 500. +5. **Routes** file in `routes/`, registered in `routes/routes.go`. +6. **Wire** it in `facade/container.go`. +7. **Test** the service with a fake repository (see `services/scan_test.go` + for the pattern) — no database needed. + +`services/scanService.go` + `controllers/scanController.go` are a complete, +current example of all seven. + +## Features with their own docs + +| Feature | For | Doc | +|---|---|---| +| Environment & deployment | everyone | `docs/ENVIRONMENT.md` | +| Scan-to-order (camera → product → nearest store with stock) | mobile app | `docs/SCAN_TO_ORDER.md` | +| Catalogue import into a store | console | `docs/CATALOGUE_IMPORT_INTEGRATION.md` | +| POS terminal ingest, login, API | POS / tills | `docs/POS_*.md` | +| Access-control audit and what is still open | everyone | `docs/SECURITY_HANDOFF.md` | + +## Things to know before you get surprised + +- **There is no auth layer.** `customerid` / `tenantid` in a request are + trusted. `docs/SECURITY_HANDOFF.md` §1 is the standing issue. +- **Login errors mean what they say.** `409 Invalid Email` = the query ran + and matched nobody. `500 Login is temporarily unavailable` = the database + could not answer (it used to be reported as Invalid Email; see + `services/userService.go` `lookupLogin`). +- **Stock is a ledger.** Live stock is always `SUM(in) − SUM(out)` of + `productstocks` at an outlet, never a stored number. Filter on the same + expression you display (`services/productVisibility.go` explains why). +- **The catalogue is a different database** and must never be reached + through the `nearledb` handle. Its per-brand tables are discovered from + `information_schema`; brands appear and columns vary. +- **Catalogue ids are not stable** across re-scrapes; `imageid` is the + durable key (`models.Products.Imageid`). +- **Migrations run on boot** and are guarded by `IF NOT EXISTS` / schema + checks, not a version table. Read the comments in `main.go` before adding + one — several have bitten before. +- **One MQTT client id per replica.** A second connection with the same id + evicts the first. Never run a local process with the production + `MQTT_URL`. +- **`scratch/` tools read production** when run with `.env.production`. + They are read-only by construction; keep them that way. + +## Operations cheat-sheet (Kubernetes, namespace `nearle`) + +```sh +kubectl get pods -n nearle # fiesta-0/1/2 (StatefulSet) +kubectl logs -n nearle fiesta-0 | grep -E 'config:|scan:|pos:' +kubectl exec -n nearle fiesta-0 -- env | grep EMBEDDING_ +kubectl set env statefulset/fiesta -n nearle KEY=value # adds a var and rolls the pods +kubectl rollout status statefulset/fiesta -n nearle +``` + +The embedding model behind scan-to-order is served by the cluster's Ollama +(`ollama.krow.svc.cluster.local:11434`); `docs/SCAN_TO_ORDER.md` has the +exact settings and why that model. diff --git a/docs/SCAN_TO_ORDER.md b/docs/SCAN_TO_ORDER.md index 6508f89..ba74af6 100644 --- a/docs/SCAN_TO_ORDER.md +++ b/docs/SCAN_TO_ORDER.md @@ -186,3 +186,76 @@ Same `ScanStore` shape as inside `stores[]` above, without options. Redis-backed short hold slots in at `Confirm` without changing the API. - **Identity** is the `customerid` in the body, like every other mobile endpoint here — there is no auth layer yet (see `SECURITY_HANDOFF.md`). + +## For backend developers + +### Where the code is + +| File | Holds | +|---|---| +| `models/scan.go` | request/response shapes (`ScanLookupRequest`, `ScanStoreOffer`, `ScanOption`, …) | +| `repositories/scanRepository.go` | all SQL: registered stores, live options, vector + text search, the two-tier cache | +| `services/scanService.go` | the pipeline: parallel reads, scoring, family grouping, ranking, confirm fallback | +| `controllers/scanController.go` | the three handlers and the error → status mapping | +| `routes/scanroutes.go` | `/v1/mob/scan/*` | +| `utils/embedding.go` | `Embedder` interface, OpenAI-compatible and Gemini clients | +| `utils/geo.go` | coordinate parsing, haversine, opening hours, label tokenising | +| `config/config.go` | `EmbeddingConfig` and its validation | +| `scratch/cataloguedims` | read-only check of the catalogue's embedding width / fill | + +### Try it locally + +```sh +go run . # with the local compose stack; EMBEDDING_* unset → text-only, still works +curl -s localhost:1122/live/api/v1/mob/scan/lookup -H 'Content-Type: application/json' \ + -d '{"customerid":1,"label":"Milk Bikis","latitude":11.03,"longitude":77.03}' | jq .details +``` + +To exercise the vector path locally, run Ollama on your Mac +(`ollama pull all-minilm`) and set `EMBEDDING_PROVIDER=openai`, +`EMBEDDING_BASE_URL=http://localhost:11434/v1`, `EMBEDDING_MODEL=all-minilm`, +`EMBEDDING_API_KEY=ollama`, `EMBEDDING_DIMENSIONS=384` in `.env.local`. The +local catalogue must carry vectors from the same model for results to mean +anything; a schema-only dump does not. + +### Tests + +`go test ./services -run 'Lookup|Confirm|Stores|CatalogueFamily'` drives +the whole pipeline through a fake repository (`services/scan_test.go`); no +database. `go test ./utils` covers both HTTP clients against `httptest` +servers, and the geo helpers. Add a case to `scan_test.go`'s fixture when +you change ranking — it is the spec. + +### Knobs (constants in `scanService.go`) + +| Constant | Default | Effect | +|---|---|---| +| `scanLookupTimeout` | 5 s | whole lookup, including the model call | +| `scanCatalogueTopK` | 15 | rows taken from each brand table and from the merge | +| `scanMinScore` | 0.30 | below this the best hit is not shown as a match | +| `embedTimeout` (`utils/embedding.go`) | 4 s | one model call | +| `scanVectorTTL` / `scanHitsTTL` (`scanRepository.go`) | 7 d / 30 min | cache lifetimes | + +Scores: vector = `1 − cosine distance`; text = 0.95 for the whole label +inside the name, else `0.8 × (label words found / label words)`; combined = +`max(vector, text) + 0.10` when both hit, capped at 1. + +### Changing the embedding model + +1. The catalogue team re-embeds `search_query` with the new model. +2. Serve it (Ollama pull, or a hosted key). +3. Change `EMBEDDING_MODEL` / `EMBEDDING_DIMENSIONS` (and provider/URL if + needed) in the cluster; roll the pods. +4. Flush the hit cache if you cannot wait 30 min: keys are + `scan:hits:v1:*` and `scan:emb:v1:*` in Redis (they are also keyed by + model name, so old entries simply stop being read). + +Nothing in Go changes. A width mismatch fails the first search with an error +naming both numbers. + +### Adding a provider + +Implement `utils.Embedder` (`Embed(ctx, text) ([]float32, error)` and +`Model() string`), add a case to `NewEmbedder`, and add the provider name to +the allow-list in `config.validate`. Keep the HTTP client timeout: the +customer is holding a phone.