README.md

@@ -0,0 +1,138 @@
+# Fiesta backend (`nearle`)
+
+The Go/Fiber API behind the Nearle Daily merchant console, the customer app,
+the rider app and the in-store POS terminals. Postgres (`nearledb`) for
+tenants, stores, products, stock and orders; a separate pgvector database for
+the global product catalogue; Redis for POS presence; MQTT for the tills.
+
+This page is the map. Each section says what a thing is, how to use it, and
+where the detail lives.
+
+## Run it
+
+```sh
+export PATH="$PATH:$HOME/go/bin"                  # Go 1.24 lives there on the dev Macs
+docker compose -f docker-compose.local.yml up -d  # postgres :5433, pgvector :5434, redis :6379
+go run .                                          # APP_ENV unset → .env.local, listens on :1122
+go test ./...
+```
+
+An empty database is not enough — startup runs migrations that assume the
+live schema. `init/README.md` explains loading a schema dump first.
+
+Startup prints what it loaded and where it is pointed:
+
+```
+config: loaded .env.local
+config: APP_ENV=local, listening on :1122, database nearle@localhost:5433/nearledb
+scan: product search uses openai/all-minilm     # or: EMBEDDING_PROVIDER not set, text-only
+```
+
+## Configuration
+
+Everything comes from environment variables, read once by `config.Load()`.
+
+| You want to… | Do this |
+|---|---|
+| Run locally | Nothing — `.env.local` is loaded by default |
+| Run against production settings | `APP_ENV=production go run .` (⚠ every write is real) |
+| See every variable and what it does | `.env.example` |
+| Add a new setting | Add it to `config.Config` + `Load()` + `.env.example`, **and to the cluster** (`nearle-config` ConfigMap or `app-secrets` Secret in namespace `nearle`) — a variable in the file and not in the cluster is unset in production |
+| Find out why it won't boot | Read the message — it lists *every* missing variable at once |
+
+Precedence is `real environment > .env.<APP_ENV> > .env`. The container gets
+no `.env` file at all (`.dockerignore`); the `Dockerfile` sets
+`APP_ENV=production` and the values come from Kubernetes.
+
+Full detail, including the committed-credentials situation:
+**`docs/ENVIRONMENT.md`**.
+
+## Layout
+
+```
+main.go            boot: config → databases → migrations → routes → MQTT → listen
+config/            env-file loading, typed Config, validation
+db/                Postgres (nearledb + catalogue), Redis, S3 image store
+facade/            wires repositories → services → controllers (add new modules here)
+routes/            one file per module; /live/api/v1/web/... (console) and /v1/mob/... (apps)
+controllers/       HTTP in, HTTP out — parse, call the service, shape the envelope
+services/          the rules; no SQL, no HTTP
+repositories/      the SQL; nothing else
+models/            request/response and table shapes
+messaging/         MQTT ingest from tills, console live stream
+utils/             small shared helpers (tokens, geo, embeddings, geocoding)
+docs/              integration specs for the frontends and handoff notes
+scratch/           one-off read/verify tools run with `go run ./scratch/<name>`
+init/              schema/seed for the local database
+```
+
+Every response uses the same envelope:
+`{ "code": 200, "status": true, "message": "…", "details": … }`.
+Business outcomes ("out of stock", "not registered") are 200s with a reason
+in the body; HTTP errors mean the request could not be served at all.
+
+## Adding an endpoint
+
+1. **Model** the request/response in `models/`.
+2. **Repository** method(s) in `repositories/` — SQL only, take a
+   `context.Context`, return `error`.
+3. **Service** in `services/` — the rules, with sentinel errors
+   (`ErrXxxBadRequest`, `ErrXxxNotFound`) the controller can map to statuses.
+4. **Controller** in `controllers/` — `BodyParser`/`Query`, call the service,
+   map sentinel errors to 400/404/503, everything else to 500.
+5. **Routes** file in `routes/`, registered in `routes/routes.go`.
+6. **Wire** it in `facade/container.go`.
+7. **Test** the service with a fake repository (see `services/scan_test.go`
+   for the pattern) — no database needed.
+
+`services/scanService.go` + `controllers/scanController.go` are a complete,
+current example of all seven.
+
+## Features with their own docs
+
+| Feature | For | Doc |
+|---|---|---|
+| Environment & deployment | everyone | `docs/ENVIRONMENT.md` |
+| Scan-to-order (camera → product → nearest store with stock) | mobile app | `docs/SCAN_TO_ORDER.md` |
+| Catalogue import into a store | console | `docs/CATALOGUE_IMPORT_INTEGRATION.md` |
+| POS terminal ingest, login, API | POS / tills | `docs/POS_*.md` |
+| Access-control audit and what is still open | everyone | `docs/SECURITY_HANDOFF.md` |
+
+## Things to know before you get surprised
+
+- **There is no auth layer.** `customerid` / `tenantid` in a request are
+  trusted. `docs/SECURITY_HANDOFF.md` §1 is the standing issue.
+- **Login errors mean what they say.** `409 Invalid Email` = the query ran
+  and matched nobody. `500 Login is temporarily unavailable` = the database
+  could not answer (it used to be reported as Invalid Email; see
+  `services/userService.go` `lookupLogin`).
+- **Stock is a ledger.** Live stock is always `SUM(in) − SUM(out)` of
+  `productstocks` at an outlet, never a stored number. Filter on the same
+  expression you display (`services/productVisibility.go` explains why).
+- **The catalogue is a different database** and must never be reached
+  through the `nearledb` handle. Its per-brand tables are discovered from
+  `information_schema`; brands appear and columns vary.
+- **Catalogue ids are not stable** across re-scrapes; `imageid` is the
+  durable key (`models.Products.Imageid`).
+- **Migrations run on boot** and are guarded by `IF NOT EXISTS` / schema
+  checks, not a version table. Read the comments in `main.go` before adding
+  one — several have bitten before.
+- **One MQTT client id per replica.** A second connection with the same id
+  evicts the first. Never run a local process with the production
+  `MQTT_URL`.
+- **`scratch/` tools read production** when run with `.env.production`.
+  They are read-only by construction; keep them that way.
+
+## Operations cheat-sheet (Kubernetes, namespace `nearle`)
+
+```sh
+kubectl get pods -n nearle                                  # fiesta-0/1/2 (StatefulSet)
+kubectl logs -n nearle fiesta-0 | grep -E 'config:|scan:|pos:'
+kubectl exec -n nearle fiesta-0 -- env | grep EMBEDDING_
+kubectl set env statefulset/fiesta -n nearle KEY=value       # adds a var and rolls the pods
+kubectl rollout status statefulset/fiesta -n nearle
+```
+
+The embedding model behind scan-to-order is served by the cluster's Ollama
+(`ollama.krow.svc.cluster.local:11434`); `docs/SCAN_TO_ORDER.md` has the
+exact settings and why that model.

docs/SCAN_TO_ORDER.md

@@ -186,3 +186,76 @@ Same `ScanStore` shape as inside `stores[]` above, without options.
   Redis-backed short hold slots in at `Confirm` without changing the API.
 - **Identity** is the `customerid` in the body, like every other mobile
   endpoint here — there is no auth layer yet (see `SECURITY_HANDOFF.md`).
+
+## For backend developers
+
+### Where the code is
+
+| File | Holds |
+|---|---|
+| `models/scan.go` | request/response shapes (`ScanLookupRequest`, `ScanStoreOffer`, `ScanOption`, …) |
+| `repositories/scanRepository.go` | all SQL: registered stores, live options, vector + text search, the two-tier cache |
+| `services/scanService.go` | the pipeline: parallel reads, scoring, family grouping, ranking, confirm fallback |
+| `controllers/scanController.go` | the three handlers and the error → status mapping |
+| `routes/scanroutes.go` | `/v1/mob/scan/*` |
+| `utils/embedding.go` | `Embedder` interface, OpenAI-compatible and Gemini clients |
+| `utils/geo.go` | coordinate parsing, haversine, opening hours, label tokenising |
+| `config/config.go` | `EmbeddingConfig` and its validation |
+| `scratch/cataloguedims` | read-only check of the catalogue's embedding width / fill |
+
+### Try it locally
+
+```sh
+go run .    # with the local compose stack; EMBEDDING_* unset → text-only, still works
+curl -s localhost:1122/live/api/v1/mob/scan/lookup -H 'Content-Type: application/json' \
+  -d '{"customerid":1,"label":"Milk Bikis","latitude":11.03,"longitude":77.03}' | jq .details
+```
+
+To exercise the vector path locally, run Ollama on your Mac
+(`ollama pull all-minilm`) and set `EMBEDDING_PROVIDER=openai`,
+`EMBEDDING_BASE_URL=http://localhost:11434/v1`, `EMBEDDING_MODEL=all-minilm`,
+`EMBEDDING_API_KEY=ollama`, `EMBEDDING_DIMENSIONS=384` in `.env.local`. The
+local catalogue must carry vectors from the same model for results to mean
+anything; a schema-only dump does not.
+
+### Tests
+
+`go test ./services -run 'Lookup|Confirm|Stores|CatalogueFamily'` drives
+the whole pipeline through a fake repository (`services/scan_test.go`); no
+database. `go test ./utils` covers both HTTP clients against `httptest`
+servers, and the geo helpers. Add a case to `scan_test.go`'s fixture when
+you change ranking — it is the spec.
+
+### Knobs (constants in `scanService.go`)
+
+| Constant | Default | Effect |
+|---|---|---|
+| `scanLookupTimeout` | 5 s | whole lookup, including the model call |
+| `scanCatalogueTopK` | 15 | rows taken from each brand table and from the merge |
+| `scanMinScore` | 0.30 | below this the best hit is not shown as a match |
+| `embedTimeout` (`utils/embedding.go`) | 4 s | one model call |
+| `scanVectorTTL` / `scanHitsTTL` (`scanRepository.go`) | 7 d / 30 min | cache lifetimes |
+
+Scores: vector = `1 − cosine distance`; text = 0.95 for the whole label
+inside the name, else `0.8 × (label words found / label words)`; combined =
+`max(vector, text) + 0.10` when both hit, capped at 1.
+
+### Changing the embedding model
+
+1. The catalogue team re-embeds `search_query` with the new model.
+2. Serve it (Ollama pull, or a hosted key).
+3. Change `EMBEDDING_MODEL` / `EMBEDDING_DIMENSIONS` (and provider/URL if
+   needed) in the cluster; roll the pods.
+4. Flush the hit cache if you cannot wait 30 min: keys are
+   `scan:hits:v1:*` and `scan:emb:v1:*` in Redis (they are also keyed by
+   model name, so old entries simply stop being read).
+
+Nothing in Go changes. A width mismatch fails the first search with an error
+naming both numbers.
+
+### Adding a provider
+
+Implement `utils.Embedder` (`Embed(ctx, text) ([]float32, error)` and
+`Model() string`), add a case to `NewEmbedder`, and add the provider name to
+the allow-list in `config.validate`. Keep the HTTP client timeout: the
+customer is holding a phone.