README: a map of the service for new developers
What it is, how to run it, how configuration works, the module layout, the seven steps to add an endpoint, the standing surprises, and a Kubernetes cheat-sheet — each pointing at the detailed doc. Plus a backend-developer section in SCAN_TO_ORDER.md: file map, local try-out, tests, tuning knobs, and how to change the embedding model or add a provider. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
138
README.md
Normal file
138
README.md
Normal file
@@ -0,0 +1,138 @@
|
||||
# Fiesta backend (`nearle`)
|
||||
|
||||
The Go/Fiber API behind the Nearle Daily merchant console, the customer app,
|
||||
the rider app and the in-store POS terminals. Postgres (`nearledb`) for
|
||||
tenants, stores, products, stock and orders; a separate pgvector database for
|
||||
the global product catalogue; Redis for POS presence; MQTT for the tills.
|
||||
|
||||
This page is the map. Each section says what a thing is, how to use it, and
|
||||
where the detail lives.
|
||||
|
||||
## Run it
|
||||
|
||||
```sh
|
||||
export PATH="$PATH:$HOME/go/bin" # Go 1.24 lives there on the dev Macs
|
||||
docker compose -f docker-compose.local.yml up -d # postgres :5433, pgvector :5434, redis :6379
|
||||
go run . # APP_ENV unset → .env.local, listens on :1122
|
||||
go test ./...
|
||||
```
|
||||
|
||||
An empty database is not enough — startup runs migrations that assume the
|
||||
live schema. `init/README.md` explains loading a schema dump first.
|
||||
|
||||
Startup prints what it loaded and where it is pointed:
|
||||
|
||||
```
|
||||
config: loaded .env.local
|
||||
config: APP_ENV=local, listening on :1122, database nearle@localhost:5433/nearledb
|
||||
scan: product search uses openai/all-minilm # or: EMBEDDING_PROVIDER not set, text-only
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Everything comes from environment variables, read once by `config.Load()`.
|
||||
|
||||
| You want to… | Do this |
|
||||
|---|---|
|
||||
| Run locally | Nothing — `.env.local` is loaded by default |
|
||||
| Run against production settings | `APP_ENV=production go run .` (⚠ every write is real) |
|
||||
| See every variable and what it does | `.env.example` |
|
||||
| Add a new setting | Add it to `config.Config` + `Load()` + `.env.example`, **and to the cluster** (`nearle-config` ConfigMap or `app-secrets` Secret in namespace `nearle`) — a variable in the file and not in the cluster is unset in production |
|
||||
| Find out why it won't boot | Read the message — it lists *every* missing variable at once |
|
||||
|
||||
Precedence is `real environment > .env.<APP_ENV> > .env`. The container gets
|
||||
no `.env` file at all (`.dockerignore`); the `Dockerfile` sets
|
||||
`APP_ENV=production` and the values come from Kubernetes.
|
||||
|
||||
Full detail, including the committed-credentials situation:
|
||||
**`docs/ENVIRONMENT.md`**.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
main.go boot: config → databases → migrations → routes → MQTT → listen
|
||||
config/ env-file loading, typed Config, validation
|
||||
db/ Postgres (nearledb + catalogue), Redis, S3 image store
|
||||
facade/ wires repositories → services → controllers (add new modules here)
|
||||
routes/ one file per module; /live/api/v1/web/... (console) and /v1/mob/... (apps)
|
||||
controllers/ HTTP in, HTTP out — parse, call the service, shape the envelope
|
||||
services/ the rules; no SQL, no HTTP
|
||||
repositories/ the SQL; nothing else
|
||||
models/ request/response and table shapes
|
||||
messaging/ MQTT ingest from tills, console live stream
|
||||
utils/ small shared helpers (tokens, geo, embeddings, geocoding)
|
||||
docs/ integration specs for the frontends and handoff notes
|
||||
scratch/ one-off read/verify tools run with `go run ./scratch/<name>`
|
||||
init/ schema/seed for the local database
|
||||
```
|
||||
|
||||
Every response uses the same envelope:
|
||||
`{ "code": 200, "status": true, "message": "…", "details": … }`.
|
||||
Business outcomes ("out of stock", "not registered") are 200s with a reason
|
||||
in the body; HTTP errors mean the request could not be served at all.
|
||||
|
||||
## Adding an endpoint
|
||||
|
||||
1. **Model** the request/response in `models/`.
|
||||
2. **Repository** method(s) in `repositories/` — SQL only, take a
|
||||
`context.Context`, return `error`.
|
||||
3. **Service** in `services/` — the rules, with sentinel errors
|
||||
(`ErrXxxBadRequest`, `ErrXxxNotFound`) the controller can map to statuses.
|
||||
4. **Controller** in `controllers/` — `BodyParser`/`Query`, call the service,
|
||||
map sentinel errors to 400/404/503, everything else to 500.
|
||||
5. **Routes** file in `routes/`, registered in `routes/routes.go`.
|
||||
6. **Wire** it in `facade/container.go`.
|
||||
7. **Test** the service with a fake repository (see `services/scan_test.go`
|
||||
for the pattern) — no database needed.
|
||||
|
||||
`services/scanService.go` + `controllers/scanController.go` are a complete,
|
||||
current example of all seven.
|
||||
|
||||
## Features with their own docs
|
||||
|
||||
| Feature | For | Doc |
|
||||
|---|---|---|
|
||||
| Environment & deployment | everyone | `docs/ENVIRONMENT.md` |
|
||||
| Scan-to-order (camera → product → nearest store with stock) | mobile app | `docs/SCAN_TO_ORDER.md` |
|
||||
| Catalogue import into a store | console | `docs/CATALOGUE_IMPORT_INTEGRATION.md` |
|
||||
| POS terminal ingest, login, API | POS / tills | `docs/POS_*.md` |
|
||||
| Access-control audit and what is still open | everyone | `docs/SECURITY_HANDOFF.md` |
|
||||
|
||||
## Things to know before you get surprised
|
||||
|
||||
- **There is no auth layer.** `customerid` / `tenantid` in a request are
|
||||
trusted. `docs/SECURITY_HANDOFF.md` §1 is the standing issue.
|
||||
- **Login errors mean what they say.** `409 Invalid Email` = the query ran
|
||||
and matched nobody. `500 Login is temporarily unavailable` = the database
|
||||
could not answer (it used to be reported as Invalid Email; see
|
||||
`services/userService.go` `lookupLogin`).
|
||||
- **Stock is a ledger.** Live stock is always `SUM(in) − SUM(out)` of
|
||||
`productstocks` at an outlet, never a stored number. Filter on the same
|
||||
expression you display (`services/productVisibility.go` explains why).
|
||||
- **The catalogue is a different database** and must never be reached
|
||||
through the `nearledb` handle. Its per-brand tables are discovered from
|
||||
`information_schema`; brands appear and columns vary.
|
||||
- **Catalogue ids are not stable** across re-scrapes; `imageid` is the
|
||||
durable key (`models.Products.Imageid`).
|
||||
- **Migrations run on boot** and are guarded by `IF NOT EXISTS` / schema
|
||||
checks, not a version table. Read the comments in `main.go` before adding
|
||||
one — several have bitten before.
|
||||
- **One MQTT client id per replica.** A second connection with the same id
|
||||
evicts the first. Never run a local process with the production
|
||||
`MQTT_URL`.
|
||||
- **`scratch/` tools read production** when run with `.env.production`.
|
||||
They are read-only by construction; keep them that way.
|
||||
|
||||
## Operations cheat-sheet (Kubernetes, namespace `nearle`)
|
||||
|
||||
```sh
|
||||
kubectl get pods -n nearle # fiesta-0/1/2 (StatefulSet)
|
||||
kubectl logs -n nearle fiesta-0 | grep -E 'config:|scan:|pos:'
|
||||
kubectl exec -n nearle fiesta-0 -- env | grep EMBEDDING_
|
||||
kubectl set env statefulset/fiesta -n nearle KEY=value # adds a var and rolls the pods
|
||||
kubectl rollout status statefulset/fiesta -n nearle
|
||||
```
|
||||
|
||||
The embedding model behind scan-to-order is served by the cluster's Ollama
|
||||
(`ollama.krow.svc.cluster.local:11434`); `docs/SCAN_TO_ORDER.md` has the
|
||||
exact settings and why that model.
|
||||
@@ -186,3 +186,76 @@ Same `ScanStore` shape as inside `stores[]` above, without options.
|
||||
Redis-backed short hold slots in at `Confirm` without changing the API.
|
||||
- **Identity** is the `customerid` in the body, like every other mobile
|
||||
endpoint here — there is no auth layer yet (see `SECURITY_HANDOFF.md`).
|
||||
|
||||
## For backend developers
|
||||
|
||||
### Where the code is
|
||||
|
||||
| File | Holds |
|
||||
|---|---|
|
||||
| `models/scan.go` | request/response shapes (`ScanLookupRequest`, `ScanStoreOffer`, `ScanOption`, …) |
|
||||
| `repositories/scanRepository.go` | all SQL: registered stores, live options, vector + text search, the two-tier cache |
|
||||
| `services/scanService.go` | the pipeline: parallel reads, scoring, family grouping, ranking, confirm fallback |
|
||||
| `controllers/scanController.go` | the three handlers and the error → status mapping |
|
||||
| `routes/scanroutes.go` | `/v1/mob/scan/*` |
|
||||
| `utils/embedding.go` | `Embedder` interface, OpenAI-compatible and Gemini clients |
|
||||
| `utils/geo.go` | coordinate parsing, haversine, opening hours, label tokenising |
|
||||
| `config/config.go` | `EmbeddingConfig` and its validation |
|
||||
| `scratch/cataloguedims` | read-only check of the catalogue's embedding width / fill |
|
||||
|
||||
### Try it locally
|
||||
|
||||
```sh
|
||||
go run . # with the local compose stack; EMBEDDING_* unset → text-only, still works
|
||||
curl -s localhost:1122/live/api/v1/mob/scan/lookup -H 'Content-Type: application/json' \
|
||||
-d '{"customerid":1,"label":"Milk Bikis","latitude":11.03,"longitude":77.03}' | jq .details
|
||||
```
|
||||
|
||||
To exercise the vector path locally, run Ollama on your Mac
|
||||
(`ollama pull all-minilm`) and set `EMBEDDING_PROVIDER=openai`,
|
||||
`EMBEDDING_BASE_URL=http://localhost:11434/v1`, `EMBEDDING_MODEL=all-minilm`,
|
||||
`EMBEDDING_API_KEY=ollama`, `EMBEDDING_DIMENSIONS=384` in `.env.local`. The
|
||||
local catalogue must carry vectors from the same model for results to mean
|
||||
anything; a schema-only dump does not.
|
||||
|
||||
### Tests
|
||||
|
||||
`go test ./services -run 'Lookup|Confirm|Stores|CatalogueFamily'` drives
|
||||
the whole pipeline through a fake repository (`services/scan_test.go`); no
|
||||
database. `go test ./utils` covers both HTTP clients against `httptest`
|
||||
servers, and the geo helpers. Add a case to `scan_test.go`'s fixture when
|
||||
you change ranking — it is the spec.
|
||||
|
||||
### Knobs (constants in `scanService.go`)
|
||||
|
||||
| Constant | Default | Effect |
|
||||
|---|---|---|
|
||||
| `scanLookupTimeout` | 5 s | whole lookup, including the model call |
|
||||
| `scanCatalogueTopK` | 15 | rows taken from each brand table and from the merge |
|
||||
| `scanMinScore` | 0.30 | below this the best hit is not shown as a match |
|
||||
| `embedTimeout` (`utils/embedding.go`) | 4 s | one model call |
|
||||
| `scanVectorTTL` / `scanHitsTTL` (`scanRepository.go`) | 7 d / 30 min | cache lifetimes |
|
||||
|
||||
Scores: vector = `1 − cosine distance`; text = 0.95 for the whole label
|
||||
inside the name, else `0.8 × (label words found / label words)`; combined =
|
||||
`max(vector, text) + 0.10` when both hit, capped at 1.
|
||||
|
||||
### Changing the embedding model
|
||||
|
||||
1. The catalogue team re-embeds `search_query` with the new model.
|
||||
2. Serve it (Ollama pull, or a hosted key).
|
||||
3. Change `EMBEDDING_MODEL` / `EMBEDDING_DIMENSIONS` (and provider/URL if
|
||||
needed) in the cluster; roll the pods.
|
||||
4. Flush the hit cache if you cannot wait 30 min: keys are
|
||||
`scan:hits:v1:*` and `scan:emb:v1:*` in Redis (they are also keyed by
|
||||
model name, so old entries simply stop being read).
|
||||
|
||||
Nothing in Go changes. A width mismatch fails the first search with an error
|
||||
naming both numbers.
|
||||
|
||||
### Adding a provider
|
||||
|
||||
Implement `utils.Embedder` (`Embed(ctx, text) ([]float32, error)` and
|
||||
`Model() string`), add a case to `NewEmbedder`, and add the provider name to
|
||||
the allow-list in `config.validate`. Keep the HTTP client timeout: the
|
||||
customer is holding a phone.
|
||||
|
||||
Reference in New Issue
Block a user