# Scan-to-order — mobile integration A customer photographs a product. Google Lens (on the phone) turns the photo into a label — `"Milk Bikis"`, `"Dabur Honey 500g"`. The app sends that label here and gets back: what the product is, which of the customer's stores sell it, in which sizes, with live stock, nearest first, and which store we recommend. When the customer taps a store and a size, a second call confirms the shelf still has it — and if it does not, names the next-nearest store that does. Base path: `/live/api/v1/mob/scan`. Every response uses the usual envelope `{ code, status, message, details }`; the shapes below are `details`. ## The flow ``` photo ──Lens──▶ label │ ▼ POST /lookup ───▶ match + stores[] (recommended first) │ customer taps a store + a size │ ▼ POST /confirm ───▶ ok:true → add to basket with existing order APIs ok:false + alternative → offer the other store ``` `GET /stores` is for the "choose another shop" sheet: the customer's registered stores, nearest first, independent of any product. ## `POST /lookup` ```json { "customerid": 5123, "label": "Milk Bikis", "latitude": 11.0290, // phone fix; optional — saved address is used without it "longitude": 77.0290, "tenantids": [1135, 1140], // optional: what the app THINKS the customer joined "limit": 0 // optional: max stores, 0 = all } ``` `tenantids` is verified, never trusted: the server intersects it with the `tenantcustomers` table. Ids the customer is not actually registered with come back in `unregistered_tenantids` — treat that as "refresh the local list". A list that matches nothing at all is treated as stale and all registered stores are used. Response: ```json { "label": "Milk Bikis", "match": { "brand": "britannia", "catalogueid": 7, "imageid": "britannia_milk_bikis_100g", "product_name": "Milk Bikis", "size": "100 g", "variant_key": "milk_bikis", "image": "https://…", "score": 0.94, "method": "vector+text" }, "catalogue_variants": [ { "…same shape…": "100 g" }, { "…": "200 g" } ], "confidence": 0.94, "available": true, "recommended_locationid": 20, "stores": [ { "tenantid": 2, "tenantname": "R Mart", "locationid": 20, "locationname": "Hopes", "latitude": 11.01, "longitude": 77.0, "distance_km": 3.8, "open": true, "deliveryradius": 5, "deliverymins": 30, "recommended": true, "available": true, "options": [ { "productid": 200, "productname": "Milk Bikis 100g", "size": "100 g", "price": 12, "stock": 6, "available": true, "is_variant": false, "matched_by": "imageid", "image": "…" }, { "productid": 201, "productname": "Milk Bikis 200g", "size": "200 g", "price": 22, "stock": 3, "available": true, "is_variant": true, "variantname": "200 g", "matched_by": "variant-of:200" } ] }, { "locationid": 10, "locationname": "Peelamedu", "distance_km": 0.9, "available": false, "recommended": false, "options": [ { "productid": 100, "stock": 0, "available": false, "…": "…" } ] } ], "unregistered_tenantids": [], "message": "Available at 1 of your stores." } ``` How to read it: - `match == null` → nothing recognised; show `message` and let them retry. `confidence` below ~0.5 → recognised but unsure; confirm the name with the customer before showing prices. `method: "text"` means no embedding model was involved (not configured, or it timed out) — be a little more cautious. - `stores` is ordered **in-stock first, then nearest**. Exactly one store has `recommended: true` — the nearest with stock — and only when `available` is true. Stores that sell it but have nothing on the shelf are still listed (so the customer understands why they are not recommended); stores that do not sell it are not. - `options` are the things that can actually go in a basket at that store — the matched product and each of its sizes — each a real product with its own `productid`, price and live `stock`. Use `productid` in the existing cart/order calls exactly as you would from the catalogue screen. - `distance_km: -1` means the distance is unknown (no fix from the phone and no saved address, or the store has no coordinates). Do not render it as 0. Send `latitude`/`longitude` on `/confirm` too if you display distance from its reply: the saved address is only consulted there when the shelf is empty and alternatives have to be ranked, so without a fix the store you tapped comes back `-1`. ## `POST /confirm` Sent when the customer taps a store and an option. Re-reads live stock — nothing is cached on this path. ```json { "customerid": 5123, "tenantid": 1, "locationid": 10, "productid": 100, "quantity": 2, "latitude": 11.029, "longitude": 77.029 } ``` ```json { "ok": false, "reason": "out_of_stock", // in_stock | insufficient_stock | out_of_stock | not_sold_here | store_not_registered "store": { "…the store they tapped…" }, // distance_km filled from the fix you send "option": { "productid": 100, "stock": 0, "…": "…" }, "requested": 2, "alternative": { // absent when nobody has enough "locationid": 20, "locationname": "Hopes", "distance_km": 3.8, "recommended": true, "available": true, "options": [ { "productid": 200, "stock": 6, "price": 12, "…": "…" } ] }, "message": "Out of stock at Peelamedu. Hopes has it (3.8 km away)." } ``` `ok: true` → proceed to the basket. `ok: false` → show `message`; if `alternative` is present offer it as a one-tap switch (it is the **same product**, not another size — the customer chose a size and we do not substitute). These are HTTP 200s: they are answers, not errors. ## `GET /stores?customerid=5123&latitude=11.029&longitude=77.029` The customer's registered stores, nearest first, `distance_km: -1` last. Same `ScanStore` shape as inside `stores[]` above, without options. ## Errors (HTTP status ≠ 200) | Status | When | |---|---| | 400 | Missing `customerid`/`label`/ids, or a body that is not JSON. `message` says which. | | 404 | `customerid` does not exist. | | 503 | The catalogue database is not reachable. Retry later; the rest of the app is unaffected. | | 500 | Anything else. Logged server-side. | ## Behind the curtain (for whoever operates it) - **Recognition** = pgvector cosine search over every `brand_*` table in the catalogue (each with its own index, merged), plus a word match on `product_name`/`title`/`search_query` that settles near-ties and works on its own when no embedding model is configured. The model is set by `EMBEDDING_PROVIDER/MODEL/API_KEY` and **must** be the one that indexed the catalogue — the first search checks the vector width and refuses a mismatch by name. - **The catalogue's model** (verified 2026-09-15 by cosine against a stored row: 1.0000): `all-MiniLM-L6-v2`, 384-d, unit-normalised, embedding the `search_query` column (brand + name + category + blurb + price range). Ollama ships it as `all-minilm`; the cluster's `ollama.krow` service serves it, so production is: ``` EMBEDDING_PROVIDER=openai EMBEDDING_BASE_URL=http://ollama.krow.svc.cluster.local:11434/v1 EMBEDDING_MODEL=all-minilm EMBEDDING_API_KEY=ollama # any non-empty value; Ollama ignores it EMBEDDING_DIMENSIONS=384 ``` A bare label ("Milk Bikis") scores ~0.92 against its product's stored vector and ~0.23 against an unrelated one, which is what the 0.50 floor in `scanService.go` is set against — the middle of that split, not the edge of the noise. It was 0.30 until a near-miss got through in production ("Paracetamol" → "Paneer Makhni 500ml", 0.304). If the catalogue team ever re-embeds with another model, change `EMBEDDING_MODEL`/`DIMENSIONS` here and nothing else. - **Speed**: the label's vector (7 days) and the ranked catalogue hits (30 min) are cached in Redis and in-process, so a popular product costs one model call platform-wide. Customer, stores and catalogue are read concurrently; the whole lookup is capped at 5 s and a slow model degrades to a text answer instead of a spinner. Live stock is one indexed query and is never cached. - **Availability** is the same rule the app's catalogue screen uses: `products.approve = 1`, `productlocations.publishedat IS NOT NULL`, stock = live `SUM(in) − SUM(out)` of `productstocks` at that outlet, price = the outlet's own price else the tenant's retail price. - **No reservation.** Confirm re-reads the ledger; a hold would give the same answer with a timer to babysit. If contention becomes real, a Redis-backed short hold slots in at `Confirm` without changing the API. - **Identity** is the `customerid` in the body, like every other mobile endpoint here — there is no auth layer yet (see `SECURITY_HANDOFF.md`). ## For backend developers ### Where the code is | File | Holds | |---|---| | `models/scan.go` | request/response shapes (`ScanLookupRequest`, `ScanStoreOffer`, `ScanOption`, …) | | `repositories/scanRepository.go` | all SQL: registered stores, live options, vector + text search, the two-tier cache | | `services/scanService.go` | the pipeline: parallel reads, scoring, family grouping, ranking, confirm fallback | | `controllers/scanController.go` | the three handlers and the error → status mapping | | `routes/scanroutes.go` | `/v1/mob/scan/*` | | `utils/embedding.go` | `Embedder` interface, OpenAI-compatible and Gemini clients | | `utils/geo.go` | coordinate parsing, haversine, opening hours, label tokenising | | `config/config.go` | `EmbeddingConfig` and its validation | | `scratch/cataloguedims` | read-only check of the catalogue's embedding width / fill | ### Try it locally ```sh go run . # with the local compose stack; EMBEDDING_* unset → text-only, still works curl -s localhost:1122/live/api/v1/mob/scan/lookup -H 'Content-Type: application/json' \ -d '{"customerid":1,"label":"Milk Bikis","latitude":11.03,"longitude":77.03}' | jq .details ``` To exercise the vector path locally, run Ollama on your Mac (`ollama pull all-minilm`) and set `EMBEDDING_PROVIDER=openai`, `EMBEDDING_BASE_URL=http://localhost:11434/v1`, `EMBEDDING_MODEL=all-minilm`, `EMBEDDING_API_KEY=ollama`, `EMBEDDING_DIMENSIONS=384` in `.env.local`. The local catalogue must carry vectors from the same model for results to mean anything; a schema-only dump does not. ### Tests `go test ./services -run 'Lookup|Confirm|Stores|CatalogueFamily'` drives the whole pipeline through a fake repository (`services/scan_test.go`); no database. `go test ./utils` covers both HTTP clients against `httptest` servers, and the geo helpers. Add a case to `scan_test.go`'s fixture when you change ranking — it is the spec. ### Knobs (constants in `scanService.go`) | Constant | Default | Effect | |---|---|---| | `scanLookupTimeout` | 5 s | whole lookup, including the model call | | `scanCatalogueTopK` | 15 | rows taken from each brand table and from the merge | | `scanMinScore` | 0.50 | below this the best hit is not shown as a match | | `embedTimeout` (`utils/embedding.go`) | 4 s | one model call | | `scanVectorTTL` / `scanHitsTTL` (`scanRepository.go`) | 7 d / 30 min | cache lifetimes | Scores: vector = `1 − cosine distance`; text = 0.95 for the whole label inside the name, else `0.8 × (label words found / label words)`; combined = `max(vector, text) + 0.10` when both hit, capped at 1. ### Changing the embedding model 1. The catalogue team re-embeds `search_query` with the new model. 2. Serve it (Ollama pull, or a hosted key). 3. Change `EMBEDDING_MODEL` / `EMBEDDING_DIMENSIONS` (and provider/URL if needed) in the cluster; roll the pods. 4. Flush the hit cache if you cannot wait 30 min: keys are `scan:hits:v1:*` and `scan:emb:v1:*` in Redis (they are also keyed by model name, so old entries simply stop being read). Nothing in Go changes. A width mismatch fails the first search with an error naming both numbers. ### Adding a provider Implement `utils.Embedder` (`Embed(ctx, text) ([]float32, error)` and `Model() string`), add a case to `NewEmbedder`, and add the provider name to the allow-list in `config.validate`. Keep the HTTP client timeout: the customer is holding a phone.