# Scan-to-order — mobile integration A customer photographs a product. Google Lens (on the phone) turns the photo into a label — `"Milk Bikis"`, `"Dabur Honey 500g"`. The app sends that label here and gets back: what the product is, which of the customer's stores sell it, in which sizes, with live stock, nearest first, and which store we recommend. When the customer taps a store and a size, a second call confirms the shelf still has it — and if it does not, names the next-nearest store that does. When the label fits several products — `"britannia"` names 258 of them — it answers with a short "did you mean?" list instead of picking one, because a confident price on the wrong biscuit is worse than one extra tap. Base path: `/live/api/v1/mob/scan`. Every response uses the usual envelope `{ code, status, message, details }`; the shapes below are `details`. ## The flow ``` photo ──Lens──▶ label │ ▼ POST /lookup ───▶ ambiguous:true + candidates[] "did you mean?" │ │ │ customer taps one candidate │ │ │ POST /lookup { brand, catalogueid } │ │ └───▶ match + stores[] (recommended first) ◀──┘ │ customer taps a store + a size │ ▼ POST /confirm ───▶ ok:true → add to basket with existing order APIs ok:false + alternative → offer the other store ``` **`/lookup` has two possible answers and the app must handle both.** A label that names one product comes back with `match` + `stores`. A label that fits several — a bare brand name like `"britannia"`, a generic word like `"biscuits"` — comes back with `ambiguous: true` and `candidates`, and the app asks the customer which one before any price is shown. Lens returns a bare wordmark often, because it is usually the biggest thing printed on a packet, so this is a normal path and not an error case. `GET /stores` is for the "choose another shop" sheet: the customer's registered stores, nearest first, independent of any product. ## `POST /lookup` Note the `//` notes below are annotations, not JSON — strip them. ```json { "customerid": 5123, "label": "Milk Bikis", "latitude": 11.0290, // phone fix; optional — saved address is used without it "longitude": 77.0290, "tenantids": [1135, 1140], // optional: what the app THINKS the customer joined "limit": 0 // optional: max stores, 0 = all // Instead of a label: name the product outright. This is how you resolve // a candidate the customer tapped, and how a deep link or a "buy again" // skips recognition. With both set, `label` is ignored. // "brand": "britannia", "catalogueid": 7 } ``` `label` is required **unless** `brand` and `catalogueid` are both given. `tenantids` is verified, never trusted: the server intersects it with the `tenantcustomers` table. Ids the customer is not actually registered with come back in `unregistered_tenantids` — treat that as "refresh the local list". A list that matches nothing at all is treated as stale and all registered stores are used. ### Response A — one product identified `ambiguous: false`, `match` set, `candidates` empty. ```json { "label": "Milk Bikis", "match": { "brand": "britannia", "catalogueid": 7, "imageid": "britannia_milk_bikis_100g", "product_name": "Milk Bikis", "size": "100 g", "variant_key": "milk_bikis", "image": "https://…", "score": 0.94, "method": "vector+text" }, "catalogue_variants": [ { "…same shape…": "100 g" }, { "…": "200 g" } ], "ambiguous": false, "candidates": [], "confidence": 0.94, "available": true, "recommended_locationid": 20, "stores": [ { "tenantid": 2, "tenantname": "R Mart", "locationid": 20, "locationname": "Hopes", "latitude": 11.01, "longitude": 77.0, "distance_km": 3.8, "open": true, "deliveryradius": 5, "deliverymins": 30, "recommended": true, "available": true, "options": [ { "productid": 200, "productname": "Milk Bikis 100g", "size": "100 g", "price": 12, "stock": 6, "available": true, "is_variant": false, "matched_by": "imageid", "image": "…" }, { "productid": 201, "productname": "Milk Bikis 200g", "size": "200 g", "price": 22, "stock": 3, "available": true, "is_variant": true, "variantname": "200 g", "matched_by": "variant-of:200" } ] }, { "locationid": 10, "locationname": "Peelamedu", "distance_km": 0.9, "available": false, "recommended": false, "options": [ { "productid": 100, "stock": 0, "available": false, "…": "…" } ] } ], "unregistered_tenantids": [], "message": "Available at 1 of your stores." } ``` ### Response B — several products fit, none clearly `ambiguous: true`, `match: null`, `stores: []`. Show a "did you mean?" list. ```json { "label": "britannia", "match": null, "ambiguous": true, "candidates": [ { "brand": "britannia", "catalogueid": 23, "product_name": "Britannia Marie Gold", "size": "250 g", "image": "https://…", "score": 0.5, "method": "text", "available": true }, { "brand": "britannia", "catalogueid": 22, "product_name": "Britannia Good Day Butter Cookies", "image": "https://…", "score": 0.333, "method": "text" }, { "brand": "britannia", "catalogueid": 21, "product_name": "Britannia Good Day Cashew Cookies", "image": "https://…", "score": 0.333, "method": "text" } ], "confidence": 0.5, "available": false, "stores": [], "catalogue_variants": [], "message": "Which one is it? 1 of these 3 are in stock near you." } ``` - **`available` on a candidate** means at least one of the customer's registered stores has it in stock right now. Candidates are ordered available-first, so the list can show what is buyable before what is not — and the field is absent (not `false`) when unavailable, so read it as falsy, not as a required key. - **To resolve a pick**, call `/lookup` again with that candidate's `brand` and `catalogueid` and no label. You get Response A for that exact product, with `method: "direct"` and `confidence: 1`. - At most 10 candidates come back. ### How to read either response - `match == null && !ambiguous` → nothing recognised; show `message` and let them retry with a clearer photo. - `ambiguous: true` → ask, do not guess. Never show a price on this path; `stores` is deliberately empty. - `confidence` below ~0.5 with a `match` → recognised but unsure; worth confirming the name before showing prices. `method: "text"` means no embedding model was involved (not configured, or it timed out) — be a little more cautious. `method: "direct"` means the caller named the product, so nothing was recognised at all. - `stores` is ordered **in-stock first, then nearest**. Exactly one store has `recommended: true` — the nearest with stock — and only when `available` is true. Stores that sell it but have nothing on the shelf are still listed (so the customer understands why they are not recommended); stores that do not sell it are not. - `options` are the things that can actually go in a basket at that store — the matched product and each of its sizes — each a real product with its own `productid`, price and live `stock`. Use `productid` in the existing cart/order calls exactly as you would from the catalogue screen. - `distance_km: -1` means the distance is unknown (no fix from the phone and no saved address, or the store has no coordinates). Do not render it as 0. ## `POST /confirm` Sent when the customer taps a store and an option. Re-reads live stock — nothing is cached on this path. ```json { "customerid": 5123, "tenantid": 1, "locationid": 10, "productid": 100, "quantity": 2, "latitude": 11.029, "longitude": 77.029 } ``` ```json { "ok": false, "reason": "out_of_stock", // in_stock | insufficient_stock | out_of_stock | not_sold_here | store_not_registered "store": { "…the store they tapped…" }, "option": { "productid": 100, "stock": 0, "…": "…" }, "requested": 2, "alternative": { // absent when nobody has enough "locationid": 20, "locationname": "Hopes", "distance_km": 3.8, "recommended": true, "available": true, "options": [ { "productid": 200, "stock": 6, "price": 12, "…": "…" } ] }, "message": "Out of stock at Peelamedu. Hopes has it (3.8 km away)." } ``` `ok: true` → proceed to the basket. `ok: false` → show `message`; if `alternative` is present offer it as a one-tap switch (it is the **same product**, not another size — the customer chose a size and we do not substitute). These are HTTP 200s: they are answers, not errors. ## `GET /stores?customerid=5123&latitude=11.029&longitude=77.029` The customer's registered stores, nearest first, `distance_km: -1` last. Same `ScanStore` shape as inside `stores[]` above, without options. ## Errors (HTTP status ≠ 200) | Status | When | |---|---| | 400 | Missing `customerid`/`label`/ids, or a body that is not JSON. `message` says which. | | 404 | `customerid` does not exist. | | 503 | The catalogue database is not reachable. Retry later; the rest of the app is unaffected. | | 500 | Anything else. Logged server-side. | ## Behind the curtain (for whoever operates it) - **Recognition** = pgvector cosine search over every `brand_*` table in the catalogue (each with its own index, merged), plus a word match on `product_name`/`title`/`search_query` that settles near-ties and works on its own when no embedding model is configured. The model is set by `EMBEDDING_PROVIDER/MODEL/API_KEY` and **must** be the one that indexed the catalogue — the first search checks the vector width and refuses a mismatch by name. - **The catalogue's model** (verified 2026-09-15 by cosine against a stored row: 1.0000): `all-MiniLM-L6-v2`, 384-d, unit-normalised, embedding the `search_query` column (brand + name + category + blurb + price range). Ollama ships it as `all-minilm`; the cluster's `ollama.krow` service serves it, so production is: ``` EMBEDDING_PROVIDER=openai EMBEDDING_BASE_URL=http://ollama.krow.svc.cluster.local:11434/v1 EMBEDDING_MODEL=all-minilm EMBEDDING_API_KEY=ollama # any non-empty value; Ollama ignores it EMBEDDING_DIMENSIONS=384 ``` A bare label ("Milk Bikis") scores ~0.92 against its product's stored vector and ~0.23 against an unrelated one, which is what the 0.30 floor in `scanService.go` is set against. If the catalogue team ever re-embeds with another model, change `EMBEDDING_MODEL`/`DIMENSIONS` here and nothing else. - **Speed**: the label's vector (7 days) and the ranked catalogue hits (30 min) are cached in Redis and in-process, so a popular product costs one model call platform-wide. Customer, stores and catalogue are read concurrently; the whole lookup is capped at 5 s and a slow model degrades to a text answer instead of a spinner. Live stock is one indexed query and is never cached. - **Availability** is the same rule the app's catalogue screen uses: `products.approve = 1`, `productlocations.publishedat IS NOT NULL`, stock = live `SUM(in) − SUM(out)` of `productstocks` at that outlet, price = the outlet's own price else the tenant's retail price. - **No reservation.** Confirm re-reads the ledger; a hold would give the same answer with a timer to babysit. If contention becomes real, a Redis-backed short hold slots in at `Confirm` without changing the API. - **Identity** is the `customerid` in the body, like every other mobile endpoint here — there is no auth layer yet (see `SECURITY_HANDOFF.md`). ## Two decisions, and why Both come from a proposal (2026-09-23) to have the app send vectors it computed on the phone. Recorded here because the next person will ask. ### The app does not send `textvector` An on-device MiniLM vector is only comparable to the catalogue's if the app ships the identical model *and* tokenizer *and* pooling *and* normalisation; a quantised tflite build usually drifts, and the failure is silent — the ranking just gets worse. There is also nothing to gain: the server-side embed is ~30 ms warm and the result is cached in Redis by label, so one model call serves every customer who scans that product. A client-supplied vector *defeats* that cache (the key would have to be the vector, not the label), and 384 floats is ~5 KB of upload against ~12 bytes for `"Milk Bikis"`. If the field ever arrives it can be accepted and validated, but the app should not be asked to compute it. **Send the full OCR text instead** if you want to give the server more to work with — ~100 bytes, no model coupling, strictly more information than a single label. ### The app does not send `imagevector` — yet The catalogue *does* carry image vectors: every `brand_*` table has `img_vector vector(1024)`, filled on 1885 of 2124 rows (empty in `brand_haldirams`, `brand_kaleesuwari`, `brand_mdh`, `brand_zzsmoketest`). That matches the proposed MobileNetV3-Small embedder, so the idea is coherent and half-built — this flow simply does not read that column. It stays unread for now because **Google Lens is already the image recogniser, and a far better one**: photo → Lens → label is Google's product recognition, trained on billions of images. Putting a 137M-parameter ImageNet backbone searching 1885 vectors *behind* that adds little where Lens succeeds, and MobileNetV3-Small — which struggles to tell one blue biscuit wrapper from another — is unlikely to rescue the cases where Lens fails. There is also an unverified dependency: the preprocessing the app would use (BGR → centre crop → 224×224 INTER_AREA → RGB → `/255.0`) has to match whatever the catalogue pipeline actually ran, or the search returns confidently-ranked noise. **What would change this:** the field data. Once live, count how often `/lookup` returns `ambiguous: true` or nothing recognised. If Lens labels are reliable, image search is polish; if that number is high, it becomes the priority — and the first task is the cosine check (embed a known catalogue product's image through the app's exact pipeline, compare with its stored `img_vector`; ≈0.99 means the contract holds), not writing the query. There is one non-recognition argument for it worth remembering: on-device inference is free and needs no Google dependency, which matters if Cloud Vision costs start to bite at volume. That is a business reason, not a quality one. ## For backend developers ### Where the code is | File | Holds | |---|---| | `models/scan.go` | request/response shapes (`ScanLookupRequest`, `ScanStoreOffer`, `ScanOption`, …) | | `repositories/scanRepository.go` | all SQL: registered stores, live options, vector + text search, the two-tier cache | | `services/scanService.go` | the pipeline: parallel reads, scoring, family grouping, ranking, confirm fallback | | `controllers/scanController.go` | the three handlers and the error → status mapping | | `routes/scanroutes.go` | `/v1/mob/scan/*` | | `utils/embedding.go` | `Embedder` interface, OpenAI-compatible and Gemini clients | | `utils/geo.go` | coordinate parsing, haversine, opening hours, label tokenising | | `config/config.go` | `EmbeddingConfig` and its validation | | `scratch/cataloguedims` | read-only check of every catalogue vector column's width and fill | ### Try it locally ```sh go run . # with the local compose stack; EMBEDDING_* unset → text-only, still works curl -s localhost:1122/live/api/v1/mob/scan/lookup -H 'Content-Type: application/json' \ -d '{"customerid":1,"label":"Milk Bikis","latitude":11.03,"longitude":77.03}' | jq .details ``` To exercise the vector path locally, run Ollama on your Mac (`ollama pull all-minilm`) and set `EMBEDDING_PROVIDER=openai`, `EMBEDDING_BASE_URL=http://localhost:11434/v1`, `EMBEDDING_MODEL=all-minilm`, `EMBEDDING_API_KEY=ollama`, `EMBEDDING_DIMENSIONS=384` in `.env.local`. The local catalogue must carry vectors from the same model for results to mean anything; a schema-only dump does not. ### Tests `go test ./services -run 'Lookup|Confirm|Stores|Brand|Ambiguous|Specific|TextScore|Distinct|Naming'` drives the whole pipeline through a fake repository (`services/scan_test.go`); no database. `go test ./utils` covers both HTTP clients against `httptest` servers, and the geo helpers. Add a case to `scan_test.go`'s fixture when you change ranking — it is the spec, and `newBrandLabelFixture` in particular is the regression guard for the brand-name bug described under Scoring. ### Knobs (constants in `scanService.go`) | Constant | Default | Effect | |---|---|---| | `scanLookupTimeout` | 5 s | whole lookup, including the model call | | `scanCatalogueTopK` | 15 | rows taken from each brand table and from the merge | | `scanMinScore` | 0.30 | below this the best hit is not shown as a match | | `scanAmbiguityMargin` | 0.06 | how close the runner-up may be before the answer becomes a question | | `scanSpecificEnough` | 0.55 | text coverage the leader needs before it counts as identified | | `scanMaxCandidates` | 10 | longest "did you mean?" list | | `embedTimeout` (`utils/embedding.go`) | 4 s | one model call | | `scanVectorTTL` / `scanHitsTTL` (`scanRepository.go`) | 7 d / 30 min | cache lifetimes | **Scoring.** Each hit carries two numbers, and they answer different questions: - `score` ranks. Vector = `1 − cosine distance`. Combined = `max(vector, text) + 0.10 × text`, capped at 1 — the confirmation bonus is proportional, so only a text match that actually names the product strengthens a vector hit. - `text` says how *specifically* the label names this product, and is the harmonic mean of two coverages: how much of the label the product accounts for, and how much of the product's name the label accounts for. Pack sizes are dropped from both sides. Why both: `"britannia"` is a substring of all 258 Britannia product names. Judging on overlap alone scored every one of them 0.95, the tie broke alphabetically, and one arbitrary biscuit came back with a price. Now they score ~0.33 *equally*, which `isAmbiguous` reads as "ask, don't guess" — via the margin test (something is level with the leader) or the specificity test (the leader may rank first on vector similarity while the label names no one product). Erring towards asking is deliberate: asking costs one tap on a picture, guessing wrong costs the customer's belief that the scanner works. An exact product name still scores ~1.0, so the common case is untouched. ### Changing the embedding model 1. The catalogue team re-embeds `search_query` with the new model. 2. Serve it (Ollama pull, or a hosted key). 3. Change `EMBEDDING_MODEL` / `EMBEDDING_DIMENSIONS` (and provider/URL if needed) in the cluster; roll the pods. 4. Flush the hit cache if you cannot wait 30 min: keys are `scan:hits:v1:*` and `scan:emb:v1:*` in Redis (they are also keyed by model name, so old entries simply stop being read). Nothing in Go changes. A width mismatch fails the first search with an error naming both numbers. ### Adding a provider Implement `utils.Embedder` (`Embed(ctx, text) ([]float32, error)` and `Model() string`), add a case to `NewEmbedder`, and add the provider name to the allow-list in `config.validate`. Keep the HTTP client timeout: the customer is holding a phone.