backend_fiesta/docs/SCAN_TO_ORDER.md

Scan-to-order — mobile integration

A customer photographs a product. Google Lens (on the phone) turns the photo into a label — "Milk Bikis", "Dabur Honey 500g". The app sends that label here and gets back: what the product is, which of the customer's stores sell it, in which sizes, with live stock, nearest first, and which store we recommend. When the customer taps a store and a size, a second call confirms the shelf still has it — and if it does not, names the next-nearest store that does.

When the label fits several products — "britannia" names 258 of them — it answers with a short "did you mean?" list instead of picking one, because a confident price on the wrong biscuit is worse than one extra tap.

Base path: /live/api/v1/mob/scan. Every response uses the usual envelope { code, status, message, details }; the shapes below are details.

The flow

photo ──Lens──▶ label
                 │
                 ▼
   POST /lookup  ───▶  ambiguous:true + candidates[]   "did you mean?"
                 │                        │
                 │      customer taps one candidate
                 │                        │
                 │      POST /lookup { brand, catalogueid }
                 │                        │
                 └───▶  match + stores[] (recommended first)  ◀──┘
                 │
   customer taps a store + a size
                 │
                 ▼
   POST /confirm ───▶  ok:true            → add to basket with existing order APIs
                       ok:false + alternative → offer the other store

/lookup has two possible answers and the app must handle both. A label that names one product comes back with match + stores. A label that fits several — a bare brand name like "britannia", a generic word like "biscuits" — comes back with ambiguous: true and candidates, and the app asks the customer which one before any price is shown. Lens returns a bare wordmark often, because it is usually the biggest thing printed on a packet, so this is a normal path and not an error case.

GET /stores is for the "choose another shop" sheet: the customer's registered stores, nearest first, independent of any product.

POST /lookup

Note the // notes below are annotations, not JSON — strip them.

{
  "customerid": 5123,
  "label": "Milk Bikis",
  "latitude": 11.0290,          // phone fix; optional — saved address is used without it
  "longitude": 77.0290,
  "tenantids": [1135, 1140],    // optional: what the app THINKS the customer joined
  "limit": 0                    // optional: max stores, 0 = all

  // Instead of a label: name the product outright. This is how you resolve
  // a candidate the customer tapped, and how a deep link or a "buy again"
  // skips recognition. With both set, `label` is ignored.
  // "brand": "britannia", "catalogueid": 7
}

label is required unless brand and catalogueid are both given.

tenantids is verified, never trusted: the server intersects it with the tenantcustomers table. Ids the customer is not actually registered with come back in unregistered_tenantids — treat that as "refresh the local list". A list that matches nothing at all is treated as stale and all registered stores are used.

Response A — one product identified

ambiguous: false, match set, candidates empty.

{
  "label": "Milk Bikis",
  "match": {
    "brand": "britannia", "catalogueid": 7, "imageid": "britannia_milk_bikis_100g",
    "product_name": "Milk Bikis", "size": "100 g", "variant_key": "milk_bikis",
    "image": "https://…", "score": 0.94, "method": "vector+text"
  },
  "catalogue_variants": [ { "…same shape…": "100 g" }, { "…": "200 g" } ],
  "ambiguous": false,
  "candidates": [],
  "confidence": 0.94,
  "available": true,
  "recommended_locationid": 20,
  "stores": [
    {
      "tenantid": 2, "tenantname": "R Mart", "locationid": 20, "locationname": "Hopes",
      "latitude": 11.01, "longitude": 77.0, "distance_km": 3.8, "open": true,
      "deliveryradius": 5, "deliverymins": 30,
      "recommended": true, "available": true,
      "options": [
        { "productid": 200, "productname": "Milk Bikis 100g", "size": "100 g", "price": 12, "stock": 6,
          "available": true, "is_variant": false, "matched_by": "imageid", "image": "…" },
        { "productid": 201, "productname": "Milk Bikis 200g", "size": "200 g", "price": 22, "stock": 3,
          "available": true, "is_variant": true, "variantname": "200 g", "matched_by": "variant-of:200" }
      ]
    },
    { "locationid": 10, "locationname": "Peelamedu", "distance_km": 0.9, "available": false, "recommended": false,
      "options": [ { "productid": 100, "stock": 0, "available": false, "…": "…" } ] }
  ],
  "unregistered_tenantids": [],
  "message": "Available at 1 of your stores."
}

Response B — several products fit, none clearly

ambiguous: true, match: null, stores: []. Show a "did you mean?" list.

{
  "label": "britannia",
  "match": null,
  "ambiguous": true,
  "candidates": [
    { "brand": "britannia", "catalogueid": 23, "product_name": "Britannia Marie Gold",
      "size": "250 g", "image": "https://…", "score": 0.95, "method": "text", "available": true },
    { "brand": "britannia", "catalogueid": 22, "product_name": "Britannia Good Day Butter Cookies",
      "image": "https://…", "score": 0.95, "method": "text" },
    { "brand": "britannia", "catalogueid": 21, "product_name": "Britannia Good Day Cashew Cookies",
      "image": "https://…", "score": 0.95, "method": "text" }
  ],
  "confidence": 0.95,
  "available": false,
  "stores": [],
  "catalogue_variants": [],
  "message": "Which one is it? 1 of these 3 are in stock near you."
}

• confidence is not low here, and that is not a bug. "britannia" really does appear in all three names, so relevance is high — what is missing is identification. Gate on ambiguous, never on confidence: an app that reads 0.95 as "sure enough to show a price" reintroduces the exact bug this path exists to prevent.
• available on a candidate means at least one of the customer's registered stores has it in stock right now. Candidates are ordered available-first, so the list can show what is buyable before what is not — and the field is absent (not false) when unavailable, so read it as falsy, not as a required key.
• To resolve a pick, call /lookup again with that candidate's brand and catalogueid and no label. You get Response A for that exact product, with method: "direct" and confidence: 1.
• At most 10 candidates come back.

How to read either response

• match == null && !ambiguous → nothing recognised; show message and let them retry with a clearer photo.
• ambiguous: true → ask, do not guess. Never show a price on this path; stores is deliberately empty.
• confidence below ~0.5 with a match → recognised but unsure; worth confirming the name before showing prices. method: "text" means no embedding model was involved (not configured, or it timed out) — be a little more cautious. method: "direct" means the caller named the product, so nothing was recognised at all.
• stores is ordered in-stock first, then nearest. Exactly one store has recommended: true — the nearest with stock — and only when available is true. Stores that sell it but have nothing on the shelf are still listed (so the customer understands why they are not recommended); stores that do not sell it are not.
• options are the things that can actually go in a basket at that store — the matched product and each of its sizes — each a real product with its own productid, price and live stock. Use productid in the existing cart/order calls exactly as you would from the catalogue screen.
• distance_km: -1 means the distance is unknown (no fix from the phone and no saved address, or the store has no coordinates). Do not render it as 0. Send latitude/longitude on /confirm too if you display distance from its reply: the saved address is only consulted there when the shelf is empty and alternatives have to be ranked, so without a fix the store you tapped comes back -1.

POST /confirm

Sent when the customer taps a store and an option. Re-reads live stock — nothing is cached on this path.

{ "customerid": 5123, "tenantid": 1, "locationid": 10, "productid": 100, "quantity": 2,
  "latitude": 11.029, "longitude": 77.029 }

{
  "ok": false,
  "reason": "out_of_stock",           // in_stock | insufficient_stock | out_of_stock | not_sold_here | store_not_registered
  "store":  { "…the store they tapped…" },   // distance_km filled from the fix you send
  "option": { "productid": 100, "stock": 0, "…": "…" },
  "requested": 2,
  "alternative": {                     // absent when nobody has enough
    "locationid": 20, "locationname": "Hopes", "distance_km": 3.8, "recommended": true, "available": true,
    "options": [ { "productid": 200, "stock": 6, "price": 12, "…": "…" } ]
  },
  "message": "Out of stock at Peelamedu. Hopes has it (3.8 km away)."
}

ok: true → proceed to the basket. ok: false → show message; if alternative is present offer it as a one-tap switch (it is the same product, not another size — the customer chose a size and we do not substitute). These are HTTP 200s: they are answers, not errors.

GET /stores?customerid=5123&latitude=11.029&longitude=77.029

The customer's registered stores, nearest first, distance_km: -1 last. Same ScanStore shape as inside stores[] above, without options.

Errors (HTTP status ≠ 200)

Status	When
400	Missing customerid/label/ids, or a body that is not JSON. message says which.
404	customerid does not exist.
503	The catalogue database is not reachable. Retry later; the rest of the app is unaffected.
500	Anything else. Logged server-side.

Behind the curtain (for whoever operates it)

• Recognition = pgvector cosine search over every brand_* table in the catalogue (each with its own index, merged), plus a word match on product_name/title/search_query that settles near-ties and works on its own when no embedding model is configured. The model is set by EMBEDDING_PROVIDER/MODEL/API_KEY and must be the one that indexed the catalogue — the first search checks the vector width and refuses a mismatch by name.
• The word match asks for most of the label, not all of it (minTokenHits: two thirds, rounded up, and both of a two-word label). Requiring every word meant one word the catalogue does not use took the right product out of the running entirely — "Dettol bottle pack" retrieved no Dettol, "Parle G biscuit pack" retrieved no Parle-G — and the vector search then answered alone, confidently and wrongly, at a score the floor could not catch. Each brand's rows are ordered by how much of the label they carry (the whole label as a substring outranks any number of loose words) so that the per-brand LIMIT keeps the best rows and not merely the first ones the planner reached. Packaging words — "pack", "bottle", "jar", "sachet" and friends, see utils.isPackaging — are dropped before any of this, like pack sizes, unless the label is nothing else.
• The catalogue's model (verified 2026-09-15 by cosine against a stored row: 1.0000): all-MiniLM-L6-v2, 384-d, unit-normalised, embedding the search_query column (brand + name + category + blurb + price range). Ollama ships it as all-minilm; the cluster's ollama.krow service serves it, so production is:

  EMBEDDING_PROVIDER=openai
  EMBEDDING_BASE_URL=http://ollama.krow.svc.cluster.local:11434/v1
  EMBEDDING_MODEL=all-minilm
  EMBEDDING_API_KEY=ollama        # any non-empty value; Ollama ignores it
  EMBEDDING_DIMENSIONS=384
  

  A bare label ("Milk Bikis") scores ~0.92 against its product's stored vector and ~0.23 against an unrelated one, which is what the 0.50 floor in scanService.go is set against — the middle of that split, not the edge of the noise. It was 0.30 until a near-miss got through in production ("Paracetamol" → "Paneer Makhni 500ml", 0.304). If the catalogue team ever re-embeds with another model, change EMBEDDING_MODEL/DIMENSIONS here and nothing else.
• Speed: the label's vector (7 days) and the ranked catalogue hits (30 min) are cached in Redis and in-process, so a popular product costs one model call platform-wide. Customer, stores and catalogue are read concurrently; the whole lookup is capped at 5 s and a slow model degrades to a text answer instead of a spinner. Live stock is one indexed query and is never cached.
• Availability is the same rule the app's catalogue screen uses: products.approve = 1, productlocations.publishedat IS NOT NULL, stock = live SUM(in) − SUM(out) of productstocks at that outlet, price = the outlet's own price else the tenant's retail price.
• No reservation. Confirm re-reads the ledger; a hold would give the same answer with a timer to babysit. If contention becomes real, a Redis-backed short hold slots in at Confirm without changing the API.
• Identity is the customerid in the body, like every other mobile endpoint here — there is no auth layer yet (see SECURITY_HANDOFF.md).

Two decisions, and why

Both come from a proposal (2026-09-23) to have the app send vectors it computed on the phone. Recorded here because the next person will ask.

The app does not send textvector

An on-device MiniLM vector is only comparable to the catalogue's if the app ships the identical model and tokenizer and pooling and normalisation; a quantised tflite build usually drifts, and the failure is silent — the ranking just gets worse. There is also nothing to gain: the server-side embed is ~30 ms warm and the result is cached in Redis by label, so one model call serves every customer who scans that product. A client-supplied vector defeats that cache (the key would have to be the vector, not the label), and 384 floats is ~5 KB of upload against ~12 bytes for "Milk Bikis". If the field ever arrives it can be accepted and validated, but the app should not be asked to compute it.

Send the full OCR text instead if you want to give the server more to work with — ~100 bytes, no model coupling, strictly more information than a single label.

The app does not send imagevector — yet

The catalogue does carry image vectors: every brand_* table has img_vector vector(1024), filled on 1885 of 2124 rows (empty in brand_haldirams, brand_kaleesuwari, brand_mdh, brand_zzsmoketest). That matches the proposed MobileNetV3-Small embedder, so the idea is coherent and half-built — this flow simply does not read that column.

It stays unread for now because Google Lens is already the image recogniser, and a far better one: photo → Lens → label is Google's product recognition, trained on billions of images. Putting a 137M-parameter ImageNet backbone searching 1885 vectors behind that adds little where Lens succeeds, and MobileNetV3-Small — which struggles to tell one blue biscuit wrapper from another — is unlikely to rescue the cases where Lens fails. There is also an unverified dependency: the preprocessing the app would use (BGR → centre crop → 224×224 INTER_AREA → RGB → /255.0) has to match whatever the catalogue pipeline actually ran, or the search returns confidently-ranked noise.

What would change this: the field data. Once live, count how often /lookup returns ambiguous: true or nothing recognised. If Lens labels are reliable, image search is polish; if that number is high, it becomes the priority — and the first task is the cosine check (embed a known catalogue product's image through the app's exact pipeline, compare with its stored img_vector; ≈0.99 means the contract holds), not writing the query.

There is one non-recognition argument for it worth remembering: on-device inference is free and needs no Google dependency, which matters if Cloud Vision costs start to bite at volume. That is a business reason, not a quality one.

For backend developers

Where the code is

File	Holds
models/scan.go	request/response shapes (ScanLookupRequest, ScanStoreOffer, ScanOption, …)
repositories/scanRepository.go	all SQL: registered stores, live options, vector + text search, the two-tier cache
services/scanService.go	the pipeline: parallel reads, scoring, family grouping, ranking, confirm fallback
controllers/scanController.go	the three handlers and the error → status mapping
routes/scanroutes.go	/v1/mob/scan/*
utils/embedding.go	Embedder interface, OpenAI-compatible and Gemini clients
utils/geo.go	coordinate parsing, haversine, opening hours, label tokenising
config/config.go	EmbeddingConfig and its validation
scratch/cataloguedims	read-only check of every catalogue vector column's width and fill

Try it locally

go run .    # with the local compose stack; EMBEDDING_* unset → text-only, still works
curl -s localhost:1122/live/api/v1/mob/scan/lookup -H 'Content-Type: application/json' \
  -d '{"customerid":1,"label":"Milk Bikis","latitude":11.03,"longitude":77.03}' | jq .details

To exercise the vector path locally, run Ollama on your Mac (ollama pull all-minilm) and set EMBEDDING_PROVIDER=openai, EMBEDDING_BASE_URL=http://localhost:11434/v1, EMBEDDING_MODEL=all-minilm, EMBEDDING_API_KEY=ollama, EMBEDDING_DIMENSIONS=384 in .env.local. The local catalogue must carry vectors from the same model for results to mean anything; a schema-only dump does not.

Tests

go test ./services -run 'Lookup|Confirm|Stores|Brand|Ambiguous|Specific|TextScore|Distinct|Naming' drives the whole pipeline through a fake repository (services/scan_test.go); no database. go test ./utils covers both HTTP clients against httptest servers, and the geo helpers. Add a case to scan_test.go's fixture when you change ranking — it is the spec, and newBrandLabelFixture in particular is the regression guard for the brand-name bug described under Scoring.

Knobs (constants in scanService.go)

Constant	Default	Effect
scanLookupTimeout	5 s	whole lookup, including the model call
scanCatalogueTopK	15	rows taken from each brand table and from the merge
scanMinScore	0.50	below this the best hit is not shown as a match
scanAmbiguityMargin	0.06	how close the runner-up may be before the answer becomes a question
scanMaxCandidates	10	longest "did you mean?" list
embedTimeout (utils/embedding.go)	4 s	one model call
scanVectorTTL / scanHitsTTL (scanRepository.go)	7 d / 30 min	cache lifetimes

Scores: vector = 1 − cosine distance; text = 0.95 for the whole label inside the name, else 0.8 × (label words found / label words); combined = max(vector, text) + 0.10 when both hit, capped at 1. Ties are broken by cosine distance — nearest first, a text-only row last — and only then by name.

The label and the product name are both separator-folded before that substring test (utils.FoldSeparators), and compared again with separators removed (utils.TightenLabel, labels of 4+ characters), so the brand's own punctuation does not decide the match: "Parle G", "Parle-G" and "ParleG" all reach Parle-G Original Glucose Biscuits. A single-character token survives tokenising when it follows a word, because it is often the whole name — the "G" of Parle-G, the "K" of Special K. It is still dropped when it stands alone or is a pack multiplier.

All three mattered at once: before this, "Parle G" tied with Parle Monaco Classic at 0.9 (the "G" was dropped, so only "parle" matched either row), and the name tie-break handed it to Monaco because a space precedes a hyphen in ASCII. A confident, wrong answer — the kind no score floor can catch.

When the substring rule ties, that tie is the answer. A bare brand name is a substring of every one of that brand's names, so all of them score 0.95 — identically, at a high score no floor would ever catch. Rather than scoring around it, isAmbiguous reads it: if the runner-up is within scanAmbiguityMargin of the leader, the reply becomes ambiguous: true with candidates instead of a match (see Response B). Erring towards asking is deliberate — one tap on a picture against the wrong biscuit. A label that names one product leaves the runner-up far behind, so the common case is untouched, and services/scan_test.go's TestABrandNameScoresItsProductsIdentically guards the tie itself: a formula that broke it on name length or word count would bring the bug back.

Changing the embedding model

1. The catalogue team re-embeds search_query with the new model.
2. Serve it (Ollama pull, or a hosted key).
3. Change EMBEDDING_MODEL / EMBEDDING_DIMENSIONS (and provider/URL if needed) in the cluster; roll the pods.
4. Flush the hit cache if you cannot wait 30 min: keys are scan:hits:v1:* and scan:emb:v1:* in Redis (they are also keyed by model name, so old entries simply stop being read).

Nothing in Go changes. A width mismatch fails the first search with an error naming both numbers.

Adding a provider

Implement utils.Embedder (Embed(ctx, text) ([]float32, error) and Model() string), add a case to NewEmbedder, and add the provider name to the allow-list in config.validate. Keep the HTTP client timeout: the customer is holding a phone.