backend_fiesta/docs/SCAN_TO_ORDER.md

# Scan-to-order — mobile integration

A customer photographs a product. Google Lens (on the phone) turns the photo
into a label — `"Milk Bikis"`, `"Dabur Honey 500g"`. The app sends that label
here and gets back: what the product is, which of the customer's stores sell
it, in which sizes, with live stock, nearest first, and which store we
recommend. When the customer taps a store and a size, a second call confirms
the shelf still has it — and if it does not, names the next-nearest store
that does.

When the label fits several products — `"britannia"` names 258 of them — it
answers with a short "did you mean?" list instead of picking one, because a
confident price on the wrong biscuit is worse than one extra tap.

Base path: `/live/api/v1/mob/scan`. Every response uses the usual envelope
`{ code, status, message, details }`; the shapes below are `details`.

## The flow

```
photo ──Lens──▶ label
                 │
                 ▼
   POST /lookup  ───▶  ambiguous:true + candidates[]   "did you mean?"
                 │                        │
                 │      customer taps one candidate
                 │                        │
                 │      POST /lookup { brand, catalogueid }
                 │                        │
                 └───▶  match + stores[] (recommended first)  ◀──┘
                 │
   customer taps a store + a size
                 │
                 ▼
   POST /confirm ───▶  ok:true            → add to basket with existing order APIs
                       ok:false + alternative → offer the other store
```

**`/lookup` has two possible answers and the app must handle both.** A label
that names one product comes back with `match` + `stores`. A label that fits
several — a bare brand name like `"britannia"`, a generic word like
`"biscuits"` — comes back with `ambiguous: true` and `candidates`, and the
app asks the customer which one before any price is shown. Lens returns a
bare wordmark often, because it is usually the biggest thing printed on a
packet, so this is a normal path and not an error case.

`GET /stores` is for the "choose another shop" sheet: the customer's
registered stores, nearest first, independent of any product.

## `POST /lookup`

Note the `//` notes below are annotations, not JSON — strip them.

```json
{
  "customerid": 5123,
  "label": "Milk Bikis",
  "latitude": 11.0290,          // phone fix; optional — saved address is used without it
  "longitude": 77.0290,
  "tenantids": [1135, 1140],    // optional: what the app THINKS the customer joined
  "limit": 0                    // optional: max stores, 0 = all

  // Instead of a label: name the product outright. This is how you resolve
  // a candidate the customer tapped, and how a deep link or a "buy again"
  // skips recognition. With both set, `label` is ignored.
  // "brand": "britannia", "catalogueid": 7
}
```

`label` is required **unless** `brand` and `catalogueid` are both given.

`tenantids` is verified, never trusted: the server intersects it with the
`tenantcustomers` table. Ids the customer is not actually registered with
come back in `unregistered_tenantids` — treat that as "refresh the local
list". A list that matches nothing at all is treated as stale and all
registered stores are used.

### Response A — one product identified

`ambiguous: false`, `match` set, `candidates` empty.

```json
{
  "label": "Milk Bikis",
  "match": {
    "brand": "britannia", "catalogueid": 7, "imageid": "britannia_milk_bikis_100g",
    "product_name": "Milk Bikis", "size": "100 g", "variant_key": "milk_bikis",
    "image": "https://…", "score": 0.94, "method": "vector+text"
  },
  "catalogue_variants": [ { "…same shape…": "100 g" }, { "…": "200 g" } ],
  "ambiguous": false,
  "candidates": [],
  "confidence": 0.94,
  "available": true,
  "recommended_locationid": 20,
  "stores": [
    {
      "tenantid": 2, "tenantname": "R Mart", "locationid": 20, "locationname": "Hopes",
      "latitude": 11.01, "longitude": 77.0, "distance_km": 3.8, "open": true,
      "deliveryradius": 5, "deliverymins": 30,
      "recommended": true, "available": true,
      "options": [
        { "productid": 200, "productname": "Milk Bikis 100g", "size": "100 g", "price": 12, "stock": 6,
          "available": true, "is_variant": false, "matched_by": "imageid", "image": "…" },
        { "productid": 201, "productname": "Milk Bikis 200g", "size": "200 g", "price": 22, "stock": 3,
          "available": true, "is_variant": true, "variantname": "200 g", "matched_by": "variant-of:200" }
      ]
    },
    { "locationid": 10, "locationname": "Peelamedu", "distance_km": 0.9, "available": false, "recommended": false,
      "options": [ { "productid": 100, "stock": 0, "available": false, "…": "…" } ] }
  ],
  "unregistered_tenantids": [],
  "message": "Available at 1 of your stores."
}
```

### Response B — several products fit, none clearly

`ambiguous: true`, `match: null`, `stores: []`. Show a "did you mean?" list.

```json
{
  "label": "britannia",
  "match": null,
  "ambiguous": true,
  "candidates": [
    { "brand": "britannia", "catalogueid": 23, "product_name": "Britannia Marie Gold",
      "size": "250 g", "image": "https://…", "score": 0.95, "method": "text", "available": true },
    { "brand": "britannia", "catalogueid": 22, "product_name": "Britannia Good Day Butter Cookies",
      "image": "https://…", "score": 0.95, "method": "text" },
    { "brand": "britannia", "catalogueid": 21, "product_name": "Britannia Good Day Cashew Cookies",
      "image": "https://…", "score": 0.95, "method": "text" }
  ],
  "confidence": 0.95,
  "available": false,
  "stores": [],
  "catalogue_variants": [],
  "message": "Which one is it? 1 of these 3 are in stock near you."
}
```

- **`confidence` is not low here, and that is not a bug.** "britannia" really
  does appear in all three names, so relevance is high — what is missing is
  *identification*. Gate on `ambiguous`, never on `confidence`: an app that
  reads 0.95 as "sure enough to show a price" reintroduces the exact bug this
  path exists to prevent.
- **`available` on a candidate** means at least one of the customer's
  registered stores has it in stock right now. Candidates are ordered
  available-first, so the list can show what is buyable before what is not
  — and the field is absent (not `false`) when unavailable, so read it as
  falsy, not as a required key.
- **To resolve a pick**, call `/lookup` again with that candidate's `brand`
  and `catalogueid` and no label. You get Response A for that exact product,
  with `method: "direct"` and `confidence: 1`.
- At most 10 candidates come back.

### How to read either response

- `match == null && !ambiguous` → nothing recognised; show `message` and let
  them retry with a clearer photo.
- `ambiguous: true` → ask, do not guess. Never show a price on this path;
  `stores` is deliberately empty.
- `confidence` below ~0.5 with a `match` → recognised but unsure; worth
  confirming the name before showing prices. `method: "text"` means no
  embedding model was involved (not configured, or it timed out) — be a
  little more cautious. `method: "direct"` means the caller named the
  product, so nothing was recognised at all.
- `stores` is ordered **in-stock first, then nearest**. Exactly one store has
  `recommended: true` — the nearest with stock — and only when `available`
  is true. Stores that sell it but have nothing on the shelf are still listed
  (so the customer understands why they are not recommended); stores that do
  not sell it are not.
- `options` are the things that can actually go in a basket at that store —
  the matched product and each of its sizes — each a real product with its
  own `productid`, price and live `stock`. Use `productid` in the existing
  cart/order calls exactly as you would from the catalogue screen.
- `distance_km: -1` means the distance is unknown (no fix from the phone and
  no saved address, or the store has no coordinates). Do not render it as 0.
  Send `latitude`/`longitude` on `/confirm` too if you display distance from
  its reply: the saved address is only consulted there when the shelf is
  empty and alternatives have to be ranked, so without a fix the store you
  tapped comes back `-1`.

## `POST /confirm`

Sent when the customer taps a store and an option. Re-reads live stock —
nothing is cached on this path.

```json
{ "customerid": 5123, "tenantid": 1, "locationid": 10, "productid": 100, "quantity": 2,
  "latitude": 11.029, "longitude": 77.029 }
```

```json
{
  "ok": false,
  "reason": "out_of_stock",           // in_stock | insufficient_stock | out_of_stock | not_sold_here | store_not_registered
  "store":  { "…the store they tapped…" },   // distance_km filled from the fix you send
  "option": { "productid": 100, "stock": 0, "…": "…" },
  "requested": 2,
  "alternative": {                     // absent when nobody has enough
    "locationid": 20, "locationname": "Hopes", "distance_km": 3.8, "recommended": true, "available": true,
    "options": [ { "productid": 200, "stock": 6, "price": 12, "…": "…" } ]
  },
  "message": "Out of stock at Peelamedu. Hopes has it (3.8 km away)."
}
```

`ok: true` → proceed to the basket. `ok: false` → show `message`; if
`alternative` is present offer it as a one-tap switch (it is the **same
product**, not another size — the customer chose a size and we do not
substitute). These are HTTP 200s: they are answers, not errors.

## `GET /stores?customerid=5123&latitude=11.029&longitude=77.029`

The customer's registered stores, nearest first, `distance_km: -1` last.
Same `ScanStore` shape as inside `stores[]` above, without options.

## Errors (HTTP status ≠ 200)

| Status | When |
|---|---|
| 400 | Missing `customerid`/`label`/ids, or a body that is not JSON. `message` says which. |
| 404 | `customerid` does not exist. |
| 503 | The catalogue database is not reachable. Retry later; the rest of the app is unaffected. |
| 500 | Anything else. Logged server-side. |

## Behind the curtain (for whoever operates it)

- **Recognition** = pgvector cosine search over every `brand_*` table in the
  catalogue (each with its own index, merged), plus a word match on
  `product_name`/`title`/`search_query` that settles near-ties and works on
  its own when no embedding model is configured. The model is set by
  `EMBEDDING_PROVIDER/MODEL/API_KEY` and **must** be the one that indexed
  the catalogue — the first search checks the vector width and refuses a
  mismatch by name.
- **The word match asks for most of the label, not all of it**
  (`minTokenHits`: two thirds, rounded up, and both of a two-word label).
  Requiring every word meant one word the catalogue does not use took the
  right product out of the running entirely — "Dettol bottle pack" retrieved
  no Dettol, "Parle G biscuit pack" retrieved no Parle-G — and the vector
  search then answered alone, confidently and wrongly, at a score the floor
  could not catch. Each brand's rows are ordered by how much of the label
  they carry (the whole label as a substring outranks any number of loose
  words) so that the per-brand `LIMIT` keeps the best rows and not merely the
  first ones the planner reached. Packaging words — "pack", "bottle", "jar",
  "sachet" and friends, see `utils.isPackaging` — are dropped before any of
  this, like pack sizes, unless the label is nothing else.
- **The catalogue's model** (verified 2026-09-15 by cosine against a stored
  row: 1.0000): `all-MiniLM-L6-v2`, 384-d, unit-normalised, embedding the
  `search_query` column (brand + name + category + blurb + price range).
  Ollama ships it as `all-minilm`; the cluster's `ollama.krow` service serves
  it, so production is:
  ```
  EMBEDDING_PROVIDER=openai
  EMBEDDING_BASE_URL=http://ollama.krow.svc.cluster.local:11434/v1
  EMBEDDING_MODEL=all-minilm
  EMBEDDING_API_KEY=ollama        # any non-empty value; Ollama ignores it
  EMBEDDING_DIMENSIONS=384
  ```
  A bare label ("Milk Bikis") scores ~0.92 against its product's stored
  vector and ~0.23 against an unrelated one, which is what the 0.50 floor in
  `scanService.go` is set against — the middle of that split, not the edge of
  the noise. It was 0.30 until a near-miss got through in production
  ("Paracetamol" → "Paneer Makhni 500ml", 0.304). If the catalogue team ever
  re-embeds with another model, change `EMBEDDING_MODEL`/`DIMENSIONS` here
  and nothing else.
- **Speed**: the label's vector (7 days) and the ranked catalogue hits
  (30 min) are cached in Redis and in-process, so a popular product costs
  one model call platform-wide. Customer, stores and catalogue are read
  concurrently; the whole lookup is capped at 5 s and a slow model degrades
  to a text answer instead of a spinner. Live stock is one indexed query and
  is never cached.
- **Availability** is the same rule the app's catalogue screen uses:
  `products.approve = 1`, `productlocations.publishedat IS NOT NULL`, stock =
  live `SUM(in) − SUM(out)` of `productstocks` at that outlet, price = the
  outlet's own price else the tenant's retail price.
- **No reservation.** Confirm re-reads the ledger; a hold would give the
  same answer with a timer to babysit. If contention becomes real, a
  Redis-backed short hold slots in at `Confirm` without changing the API.
- **Identity** is the `customerid` in the body, like every other mobile
  endpoint here — there is no auth layer yet (see `SECURITY_HANDOFF.md`).

## Two decisions, and why

Both come from a proposal (2026-09-23) to have the app send vectors it
computed on the phone. Recorded here because the next person will ask.

### The app does not send `textvector`

An on-device MiniLM vector is only comparable to the catalogue's if the app
ships the identical model *and* tokenizer *and* pooling *and* normalisation;
a quantised tflite build usually drifts, and the failure is silent — the
ranking just gets worse. There is also nothing to gain: the server-side
embed is ~30 ms warm and the result is cached in Redis by label, so one
model call serves every customer who scans that product. A client-supplied
vector *defeats* that cache (the key would have to be the vector, not the
label), and 384 floats is ~5 KB of upload against ~12 bytes for
`"Milk Bikis"`. If the field ever arrives it can be accepted and validated,
but the app should not be asked to compute it.

**Send the full OCR text instead** if you want to give the server more to
work with — ~100 bytes, no model coupling, strictly more information than a
single label.

### The app does not send `imagevector` — yet

The catalogue *does* carry image vectors: every `brand_*` table has
`img_vector vector(1024)`, filled on 1885 of 2124 rows (empty in
`brand_haldirams`, `brand_kaleesuwari`, `brand_mdh`, `brand_zzsmoketest`).
That matches the proposed MobileNetV3-Small embedder, so the idea is
coherent and half-built — this flow simply does not read that column.

It stays unread for now because **Google Lens is already the image
recogniser, and a far better one**: photo → Lens → label is Google's product
recognition, trained on billions of images. Putting a 137M-parameter
ImageNet backbone searching 1885 vectors *behind* that adds little where
Lens succeeds, and MobileNetV3-Small — which struggles to tell one blue
biscuit wrapper from another — is unlikely to rescue the cases where Lens
fails. There is also an unverified dependency: the preprocessing the app
would use (BGR → centre crop → 224×224 INTER_AREA → RGB → `/255.0`) has to
match whatever the catalogue pipeline actually ran, or the search returns
confidently-ranked noise.

**What would change this:** the field data. Once live, count how often
`/lookup` returns `ambiguous: true` or nothing recognised. If Lens labels are
reliable, image search is polish; if that number is high, it becomes the
priority — and the first task is the cosine check (embed a known catalogue
product's image through the app's exact pipeline, compare with its stored
`img_vector`; ≈0.99 means the contract holds), not writing the query.

There is one non-recognition argument for it worth remembering: on-device
inference is free and needs no Google dependency, which matters if Cloud
Vision costs start to bite at volume. That is a business reason, not a
quality one.

## For backend developers

### Where the code is

| File | Holds |
|---|---|
| `models/scan.go` | request/response shapes (`ScanLookupRequest`, `ScanStoreOffer`, `ScanOption`, …) |
| `repositories/scanRepository.go` | all SQL: registered stores, live options, vector + text search, the two-tier cache |
| `services/scanService.go` | the pipeline: parallel reads, scoring, family grouping, ranking, confirm fallback |
| `controllers/scanController.go` | the three handlers and the error → status mapping |
| `routes/scanroutes.go` | `/v1/mob/scan/*` |
| `utils/embedding.go` | `Embedder` interface, OpenAI-compatible and Gemini clients |
| `utils/geo.go` | coordinate parsing, haversine, opening hours, label tokenising |
| `config/config.go` | `EmbeddingConfig` and its validation |
| `scratch/cataloguedims` | read-only check of every catalogue vector column's width and fill |

### Try it locally

```sh
go run .    # with the local compose stack; EMBEDDING_* unset → text-only, still works
curl -s localhost:1122/live/api/v1/mob/scan/lookup -H 'Content-Type: application/json' \
  -d '{"customerid":1,"label":"Milk Bikis","latitude":11.03,"longitude":77.03}' | jq .details
```

To exercise the vector path locally, run Ollama on your Mac
(`ollama pull all-minilm`) and set `EMBEDDING_PROVIDER=openai`,
`EMBEDDING_BASE_URL=http://localhost:11434/v1`, `EMBEDDING_MODEL=all-minilm`,
`EMBEDDING_API_KEY=ollama`, `EMBEDDING_DIMENSIONS=384` in `.env.local`. The
local catalogue must carry vectors from the same model for results to mean
anything; a schema-only dump does not.

### Tests

`go test ./services -run 'Lookup|Confirm|Stores|Brand|Ambiguous|Specific|TextScore|Distinct|Naming'`
drives the whole pipeline through a fake repository
(`services/scan_test.go`); no database. `go test ./utils` covers both HTTP
clients against `httptest` servers, and the geo helpers. Add a case to
`scan_test.go`'s fixture when you change ranking — it is the spec, and
`newBrandLabelFixture` in particular is the regression guard for the
brand-name bug described under Scoring.

### Knobs (constants in `scanService.go`)

| Constant | Default | Effect |
|---|---|---|
| `scanLookupTimeout` | 5 s | whole lookup, including the model call |
| `scanCatalogueTopK` | 15 | rows taken from each brand table and from the merge |
| `scanMinScore` | 0.50 | below this the best hit is not shown as a match |
| `scanAmbiguityMargin` | 0.06 | how close the runner-up may be before the answer becomes a question |
| `scanMaxCandidates` | 10 | longest "did you mean?" list |
| `embedTimeout` (`utils/embedding.go`) | 4 s | one model call |
| `scanVectorTTL` / `scanHitsTTL` (`scanRepository.go`) | 7 d / 30 min | cache lifetimes |

Scores: vector = `1 − cosine distance`; text = 0.95 for the whole label
inside the name, else `0.8 × (label words found / label words)`; combined =
`max(vector, text) + 0.10` when both hit, capped at 1. Ties are broken by
cosine distance — nearest first, a text-only row last — and only then by
name.

The label and the product name are both separator-folded before that
substring test (`utils.FoldSeparators`), and compared again with separators
removed (`utils.TightenLabel`, labels of 4+ characters), so the brand's own
punctuation does not decide the match: "Parle G", "Parle-G" and "ParleG" all
reach *Parle-G Original Glucose Biscuits*. A single-character token survives
tokenising when it follows a word, because it is often the whole name — the
"G" of Parle-G, the "K" of Special K. It is still dropped when it stands
alone or is a pack multiplier.

All three mattered at once: before this, "Parle G" tied with *Parle Monaco
Classic* at 0.9 (the "G" was dropped, so only "parle" matched either row),
and the name tie-break handed it to Monaco because a space precedes a hyphen
in ASCII. A confident, wrong answer — the kind no score floor can catch.

**When the substring rule ties, that tie is the answer.** A bare brand name
is a substring of every one of that brand's names, so all of them score 0.95
— identically, at a high score no floor would ever catch. Rather than
scoring around it, `isAmbiguous` reads it: if the runner-up is within
`scanAmbiguityMargin` of the leader, the reply becomes `ambiguous: true`
with `candidates` instead of a match (see Response B). Erring towards asking
is deliberate — one tap on a picture against the wrong biscuit. A label that
names one product leaves the runner-up far behind, so the common case is
untouched, and `services/scan_test.go`'s
`TestABrandNameScoresItsProductsIdentically` guards the tie itself: a
formula that broke it on name length or word count would bring the bug
back.

### Changing the embedding model

1. The catalogue team re-embeds `search_query` with the new model.
2. Serve it (Ollama pull, or a hosted key).
3. Change `EMBEDDING_MODEL` / `EMBEDDING_DIMENSIONS` (and provider/URL if
   needed) in the cluster; roll the pods.
4. Flush the hit cache if you cannot wait 30 min: keys are
   `scan:hits:v1:*` and `scan:emb:v1:*` in Redis (they are also keyed by
   model name, so old entries simply stop being read).

Nothing in Go changes. A width mismatch fails the first search with an error
naming both numbers.

### Adding a provider

Implement `utils.Embedder` (`Embed(ctx, text) ([]float32, error)` and
`Model() string`), add a case to `NewEmbedder`, and add the provider name to
the allow-list in `config.validate`. Keep the HTTP client timeout: the
customer is holding a phone.