feat/env-login-scan-to-order #5

Suriya · 2026-09-23T05:38:30Z

Suriya commented

2026-09-23 05:38:30 +00:00

Scan-to-order: ask when a label fits several products

A bare brand label resolved to one arbitrary product and quoted a price for it.
This makes /lookup return a "did you mean?" list instead. textScore is
untouched — the fix reads the tie it already produces.

Found from the example request in the mobile team's own proposal:
{"label": "britannia"}.

The bug

"britannia" is a substring of all 258 Britannia product names, so textScore
scored every one of them 0.95. Nothing downstream read that tie, so the sort
order picked a winner and the customer was shown one arbitrary biscuit with
"confidence": 0.95 and a price on it.

Lens returns a bare wordmark often — it is usually the biggest thing printed on
a packet — so this was the common path, not an edge case.

What changed

isAmbiguous: if the runner-up is within scanAmbiguityMargin (0.06) of the
leader, the reply becomes ambiguous: true with candidates instead of a
match. match is nil and stores empty on that path — no price for a product
nobody chose.

Candidates are distinct products, not pack sizes (distinctProducts, keyed
on variant_key else name within a brand), capped at 10, and each is marked
with whether one of that customer's registered stores has it in stock —
available ones first. That availability read reuses the same StoreOptions
query the confident path already runs, widened to every candidate, so the list
can show what is buyable before what is not.

To act on a pick, /lookup now accepts brand + catalogueid instead of a
label and skips recognition entirely (method: "direct", confidence: 1).
Also serves deep links and "buy again". New: ScanRepository.CatalogueRef,
resolving the brand through the tables discovered from information_schema,
never a name built from the request.

What deliberately did NOT change

textScore, the 0.50 floor, minTokenHits, separator folding, and the
cosine tie-break are all exactly as they are on main.

This branch originally replaced the substring rule with a coverage formula
(harmonic mean of label-coverage and name-coverage) to stop the arbitrary pick.
That was the wrong half to change, and the merge drops it:

The substring rule scoring a brand's products identically is not the bug,
it is the signal. isAmbiguous reads it, so a scoring change adds nothing.
The coverage formula broke "Parle G": SearchTokens drops the single
character, so coverage alone cannot see Parle-G Original Glucose Biscuits
at all. The folding on main handles it properly.

Dropped with it: scanSpecificEnough, the per-hit text score, and a
proportional confirmation bonus — the flat +0.10 is back. Net result is less
code than the branch started with, and none of main's tuning is touched.

"britannia"  → all rows tie at 0.95 → ambiguous: true + candidates
"Parle G"    → folding + the single-character token still land it
a real name  → runner-up far behind → match, unchanged

API change (mobile)

/lookup has two possible answers now and the app must handle both. Fully
specified in docs/SCAN_TO_ORDER.md (Response A / Response B).

One thing worth flagging to the app developer: confidence stays high on the
ambiguous path — 0.95, because "britannia" genuinely does appear in all
those names. What is missing is identification, not relevance. Gate on
ambiguous, never on confidence; an app that reads 0.95 as "safe to show a
price" reintroduces exactly this bug. The doc says so in those words.

Tests

25 scan tests pass together, both sides' included. Full suite, go build and
go vet clean.

TestTextScoreRewardsSpecificityNotJustOverlap tested the removed formula and
is replaced by TestABrandNameScoresItsProductsIdentically, which guards the
tie itself — a formula that broke it on name length or word count would bring
the bug back. newBrandLabelFixture guards the end-to-end behaviour.

go test ./services -run 'Lookup|Confirm|Stores|Brand|Ambiguous|Specific|Distinct|Naming|SortHits|Hyphen' -v

Notes for review

@abhishek — the merge commit (24339a8) is the interesting one; it explains why
your textScore survived unchanged. Your two production findings are both
preserved and now documented in docs/SCAN_TO_ORDER.md: the 0.304
"Paracetamol" → "Paneer Makhni 500ml" near-miss behind the 0.50 floor, and
"Parle G" losing to "Parle Monaco Classic" on an ASCII tie-break.

Two things found on the side, neither acted on in this PR:

The catalogue also carries img_vector vector(1024) — filled on 1885 of
2124 rows, empty in brand_haldirams, brand_kaleesuwari, brand_mdh,
brand_zzsmoketest. This flow does not read it. The reasoning, and the
specific field measurement that would reverse the decision, is under "Two
decisions, and why" in the doc — worth reading before that thread is
reopened, since the mobile team proposed sending image vectors from the
phone.
scratch/cataloguedims now reports every vector column rather than only
embedding, which is how (1) came to light.

Scan-to-order: ask when a label fits several products --- A bare brand label resolved to one arbitrary product and quoted a price for it. This makes `/lookup` return a "did you mean?" list instead. `textScore` is untouched — the fix reads the tie it already produces. Found from the example request in the mobile team's own proposal: `{"label": "britannia"}`. ## The bug `"britannia"` is a substring of all 258 Britannia product names, so `textScore` scored every one of them 0.95. Nothing downstream read that tie, so the sort order picked a winner and the customer was shown one arbitrary biscuit with `"confidence": 0.95` and a price on it. Lens returns a bare wordmark often — it is usually the biggest thing printed on a packet — so this was the common path, not an edge case. ## What changed `isAmbiguous`: if the runner-up is within `scanAmbiguityMargin` (0.06) of the leader, the reply becomes `ambiguous: true` with `candidates` instead of a match. `match` is nil and `stores` empty on that path — no price for a product nobody chose. Candidates are **distinct products, not pack sizes** (`distinctProducts`, keyed on `variant_key` else name within a brand), capped at 10, and each is marked with whether one of *that customer's* registered stores has it in stock — available ones first. That availability read reuses the same `StoreOptions` query the confident path already runs, widened to every candidate, so the list can show what is buyable before what is not. To act on a pick, `/lookup` now accepts `brand` + `catalogueid` instead of a label and skips recognition entirely (`method: "direct"`, `confidence: 1`). Also serves deep links and "buy again". New: `ScanRepository.CatalogueRef`, resolving the brand through the tables discovered from `information_schema`, never a name built from the request. ## What deliberately did NOT change **`textScore`, the 0.50 floor, `minTokenHits`, separator folding, and the cosine tie-break are all exactly as they are on main.** This branch originally replaced the substring rule with a coverage formula (harmonic mean of label-coverage and name-coverage) to stop the arbitrary pick. That was the wrong half to change, and the merge drops it: - The substring rule scoring a brand's products **identically** is not the bug, it is the signal. `isAmbiguous` reads it, so a scoring change adds nothing. - The coverage formula **broke "Parle G"**: `SearchTokens` drops the single character, so coverage alone cannot see `Parle-G Original Glucose Biscuits` at all. The folding on main handles it properly. Dropped with it: `scanSpecificEnough`, the per-hit text score, and a proportional confirmation bonus — the flat `+0.10` is back. Net result is less code than the branch started with, and none of main's tuning is touched. ``` "britannia" → all rows tie at 0.95 → ambiguous: true + candidates "Parle G" → folding + the single-character token still land it a real name → runner-up far behind → match, unchanged ``` ## API change (mobile) `/lookup` has two possible answers now and the app must handle both. Fully specified in `docs/SCAN_TO_ORDER.md` (Response A / Response B). One thing worth flagging to the app developer: **`confidence` stays high on the ambiguous path** — 0.95, because `"britannia"` genuinely does appear in all those names. What is missing is identification, not relevance. Gate on `ambiguous`, never on `confidence`; an app that reads 0.95 as "safe to show a price" reintroduces exactly this bug. The doc says so in those words. ## Tests 25 scan tests pass together, both sides' included. Full suite, `go build` and `go vet` clean. `TestTextScoreRewardsSpecificityNotJustOverlap` tested the removed formula and is replaced by `TestABrandNameScoresItsProductsIdentically`, which guards the tie itself — a formula that broke it on name length or word count would bring the bug back. `newBrandLabelFixture` guards the end-to-end behaviour. ``` go test ./services -run 'Lookup|Confirm|Stores|Brand|Ambiguous|Specific|Distinct|Naming|SortHits|Hyphen' -v ``` ## Notes for review @abhishek — the merge commit (`24339a8`) is the interesting one; it explains why your `textScore` survived unchanged. Your two production findings are both preserved and now documented in `docs/SCAN_TO_ORDER.md`: the 0.304 "Paracetamol" → "Paneer Makhni 500ml" near-miss behind the 0.50 floor, and "Parle G" losing to "Parle Monaco Classic" on an ASCII tie-break. Two things found on the side, neither acted on in this PR: 1. The catalogue also carries **`img_vector vector(1024)`** — filled on 1885 of 2124 rows, empty in `brand_haldirams`, `brand_kaleesuwari`, `brand_mdh`, `brand_zzsmoketest`. This flow does not read it. The reasoning, and the specific field measurement that would reverse the decision, is under "Two decisions, and why" in the doc — worth reading before that thread is reopened, since the mobile team proposed sending image vectors from the phone. 2. `scratch/cataloguedims` now reports **every** vector column rather than only `embedding`, which is how (1) came to light.

Suriya added 2 commits 2026-09-23 05:38:31 +00:00

Ask instead of guessing when a label fits several products 01bc89ab77

`"britannia"` is a substring of all 258 Britannia product names, and
textScore returned 0.95 for any product whose name contained the label.
So every one of them tied, the tie broke alphabetically, and the customer
was shown one arbitrary biscuit with "confidence": 0.95 and a price. Lens
hands back a bare wordmark often — it is usually the biggest thing printed
on a packet — so this was the common case, not an edge one. Found via the
example request in the mobile team's own proposal.

Scoring now asks both questions. A hit carries `score` (ranks) and `text`
(how specifically the label names THIS product: the harmonic mean of how
much of the label the product explains and how much of the product's name
the label explains, pack sizes dropped from both sides). A brand name
scores its products ~0.33 equally instead of 0.95 arbitrarily. The
"vector and text agree" bonus is now proportional to the text score, so a
weak match can no longer inflate a whole brand.

isAmbiguous reads that: the leader is a guess if anything is level with it
(margin) or if the label names no one product (specificity), and then the
response carries `ambiguous: true` with `candidates` — distinct products,
not pack sizes, at most ten, each marked with whether one of the
customer's stores has it in stock, available ones first. `match` is nil
and `stores` empty on that path: no price for a product nobody chose.
Erring towards asking is deliberate — a tap versus the wrong biscuit.

To act on a pick, /lookup now accepts `brand` + `catalogueid` instead of a
label and skips recognition entirely (also serves deep links and re-order).
New: ScanRepository.CatalogueRef, resolving via the brand tables discovered
from information_schema, never a name built from the request.

Also: scratch/cataloguedims now reports every vector column, not just
`embedding` — which is how we learned the catalogue also carries
img_vector(1024), filled on 1885 of 2124 rows. SCAN_TO_ORDER.md records
why that column stays unread for now and what would change it, alongside
why the app is not asked to compute vectors on the phone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Merge origin/main: keep the substring rule, read its tie 24339a8b51

main had moved on with retrieval work validated against real queries —
minTokenHits (the word match needs two thirds of the label, not all of
it), separator folding so "Parle G"/"Parle-G"/"ParleG" all reach Parle-G,
the floor at 0.50 after "Paracetamol" came back as "Paneer Makhni 500ml"
at 0.304, and ties broken on cosine distance instead of name. All of that
is kept exactly as it was.

The conflict was in textScore: this branch replaced the substring rule
with a coverage formula to stop a bare brand name resolving to one
arbitrary product. That is the wrong half to change. The substring rule
scores every product of a brand 0.95 IDENTICALLY, and that tie is not the
bug — it is the signal. isAmbiguous reads it, so the branch's coverage
rewrite is dropped and the ambiguity layer alone does the work:

  "britannia" → all 258 rows tie at 0.95 → ambiguous: true + candidates
  "Parle G"   → folding and the single-character token still land it
  a real name → runner-up far behind → match, unchanged

Dropped with it: scanSpecificEnough, the per-hit text score, and the
proportional confirmation bonus — the flat +0.10 is back. Simpler, and it
leaves main's tuning untouched.

TestTextScoreRewardsSpecificityNotJustOverlap tested the removed formula
and is replaced by TestABrandNameScoresItsProductsIdentically, which
guards the tie itself: a formula that broke it on name length or word
count would bring the bug back.

Docs carry both rationales, and now say plainly that confidence stays
high on the ambiguous path — gate on `ambiguous`, never on `confidence`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This branch is already included in the target branch. There is nothing to merge.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feat/env-login-scan-to-order:feat/env-login-scan-to-order

git checkout feat/env-login-scan-to-order

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: nearle_daily/backend_fiesta#5