Compare commits

..

2 Commits

Author SHA1 Message Date
24339a8b51 Merge origin/main: keep the substring rule, read its tie
main had moved on with retrieval work validated against real queries —
minTokenHits (the word match needs two thirds of the label, not all of
it), separator folding so "Parle G"/"Parle-G"/"ParleG" all reach Parle-G,
the floor at 0.50 after "Paracetamol" came back as "Paneer Makhni 500ml"
at 0.304, and ties broken on cosine distance instead of name. All of that
is kept exactly as it was.

The conflict was in textScore: this branch replaced the substring rule
with a coverage formula to stop a bare brand name resolving to one
arbitrary product. That is the wrong half to change. The substring rule
scores every product of a brand 0.95 IDENTICALLY, and that tie is not the
bug — it is the signal. isAmbiguous reads it, so the branch's coverage
rewrite is dropped and the ambiguity layer alone does the work:

  "britannia" → all 258 rows tie at 0.95 → ambiguous: true + candidates
  "Parle G"   → folding and the single-character token still land it
  a real name → runner-up far behind → match, unchanged

Dropped with it: scanSpecificEnough, the per-hit text score, and the
proportional confirmation bonus — the flat +0.10 is back. Simpler, and it
leaves main's tuning untouched.

TestTextScoreRewardsSpecificityNotJustOverlap tested the removed formula
and is replaced by TestABrandNameScoresItsProductsIdentically, which
guards the tie itself: a formula that broke it on name length or word
count would bring the bug back.

Docs carry both rationales, and now say plainly that confidence stays
high on the ambiguous path — gate on `ambiguous`, never on `confidence`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-23 11:05:03 +05:30
01bc89ab77 Ask instead of guessing when a label fits several products
`"britannia"` is a substring of all 258 Britannia product names, and
textScore returned 0.95 for any product whose name contained the label.
So every one of them tied, the tie broke alphabetically, and the customer
was shown one arbitrary biscuit with "confidence": 0.95 and a price. Lens
hands back a bare wordmark often — it is usually the biggest thing printed
on a packet — so this was the common case, not an edge one. Found via the
example request in the mobile team's own proposal.

Scoring now asks both questions. A hit carries `score` (ranks) and `text`
(how specifically the label names THIS product: the harmonic mean of how
much of the label the product explains and how much of the product's name
the label explains, pack sizes dropped from both sides). A brand name
scores its products ~0.33 equally instead of 0.95 arbitrarily. The
"vector and text agree" bonus is now proportional to the text score, so a
weak match can no longer inflate a whole brand.

isAmbiguous reads that: the leader is a guess if anything is level with it
(margin) or if the label names no one product (specificity), and then the
response carries `ambiguous: true` with `candidates` — distinct products,
not pack sizes, at most ten, each marked with whether one of the
customer's stores has it in stock, available ones first. `match` is nil
and `stores` empty on that path: no price for a product nobody chose.
Erring towards asking is deliberate — a tap versus the wrong biscuit.

To act on a pick, /lookup now accepts `brand` + `catalogueid` instead of a
label and skips recognition entirely (also serves deep links and re-order).
New: ScanRepository.CatalogueRef, resolving via the brand tables discovered
from information_schema, never a name built from the request.

Also: scratch/cataloguedims now reports every vector column, not just
`embedding` — which is how we learned the catalogue also carries
img_vector(1024), filled on 1885 of 2124 rows. SCAN_TO_ORDER.md records
why that column stays unread for now and what would change it, alongside
why the app is not asked to compute vectors on the phone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-23 10:56:02 +05:30
6 changed files with 729 additions and 55 deletions

View File

@@ -8,6 +8,10 @@ recommend. When the customer taps a store and a size, a second call confirms
the shelf still has it — and if it does not, names the next-nearest store the shelf still has it — and if it does not, names the next-nearest store
that does. that does.
When the label fits several products — `"britannia"` names 258 of them — it
answers with a short "did you mean?" list instead of picking one, because a
confident price on the wrong biscuit is worse than one extra tap.
Base path: `/live/api/v1/mob/scan`. Every response uses the usual envelope Base path: `/live/api/v1/mob/scan`. Every response uses the usual envelope
`{ code, status, message, details }`; the shapes below are `details`. `{ code, status, message, details }`; the shapes below are `details`.
@@ -17,7 +21,13 @@ Base path: `/live/api/v1/mob/scan`. Every response uses the usual envelope
photo ──Lens──▶ label photo ──Lens──▶ label
│ │
▼ ▼
POST /lookup ───▶ match + stores[] (recommended first) POST /lookup ───▶ ambiguous:true + candidates[] "did you mean?"
│ │
│ customer taps one candidate
│ │
│ POST /lookup { brand, catalogueid }
│ │
└───▶ match + stores[] (recommended first) ◀──┘
│ │
customer taps a store + a size customer taps a store + a size
│ │
@@ -26,11 +36,21 @@ photo ──Lens──▶ label
ok:false + alternative → offer the other store ok:false + alternative → offer the other store
``` ```
**`/lookup` has two possible answers and the app must handle both.** A label
that names one product comes back with `match` + `stores`. A label that fits
several — a bare brand name like `"britannia"`, a generic word like
`"biscuits"` — comes back with `ambiguous: true` and `candidates`, and the
app asks the customer which one before any price is shown. Lens returns a
bare wordmark often, because it is usually the biggest thing printed on a
packet, so this is a normal path and not an error case.
`GET /stores` is for the "choose another shop" sheet: the customer's `GET /stores` is for the "choose another shop" sheet: the customer's
registered stores, nearest first, independent of any product. registered stores, nearest first, independent of any product.
## `POST /lookup` ## `POST /lookup`
Note the `//` notes below are annotations, not JSON — strip them.
```json ```json
{ {
"customerid": 5123, "customerid": 5123,
@@ -39,16 +59,25 @@ registered stores, nearest first, independent of any product.
"longitude": 77.0290, "longitude": 77.0290,
"tenantids": [1135, 1140], // optional: what the app THINKS the customer joined "tenantids": [1135, 1140], // optional: what the app THINKS the customer joined
"limit": 0 // optional: max stores, 0 = all "limit": 0 // optional: max stores, 0 = all
// Instead of a label: name the product outright. This is how you resolve
// a candidate the customer tapped, and how a deep link or a "buy again"
// skips recognition. With both set, `label` is ignored.
// "brand": "britannia", "catalogueid": 7
} }
``` ```
`label` is required **unless** `brand` and `catalogueid` are both given.
`tenantids` is verified, never trusted: the server intersects it with the `tenantids` is verified, never trusted: the server intersects it with the
`tenantcustomers` table. Ids the customer is not actually registered with `tenantcustomers` table. Ids the customer is not actually registered with
come back in `unregistered_tenantids` — treat that as "refresh the local come back in `unregistered_tenantids` — treat that as "refresh the local
list". A list that matches nothing at all is treated as stale and all list". A list that matches nothing at all is treated as stale and all
registered stores are used. registered stores are used.
Response: ### Response A — one product identified
`ambiguous: false`, `match` set, `candidates` empty.
```json ```json
{ {
@@ -59,6 +88,8 @@ Response:
"image": "https://…", "score": 0.94, "method": "vector+text" "image": "https://…", "score": 0.94, "method": "vector+text"
}, },
"catalogue_variants": [ { "…same shape…": "100 g" }, { "…": "200 g" } ], "catalogue_variants": [ { "…same shape…": "100 g" }, { "…": "200 g" } ],
"ambiguous": false,
"candidates": [],
"confidence": 0.94, "confidence": 0.94,
"available": true, "available": true,
"recommended_locationid": 20, "recommended_locationid": 20,
@@ -83,12 +114,57 @@ Response:
} }
``` ```
How to read it: ### Response B — several products fit, none clearly
- `match == null` → nothing recognised; show `message` and let them retry. `ambiguous: true`, `match: null`, `stores: []`. Show a "did you mean?" list.
`confidence` below ~0.5 → recognised but unsure; confirm the name with the
customer before showing prices. `method: "text"` means no embedding model ```json
was involved (not configured, or it timed out) — be a little more cautious. {
"label": "britannia",
"match": null,
"ambiguous": true,
"candidates": [
{ "brand": "britannia", "catalogueid": 23, "product_name": "Britannia Marie Gold",
"size": "250 g", "image": "https://…", "score": 0.95, "method": "text", "available": true },
{ "brand": "britannia", "catalogueid": 22, "product_name": "Britannia Good Day Butter Cookies",
"image": "https://…", "score": 0.95, "method": "text" },
{ "brand": "britannia", "catalogueid": 21, "product_name": "Britannia Good Day Cashew Cookies",
"image": "https://…", "score": 0.95, "method": "text" }
],
"confidence": 0.95,
"available": false,
"stores": [],
"catalogue_variants": [],
"message": "Which one is it? 1 of these 3 are in stock near you."
}
```
- **`confidence` is not low here, and that is not a bug.** "britannia" really
does appear in all three names, so relevance is high — what is missing is
*identification*. Gate on `ambiguous`, never on `confidence`: an app that
reads 0.95 as "sure enough to show a price" reintroduces the exact bug this
path exists to prevent.
- **`available` on a candidate** means at least one of the customer's
registered stores has it in stock right now. Candidates are ordered
available-first, so the list can show what is buyable before what is not
— and the field is absent (not `false`) when unavailable, so read it as
falsy, not as a required key.
- **To resolve a pick**, call `/lookup` again with that candidate's `brand`
and `catalogueid` and no label. You get Response A for that exact product,
with `method: "direct"` and `confidence: 1`.
- At most 10 candidates come back.
### How to read either response
- `match == null && !ambiguous` → nothing recognised; show `message` and let
them retry with a clearer photo.
- `ambiguous: true` → ask, do not guess. Never show a price on this path;
`stores` is deliberately empty.
- `confidence` below ~0.5 with a `match` → recognised but unsure; worth
confirming the name before showing prices. `method: "text"` means no
embedding model was involved (not configured, or it timed out) — be a
little more cautious. `method: "direct"` means the caller named the
product, so nothing was recognised at all.
- `stores` is ordered **in-stock first, then nearest**. Exactly one store has - `stores` is ordered **in-stock first, then nearest**. Exactly one store has
`recommended: true` — the nearest with stock — and only when `available` `recommended: true` — the nearest with stock — and only when `available`
is true. Stores that sell it but have nothing on the shelf are still listed is true. Stores that sell it but have nothing on the shelf are still listed
@@ -205,6 +281,59 @@ Same `ScanStore` shape as inside `stores[]` above, without options.
- **Identity** is the `customerid` in the body, like every other mobile - **Identity** is the `customerid` in the body, like every other mobile
endpoint here — there is no auth layer yet (see `SECURITY_HANDOFF.md`). endpoint here — there is no auth layer yet (see `SECURITY_HANDOFF.md`).
## Two decisions, and why
Both come from a proposal (2026-09-23) to have the app send vectors it
computed on the phone. Recorded here because the next person will ask.
### The app does not send `textvector`
An on-device MiniLM vector is only comparable to the catalogue's if the app
ships the identical model *and* tokenizer *and* pooling *and* normalisation;
a quantised tflite build usually drifts, and the failure is silent — the
ranking just gets worse. There is also nothing to gain: the server-side
embed is ~30 ms warm and the result is cached in Redis by label, so one
model call serves every customer who scans that product. A client-supplied
vector *defeats* that cache (the key would have to be the vector, not the
label), and 384 floats is ~5 KB of upload against ~12 bytes for
`"Milk Bikis"`. If the field ever arrives it can be accepted and validated,
but the app should not be asked to compute it.
**Send the full OCR text instead** if you want to give the server more to
work with — ~100 bytes, no model coupling, strictly more information than a
single label.
### The app does not send `imagevector` — yet
The catalogue *does* carry image vectors: every `brand_*` table has
`img_vector vector(1024)`, filled on 1885 of 2124 rows (empty in
`brand_haldirams`, `brand_kaleesuwari`, `brand_mdh`, `brand_zzsmoketest`).
That matches the proposed MobileNetV3-Small embedder, so the idea is
coherent and half-built — this flow simply does not read that column.
It stays unread for now because **Google Lens is already the image
recogniser, and a far better one**: photo → Lens → label is Google's product
recognition, trained on billions of images. Putting a 137M-parameter
ImageNet backbone searching 1885 vectors *behind* that adds little where
Lens succeeds, and MobileNetV3-Small — which struggles to tell one blue
biscuit wrapper from another — is unlikely to rescue the cases where Lens
fails. There is also an unverified dependency: the preprocessing the app
would use (BGR → centre crop → 224×224 INTER_AREA → RGB → `/255.0`) has to
match whatever the catalogue pipeline actually ran, or the search returns
confidently-ranked noise.
**What would change this:** the field data. Once live, count how often
`/lookup` returns `ambiguous: true` or nothing recognised. If Lens labels are
reliable, image search is polish; if that number is high, it becomes the
priority — and the first task is the cosine check (embed a known catalogue
product's image through the app's exact pipeline, compare with its stored
`img_vector`; ≈0.99 means the contract holds), not writing the query.
There is one non-recognition argument for it worth remembering: on-device
inference is free and needs no Google dependency, which matters if Cloud
Vision costs start to bite at volume. That is a business reason, not a
quality one.
## For backend developers ## For backend developers
### Where the code is ### Where the code is
@@ -219,7 +348,7 @@ Same `ScanStore` shape as inside `stores[]` above, without options.
| `utils/embedding.go` | `Embedder` interface, OpenAI-compatible and Gemini clients | | `utils/embedding.go` | `Embedder` interface, OpenAI-compatible and Gemini clients |
| `utils/geo.go` | coordinate parsing, haversine, opening hours, label tokenising | | `utils/geo.go` | coordinate parsing, haversine, opening hours, label tokenising |
| `config/config.go` | `EmbeddingConfig` and its validation | | `config/config.go` | `EmbeddingConfig` and its validation |
| `scratch/cataloguedims` | read-only check of the catalogue's embedding width / fill | | `scratch/cataloguedims` | read-only check of every catalogue vector column's width and fill |
### Try it locally ### Try it locally
@@ -238,11 +367,13 @@ anything; a schema-only dump does not.
### Tests ### Tests
`go test ./services -run 'Lookup|Confirm|Stores|CatalogueFamily'` drives `go test ./services -run 'Lookup|Confirm|Stores|Brand|Ambiguous|Specific|TextScore|Distinct|Naming'`
the whole pipeline through a fake repository (`services/scan_test.go`); no drives the whole pipeline through a fake repository
database. `go test ./utils` covers both HTTP clients against `httptest` (`services/scan_test.go`); no database. `go test ./utils` covers both HTTP
servers, and the geo helpers. Add a case to `scan_test.go`'s fixture when clients against `httptest` servers, and the geo helpers. Add a case to
you change ranking — it is the spec. `scan_test.go`'s fixture when you change ranking — it is the spec, and
`newBrandLabelFixture` in particular is the regression guard for the
brand-name bug described under Scoring.
### Knobs (constants in `scanService.go`) ### Knobs (constants in `scanService.go`)
@@ -251,6 +382,8 @@ you change ranking — it is the spec.
| `scanLookupTimeout` | 5 s | whole lookup, including the model call | | `scanLookupTimeout` | 5 s | whole lookup, including the model call |
| `scanCatalogueTopK` | 15 | rows taken from each brand table and from the merge | | `scanCatalogueTopK` | 15 | rows taken from each brand table and from the merge |
| `scanMinScore` | 0.50 | below this the best hit is not shown as a match | | `scanMinScore` | 0.50 | below this the best hit is not shown as a match |
| `scanAmbiguityMargin` | 0.06 | how close the runner-up may be before the answer becomes a question |
| `scanMaxCandidates` | 10 | longest "did you mean?" list |
| `embedTimeout` (`utils/embedding.go`) | 4 s | one model call | | `embedTimeout` (`utils/embedding.go`) | 4 s | one model call |
| `scanVectorTTL` / `scanHitsTTL` (`scanRepository.go`) | 7 d / 30 min | cache lifetimes | | `scanVectorTTL` / `scanHitsTTL` (`scanRepository.go`) | 7 d / 30 min | cache lifetimes |
@@ -274,6 +407,19 @@ Classic* at 0.9 (the "G" was dropped, so only "parle" matched either row),
and the name tie-break handed it to Monaco because a space precedes a hyphen and the name tie-break handed it to Monaco because a space precedes a hyphen
in ASCII. A confident, wrong answer — the kind no score floor can catch. in ASCII. A confident, wrong answer — the kind no score floor can catch.
**When the substring rule ties, that tie is the answer.** A bare brand name
is a substring of every one of that brand's names, so all of them score 0.95
— identically, at a high score no floor would ever catch. Rather than
scoring around it, `isAmbiguous` reads it: if the runner-up is within
`scanAmbiguityMargin` of the leader, the reply becomes `ambiguous: true`
with `candidates` instead of a match (see Response B). Erring towards asking
is deliberate — one tap on a picture against the wrong biscuit. A label that
names one product leaves the runner-up far behind, so the common case is
untouched, and `services/scan_test.go`'s
`TestABrandNameScoresItsProductsIdentically` guards the tie itself: a
formula that broke it on name length or word count would bring the bug
back.
### Changing the embedding model ### Changing the embedding model
1. The catalogue team re-embeds `search_query` with the new model. 1. The catalogue team re-embeds `search_query` with the new model.

View File

@@ -11,8 +11,16 @@ package models
type ScanLookupRequest struct { type ScanLookupRequest struct {
Customerid int `json:"customerid"` Customerid int `json:"customerid"`
// What Lens read: "Milk Bikis", "Dabur Honey 500g". Free text, trimmed // What Lens read: "Milk Bikis", "Dabur Honey 500g". Free text, trimmed
// and capped by the service. // and capped by the service. Not required when Brand and Catalogueid
// name a product outright.
Label string `json:"label"` Label string `json:"label"`
// A product the customer has already chosen, by its catalogue key —
// which is how the app resolves a `candidates` list from an earlier
// ambiguous lookup, and how a deep link or a re-order skips recognition
// altogether. When both are set the label is ignored and no catalogue
// search runs.
Brand string `json:"brand"`
Catalogueid int64 `json:"catalogueid"`
// Where the customer is right now. Optional: without it the customer's // Where the customer is right now. Optional: without it the customer's
// saved primary address is used, and without that stores are listed in // saved primary address is used, and without that stores are listed in
// registration order with no distance. // registration order with no distance.
@@ -86,9 +94,15 @@ type ScanCatalogueMatch struct {
VariantKey string `json:"variant_key,omitempty"` VariantKey string `json:"variant_key,omitempty"`
Image string `json:"image,omitempty"` Image string `json:"image,omitempty"`
Score float64 `json:"score"` Score float64 `json:"score"`
// "vector", "vector+text" or "text" — how the score was produced. The app // "vector+text", "text" or "direct" — how the score was produced. The app
// can be more cautious with a text-only match. // can be more cautious with a text-only match; "direct" means the caller
// named the product by its catalogue key and nothing was recognised.
Method string `json:"method"` Method string `json:"method"`
// Set only on entries of `candidates`: at least one of the customer's
// registered stores has this product in stock right now. Candidates are
// ordered with the available ones first, so a "did you mean?" list can
// show what is actually buyable before what is not.
Available bool `json:"available,omitempty"`
} }
// ScanLookupResponse is the answer to a scan. // ScanLookupResponse is the answer to a scan.
@@ -96,10 +110,27 @@ type ScanLookupResponse struct {
Label string `json:"label"` Label string `json:"label"`
// The best catalogue product for the label, and the sizes of it the // The best catalogue product for the label, and the sizes of it the
// catalogue knows about (each a separate catalogue row). // catalogue knows about (each a separate catalogue row).
//
// Match is nil when nothing was recognised, and also when several
// products matched equally well — see Ambiguous.
Match *ScanCatalogueMatch `json:"match"` Match *ScanCatalogueMatch `json:"match"`
Variants []ScanCatalogueMatch `json:"catalogue_variants"` Variants []ScanCatalogueMatch `json:"catalogue_variants"`
// Several products fit the label and no one of them is a clear winner —
// which is what a bare brand name ("britannia") or a generic word
// ("biscuits") produces, and Lens returns those often because a
// wordmark is the most legible thing on a packet.
//
// When true: Match is nil, Stores is empty, and Candidates holds the
// products to offer as "did you mean?". Picking one means calling
// /lookup again with that candidate's `brand` and `catalogueid`.
//
// Guessing instead would mean showing a confident price for a product
// the customer did not photograph.
Ambiguous bool `json:"ambiguous"`
Candidates []ScanCatalogueMatch `json:"candidates"`
// 0..1. Below ~0.5 the app should confirm with the customer before // 0..1. Below ~0.5 the app should confirm with the customer before
// showing prices. // showing prices. With Ambiguous set this is the leader's score, which
// by definition the runner-up nearly equals.
Confidence float64 `json:"confidence"` Confidence float64 `json:"confidence"`
// Registered stores that stock the product, nearest first, in-stock // Registered stores that stock the product, nearest first, in-stock
// first. Empty with Available=false when none does. // first. Empty with Available=false when none does.

View File

@@ -94,6 +94,9 @@ type ScanRepository interface {
VectorSearch(ctx context.Context, vector []float32, limit int) ([]CatalogueHit, error) VectorSearch(ctx context.Context, vector []float32, limit int) ([]CatalogueHit, error)
TextSearch(ctx context.Context, label string, limit int) ([]CatalogueHit, error) TextSearch(ctx context.Context, label string, limit int) ([]CatalogueHit, error)
VectorSearchAvailable() bool VectorSearchAvailable() bool
// CatalogueRef is one product named by its catalogue key, with its other
// pack sizes after it. Nothing is recognised or scored.
CatalogueRef(ctx context.Context, brand string, id int64) ([]CatalogueHit, error)
// cache // cache
CachedVector(ctx context.Context, model, label string) ([]float32, bool) CachedVector(ctx context.Context, model, label string) ([]float32, bool)
@@ -546,6 +549,72 @@ func (r *scanRepository) TextSearch(ctx context.Context, label string, limit int
return hits, nil return hits, nil
} }
// tableFor resolves a brand the caller named to a real catalogue table.
//
// The lookup is against the tables discovered from information_schema, never
// a string built from the request: table names cannot be parameterised in
// SQL, so the discovered map is what keeps this from being an injection
// point. Both the table suffix ("britannia") and a display name ("24 Mantra"
// → brand_24_mantra) resolve.
func (r *scanRepository) tableFor(ctx context.Context, brand string) (string, map[string]bool, error) {
tables, err := r.brandTables(ctx)
if err != nil {
return "", nil, err
}
for _, candidate := range []string{
"brand_" + strings.ToLower(strings.TrimSpace(brand)),
"brand_" + normaliseBrandKey(brand),
} {
if cols, ok := tables[candidate]; ok {
return candidate, cols, nil
}
}
return "", nil, ErrUnknownBrand
}
// CatalogueRef reads one product by (brand, id) and appends its other pack
// sizes — same variant_key where the catalogue assigned one, same name
// otherwise, matching how the search groups a family.
//
// Distance is 0 on every row: nothing here was ranked, the caller said which
// product they meant.
func (r *scanRepository) CatalogueRef(ctx context.Context, brand string, id int64) ([]CatalogueHit, error) {
table, cols, err := r.tableFor(ctx, brand)
if err != nil {
return nil, err
}
suffix := strings.TrimPrefix(table, "brand_")
columns := hitColumns(suffix, cols)
var self []CatalogueHit
err = r.catalogue.WithContext(ctx).Raw(fmt.Sprintf(
`SELECT %s, 0::float8 AS distance FROM %s WHERE id = ?`, columns, table), id).Scan(&self).Error
if err != nil {
return nil, err
}
if len(self) == 0 {
return nil, nil
}
var siblings []CatalogueHit
if cols["variant_key"] && strings.TrimSpace(self[0].VariantKey) != "" {
err = r.catalogue.WithContext(ctx).Raw(fmt.Sprintf(
`SELECT %s, 0::float8 AS distance FROM %s WHERE variant_key = ? AND id <> ? ORDER BY id`,
columns, table), self[0].VariantKey, id).Scan(&siblings).Error
} else {
err = r.catalogue.WithContext(ctx).Raw(fmt.Sprintf(
`SELECT %s, 0::float8 AS distance FROM %s WHERE LOWER(product_name) = LOWER(?) AND id <> ? ORDER BY id`,
columns, table), self[0].ProductName, id).Scan(&siblings).Error
}
if err != nil {
// The product itself was found; losing its other sizes is the smaller
// failure and the caller asked for this one.
log.Printf("scan: could not read pack sizes of %s#%d: %v", brand, id, err)
return self, nil
}
return append(self, siblings...), nil
}
func sortedKeys(m map[string]map[string]bool) []string { func sortedKeys(m map[string]map[string]bool) []string {
keys := make([]string, 0, len(m)) keys := make([]string, 0, len(m))
for k := range m { for k := range m {

View File

@@ -53,28 +53,29 @@ func main() {
var cols []struct { var cols []struct {
Relname string Relname string
Attname string
Typname string Typname string
Atttypmod int Atttypmod int
} }
if err := db.Raw(` if err := db.Raw(`
SELECT c.relname, t.typname, a.atttypmod SELECT c.relname, a.attname, t.typname, a.atttypmod
FROM pg_attribute a FROM pg_attribute a
JOIN pg_class c ON c.oid = a.attrelid JOIN pg_class c ON c.oid = a.attrelid
JOIN pg_type t ON t.oid = a.atttypid JOIN pg_type t ON t.oid = a.atttypid
WHERE a.attname = 'embedding' AND c.relname LIKE 'brand\_%' WHERE t.typname = 'vector' AND a.attnum > 0 AND c.relname LIKE 'brand\_%'
ORDER BY c.relname`).Scan(&cols).Error; err != nil { ORDER BY c.relname, a.attname`).Scan(&cols).Error; err != nil {
log.Fatal(err) log.Fatal(err)
} }
if len(cols) == 0 { if len(cols) == 0 {
fmt.Println("no brand_* table has an embedding column") fmt.Println("no brand_* table has a vector column")
return return
} }
fmt.Printf("%-28s %-8s %5s %5s %5s\n", "table", "type", "dims", "rows", "embd") fmt.Printf("%-24s %-16s %-8s %5s %5s %5s\n", "table", "column", "type", "dims", "rows", "filled")
for _, c := range cols { for _, c := range cols {
var total, filled int64 var total, filled int64
db.Raw(fmt.Sprintf(`SELECT COUNT(1) FROM %s`, c.Relname)).Scan(&total) db.Raw(fmt.Sprintf(`SELECT COUNT(1) FROM %s`, c.Relname)).Scan(&total)
db.Raw(fmt.Sprintf(`SELECT COUNT(1) FROM %s WHERE embedding IS NOT NULL`, c.Relname)).Scan(&filled) db.Raw(fmt.Sprintf(`SELECT COUNT(1) FROM %s WHERE %s IS NOT NULL`, c.Relname, c.Attname)).Scan(&filled)
fmt.Printf("%-28s %-8s %5d %5d %5d\n", c.Relname, c.Typname, c.Atttypmod, total, filled) fmt.Printf("%-24s %-16s %-8s %5d %5d %5d\n", c.Relname, c.Attname, c.Typname, c.Atttypmod, total, filled)
} }
// nomic/bge emit unit vectors; a norm far from 1 means another pipeline. // nomic/bge emit unit vectors; a norm far from 1 means another pipeline.

View File

@@ -10,6 +10,7 @@ import (
"nearle/repositories" "nearle/repositories"
"nearle/utils" "nearle/utils"
"sort" "sort"
"strconv"
"strings" "strings"
"sync" "sync"
"time" "time"
@@ -54,6 +55,12 @@ const (
// came back as "Paneer Makhni 500ml" (0.304) — a near-miss on an // came back as "Paneer Makhni 500ml" (0.304) — a near-miss on an
// unrelated row clears a floor set that close to the noise. // unrelated row clears a floor set that close to the noise.
scanMinScore = 0.50 scanMinScore = 0.50
// How close the runner-up may be before the leader stops being an answer
// and the two become a question. See isAmbiguous.
scanAmbiguityMargin = 0.06
// A "did you mean?" list longer than this is not a choice, it is a
// catalogue — the customer is standing in a shop holding a packet.
scanMaxCandidates = 10
) )
// ScanErrors the controller maps to statuses. Everything else is a 500. // ScanErrors the controller maps to statuses. Everything else is a 500.
@@ -82,11 +89,15 @@ func NewScanService(repo repositories.ScanRepository, embedder utils.Embedder) S
func (s *scanService) Lookup(ctx context.Context, req models.ScanLookupRequest) (*models.ScanLookupResponse, error) { func (s *scanService) Lookup(ctx context.Context, req models.ScanLookupRequest) (*models.ScanLookupResponse, error) {
label := strings.TrimSpace(req.Label) label := strings.TrimSpace(req.Label)
// The caller can name the product outright instead of describing it —
// how the app resolves a candidate the customer picked.
direct := strings.TrimSpace(req.Brand) != "" && req.Catalogueid > 0
if req.Customerid <= 0 { if req.Customerid <= 0 {
return nil, fmt.Errorf("%w: customerid is required", ErrScanBadRequest) return nil, fmt.Errorf("%w: customerid is required", ErrScanBadRequest)
} }
if label == "" { if label == "" && !direct {
return nil, fmt.Errorf("%w: label is required", ErrScanBadRequest) return nil, fmt.Errorf("%w: label, or brand and catalogueid, is required", ErrScanBadRequest)
} }
if len(label) > scanMaxLabelLen { if len(label) > scanMaxLabelLen {
label = label[:scanMaxLabelLen] label = label[:scanMaxLabelLen]
@@ -126,6 +137,10 @@ func (s *scanService) Lookup(ctx context.Context, req models.ScanLookupRequest)
}() }()
go func() { go func() {
defer wg.Done() defer wg.Done()
if direct {
hits, method, matchErr = s.resolveRef(ctx, req.Brand, req.Catalogueid)
return
}
hits, method, matchErr = s.searchCatalogue(ctx, label) hits, method, matchErr = s.searchCatalogue(ctx, label)
}() }()
wg.Wait() wg.Wait()
@@ -149,18 +164,32 @@ func (s *scanService) Lookup(ctx context.Context, req models.ScanLookupRequest)
Label: label, Label: label,
Stores: []models.ScanStoreOffer{}, Stores: []models.ScanStoreOffer{},
Variants: []models.ScanCatalogueMatch{}, Variants: []models.ScanCatalogueMatch{},
Candidates: []models.ScanCatalogueMatch{},
} }
// Verify the app's idea of the customer's tenants against the truth. // Verify the app's idea of the customer's tenants against the truth.
stores, resp.UnregisteredTenantids = restrictToTenants(stores, req.Tenantids) stores, resp.UnregisteredTenantids = restrictToTenants(stores, req.Tenantids)
if len(hits) == 0 || hits[0].score < scanMinScore { switch {
case len(hits) == 0 && direct:
resp.Message = "That product is no longer in the catalogue."
return resp, nil
case len(hits) == 0, hits[0].score < scanMinScore:
resp.Message = "We couldn't recognise that product. Try a clearer photo of the front of the pack." resp.Message = "We couldn't recognise that product. Try a clearer photo of the front of the pack."
return resp, nil return resp, nil
} }
best := hits[0] distinct := distinctProducts(hits)
family := catalogueFamily(hits)
// Several products fit and none of them clearly wins — a bare brand name
// or a generic word. Ask rather than guess: naming one of them would put
// a confident price on a product the customer did not photograph.
if !direct && isAmbiguous(distinct) {
return s.candidatesResponse(ctx, resp, hits, distinct, method, stores)
}
best := distinct[0]
family := catalogueFamily(hits, best)
resp.Match = ptr(best.toMatch(method)) resp.Match = ptr(best.toMatch(method))
resp.Confidence = round3(best.score) resp.Confidence = round3(best.score)
for _, h := range family { for _, h := range family {
@@ -179,11 +208,7 @@ func (s *scanService) Lookup(ctx context.Context, req models.ScanLookupRequest)
keys = append(keys, repositories.CatalogueKey{Brand: h.Brand, Catalogueid: h.ID, Imageid: h.ImageID}) keys = append(keys, repositories.CatalogueKey{Brand: h.Brand, Catalogueid: h.ID, Imageid: h.ImageID})
names = append(names, h.ProductName) names = append(names, h.ProductName)
} }
locationids := make([]int, 0, len(stores)) rows, err := s.repo.StoreOptions(ctx, locationIDs(stores), keys, names)
for _, st := range stores {
locationids = append(locationids, st.Locationid)
}
rows, err := s.repo.StoreOptions(ctx, locationids, keys, names)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -213,6 +238,137 @@ func (s *scanService) Lookup(ctx context.Context, req models.ScanLookupRequest)
return resp, nil return resp, nil
} }
// candidatesResponse answers an ambiguous label with the products to choose
// between, marking which of them the customer can actually buy right now.
//
// The availability read is the same StoreOptions query the confident path
// runs, widened to every candidate — so a "did you mean?" list can put the
// three that are in stock above the five that are not, instead of sending
// somebody to a shelf that has none of them.
func (s *scanService) candidatesResponse(ctx context.Context, resp *models.ScanLookupResponse,
hits, distinct []scoredHit, method string, stores []models.ScanStore) (*models.ScanLookupResponse, error) {
candidates := distinct
if len(candidates) > scanMaxCandidates {
candidates = candidates[:scanMaxCandidates]
}
resp.Ambiguous = true
resp.Confidence = round3(distinct[0].score)
// Every catalogue row belonging to a candidate, and a way back from what
// a tenant's product row carries to the candidate it stands for.
inCandidates := make(map[string]bool, len(candidates))
for _, c := range candidates {
inCandidates[c.productKey()] = true
}
var keys []repositories.CatalogueKey
var names []string
byImage := make(map[string]string)
byRef := make(map[string]string)
byName := make(map[string]string)
for _, h := range hits {
key := h.productKey()
if !inCandidates[key] {
continue
}
keys = append(keys, repositories.CatalogueKey{Brand: h.Brand, Catalogueid: h.ID, Imageid: h.ImageID})
names = append(names, h.ProductName)
if h.ImageID != "" {
byImage[h.ImageID] = key
}
byRef[refKey(h.Brand, h.ID)] = key
if n := strings.ToLower(strings.TrimSpace(h.ProductName)); n != "" {
byName[n] = key
}
}
stocked := make(map[string]bool)
if len(stores) > 0 && len(keys) > 0 {
rows, err := s.repo.StoreOptions(ctx, locationIDs(stores), keys, names)
if err != nil {
return nil, err
}
for _, row := range rows {
if row.Stock <= 0 {
continue
}
// Same precedence as optionFromRow: the stable key first.
if row.Imageid != "" {
if key, ok := byImage[row.Imageid]; ok {
stocked[key] = true
continue
}
}
if row.Catalogueid > 0 {
if key, ok := byRef[refKey(row.Productbrand, row.Catalogueid)]; ok {
stocked[key] = true
continue
}
}
if key, ok := byName[strings.ToLower(strings.TrimSpace(row.Productname))]; ok {
stocked[key] = true
}
}
}
for _, c := range candidates {
m := c.toMatch(method)
m.Available = stocked[c.productKey()]
resp.Candidates = append(resp.Candidates, m)
}
// Buyable first; within each group the search's own ranking stands.
sort.SliceStable(resp.Candidates, func(i, j int) bool {
return resp.Candidates[i].Available && !resp.Candidates[j].Available
})
available := 0
for _, c := range resp.Candidates {
if c.Available {
available++
}
}
if available > 0 {
resp.Message = fmt.Sprintf("Which one is it? %d of these %d are in stock near you.",
available, len(resp.Candidates))
} else {
resp.Message = fmt.Sprintf("Which one is it? We found %d products that could match.",
len(resp.Candidates))
}
return resp, nil
}
// resolveRef reads the product the caller named, with its pack sizes. No
// recognition, so every row scores 1 and the method says so.
func (s *scanService) resolveRef(ctx context.Context, brand string, id int64) ([]scoredHit, string, error) {
rows, err := s.repo.CatalogueRef(ctx, brand, id)
if err != nil {
switch {
case errors.Is(err, repositories.ErrCatalogueDBUnavailable):
return nil, "direct", ErrScanCatalogueDown
case errors.Is(err, repositories.ErrUnknownBrand):
return nil, "direct", fmt.Errorf("%w: unknown brand %q", ErrScanBadRequest, brand)
}
return nil, "direct", err
}
hits := make([]scoredHit, 0, len(rows))
for _, row := range rows {
hits = append(hits, scoredHit{CatalogueHit: row, score: 1})
}
return hits, "direct", nil
}
func refKey(brand string, id int64) string {
return strings.ToLower(strings.TrimSpace(brand)) + "#" + strconv.FormatInt(id, 10)
}
func locationIDs(stores []models.ScanStore) []int {
ids := make([]int, 0, len(stores))
for _, st := range stores {
ids = append(ids, st.Locationid)
}
return ids
}
// ── Confirm ───────────────────────────────────────────────────────────────── // ── Confirm ─────────────────────────────────────────────────────────────────
func (s *scanService) Confirm(ctx context.Context, req models.ScanConfirmRequest) (*models.ScanConfirmResponse, error) { func (s *scanService) Confirm(ctx context.Context, req models.ScanConfirmRequest) (*models.ScanConfirmResponse, error) {
@@ -530,6 +686,11 @@ func (s *scanService) embed(ctx context.Context, label string) ([]float32, error
// The whole label as a substring of the name is near-certain; otherwise the // The whole label as a substring of the name is near-certain; otherwise the
// share of label words found in name+title, scaled so that "all of them" // share of label words found in name+title, scaled so that "all of them"
// stops short of the substring case. // stops short of the substring case.
//
// A label that is a substring of MANY names — a bare brand, "britannia" —
// therefore scores them all 0.95, identically. That tie is not a flaw to
// score around: it is the signal, and isAmbiguous reads it to answer "did
// you mean?" rather than letting the sort order pick a winner.
func textScore(h repositories.CatalogueHit, label string, tokens []string) float64 { func textScore(h repositories.CatalogueHit, label string, tokens []string) float64 {
name := strings.ToLower(h.ProductName) name := strings.ToLower(h.ProductName)
hay := name + " " + strings.ToLower(h.Title) hay := name + " " + strings.ToLower(h.Title)
@@ -561,27 +722,64 @@ func textScore(h repositories.CatalogueHit, label string, tokens []string) float
return 0.8 * float64(found) / float64(len(tokens)) return 0.8 * float64(found) / float64(len(tokens))
} }
// catalogueFamily is the best hit and its other pack sizes: same brand, and // productKey identifies a product across its pack sizes: the catalogue's own
// the same variant_key when the catalogue assigned one, else the same name. // variant_key where it assigned one, the name otherwise, always within a
// Every member is a separate catalogue row a shop may have imported. // brand. Two rows sharing it are 100 g and 200 g of one thing; two rows that
func catalogueFamily(hits []scoredHit) []scoredHit { // do not are different products to choose between.
if len(hits) == 0 { func productKey(h repositories.CatalogueHit) string {
return nil if k := strings.TrimSpace(h.VariantKey); k != "" {
return h.Brand + "/" + strings.ToLower(k)
} }
best := hits[0] return h.Brand + "/" + strings.ToLower(strings.TrimSpace(h.ProductName))
family := []scoredHit{best} }
for _, h := range hits[1:] {
if h.Brand != best.Brand { // productKey of a scored hit — scoredHit embeds the row it scored.
func (h scoredHit) productKey() string { return productKey(h.CatalogueHit) }
// distinctProducts keeps the best-scoring row of each product, in rank
// order — the list of things the customer could actually be shown to choose
// between, as opposed to the same product listed four times in four sizes.
func distinctProducts(hits []scoredHit) []scoredHit {
seen := make(map[string]bool, len(hits))
out := make([]scoredHit, 0, len(hits))
for _, h := range hits {
key := h.productKey()
if seen[key] {
continue continue
} }
switch { seen[key] = true
case best.VariantKey != "" && h.VariantKey != "": out = append(out, h)
if h.VariantKey == best.VariantKey { }
return out
}
// isAmbiguous reports that naming the leader as THE match would be a guess
// dressed up as an answer, because something else is level with it.
//
// A margin rather than an absolute threshold: what matters is not how high
// the best score is but whether anything is tied with it. A bare brand name
// is a substring of every one of that brand's names, so textScore gives them
// all 0.95 — a perfect tie at a HIGH score, which no floor would catch.
//
// Erring towards asking is deliberate. Asking costs the customer one tap on
// a picture; guessing wrong costs them the wrong biscuit and costs us the
// belief that the scanner works. A label that names one product leaves the
// runner-up far behind, so the common case is unaffected.
func isAmbiguous(distinct []scoredHit) bool {
return len(distinct) >= 2 && distinct[1].score >= distinct[0].score-scanAmbiguityMargin
}
// catalogueFamily is `of` and its other pack sizes, drawn from hits.
func catalogueFamily(hits []scoredHit, of scoredHit) []scoredHit {
key := of.productKey()
family := make([]scoredHit, 0, 4)
for _, h := range hits {
if h.productKey() == key {
family = append(family, h) family = append(family, h)
} }
case strings.EqualFold(strings.TrimSpace(h.ProductName), strings.TrimSpace(best.ProductName)):
family = append(family, h)
} }
if len(family) == 0 {
return []scoredHit{of}
} }
return family return family
} }

View File

@@ -3,10 +3,13 @@ package services
import ( import (
"context" "context"
"errors" "errors"
"fmt"
"strings"
"testing" "testing"
"nearle/models" "nearle/models"
"nearle/repositories" "nearle/repositories"
"nearle/utils"
) )
/* /*
@@ -30,9 +33,13 @@ type fakeScanRepo struct {
options []repositories.StoreOptionRow options []repositories.StoreOptionRow
at map[int]*repositories.StoreOptionRow // productid → row at map[int]*repositories.StoreOptionRow // productid → row
ref []repositories.CatalogueHit
refErr error
askedKeys []repositories.CatalogueKey askedKeys []repositories.CatalogueKey
askedNames []string askedNames []string
askedLocs []int askedLocs []int
askedRef string
cachedHits map[string][]repositories.CatalogueHit cachedHits map[string][]repositories.CatalogueHit
} }
@@ -69,6 +76,10 @@ func (f *fakeScanRepo) TextSearch(context.Context, string, int) ([]repositories.
return f.text, nil return f.text, nil
} }
func (f *fakeScanRepo) VectorSearchAvailable() bool { return f.hasVec } func (f *fakeScanRepo) VectorSearchAvailable() bool { return f.hasVec }
func (f *fakeScanRepo) CatalogueRef(_ context.Context, brand string, id int64) ([]repositories.CatalogueHit, error) {
f.askedRef = fmt.Sprintf("%s#%d", brand, id)
return f.ref, f.refErr
}
func (f *fakeScanRepo) CachedVector(context.Context, string, string) ([]float32, bool) { func (f *fakeScanRepo) CachedVector(context.Context, string, string) ([]float32, bool) {
return nil, false return nil, false
} }
@@ -423,7 +434,7 @@ func TestCatalogueFamilyGroupsByVariantKeyThenName(t *testing.T) {
{CatalogueHit: milkBikis200, score: 0.88}, {CatalogueHit: milkBikis200, score: 0.88},
{CatalogueHit: repositories.CatalogueHit{Brand: "parle", ProductName: "Milk Bikis", VariantKey: "milk_bikis"}, score: 0.5}, {CatalogueHit: repositories.CatalogueHit{Brand: "parle", ProductName: "Milk Bikis", VariantKey: "milk_bikis"}, score: 0.5},
} }
family := catalogueFamily(hits) family := catalogueFamily(hits, hits[0])
if len(family) != 2 || family[1].ID != 8 { if len(family) != 2 || family[1].ID != 8 {
t.Fatalf("family should be the two britannia sizes, got %+v", family) t.Fatalf("family should be the two britannia sizes, got %+v", family)
} }
@@ -432,11 +443,229 @@ func TestCatalogueFamilyGroupsByVariantKeyThenName(t *testing.T) {
a := scoredHit{CatalogueHit: repositories.CatalogueHit{Brand: "b", ID: 1, ProductName: "Honey"}} a := scoredHit{CatalogueHit: repositories.CatalogueHit{Brand: "b", ID: 1, ProductName: "Honey"}}
b := scoredHit{CatalogueHit: repositories.CatalogueHit{Brand: "b", ID: 2, ProductName: "honey "}} b := scoredHit{CatalogueHit: repositories.CatalogueHit{Brand: "b", ID: 2, ProductName: "honey "}}
c := scoredHit{CatalogueHit: repositories.CatalogueHit{Brand: "b", ID: 3, ProductName: "Honey Lite"}} c := scoredHit{CatalogueHit: repositories.CatalogueHit{Brand: "b", ID: 3, ProductName: "Honey Lite"}}
if family := catalogueFamily([]scoredHit{a, b, c}); len(family) != 2 { if family := catalogueFamily([]scoredHit{a, b, c}, a); len(family) != 2 {
t.Errorf("name match should join 1 and 2 only, got %+v", family) t.Errorf("name match should join 1 and 2 only, got %+v", family)
} }
} }
/*
Ambiguity.
Lens hands back whatever was most legible on the packet, and on a packet that
is very often the brand wordmark alone. "britannia" fits 258 catalogue rows
equally well, so there is no best one. textScore gives all of them 0.95 —
correctly, the label IS in every one of those names — and with nothing to
read that tie, the sort order picked a winner and the customer was shown one
arbitrary biscuit with "confidence": 0.95 and a price. These tests are the
contract that it asks instead.
*/
// Three different Britannia products, of which the customer's stores stock
// one. Text-only: no embedder, which is also how production runs until the
// model is configured.
func newBrandLabelFixture() *fakeScanRepo {
cashew := repositories.CatalogueHit{Brand: "britannia", ID: 21, ProductName: "Britannia Good Day Cashew Cookies", VariantKey: "good_day_cashew", ImageID: "britannia_good_day_cashew"}
butter := repositories.CatalogueHit{Brand: "britannia", ID: 22, ProductName: "Britannia Good Day Butter Cookies", VariantKey: "good_day_butter", ImageID: "britannia_good_day_butter"}
marie := repositories.CatalogueHit{Brand: "britannia", ID: 23, ProductName: "Britannia Marie Gold", VariantKey: "marie_gold", ImageID: "britannia_marie_gold"}
return &fakeScanRepo{
exists: true,
stores: fixtureStores(),
text: []repositories.CatalogueHit{cashew, butter, marie},
options: []repositories.StoreOptionRow{
{Tenantid: 2, Locationid: 20, Productid: 220, Productname: "Britannia Marie Gold",
Productbrand: "britannia", Catalogueid: 23, Imageid: "britannia_marie_gold", Price: 30, Stock: 4},
},
}
}
func TestABareBrandNameAsksInsteadOfGuessing(t *testing.T) {
repo := newBrandLabelFixture()
svc := NewScanService(repo, nil)
resp, err := svc.Lookup(context.Background(), models.ScanLookupRequest{
Customerid: 5, Label: "britannia", Latitude: "11.035", Longitude: "77.035",
})
if err != nil {
t.Fatal(err)
}
if !resp.Ambiguous {
t.Fatalf("a bare brand name must not resolve to one product, got match %+v", resp.Match)
}
if resp.Match != nil {
t.Errorf("Match must be nil while ambiguous, got %+v", resp.Match)
}
if len(resp.Stores) != 0 {
t.Errorf("no store or price may be quoted for a product the customer has not chosen, got %d offers", len(resp.Stores))
}
if len(resp.Variants) != 0 {
t.Errorf("pack sizes belong to a chosen product, got %+v", resp.Variants)
}
if len(resp.Candidates) != 3 {
t.Fatalf("want the three distinct Britannia products, got %d: %+v", len(resp.Candidates), resp.Candidates)
}
// The one the customer can actually buy is offered first.
if !resp.Candidates[0].Available || resp.Candidates[0].Catalogueid != 23 {
t.Errorf("the stocked product should lead the list, got %+v", resp.Candidates[0])
}
for _, c := range resp.Candidates[1:] {
if c.Available {
t.Errorf("only Marie Gold is stocked, but %s reports available", c.ProductName)
}
}
// Note what confidence does NOT say here. The label appears verbatim in
// all three names, so relevance is high — and the answer is still a
// question. An app that gated on `confidence` instead of `ambiguous`
// would show a price for the wrong biscuit, which is the whole bug.
if resp.Confidence < 0.9 {
t.Errorf("a verbatim brand match scores high; %v suggests the scoring changed", resp.Confidence)
}
if !strings.Contains(resp.Message, "Which one") {
t.Errorf("the message should ask, got %q", resp.Message)
}
}
func TestAnAmbiguousLabelWithNoStockStillLists(t *testing.T) {
repo := newBrandLabelFixture()
repo.options = nil
svc := NewScanService(repo, nil)
resp, err := svc.Lookup(context.Background(), models.ScanLookupRequest{Customerid: 5, Label: "britannia"})
if err != nil {
t.Fatal(err)
}
if !resp.Ambiguous || len(resp.Candidates) != 3 {
t.Fatalf("want three candidates, got ambiguous=%v %d", resp.Ambiguous, len(resp.Candidates))
}
for _, c := range resp.Candidates {
if c.Available {
t.Errorf("%s cannot be available with no stock anywhere", c.ProductName)
}
}
}
// The other half of the contract: a label that does name a product must not
// start asking questions.
func TestASpecificLabelStillWinsOutright(t *testing.T) {
repo := newBrandLabelFixture()
svc := NewScanService(repo, nil)
resp, err := svc.Lookup(context.Background(), models.ScanLookupRequest{
Customerid: 5, Label: "good day cashew",
})
if err != nil {
t.Fatal(err)
}
if resp.Ambiguous {
t.Fatalf("a label naming one product should resolve, got candidates %+v", resp.Candidates)
}
if resp.Match == nil || resp.Match.Catalogueid != 21 {
t.Fatalf("want the cashew cookies, got %+v", resp.Match)
}
if len(resp.Candidates) != 0 {
t.Errorf("candidates belong to an ambiguous answer, got %+v", resp.Candidates)
}
}
// The property the ambiguity check rests on: a label that is a substring of
// several names scores them EQUALLY. Nothing downstream can tell "did you
// mean?" from "found it" if a formula breaks that tie on name length, word
// count or anything else incidental — which is how one arbitrary Britannia
// biscuit used to come back with a price on it.
func TestABrandNameScoresItsProductsIdentically(t *testing.T) {
cashew := repositories.CatalogueHit{ProductName: "Britannia Good Day Cashew Cookies 200g"}
butter := repositories.CatalogueHit{ProductName: "Britannia Good Day Butter Cookies 100g"}
// Deliberately a much shorter name: length must not become a tie-breaker.
marie := repositories.CatalogueHit{ProductName: "Britannia Marie Gold"}
tokens := utils.SearchTokens("britannia")
a, b, c := textScore(cashew, "britannia", tokens), textScore(butter, "britannia", tokens), textScore(marie, "britannia", tokens)
if a != b || b != c {
t.Fatalf("a brand must score its products equally, got %.3f / %.3f / %.3f", a, b, c)
}
if a == 0 {
t.Fatal("the brand name is in every one of those names; scoring it 0 would hide them all")
}
// And a label that does name a product must NOT tie with its siblings,
// or everything would be a question.
specific := utils.SearchTokens("good day cashew")
if textScore(cashew, "good day cashew", specific) <= textScore(butter, "good day cashew", specific) {
t.Error("a label naming one product must outscore its siblings")
}
if none := textScore(cashew, "dabur honey", utils.SearchTokens("dabur honey")); none != 0 {
t.Errorf("nothing in common should score 0, got %.3f", none)
}
}
func TestDistinctProductsCollapsesPackSizes(t *testing.T) {
hits := []scoredHit{
{CatalogueHit: milkBikis, score: 1},
{CatalogueHit: milkBikis200, score: 0.9},
{CatalogueHit: goodDay, score: 0.5},
}
distinct := distinctProducts(hits)
if len(distinct) != 2 || distinct[0].ID != 7 || distinct[1].ID != 9 {
t.Fatalf("two sizes of one product are one choice, got %+v", distinct)
}
}
// Picking a candidate: the app sends the key instead of a description, and
// nothing is recognised at all.
func TestNamingTheProductSkipsRecognition(t *testing.T) {
repo := newLookupFixture()
repo.ref = []repositories.CatalogueHit{milkBikis, milkBikis200}
// If recognition ran, these would decide the answer instead.
repo.text = []repositories.CatalogueHit{goodDay}
repo.vector = []repositories.CatalogueHit{goodDay}
svc := NewScanService(repo, fakeEmbedder{vec: []float32{0.1}})
resp, err := svc.Lookup(context.Background(), models.ScanLookupRequest{
Customerid: 5, Brand: "britannia", Catalogueid: 7,
Latitude: "11.035", Longitude: "77.035",
})
if err != nil {
t.Fatal(err)
}
if repo.askedRef != "britannia#7" {
t.Fatalf("the catalogue should have been asked for that exact product, got %q", repo.askedRef)
}
if resp.Match == nil || resp.Match.Catalogueid != 7 || resp.Match.Method != "direct" {
t.Fatalf("want a direct match on 7, got %+v", resp.Match)
}
if resp.Ambiguous || resp.Confidence != 1 {
t.Errorf("a named product is not a guess: ambiguous=%v confidence=%v", resp.Ambiguous, resp.Confidence)
}
if len(resp.Variants) != 2 {
t.Errorf("its pack sizes should come with it, got %+v", resp.Variants)
}
if !resp.Available || resp.RecommendedLocationid != 20 {
t.Errorf("stores are resolved exactly as for a recognised product, got %+v", resp.Stores)
}
}
func TestAMissingCatalogueRefIsNotAMatch(t *testing.T) {
repo := newLookupFixture()
repo.ref = nil
svc := NewScanService(repo, nil)
resp, err := svc.Lookup(context.Background(), models.ScanLookupRequest{
Customerid: 5, Brand: "britannia", Catalogueid: 999,
})
if err != nil {
t.Fatal(err)
}
if resp.Match != nil || resp.Ambiguous || len(resp.Stores) != 0 {
t.Fatalf("a product that is gone is not a match, got %+v", resp)
}
if !strings.Contains(resp.Message, "no longer") {
t.Errorf("the message should say the product is gone, got %q", resp.Message)
}
}
// A vector neighbour that is merely not-quite-unrelated used to clear the old // A vector neighbour that is merely not-quite-unrelated used to clear the old
// 0.30 floor: in production "Paracetamol" came back as "Paneer Makhni 500ml" // 0.30 floor: in production "Paracetamol" came back as "Paneer Makhni 500ml"
// on a 0.304 similarity. Correct labels land near 0.92, so nothing this weak // on a 0.304 similarity. Correct labels land near 0.92, so nothing this weak