Merge origin/main: keep the substring rule, read its tie

main had moved on with retrieval work validated against real queries —
minTokenHits (the word match needs two thirds of the label, not all of
it), separator folding so "Parle G"/"Parle-G"/"ParleG" all reach Parle-G,
the floor at 0.50 after "Paracetamol" came back as "Paneer Makhni 500ml"
at 0.304, and ties broken on cosine distance instead of name. All of that
is kept exactly as it was.

The conflict was in textScore: this branch replaced the substring rule
with a coverage formula to stop a bare brand name resolving to one
arbitrary product. That is the wrong half to change. The substring rule
scores every product of a brand 0.95 IDENTICALLY, and that tie is not the
bug — it is the signal. isAmbiguous reads it, so the branch's coverage
rewrite is dropped and the ambiguity layer alone does the work:

  "britannia" → all 258 rows tie at 0.95 → ambiguous: true + candidates
  "Parle G"   → folding and the single-character token still land it
  a real name → runner-up far behind → match, unchanged

Dropped with it: scanSpecificEnough, the per-hit text score, and the
proportional confirmation bonus — the flat +0.10 is back. Simpler, and it
leaves main's tuning untouched.

TestTextScoreRewardsSpecificityNotJustOverlap tested the removed formula
and is replaced by TestABrandNameScoresItsProductsIdentically, which
guards the tie itself: a formula that broke it on name length or word
count would bring the bug back.

Docs carry both rationales, and now say plainly that confidence stays
high on the ambiguous path — gate on `ambiguous`, never on `confidence`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-23 11:05:03 +05:30
25 changed files with 1661 additions and 228 deletions

View File

@@ -470,9 +470,22 @@ func (r *scanRepository) VectorSearch(ctx context.Context, vector []float32, lim
return hits, nil
}
// minTokenHits is how many of the label's words a row must carry to be worth
// looking at. Every word was once required, which meant a single word the
// catalogue does not use — "Parle G biscuit pack", "Milk Bikis pack" — kept
// the right product out of the result entirely, leaving the vector search to
// answer alone and confidently wrong. Most of them is enough; scoring sorts
// out the rest.
func minTokenHits(n int) int {
if n <= 2 {
return n
}
return (n*2 + 2) / 3 // two thirds, rounded up; never below 2 for n >= 3
}
// TextSearch is the fallback when there is no embedder, and the tie-breaker
// beside it when there is: rows whose name or title contains the label, or
// contains every word of it.
// carry most of its words.
func (r *scanRepository) TextSearch(ctx context.Context, label string, limit int) ([]CatalogueHit, error) {
tables, err := r.brandTables(ctx)
if err != nil {
@@ -498,18 +511,32 @@ func (r *scanRepository) TextSearch(ctx context.Context, label string, limit int
hay = "LOWER(COALESCE(product_name, '') || ' ' || COALESCE(title, '') || ' ' || COALESCE(search_query, ''))"
}
conds := []string{hay + " LIKE ?"}
args = append(args, "%"+label+"%")
all := make([]string, 0, len(tokens))
for _, tok := range tokens {
all = append(all, hay+" LIKE ?")
args = append(args, "%"+tok+"%")
// How well a row matches, as a number: the whole label as a substring
// outweighs any number of loose words, then one point per word found.
hits := make([]string, 0, len(tokens)+1)
hits = append(hits, "(CASE WHEN "+hay+" LIKE ? THEN 100 ELSE 0 END)")
for range tokens {
hits = append(hits, "(CASE WHEN "+hay+" LIKE ? THEN 1 ELSE 0 END)")
}
conds = append(conds, "("+strings.Join(all, " AND ")+")")
rank := strings.Join(hits, " + ")
// The expression appears twice in the SQL — once to filter, once to
// order — so its arguments are bound twice, in that order.
bind := func() {
args = append(args, "%"+label+"%")
for _, tok := range tokens {
args = append(args, "%"+tok+"%")
}
}
bind()
bind()
// Ordering matters as much as the threshold: a looser WHERE lets more
// rows qualify, and an unordered LIMIT would then be free to return
// the wrong ones. Best match per brand first, id to keep it stable.
branches = append(branches, fmt.Sprintf(
`(SELECT %s, -1::float8 AS distance FROM %s WHERE %s LIMIT %d)`,
hitColumns(brand, cols), table, strings.Join(conds, " OR "), limit))
`(SELECT %s, -1::float8 AS distance FROM %s WHERE (%s) >= %d ORDER BY (%s) DESC, id LIMIT %d)`,
hitColumns(brand, cols), table, rank, minTokenHits(len(tokens)), rank, limit))
}
if len(branches) == 0 {
return nil, nil