# Barcode enrichment

`app/services/enrichment/barcode/` — how a catalog row acquires a barcode, what
the barcode is then allowed to claim, and which knob to turn.

Referenced from `.env.example` and `app/services/enrichment/pipeline.py`, both
of which pointed at this file for a long time before it existed.

---

## The four ways a row gets a barcode

| # | Path | When | Cost | Confidence |
|---|---|---|---|---|
| 1 | **The sheet** | The merchant typed it | free | highest — they are holding the pack |
| 2 | **Identity derivation** | Always, inline | free, offline | derives `gtin`/`ean13`/`upc`/`barcode_type` from a barcode already present |
| 3 | **Bulk brand corpus** | After upload, per brand | ~5 requests **per brand** | name-matched at ≥ 0.88 |
| 4 | **Per-product cascade** | Only if enabled | 1 request **per product**, 10/min cap | brand + size + name matched |

Paths 1 and 2 are always on. Path 3 is on by default. Path 4 is off by default.

### Why path 4 is off and path 3 is on

They differ in cost, not in appetite for risk.

The Open Food Facts per-product search endpoint is capped at **10 requests per
minute**. A 200-row upload on path 4 is twenty minutes of a held HTTP request,
on a shared 8 GB host — which is what `settings.py:420-423` is about.

Path 3 fetches a brand's *entire* India catalogue in about five requests, caches
it to `data/cache/off_brand_corpus/<slug>.json`, and matches offline. A brand
costs the same whether it has 3 rows or 300.

```
ENABLE_BARCODE_LOOKUP=false        # path 4: per product, inline
ENRICH_BARCODES_ON_UPLOAD=true     # path 3: per brand, after the upload settles
```

> `.env.example` shipped `ENABLE_BARCODE_LOOKUP=true` while `settings.py`
> defaulted it `false`, for as long as both existed. Anyone copying the example
> got a materially different pipeline from anyone relying on defaults. Fixed;
> if you see the two disagree again, `settings.py` is authoritative.

---

## Two thresholds, and why they are different numbers

| Constant | Value | Direction | Used by |
|---|---|---|---|
| `BARCODE_MIN_NAME_SIMILARITY` | 0.78 | reverse — barcode known, name is a sanity check | `fetch_verified_nutrition_by_barcode` |
| `OFF bulk review_min` | 0.88 | forward — name carries the whole decision | `post_ingest_barcodes`, `backfill_barcodes_from_off` |

`settings.py:449-477` records the measurement behind 0.78: at 0.45 the cascade
accepted 15 candidates of which 8 were wrong; at 0.78 it accepted 2 and none
were wrong. **Do not lower it to fix a coverage complaint.**

### The containment rule

Open Food Facts stores short names. We store long ones. Measured over the
catalog on 2026-09-08, 149 of 300 barcoded rows were refused as "found, wrong
product" when the barcode had resolved perfectly:

```
ours "Nestle Munch 8.9g"      OFF "Munch"    similarity 0.332
ours "Coca-Cola Maaza 750ml"  OFF "Maaza"    similarity 0.304
```

`name_similarity` divides overlap by *our* token count, so a one-token candidate
cannot exceed ~0.33 however right it is. But the same run correctly refused:

```
ours "Pepsico Lays 1kg"       OFF "Spanish tomato tango"  0.133
ours "Coca-Cola Fanta 750ml"  OFF "Orange"                0.089
```

A threshold low enough to admit the first group admits the second. So
`matching.name_is_contained` separates them **structurally**: every candidate
token must be one of ours once brand and size tokens are discounted, and a bare
brand name ("Colgate", "godrej" — both real OFF titles) never matches.

### `barcode_is_identity` — and the gate that actually blocked most of them

The name gate was the *visible* symptom. When the fix was measured it moved only
3 rows to 8, and the reason is that `is_match` applies its rules in order and the
**size** gate rejects first:

```
Nestle Munch 38.5 g   <- OFF "Munch"   blocked by SIZE (off quantity = None)
Coca-Cola Maaza 750ml <- OFF "Maaza"   blocked by SIZE (off quantity = None)
Cadbury Perk 22 g     <- OFF "Perk"    blocked by SIZE (off quantity = None)
```

`size_matches` returns False whenever *either* side is blank, and Open Food
Facts leaves `quantity` null on a large share of records — 57 of 146 Amul hits.
`off_bulk` had already documented this and worked around it by treating size as
a ranking bonus rather than a veto.

So the flag is `barcode_is_identity`, not `allow_containment`, because it
describes the precondition rather than one of its consequences: **the caller
already knows which product this is, because it fetched by GTIN.** Under it,
name and size stop being evidence of identity and become sanity checks against
our barcode being on the wrong row — and a sanity check cannot fail on
information the source does not have:

| Rule | Normally | Under `barcode_is_identity` |
|---|---|---|
| 1. brand | must match | unchanged |
| 2. size | blank ⇒ reject | **blank ⇒ no information, allowed.** Present-and-different still rejects |
| 3. variant terms | must not conflict | unchanged |
| 4. name similarity | ≥ floor | floor, **or** containment |

**Defaults off.** On the search path many candidates compete and name and size
are the only things telling them apart — "Munch" with no size would match every
Nestle product containing that word. Pass it only where a single candidate was
fetched by barcode: `fetch_verified_nutrition_by_barcode` and
`scripts/backfill_nutrition_from_barcodes`, and nothing else today.

---

## What a barcode is allowed to claim

`barcode_verified = true` means brand, size **and** name were matched against a
source record. It is not a synonym for "we have a barcode".

| `barcode_lookup_status` | Meaning |
|---|---|
| `verified` | the cascade matched brand + size + name |
| `name_matched` | bulk corpus match ≥ 0.88; the pack is **not** confirmed |
| `sheet_validated` | the merchant supplied it and it passes the GTIN checksum |
| `not_found` / `error` / `disabled` | no barcode stored |

A `name_matched` code is a real GS1 barcode for *that brand*, written to every
size variant of a title. Good enough for catalog matching, dedup and nutrition
lookups. **Not** good enough for logistics, invoicing, or anything a scanner
drives. Filter on `barcode_verified = false` to select, correct or revert them.

---

## Validation is not optional and not per-source

`validators.validate_barcode` is the single gate: digits only → a legal GTIN
length (8/12/13/14) → recomputed check digit. A source saying "this is the
barcode" is never sufficient. Trust affects the *order* sources are tried, never
whether validation runs.

Derived fields follow from the digits alone:

- `gtin` — the validated code
- `ean13` — a UPC-A zero-padded to 13. **GTIN-8 is not padded**: an 8-digit GTIN
  is its own symbology, not a truncated EAN-13.
- `upc` — 12-digit codes only

> **`upc` will always be 0% in this catalog, and that is correct.** Every code
> here is GS1 India (prefix `890`), which issues EAN-13 and GTIN-8. UPC-A is
> North American. Measured: 0 of 300. The coverage report counts it *not
> applicable*, not missing.

---

## Where the fields go

A field computed by a stage reaches Postgres only if it is named in **all three**
of these. Two of them were missing the barcode fields for a long time, which is
why `upc` sat at 0% and `gtin` at 8.7% while the code that produced them ran on
every ingestion:

1. `store_catalog_pipeline._to_storage_row` — the projection
2. `vector_store.upsert_brand_products` — the INSERT column list
3. `vector_store._ensure_columns` — the only migration mechanism (no Alembic)

Plus `brand_sync.EXPORT_COLUMNS` for the seed-file round trip.

Every enrichment column uses `COALESCE(EXCLUDED.x, table.x)` in the
`ON CONFLICT` clause, including `barcode` itself. Without it a bare re-seed —
which carries no enrichment keys — sets them to NULL. Proven against a live
table: it kept `gtin` and blanked `barcode`, leaving a row claiming a GTIN with
no barcode.

---

## Running it

```bash
# What is filled, and where each value came from
python -m scripts.catalog_coverage --by-provenance

# Derive gtin/ean13/upc/barcode_type from barcodes already stored (offline)
python -m scripts.backfill_barcode_identity --apply

# Bulk-match barcodes from the OFF brand corpora
python -m scripts.backfill_barcodes_from_off --apply

# Upgrade nutrition from a name match to an exact barcode match
python -m scripts.backfill_nutrition_from_barcodes --apply
```

Every script is **dry-run by default** and prints its target database on
startup. `backend/.env` points at production.