loyaly-catalogue/README.md

# Electronics Catalog (local)

A local catalogue of **real** electronics products sold in India, with a Tamil Nadu focus:
the products, their prices, images and the e-commerce platforms that list them. It is a
converted copy of `Project_Deploy`, which stays untouched. It runs only on this machine
and is not pushed anywhere.

## How data is collected (search-first)

1. **Discover**: web search (DuckDuckGo via `ddgs`; Google Programmable Search too when a
   key is set) runs `site:<platform> <brand> <variant terms>` on every registered
   platform. Only single-product URLs are kept.
2. **Gate each platform** (`probe`), grading it A, B or C:
   - **A**: robots.txt allows the page, it returns HTTP 200 with no bot check, and it
     carries schema.org Product JSON-LD with an INR price. The product page is read.
   - **B**: fetchable, but only the page HTML/meta is readable.
   - **C**: blocked, CAPTCHA, robots.txt disallows, or no data without JavaScript.
     **Web search results only**, with nothing fetched from the site.
   - Amazon.in and Flipkart are `serp_only` by policy and are never fetched directly.
3. **Collect**:
   - A/B pages are read politely: robots.txt obeyed, an honest User-Agent, 3 s between
     requests to a site, and a circuit breaker on any 403/429/CAPTCHA.
   - C platforms contribute what their search result shows: the title (with variant) and
     URL, plus a price or stock status only when the snippet or the search engine's own
     structured data states it.
4. **Match** each listing to one canonical variant (brand + model + RAM + storage; MPN for
   laptops). A different model number is never merged. Uncertain matches go to a review
   queue.
5. **Verify**: a product is shown only when listings on **at least two platforms** (at
   least one a retailer) confirm it.

**Anti-fabrication rules:**
- Every listing and price row stores its source URL and the text the value was read from.
- Prices come only from parsers, never from the LLM. EMI, "₹X off", exchange and bank-offer
  amounts are rejected.
- The local LLM (qwen2.5:1.5b) only fills missing *spec* fields from fetched text. Every
  value it returns must appear in that text, or it is dropped.
- Images come only from a product's own listings and are checked live. Only URLs are
  stored.
- Unknown stays unknown: `in_stock` is NULL and the price is "not stated".
- `pincode_applied` is true only if a site actually accepted the pincode. Currently none
  does, so prices are national listing prices.

## Platforms

| Platform | Region | Mode (as probed on 29 Sep 2026) |
|---|---|---|
| Amazon.in, Flipkart | national | C: search only (policy) |
| Croma | national | C: blocked (HTTP 403), search only |
| Tata CLiQ | national | C: bot-check page, search only |
| Reliance Digital, Vijay Sales | national | A: product pages read |
| Poorvika, Vasanth & Co | Tamil Nadu | A: product pages read |
| Sangeetha Mobiles | Tamil Nadu | C: no data without JavaScript, search only |
| Viveks | Tamil Nadu | C: few product pages indexed |
| Brand official sites | - | probed per brand |

Grades are re-checked every 7 days (`probe`). A tripped breaker pauses a site for 24 h.

## Setup (once)

```powershell
# 1. Local database (container elec_catalog_pg, 127.0.0.1:5433, DB electronics_catalog)
copy .env.example .env                  # set POSTGRES_PASSWORD
docker compose up -d

# 2. Backend
cd backend
copy .env.example .env                  # same DB_PASSWORD; set ELEC_CONTACT to a real email
py -3.13 -m venv .venv
.venv\Scripts\pip install torch --index-url https://download.pytorch.org/whl/cpu
.venv\Scripts\pip install -r requirements.txt -r requirements-dev.txt
.venv\Scripts\python -m app.electronics.cli migrate
.venv\Scripts\python -m app.electronics.cli seed-reference

# 3. Frontend
cd ..\frontend
npm install
```

## Run

```powershell
start_app.bat                     # database + API (127.0.0.1:8000) + UI (http://localhost:5173)
```

`AUTH_ALLOW_ANY_LOGIN=true` in `backend/.env` accepts any password for user `admin`. That
is fine locally because the API binds to 127.0.0.1.

## Collect data (CLI, from `backend/`)

```powershell
.venv\Scripts\python -m app.electronics.cli probe                       # grade platforms A/B/C
.venv\Scripts\python -m app.electronics.cli collect --category mobiles --brand samsung --brand xiaomi --limit 10
.venv\Scripts\python -m app.electronics.cli collect --category laptops --brand hp --brand lenovo --limit 10
.venv\Scripts\python -m app.electronics.cli report                      # what is in the catalogue
.venv\Scripts\python -m app.electronics.cli review                      # uncertain matches
.venv\Scripts\python -m app.electronics.cli verify-grounding            # audit: every price has evidence
```

Useful flags:
- `--no-fetch`: search results only.
- `--no-llm`: fully deterministic.
- `--budget N`: cap on search queries.
- `--reprobe`: re-grade the sites.

The same run can be started from the Admin page.

**Prices for Amazon, Flipkart and Croma.** Free search snippets almost never show a price
for these platforms, so their listings usually record availability only. To get their
prices without ever fetching them, set `GOOGLE_API_KEY` and `GOOGLE_CSE_ID` in
`backend/.env` (Google Programmable Search, 100 free queries a day). The Google Cloud
project behind the key must have the **Custom Search API** enabled. Then run:

```powershell
.venv\Scripts\python -m app.electronics.cli prices --limit 40
```

- Google queries are kept for this price pass. Discovery uses DuckDuckGo and falls back to
  Google only when DuckDuckGo gives no answer.
- A price is taken only from Google's structured offer data for the **same product page**
  (same site product ID), and it is stored with that data as evidence.
- A rejected key switches Google off for the rest of the run and reports why.

## Tests

```powershell
cd backend
.venv\Scripts\python -m pytest -q
```

The tests need no network and never touch real data. The database tests use a separate
`electronics_catalog_test` database on the same local server, and they are skipped if the
container is down.

## Layout

```
docker-compose.yml               local Postgres + pgvector only
backend/app/electronics/
  reference/*.yaml               brand allow-list + aliases, platforms, spec dictionary
  db/migrations/*.sql            schema `elec` (tables + views), applied by `cli migrate`
  net/                           polite HTTP client, circuit breaker
  search/                        DuckDuckGo / Google CSE providers, cache + budget
  probe/                         A/B/C platform grading
  extract/                       JSON-LD, HTML meta/spec tables, search-snippet prices
  normalise/                     brand aliases, title parser, spec units, grounding, LLM gap-fill
  match/                         listing -> canonical variant
  collector.py                   the pipeline
  cli.py                         command line
backend/app/api/routers/elec*.py read API + admin API
frontend/src                     catalogue UI (category -> brand -> product -> platforms)
```