loyaly-catalogue/README.md

Electronics Catalog (local)

A local catalogue of real electronics products sold in India, with a Tamil Nadu focus: the products, their prices, images and the e-commerce platforms that list them. It is a converted copy of Project_Deploy, which stays untouched. It runs only on this machine and is not pushed anywhere.

How data is collected (search-first)

1. Discover: web search (DuckDuckGo via ddgs; Google Programmable Search too when a key is set) runs site:<platform> <brand> <variant terms> on every registered platform. Only single-product URLs are kept.
2. Gate each platform (probe), grading it A, B or C:
   • A: robots.txt allows the page, it returns HTTP 200 with no bot check, and it carries schema.org Product JSON-LD with an INR price. The product page is read.
   • B: fetchable, but only the page HTML/meta is readable.
   • C: blocked, CAPTCHA, robots.txt disallows, or no data without JavaScript. Web search results only, with nothing fetched from the site.
   • Amazon.in and Flipkart are serp_only by policy and are never fetched directly.
3. Collect:
   • A/B pages are read politely: robots.txt obeyed, an honest User-Agent, 3 s between requests to a site, and a circuit breaker on any 403/429/CAPTCHA.
   • C platforms contribute what their search result shows: the title (with variant) and URL, plus a price or stock status only when the snippet or the search engine's own structured data states it.
4. Match each listing to one canonical variant (brand + model + RAM + storage; MPN for laptops). A different model number is never merged. Uncertain matches go to a review queue.
5. Verify: a product is shown only when listings on at least two platforms (at least one a retailer) confirm it.

Anti-fabrication rules:

• Every listing and price row stores its source URL and the text the value was read from.
• Prices come only from parsers, never from the LLM. EMI, "₹X off", exchange and bank-offer amounts are rejected.
• The local LLM (qwen2.5:1.5b) only fills missing spec fields from fetched text. Every value it returns must appear in that text, or it is dropped.
• Images come only from a product's own listings and are checked live. Only URLs are stored.
• Unknown stays unknown: in_stock is NULL and the price is "not stated".
• pincode_applied is true only if a site actually accepted the pincode. Currently none does, so prices are national listing prices.

Platforms

Platform	Region	Mode (as probed on 29 Sep 2026)
Amazon.in, Flipkart	national	C: search only (policy)
Croma	national	C: blocked (HTTP 403), search only
Tata CLiQ	national	C: bot-check page, search only
Reliance Digital, Vijay Sales	national	A: product pages read
Poorvika, Vasanth & Co	Tamil Nadu	A: product pages read
Sangeetha Mobiles	Tamil Nadu	C: no data without JavaScript, search only
Viveks	Tamil Nadu	C: few product pages indexed
Brand official sites	-	probed per brand

Grades are re-checked every 7 days (probe). A tripped breaker pauses a site for 24 h.

Setup (once)

# 1. Local database (container elec_catalog_pg, 127.0.0.1:5433, DB electronics_catalog)
copy .env.example .env                  # set POSTGRES_PASSWORD
docker compose up -d

# 2. Backend
cd backend
copy .env.example .env                  # same DB_PASSWORD; set ELEC_CONTACT to a real email
py -3.13 -m venv .venv
.venv\Scripts\pip install torch --index-url https://download.pytorch.org/whl/cpu
.venv\Scripts\pip install -r requirements.txt -r requirements-dev.txt
.venv\Scripts\python -m app.electronics.cli migrate
.venv\Scripts\python -m app.electronics.cli seed-reference

# 3. Frontend
cd ..\frontend
npm install

Run

start_app.bat                     # database + API (127.0.0.1:8000) + UI (http://localhost:5173)

AUTH_ALLOW_ANY_LOGIN=true in backend/.env accepts any password for user admin. That is fine locally because the API binds to 127.0.0.1.

Collect data (CLI, from backend/)

.venv\Scripts\python -m app.electronics.cli probe                       # grade platforms A/B/C
.venv\Scripts\python -m app.electronics.cli collect --category mobiles --brand samsung --brand xiaomi --limit 10
.venv\Scripts\python -m app.electronics.cli collect --category laptops --brand hp --brand lenovo --limit 10
.venv\Scripts\python -m app.electronics.cli report                      # what is in the catalogue
.venv\Scripts\python -m app.electronics.cli review                      # uncertain matches
.venv\Scripts\python -m app.electronics.cli verify-grounding            # audit: every price has evidence

Useful flags:

• --no-fetch: search results only.
• --no-llm: fully deterministic.
• --budget N: cap on search queries.
• --reprobe: re-grade the sites.

The same run can be started from the Admin page.

Prices for Amazon, Flipkart and Croma. Free search snippets almost never show a price for these platforms, so their listings usually record availability only. To get their prices without ever fetching them, set GOOGLE_API_KEY and GOOGLE_CSE_ID in backend/.env (Google Programmable Search, 100 free queries a day). The Google Cloud project behind the key must have the Custom Search API enabled. Then run:

.venv\Scripts\python -m app.electronics.cli prices --limit 40

• Google queries are kept for this price pass. Discovery uses DuckDuckGo and falls back to Google only when DuckDuckGo gives no answer.
• A price is taken only from Google's structured offer data for the same product page (same site product ID), and it is stored with that data as evidence.
• A rejected key switches Google off for the rest of the run and reports why.

Tests

cd backend
.venv\Scripts\python -m pytest -q

The tests need no network and never touch real data. The database tests use a separate electronics_catalog_test database on the same local server, and they are skipped if the container is down.

Layout

docker-compose.yml               local Postgres + pgvector only
backend/app/electronics/
  reference/*.yaml               brand allow-list + aliases, platforms, spec dictionary
  db/migrations/*.sql            schema `elec` (tables + views), applied by `cli migrate`
  net/                           polite HTTP client, circuit breaker
  search/                        DuckDuckGo / Google CSE providers, cache + budget
  probe/                         A/B/C platform grading
  extract/                       JSON-LD, HTML meta/spec tables, search-snippet prices
  normalise/                     brand aliases, title parser, spec units, grounding, LLM gap-fill
  match/                         listing -> canonical variant
  collector.py                   the pipeline
  cli.py                         command line
backend/app/api/routers/elec*.py read API + admin API
frontend/src                     catalogue UI (category -> brand -> product -> platforms)