Loyaly/loyaly-catalogue
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Electronics Catalog: API, MCP server, frontend and deployment

Browse Source 
Verified catalogue of mobiles and laptops sold in India, collected from
real retail listings (FastAPI backend, React frontend, Postgres/pgvector).

- REST API under /api/elec (read-only catalogue; admin endpoints need login)
- MCP server (FastMCP) at /mcp/ with list_categories, search_products,
  get_product and price_history tools
- Real ratings and reviews read from product pages and search results
- Production Dockerfile (requirements-api.txt, no PyTorch) and
  .env.production.example; remote database only via an explicit
  ELEC_ALLOW_REMOTE_DB host/name allowlist
- docs/API.md: endpoint and MCP reference with live examples

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
sriram
2026-10-01 12:17:42 +05:30
commit c7e4d59188
115 changed files with 14329 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

152
README.md Normal file
View File

@@ -0,0 +1,152 @@
# Electronics Catalog (local)
A local catalogue of **real** electronics products sold in India, with a Tamil Nadu focus:
the products, their prices, images and the e-commerce platforms that list them. It is a
converted copy of `Project_Deploy`, which stays untouched. It runs only on this machine
and is not pushed anywhere.
## How data is collected (search-first)
1. **Discover**: web search (DuckDuckGo via `ddgs`; Google Programmable Search too when a
key is set) runs `site:<platform> <brand> <variant terms>` on every registered
platform. Only single-product URLs are kept.
2. **Gate each platform** (`probe`), grading it A, B or C:
- **A**: robots.txt allows the page, it returns HTTP 200 with no bot check, and it
carries schema.org Product JSON-LD with an INR price. The product page is read.
- **B**: fetchable, but only the page HTML/meta is readable.
- **C**: blocked, CAPTCHA, robots.txt disallows, or no data without JavaScript.
**Web search results only**, with nothing fetched from the site.
- Amazon.in and Flipkart are `serp_only` by policy and are never fetched directly.
3. **Collect**:
- A/B pages are read politely: robots.txt obeyed, an honest User-Agent, 3 s between
requests to a site, and a circuit breaker on any 403/429/CAPTCHA.
- C platforms contribute what their search result shows: the title (with variant) and
URL, plus a price or stock status only when the snippet or the search engine's own
structured data states it.
4. **Match** each listing to one canonical variant (brand + model + RAM + storage; MPN for
laptops). A different model number is never merged. Uncertain matches go to a review
queue.
5. **Verify**: a product is shown only when listings on **at least two platforms** (at
least one a retailer) confirm it.
**Anti-fabrication rules:**
- Every listing and price row stores its source URL and the text the value was read from.
- Prices come only from parsers, never from the LLM. EMI, "₹X off", exchange and bank-offer
amounts are rejected.
- The local LLM (qwen2.5:1.5b) only fills missing *spec* fields from fetched text. Every
value it returns must appear in that text, or it is dropped.
- Images come only from a product's own listings and are checked live. Only URLs are
stored.
- Unknown stays unknown: `in_stock` is NULL and the price is "not stated".
- `pincode_applied` is true only if a site actually accepted the pincode. Currently none
does, so prices are national listing prices.
## Platforms
| Platform | Region | Mode (as probed on 29 Sep 2026) |
|---|---|---|
| Amazon.in, Flipkart | national | C: search only (policy) |
| Croma | national | C: blocked (HTTP 403), search only |
| Tata CLiQ | national | C: bot-check page, search only |
| Reliance Digital, Vijay Sales | national | A: product pages read |
| Poorvika, Vasanth & Co | Tamil Nadu | A: product pages read |
| Sangeetha Mobiles | Tamil Nadu | C: no data without JavaScript, search only |
| Viveks | Tamil Nadu | C: few product pages indexed |
| Brand official sites | - | probed per brand |
Grades are re-checked every 7 days (`probe`). A tripped breaker pauses a site for 24 h.
## Setup (once)
```powershell
# 1. Local database (container elec_catalog_pg, 127.0.0.1:5433, DB electronics_catalog)
copy .env.example .env # set POSTGRES_PASSWORD
docker compose up -d
# 2. Backend
cd backend
copy .env.example .env # same DB_PASSWORD; set ELEC_CONTACT to a real email
py -3.13 -m venv .venv
.venv\Scripts\pip install torch --index-url https://download.pytorch.org/whl/cpu
.venv\Scripts\pip install -r requirements.txt -r requirements-dev.txt
.venv\Scripts\python -m app.electronics.cli migrate
.venv\Scripts\python -m app.electronics.cli seed-reference
# 3. Frontend
cd ..\frontend
npm install
```
## Run
```powershell
start_app.bat # database + API (127.0.0.1:8000) + UI (http://localhost:5173)
```
`AUTH_ALLOW_ANY_LOGIN=true` in `backend/.env` accepts any password for user `admin`. That
is fine locally because the API binds to 127.0.0.1.
## Collect data (CLI, from `backend/`)
```powershell
.venv\Scripts\python -m app.electronics.cli probe # grade platforms A/B/C
.venv\Scripts\python -m app.electronics.cli collect --category mobiles --brand samsung --brand xiaomi --limit 10
.venv\Scripts\python -m app.electronics.cli collect --category laptops --brand hp --brand lenovo --limit 10
.venv\Scripts\python -m app.electronics.cli report # what is in the catalogue
.venv\Scripts\python -m app.electronics.cli review # uncertain matches
.venv\Scripts\python -m app.electronics.cli verify-grounding # audit: every price has evidence
```
Useful flags:
- `--no-fetch`: search results only.
- `--no-llm`: fully deterministic.
- `--budget N`: cap on search queries.
- `--reprobe`: re-grade the sites.
The same run can be started from the Admin page.
**Prices for Amazon, Flipkart and Croma.** Free search snippets almost never show a price
for these platforms, so their listings usually record availability only. To get their
prices without ever fetching them, set `GOOGLE_API_KEY` and `GOOGLE_CSE_ID` in
`backend/.env` (Google Programmable Search, 100 free queries a day). The Google Cloud
project behind the key must have the **Custom Search API** enabled. Then run:
```powershell
.venv\Scripts\python -m app.electronics.cli prices --limit 40
```
- Google queries are kept for this price pass. Discovery uses DuckDuckGo and falls back to
Google only when DuckDuckGo gives no answer.
- A price is taken only from Google's structured offer data for the **same product page**
(same site product ID), and it is stored with that data as evidence.
- A rejected key switches Google off for the rest of the run and reports why.
## Tests
```powershell
cd backend
.venv\Scripts\python -m pytest -q
```
The tests need no network and never touch real data. The database tests use a separate
`electronics_catalog_test` database on the same local server, and they are skipped if the
container is down.
## Layout
```
docker-compose.yml local Postgres + pgvector only
backend/app/electronics/
reference/*.yaml brand allow-list + aliases, platforms, spec dictionary
db/migrations/*.sql schema `elec` (tables + views), applied by `cli migrate`
net/ polite HTTP client, circuit breaker
search/ DuckDuckGo / Google CSE providers, cache + budget
probe/ A/B/C platform grading
extract/ JSON-LD, HTML meta/spec tables, search-snippet prices
normalise/ brand aliases, title parser, spec units, grounding, LLM gap-fill
match/ listing -> canonical variant
collector.py the pipeline
cli.py command line
backend/app/api/routers/elec*.py read API + admin API
frontend/src catalogue UI (category -> brand -> product -> platforms)
```