nearle_daily/catalogue_backend
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

upload files without API-Key

Browse Source
This commit is contained in:
sriram
2026-08-28 17:13:31 +05:30
parent 52f5d3be1d
commit 5aa2669f7d
9 changed files with 1396 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files

426
docs/INGESTION_API.md Normal file
View File

@@ -0,0 +1,426 @@
# Sending spreadsheets to the catalogue
**Base:** `https://mcp.nearle.ai.in` · **Formats:** `.xlsx` `.xls` `.csv` `.tsv`
Four endpoints accept a spreadsheet. One is **open — no credential needed** — and
stages files for review; three are operator imports that require a credential and
write directly to the store, sales and nutrition tables.
> **Deployment status.** The review-inbox behaviour in §1 is committed but **not yet
> deployed**. Until it is, `POST /api/uploads/catalog` still answers `401` without a
> credential. Confirm before integrating — see
> [§7 Checking what is deployed](#7-checking-what-is-deployed).
---
## Read this first
These endpoints used to fill a missing column with a plausible constant. A sheet whose headers didn't match still returned `200 OK` — having written rows attributed to brand `amul` in store `store_mumbai_1` at MRP 100. Nutrition rows were worse: absent nutrients became invented numbers and an absent allergens column became `allergens = ['None']`, stored as `data_status: "verified"`.
The rule now: **nothing is invented.** A missing value is stored as NULL. A row missing the fields that identify it is skipped and reported back with its row number. An upload where nothing could be imported is a `422`, not a `200` with `rows_imported: 0`.
A blank allergens cell records *"we were not told"* — which is not the claim *"contains no allergens"*. Leave the column out rather than writing `None`.
**Arithmetic is not invention.** `line_total` is still derived from `quantity × unit_price` when your sheet omits it, because both were supplied. That distinction is the whole design: compute from given values, never substitute for absent ones.
---
## Authentication
**The catalogue drop (§1) needs no credential.** Anyone who can reach the host can
send it a spreadsheet. That is only safe because nothing sent there runs on arrival:
files wait in an admin review inbox, so the cost of an unwanted drop is disk until
somebody declines it — never products in the live catalogue.
The three operator imports (§2) write straight to the database and stay credentialed.
Two credential types are accepted there:
```
X-API-Key: <your secret> # machine clients
Authorization: Bearer <token> # from POST /api/auth/login, 12h lifetime
```
Send one or the other. Sending both is not rejected — the bearer token is tried first
and the API key ignored — but relying on that is a way to spend an hour debugging the
wrong credential.
| Endpoint | Permission | Who holds it |
| --- | --- | --- |
| `POST /api/uploads/catalog` | **none** | anyone |
| `GET /api/uploads/catalog/{batch_id}` | **none** — the id is the credential | anyone holding that id |
| `GET /api/uploads/catalog` (list) | `upload_catalog` | `uploader`, `admin` |
| `/api/upload/stores` | `upload_store_inventory` | `user`, `admin` |
| `/api/upload/analytics` | `manage_analytics` | `admin` |
| `/api/upload/nutrition` | `manage_nutrition` | `admin` |
`admin` is a superuser: `Principal.has_permission` grants it every permission rather
than requiring each to be listed. On the credentialed endpoints, no credential → `401`
and a valid credential without the permission → `403`.
A credential that is **present but wrong** is a `401` everywhere, including on the open
drop. Anonymous is allowed; mistyped is not — a typo must not silently downgrade to an
anonymous submission the sender then cannot find.
---
## 1. The catalogue drop
This is the one to give a colleague. No credential, up to 20 sheets per request,
answers immediately with a `batch_id`.
**Nothing you send here runs on arrival.** The files are stored and an admin sees them
in the review inbox; only when they select the sheets and press Start does anything
reach the 11-stage pipeline or the catalogue.
### `POST /api/uploads/catalog` → `202`
```bash
curl -X POST https://mcp.nearle.ai.in/api/uploads/catalog \
-F 'files=@store-catalog.xlsx' \
-F 'files=@second-store.xlsx' \
-F 'sender=priya'
```
| Form field | | Purpose |
| --- | --- | --- |
| `files` | **required** | The spreadsheets. Repeat the field for more than one; up to 20. |
| `sender` | optional | A label for the inbox, so the admin can see who sent what. Free text, trimmed to 60 chars. Defaults to `anonymous`. |
`use_llm` and `fetch_images` are **no longer accepted here.** They decide how a run
behaves and commit the host to outbound work, so the choice belongs to the admin
pressing Start — not to the sender. Passing them is inert.
### Which columns are read
Headers match case-insensitively by keyword, so `Product Name`, `ITEM` and `variant` all resolve to the same field. **A product-name column is the only hard requirement** — a sheet without one is rejected during the request.
| Field | | Headers that match |
| --- | --- | --- |
| `product_name` | **required** | `product` · `item` · `variant` · `name` |
| `brand` | optional | `brand` · `manufacturer` · `company` |
| `category` | optional | `category` · `segment` |
| `size_variants` | optional | `size` · `pack` · `weight` · `volume` · `net qty` · `quantity` |
| `final_selling_price` | optional | `final price` · `selling price` · `price` · `mrp` · `rate` · `cost` |
| `barcode` | optional | `barcode` · `bar code` · `gtin` · `ean` · `upc` |
| `fssai_license` | optional | any header containing `fssai` |
| `hsn_code` | optional | any header containing `hsn` |
| `image_url` | optional | `image` · `photo` · `picture` · `url` · `link` |
| `description` | optional | `description` · `desc` · `detail` |
| `title` | optional | `title` |
| `product_sku` | optional | `sku` |
| `price_range` | optional | `price range` · `range` |
| `providers` | optional | `provider` · `platform` · `marketplace` · `available at` |
| `highlights` | optional | `highlight` · `feature` · `benefit` |
| `nutrients` | optional | `nutrient` · `nutrition` |
Rules are evaluated in the order above, so a header matching two of them takes the
earlier field. Where two *columns* map to the same field, the **leftmost wins** and the
other is reported as ignored. Unrecognised columns are listed back, not an error.
### Response — `202`, one good file and one bad
The bad file is kept as a failed member rather than dropped, so a sender who submitted two files and sees one knows what happened to the other.
```jsonc
{
"batch_id": "49a82536866a483a9189954d3c749243",
"status": "pending",
"detail": "Waiting for review. Nothing runs until an admin starts it.",
"submitted_by": "priya",
"created_at": 1756370000.0,
"updated_at": 1756370000.0,
"files_total": 2,
"files_done": 0,
"files_failed": 1,
"current_file": null,
"use_llm": false,
"fetch_images": false,
"totals": { "rows_total": 0, "products_built": 0, "inserted": 0,
"backfilled": 0, "skipped_existing": 0, "rejected": 0 },
"brands": [],
"files": [
{ "index": 0, "filename": "catalog.csv", "status": "queued",
"total_stages": 11, "rows_total": 1, "size_bytes": 57 },
{ "index": 1, "filename": "notes.txt", "status": "failed",
"detail": "The file has no data rows." }
],
"message": "1 file(s) received and waiting for review. 1 could not be read - see 'files' for the reason on each, and resend those."
}
```
Note the two levels of status. The **batch** is `pending` — waiting on a person. An
individual **file** inside it reads `queued`, meaning it is intact and eligible to be
run; it is waiting behind a decision, not behind the worker.
### Polling
`GET /api/uploads/catalog/{batch_id}` — the same body, live. No credential: the
`batch_id` is a `uuid4` handed only to whoever sent the drop, so holding it is the
proof of having sent it. An id nobody issued is a `404`.
Expect `pending` until an admin acts. After that the batch you sent stops existing
under that id — the files move into a run with an id of its own, and your drop
reports `404`. **A `404` after `pending` means your files were accepted and started,**
not that they were lost. Ask the admin for the run id if you need to follow it.
`GET /api/uploads/catalog?limit=20` — your recent submissions, newest first, under
`{"batches": [...]}`. This one **does** need a credential: letting an anonymous caller
enumerate every sender's drops is a different thing from letting one check the id they
hold.
### Status values
| Status | Meaning |
| --- | --- |
| `pending` | In the review inbox. Nothing has run. |
| `queued` | Approved and waiting for the worker |
| `running` | In the pipeline now |
| `done` | Every file completed |
| `partial` | Some landed, some failed. Deliberately not `done` — four of five succeeding must not read as flat success |
| `failed` | No file completed |
| `interrupted` | A restart cut a run short. An admin resumes it; it never auto-restarts |
| `cancelled` | Stopped by an admin before the remaining files began |
### The eleven stages
1. Brand Resolution & FSSAI Licence Mapping
2. Row Intake & Normalisation
3. Title & Category Consistency Guard
4. Pack-Size Variant Explosion & Unit Safety
5. Pricing Band Estimation
6. Image Search & Contamination Filtering
7. Marketplace & Internal SKU Resolution
8. Barcode Retrieval & Enrichment
9. HSN / GST Tax Enrichment
10. Deterministic Product Validation Gate
11. Vector Embedding & Storage
### Limits
| Limit | Value | Env var | Exceeded |
| --- | --- | --- | --- |
| Files per request | 20 | `BATCH_MAX_FILES` | `413` |
| Bytes per file | 10 MB | — | `413` |
| Rows per file | 2,000 | — | file marked `failed`, others accepted |
| Bytes per request | 50 MB | `BATCH_MAX_TOTAL_BYTES` | `413` |
| Rows per request | 20,000 | `BATCH_MAX_TOTAL_ROWS` | `413` |
| Files awaiting review | 200 | `INBOX_MAX_PENDING_FILES` | `429` — nothing stored |
| Bytes awaiting review | 200 MB | `INBOX_MAX_PENDING_BYTES` | `429` — nothing stored |
The last two are the ceiling on the inbox as a whole, across every sender. They count
only what is still *awaiting review*, so starting or dismissing a drop frees its share
immediately. A `429` here stores nothing — resend once an admin has cleared space.
Drops nobody acts on are deleted after `BATCH_RETENTION_DAYS` (7).
---
## 2. Operator imports
Three endpoints that write straight to the store, sales and nutrition tables — no pipeline, no review, no polling. Synchronous, with a per-row report. All three share one response shape. **Form field name is `file`** (singular).
Each also answers at an `/upload` alias — `/api/upload/stores/upload`, `/api/upload/analytics/upload`, `/api/upload/nutrition/upload` — carrying the same guard. Prefer the short form.
```jsonc
// success
{
"status": "success",
"filename": "sheet.csv",
"rows_total": 1,
"rows_imported": 1,
"rows_skipped": 0,
"errors": [],
"message": "Successfully imported 1 store inventory items."
}
// partial — row 3 had no brand
{
"status": "partial",
"rows_total": 2,
"rows_imported": 1,
"rows_skipped": 1,
"errors": [
{ "row": 3, "error": "missing required field(s): brand" }
],
"message": "Imported 1 store inventory items; skipped 1 row(s) that were missing required fields - see 'errors'."
}
```
`row` is the number your spreadsheet shows — 1-based, counting the header — so row 3 is the second data row. At most **50** errors are returned; the rest are still skipped.
### `POST /api/upload/stores`
| Column | | When absent |
| --- | --- | --- |
| `store_id` | **required** | Row skipped and reported |
| `brand` | **required** | Row skipped and reported (lowercased on write) |
| `product_name` | **required** | Row skipped and reported |
| `image_id` | optional | Derived from brand + product name, so a re-upload lands on the same key |
| `store_name` | optional | Falls back to the `store_id`, underscores replaced and title-cased |
| `category`, `city` | optional | Stored NULL |
| `available_stock`, `reserved_stock`, `reorder_level`, `safety_stock` | optional | Stored `0` — the schema's own default; columns are NOT NULL |
| `mrp`, `cost_price`, `selling_price` | **all three** | Prices written only when all three present. Inventory still imports; counted in `prices_skipped` |
Extra response fields: `stores_affected`, `prices_written`, `prices_skipped`.
### `POST /api/upload/analytics`
> **This endpoint never worked before.** Its insert named a `total_price` column that doesn't exist on `order_items` (the column is `line_total`) and omitted the NOT NULL `store_id`. Every call raised and returned `500 Database import failed`. If you have an integration that gave up on it, retry.
| Column | | When absent |
| --- | --- | --- |
| `store_id`, `brand`, `customer_id`, `quantity`, `unit_price` | **required** | Row skipped and reported |
| `image_id` **or** `product_name` | one of | Row skipped. `image_id` derived from the name when only the name is given |
| `order_id` | optional | Generated. Each row then becomes its own order |
| `total_price` | optional | Derived as `quantity × unit_price`. Accepted under this name; stored in `line_total` |
| `order_date` | optional | Import time, counted in `dates_defaulted_to_now` so you can see how much of the file isn't really dated |
| `payment_method`, `delivery_status` | optional | Stored NULL |
Extra response fields: `total_revenue`, `dates_defaulted_to_now`.
`customer_id` is required rather than defaulted: it used to fall back to a shared `cust_imported`, which merges every buyer in a file into one customer and corrupts exactly the per-customer models this table feeds. Rows sharing an `order_id` become one order, whose `order_value` is recomputed as the **sum of its lines**.
**Import each file once.** `order_items` has a surrogate key and nothing to conflict on, so re-uploading appends the lines again.
### `POST /api/upload/nutrition`
Only `brand`, plus one of `image_id` / `product_name`, is required. Every nutrient column is optional and stays NULL when omitted. Recognised: `calories_kcal`, `protein_g`, `carbohydrates_g`, `total_sugar_g`, `dietary_fiber_g`, `total_fat_g`, `sodium_mg`, `calcium_mg`, `iron_mg`, `vitamin_c_mg`, `health_score`, `diet_tags`, `allergens` (plus `category`).
What you send decides the row's `data_status`, which every downstream reader must check before treating a NULL as zero:
| Status | Awarded when |
| --- | --- |
| `verified` | All seven core nutrients present — calories, protein, carbohydrates, sugar, fibre, fat, sodium |
| `partial` | Some of the seven present |
| `unavailable` | None present. Row still created, carrying only its identity |
```jsonc
// two of seven nutrients
{ "rows_imported": 1,
"data_status_counts": { "verified": 0, "partial": 1, "unavailable": 0 } }
// full core set
{ "rows_imported": 1,
"data_status_counts": { "verified": 1, "partial": 0, "unavailable": 0 } }
```
> **Never write `"None"` in the allergens column.** An omitted allergens column stores NULL with `allergen_source: "unavailable"`. A literal `None` stores the string as an allergen name. The two must stay distinguishable: one says nobody told us, the other claims the product is allergen-free — and that claim is served by the public `/api/nutrition` endpoints.
Upserts use `COALESCE` throughout, so a later, thinner sheet can never blank a value an earlier trusted source established, and a row already `verified` is never downgraded by a `partial` upload.
---
## 3. Errors
| Code | Cause | What to do |
| --- | --- | --- |
| `400` | File unparseable, no data rows, or (catalogue drop) no product-name column in any file | Read `detail`; it names the headers it found |
| `401` | Credential present but invalid; or absent on an endpoint that needs one | Check the table in Authentication |
| `403` | Valid credential, wrong permission | Check the role table |
| `413` | Over a file-count, byte or row ceiling | Split the drop |
| `422` | **Operator imports only:** not one row could be imported | `detail.errors` lists row numbers and what each was missing |
| `429` | **Catalogue drop:** the review inbox is full | Nothing was stored. Ask an admin to clear it, then resend |
```jsonc
// 400 — wrong columns
{ "detail": "None of the uploaded files could be ingested. wrong.csv: No product name column was found. Headers read: colour, size. Recognised fields: size_variants." }
// 422 — nothing importable
{ "detail": {
"message": "No store inventory items could be imported from 'sheet.csv'. Check the column headers against the sample template (GET /api/upload/template/...).",
"rows_total": 1,
"errors": [ { "row": 2, "error": "missing required field(s): store_id, brand" } ] } }
```
Note the shape difference: on `400`, `413` and `429`, `detail` is a **string**; on `422` it is an **object**. A client that renders `detail` directly will print `[object Object]` for a 422 unless it handles both.
Sample sheets with the correct headers: `GET /api/upload/template/stores`, `/analytics`, `/nutrition` — no credential needed.
---
## 4. If you already integrated
| Was | Is now |
| --- | --- |
| `POST /api/uploads/catalog` needed an `X-API-Key` | No credential. Send the file; drop the header |
| Files ran on arrival | Files wait for review. Expect `pending`, not `queued` |
| `?use_llm` / `?fetch_images` on the drop | Ignored. The admin chooses at Start |
| `429` meant the worker queue was full | `429` now means the review inbox is full |
| `200` with `rows_imported: 0` | `422` with per-row reasons. Handle as a client error, not a server one |
| Missing `customer_id` became `cust_imported` | Row is skipped. Supply a real customer id |
| Missing `cost_price` invented as `mrp × 0.7` | Prices skipped for that row; inventory still lands. Send all three |
| Missing nutrients and allergens got plausible defaults | Stored NULL, and the row is `partial`/`unavailable` rather than `verified` |
---
## 5. API keys are now optional
Nobody needs a key to send catalogue spreadsheets. Issue one only for a machine client
that wants the credentialed reads (`GET /api/uploads/catalog`) or the operator imports.
```bash
python scripts/make_auth_secrets.py --api-key catalog-drop:uploader
```
Put the resulting `name:role:secret` triple in the **Dokploy Environment tab**, not in
`.env.production`, and restart the service. Two reasons, both recorded in
`.env.production`'s own comments:
1. `.env.production` is committed. A per-consumer key is the one credential that gets
issued and revoked often, and it does not belong in git.
2. `backend/Dockerfile` does `COPY .env.production .env`, so a value there is baked at
**build** time — issuing or revoking would mean rebuilding an image that installs CPU
torch, a build that has already failed once on disk space. `settings.py` calls
`load_dotenv()` **without** `override=True`, so the process environment wins.
Container environment is fixed at creation, so the service must be **recreated**, not
merely restarted. `/api/health` will then report `api_keys_source: "process-env"`.
Constraints enforced at boot, before any request is served:
- Format `name:role:secret`, comma-separated between entries.
- `role` is `admin`, `user` or `uploader`. Keys never expire — treat one as a long-lived
secret and rotate it deliberately. Two entries can be live at once, which is how you
rotate without a cutover window.
- The secret must be at least **32 characters**, because `/api/health` publishes a digest.
- **Name the key for its function, not the person holding it.** `/api/health` is public
and reports `{name, role, fingerprint}` for every configured key.
---
## 6. What an open drop costs
Disk, and nothing else, until somebody looks at it. The endpoint queues nothing, so it
cannot occupy the ingestion worker and cannot reach the catalogue on its own — the
review gate is what makes accepting files from anyone acceptable.
`INBOX_MAX_PENDING_FILES` and `INBOX_MAX_PENDING_BYTES` bound what unreviewed
submissions can occupy; past either the endpoint answers `429` and stores nothing.
Dismissing a drop deletes its bytes immediately, and retention reclaims anything nobody
ever looks at.
If the host is reachable from the open internet, consider an IP allow-list at the proxy
as defence in depth. The controls above bound the damage; they do not stop a stranger
from filling the inbox with sheets an admin then has to decline.
---
## 7. Checking what is deployed
`GET /api/health` is public and answers `200` even when dependencies are down.
```bash
curl -s https://mcp.nearle.ai.in/api/health
```
- **`POST /api/uploads/catalog` with no credential and no file returns `400`/`422`** →
the open drop is live. A `401` means the old, credentialed build is still running.
- **`auth.api_keys_count` / `api_keys` / `api_keys_source` present** → the build
includes the auth diagnostics. Absent → the deployment predates them.
- **`auth.api_keys[].fingerprint`** answers *"is my key on this deployment?"* without
anyone sending the secret — the question a `401` cannot answer, since an undeployed
key and a wrong key fail identically. Compare against
`scripts/make_auth_secrets.py --fingerprint`.
`status: "degraded"` with `ollama: false` is expected in production: `USE_OLLAMA` is
`false` there.