nearle_daily/catalogue_backend
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Backend upload-automation file

Browse Source
This commit is contained in:
sriram
2026-08-31 14:59:14 +05:30
parent 3b1352b99d
commit 3df2dc5991
9 changed files with 569 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files

150
docs/INGESTION_API.md
View File

@@ -10,12 +10,17 @@ If you only need to **send sheets and watch them run**, §1 is the whole story a
need no credential. §2 covers the admin side — the review inbox, starting a batch, and
picking who executes it.
> **Deployment status.** Everything in §1 and §2 is **live on
> `https://mcp.nearle.ai.in`** — verified 2026-08-31 by probing the deployment
> itself, not by reading the source. The open drop takes no credential, and the
> stage timeline in [Stage-by-stage progress](#stage-by-stage-progress) is
> present in the served schema. If you want to re-confirm before integrating,
> [§8 Checking what is deployed](#8-checking-what-is-deployed) is a few curls.
> **Deployment status.** The endpoints, the open drop and the stage timeline in
> [Stage-by-stage progress](#stage-by-stage-progress) are **live on
> `https://mcp.nearle.ai.in`** — verified 2026-08-31 by probing the deployment,
> not by reading the source.
>
> **`UPLOAD_AUTORUN` is the exception: committed, not yet deployed.** Until it
> ships, an upload still lands in the review inbox and answers `pending`. Both
> modes are documented below and both are supported; if you are integrating
> right now, write the client so it does not care which is running —
> [§8 Checking what is deployed](#8-checking-what-is-deployed) shows how to tell
> in one curl.
---
@@ -34,9 +39,18 @@ A blank allergens cell records *"we were not told"* — which is not the claim *
## Authentication
**The catalogue drop (§1) needs no credential.** Anyone who can reach the host can
send it a spreadsheet. That is only safe because nothing sent there runs on arrival:
files wait in an admin review inbox, so the cost of an unwanted drop is disk until
somebody declines it — never products in the live catalogue.
send it a spreadsheet — and with `UPLOAD_AUTORUN` on, that spreadsheet runs through
the pipeline immediately and its products land in the live catalogue. There is no
undo; ingestion is an upsert.
That is a deliberate trade, not an oversight. The requirement was uploads that run
without manual intervention, and a review queue that needs an admin to press a button
is not that. What bounds the endpoint is throughput rather than identity: the limits
in [Limits](#limits), and a single worker thread behind a queue of `BATCH_QUEUE_MAX`.
A sender can occupy the ingestion worker; they cannot multiply it.
With `UPLOAD_AUTORUN=false` the older behaviour returns — files wait in an admin
review inbox and the cost of an unwanted drop is disk until somebody declines it.
The orchestration routes (§2) and the three operator imports (§3) write straight to
the database and stay credentialed. Two credential types are accepted there:
@@ -75,9 +89,18 @@ anonymous submission the sender then cannot find.
This is the one to give a colleague. No credential, up to 20 sheets per request,
answers immediately with a `batch_id`.
**Nothing you send here runs on arrival.** The files are stored and an admin sees them
in the review inbox; only when they select the sheets and press Start does anything
reach the 11-stage pipeline or the catalogue.
**What happens on arrival depends on one setting, `UPLOAD_AUTORUN`:**
| | `true` (the intended production mode) | `false` |
| --- | --- | --- |
| On arrival | The batch is queued and the 11 stages run | Files wait in the admin review inbox |
| Batch `status` | `queued`, then `running` | `pending` |
| The id you get back | **is the run** — poll it and watch `stages[]` | is a *drop* id; the run gets a different one, reached via the file's `released_to` |
| Anything to do | No | An admin must tick the files and press Start |
**Write your client so it does not care which is running.** Both answer `202` with an
id that `GET /api/uploads/catalog/{id}` understands; only the number of hops differs.
Poll the id you were given, and follow `released_to` if a file ever grows one.
### `POST /api/uploads/catalog` → `202`
@@ -131,8 +154,8 @@ The bad file is kept as a failed member rather than dropped, so a sender who sub
```jsonc
{
"batch_id": "49a82536866a483a9189954d3c749243",
"status": "pending",
"detail": "Waiting for review. Nothing runs until an admin starts it.",
"status": "queued", // "pending" when UPLOAD_AUTORUN=false
"detail": null,
"submitted_by": "priya",
"created_at": 1756370000.0,
"updated_at": 1756370000.0,
@@ -140,8 +163,9 @@ The bad file is kept as a failed member rather than dropped, so a sender who sub
"files_done": 0,
"files_failed": 1,
"current_file": null,
"use_llm": false,
"fetch_images": false,
"use_llm": false, // UPLOAD_AUTORUN_USE_LLM
"fetch_images": true, // UPLOAD_AUTORUN_FETCH_IMAGES
"runner": "inprocess",
"totals": { "rows_total": 0, "products_built": 0, "inserted": 0,
"backfilled": 0, "skipped_existing": 0, "rejected": 0 },
"brands": [],
@@ -151,13 +175,18 @@ The bad file is kept as a failed member rather than dropped, so a sender who sub
{ "index": 1, "filename": "notes.txt", "status": "failed",
"detail": "The file has no data rows." }
],
"message": "1 file(s) received and waiting for review. 1 could not be read - see 'files' for the reason on each, and resend those."
"message": "1 file(s) accepted and queued for ingestion. 1 could not be read - see 'files' for the reason on each, and resend those."
}
```
Note the two levels of status. The **batch** is `pending` — waiting on a person. An
individual **file** inside it reads `queued`, meaning it is intact and eligible to be
run; it is waiting behind a decision, not behind the worker.
`use_llm` and `fetch_images` are reported, never accepted. They decide how much
outbound work a run commits the host to, and this endpoint's caller is anonymous, so
they come from settings — sending them in the request has no effect.
There are two levels of status, and they are not the same question. `status: "queued"`
on the **batch** means it is behind the worker; on a **file** it means intact and not
started yet. On the review-inbox path a `pending` batch containing `queued` files means
something different again: those files are waiting behind a *decision*, not the worker.
### Polling
@@ -165,6 +194,11 @@ run; it is waiting behind a decision, not behind the worker.
`batch_id` is a `uuid4` handed only to whoever sent the drop, so holding it is the
proof of having sent it. An id nobody issued is a `404`.
Under autorun that id is already the run, so the rest of this subsection does not
apply — go straight to [Stage-by-stage progress](#stage-by-stage-progress). The
release hop below is the `UPLOAD_AUTORUN=false` path, and is kept because that mode
is still supported and a client that handles both needs no branch.
**Your drop id stays valid for the whole lifecycle.** Poll it and read the per-file
`status`:
@@ -273,9 +307,9 @@ below.
| Status | Meaning |
| --- | --- |
| `pending` | In the review inbox. Nothing has run. |
| `pending` | In the review inbox. Nothing has run. `UPLOAD_AUTORUN=false` only |
| `retired` | Every file in this drop has been released or dismissed. Read the per-file `status` |
| `queued` | Approved and waiting for the worker |
| `queued` | Waiting for the worker. Under autorun this is the first status an upload gets |
| `running` | In the pipeline now |
| `done` | Every file completed |
| `partial` | Some landed, some failed. Deliberately not `done` — four of five succeeding must not read as flat success |
@@ -368,12 +402,19 @@ But that stability is per `image_id`: change the product name and you get a new
| Rows per file | 2,000 | — | file marked `failed`, others accepted |
| Bytes per request | 50 MB | `BATCH_MAX_TOTAL_BYTES` | `413` |
| Rows per request | 20,000 | `BATCH_MAX_TOTAL_ROWS` | `413` |
| Files awaiting review | 200 | `INBOX_MAX_PENDING_FILES` | `429` — nothing stored |
| Bytes awaiting review | 200 MB | `INBOX_MAX_PENDING_BYTES` | `429` — nothing stored |
| Batches queued behind the running one | 4 | `BATCH_QUEUE_MAX` | `429` — **autorun only** |
| Files awaiting review | 200 | `INBOX_MAX_PENDING_FILES` | `429` — `UPLOAD_AUTORUN=false` only |
| Bytes awaiting review | 200 MB | `INBOX_MAX_PENDING_BYTES` | `429` — `UPLOAD_AUTORUN=false` only |
The last two are the ceiling on the inbox as a whole, across every sender. They count
only what is still *awaiting review*, so starting or dismissing a drop frees its share
immediately. A `429` here stores nothing — resend once an admin has cleared space.
**Under autorun, expect a `429` occasionally and retry.** One batch runs at a time and
only four may wait behind it, so a handful of uploads in quick succession will hit the
ceiling. The files *are* staged when this happens and the message names the batch id,
so an admin can resume that one instead of you resending — but a client that treats
429 as a hard failure will report a working system as broken. Back off and retry.
The two inbox ceilings apply only when `UPLOAD_AUTORUN=false`. They count what is still
*awaiting review*, so starting or dismissing a drop frees its share immediately, and a
`429` from them stores nothing — resend once an admin has cleared space.
Drops nobody acts on are deleted after `BATCH_RETENTION_DAYS` (7).
@@ -595,7 +636,8 @@ Upserts use `COALESCE` throughout, so a later, thinner sheet can never blank a v
| `403` | Valid credential, wrong permission | Check the role table |
| `413` | Over a file-count, byte or row ceiling | Split the drop |
| `422` | **Operator imports only:** not one row could be imported | `detail.errors` lists row numbers and what each was missing |
| `429` | **Catalogue drop:** the review inbox is full | Nothing was stored. Ask an admin to clear it, then resend |
| `429` | **Catalogue drop, autorun:** four batches already queued | Normal under load. Back off and retry; `detail` names the staged batch id |
| `429` | **Catalogue drop, `UPLOAD_AUTORUN=false`:** the review inbox is full | Nothing was stored. Ask an admin to clear it, then resend |
```jsonc
// 400 — wrong columns
@@ -619,12 +661,12 @@ Sample sheets with the correct headers: `GET /api/upload/template/stores`, `/ana
| Was | Is now |
| --- | --- |
| `POST /api/uploads/catalog` needed an `X-API-Key` | No credential. Send the file; drop the header |
| Files ran on arrival | Files wait for review. Expect `pending`, not `queued` |
| Files waited for review (28 Aug – 31 Aug 2026) | **They run on arrival again.** Expect `queued`, not `pending`, and no `released_to` hop — the id you get back is the run. `UPLOAD_AUTORUN=false` restores the review inbox |
| The drop id 404'd once an admin started it | It stays valid. The file reads `released` and carries `released_to` |
| The result was counts only | It also lists `products` with `image_id` / `product_sku` / `disposition` |
| Progress was one `stage_index` scalar | Each file also carries a `stages[]` timeline, and the batch carries `stage_names` and `runner` — see [Stage-by-stage progress](#stage-by-stage-progress) |
| `?use_llm` / `?fetch_images` on the drop | Ignored. The admin chooses at Start |
| `429` meant the worker queue was full | `429` now means the review inbox is full |
| `?use_llm` / `?fetch_images` on the drop | Still ignored. Under autorun they come from `UPLOAD_AUTORUN_USE_LLM` / `UPLOAD_AUTORUN_FETCH_IMAGES`; the response reports what was used |
| `429` meant the review inbox was full | Under autorun it means the worker queue is full again — retryable, and the files were staged |
| `200` with `rows_imported: 0` | `422` with per-row reasons. Handle as a client error, not a server one |
| Missing `customer_id` became `cust_imported` | Row is skipped. Supply a real customer id |
| Missing `cost_price` invented as `mrp × 0.7` | Prices skipped for that row; inventory still lands. Send all three |
@@ -669,18 +711,32 @@ Constraints enforced at boot, before any request is served:
## 7. What an open drop costs
Disk, and nothing else, until somebody looks at it. The endpoint queues nothing, so it
cannot occupy the ingestion worker and cannot reach the catalogue on its own — the
review gate is what makes accepting files from anyone acceptable.
**With `UPLOAD_AUTORUN=true`, it costs CPU and it reaches the catalogue.** This is the
part to be clear-eyed about: the endpoint takes no credential, so anyone who can reach
the host can cause products to be written to the live catalogue, and an ingest is an
upsert with no undo. That was chosen knowingly — the requirement was uploads that run
without manual intervention — but it should never be a surprise to whoever operates
this next.
`INBOX_MAX_PENDING_FILES` and `INBOX_MAX_PENDING_BYTES` bound what unreviewed
submissions can occupy; past either the endpoint answers `429` and stores nothing.
Dismissing a drop deletes its bytes immediately, and retention reclaims anything nobody
ever looks at.
What still bounds it is throughput, not identity:
If the host is reachable from the open internet, consider an IP allow-list at the proxy
as defence in depth. The controls above bound the damage; they do not stop a stranger
from filling the inbox with sheets an admin then has to decline.
- the per-request ceilings in [Limits](#limits) — 20 files, 50 MB, 20,000 rows;
- one worker thread running a single batch at a time, with `BATCH_QUEUE_MAX` waiting
behind it and a `429` past that. A sender can occupy the ingestion worker — that is
what it is for — but cannot multiply it, and cannot touch the request path the
healthcheck reads;
- `BATCH_RETENTION_DAYS`, which reclaims staged bytes either way.
Nothing here bounds *who*. **If the host is reachable from the open internet, put an IP
allow-list on this route at the proxy** — with autorun on, that is no longer defence in
depth, it is the only control over who may write.
**With `UPLOAD_AUTORUN=false`** the cost is disk and nothing else until somebody looks:
the endpoint queues nothing, so it cannot occupy the worker or reach the catalogue on
its own. `INBOX_MAX_PENDING_FILES` and `INBOX_MAX_PENDING_BYTES` bound what unreviewed
submissions occupy, past either it answers `429` and stores nothing, and dismissing a
drop deletes its bytes immediately. The worst a stranger can then do is fill an inbox
an admin has to decline.
---
@@ -697,6 +753,18 @@ curl -s https://mcp.nearle.ai.in/api/health
old, credentialed build had been rolled back.
- **`GET /api/uploads/catalog/<32 random hex chars>` returns `404`, not `401`** → the
anonymous read is live and the id really is the credential.
- **Which mode is running.** There is no settings endpoint, so send one small sheet and
read the batch `status` it comes back with:
```bash
printf 'Product Name\nAmul Butter 100g\n' > /tmp/probe.csv
curl -s -X POST https://mcp.nearle.ai.in/api/uploads/catalog \
-F 'files=@/tmp/probe.csv' -F 'sender=deployment-probe' \
| python -c "import json,sys; print(json.load(sys.stdin)['status'])"
# queued -> UPLOAD_AUTORUN is on; that row is being ingested now
# pending -> the review inbox is on; nothing runs until an admin starts it
```
Note what the first answer means: the probe row **is ingested**. Use a product name
you are willing to see in the catalogue, or run this against a staging host.
- **The served schema carries the stage timeline** → `stages[]` and `stage_names` are
there, so a client can render the eleven stages:
```bash