nearle_daily/catalogue_backend
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

md file document updation

Browse Source
This commit is contained in:
sriram
2026-08-31 12:10:54 +05:30
parent 8f25842281
commit 3b1352b99d
1 changed files with 223 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

237
docs/INGESTION_API.md
View File

@@ -6,10 +6,16 @@ Four endpoints accept a spreadsheet. One is **open — no credential needed**
stages files for review; three are operator imports that require a credential and
write directly to the store, sales and nutrition tables.
> **Deployment status.** The review-inbox behaviour in §1 is committed but **not yet
> deployed**. Until it is, `POST /api/uploads/catalog` still answers `401` without a
> credential. Confirm before integrating — see
> [§7 Checking what is deployed](#7-checking-what-is-deployed).
If you only need to **send sheets and watch them run**, §1 is the whole story and you
need no credential. §2 covers the admin side — the review inbox, starting a batch, and
picking who executes it.
> **Deployment status.** Everything in §1 and §2 is **live on
> `https://mcp.nearle.ai.in`** — verified 2026-08-31 by probing the deployment
> itself, not by reading the source. The open drop takes no credential, and the
> stage timeline in [Stage-by-stage progress](#stage-by-stage-progress) is
> present in the served schema. If you want to re-confirm before integrating,
> [§8 Checking what is deployed](#8-checking-what-is-deployed) is a few curls.
---
@@ -32,8 +38,8 @@ send it a spreadsheet. That is only safe because nothing sent there runs on arri
files wait in an admin review inbox, so the cost of an unwanted drop is disk until
somebody declines it — never products in the live catalogue.
The three operator imports (§2) write straight to the database and stay credentialed.
Two credential types are accepted there:
The orchestration routes (§2) and the three operator imports (§3) write straight to
the database and stay credentialed. Two credential types are accepted there:
```
X-API-Key: <your secret> # machine clients
@@ -49,6 +55,7 @@ wrong credential.
| `POST /api/uploads/catalog` | **none** | anyone |
| `GET /api/uploads/catalog/{batch_id}` | **none** — the id is the credential | anyone holding that id |
| `GET /api/uploads/catalog` (list) | `upload_catalog` | `uploader`, `admin` |
| `/api/admin/catalog-batch/*` (§2) | `admin` | `admin` only |
| `/api/upload/stores` | `upload_store_inventory` | `user`, `admin` |
| `/api/upload/analytics` | `manage_analytics` | `admin` |
| `/api/upload/nutrition` | `manage_nutrition` | `admin` |
@@ -182,7 +189,85 @@ see filenames batched alongside your own.
`GET /api/uploads/catalog?limit=20` — your recent submissions, newest first, under
`{"batches": [...]}`. This one **does** need a credential: letting an anonymous caller
enumerate every sender's drops is a different thing from letting one check the id they
hold.
hold. Note that production currently has **no API keys configured** at all
(`auth.api_keys_count` is `0`), so today this list is reachable only with an admin
bearer token. Ask for a key to be issued if you need it; the two endpoints above
need none.
#### Poll anonymously, even if you hold a credential
Ownership is matched on the sender name, exactly. A run an admin assembled from
several drops carries a joined list — `"alice, bob"` — so a caller presenting a
credential as `alice` gets a **`404` on a run containing their own file**, while the
identical request *without* the credential returns `200`. That is not a bug you can
work around from the outside; it is the ownership filter doing what it says. Hold the
id, send no credential.
### Stage-by-stage progress
The poll body carries enough to draw the whole pipeline, not just a percentage.
**On the batch:**
| Field | Meaning |
| --- | --- |
| `stage_names` | All eleven names, in order — see [The eleven stages](#the-eleven-stages) |
| `runner` | `"inprocess"` or `"dagster"` — who is executing it (§2) |
`stage_names` is served rather than left for you to hardcode, deliberately: draw the
pipeline from it and your client cannot drift out of step when a stage is added or
renamed. Read it once per poll; it is eleven short strings.
**On each file:**
| Field | Meaning |
| --- | --- |
| `stage_index` / `stage_name` | Where this file is **right now** (`0` = not started) |
| `total_stages` | `11`, so a progress bar needs no constant |
| `rows_done` / `rows_total` | Progress *within* the current stage |
| `stages[]` | The timeline — one entry per stage this file has entered |
Each `stages[]` entry is `{index, name, rows_done, rows_total, started_at,
finished_at}`. **`finished_at: null` is the stage running now.** The array persists
after the run ends, so a finished file can still show its full timeline — the scalars
above only ever describe the present moment, which is why they are not enough on their
own.
```jsonc
// GET /api/uploads/catalog/{released_to} — mid-run
{
"batch_id": "8dcef8a2ad94...",
"status": "running",
"runner": "inprocess",
"stage_names": ["Brand Resolution & FSSAI Licence Mapping", "Row Intake & Normalisation", "..."],
"files": [
{
"filename": "catalog.csv",
"status": "running",
"stage_index": 6,
"stage_name": "Image Search & Contamination Filtering",
"total_stages": 11,
"rows_done": 120,
"rows_total": 400,
"stages": [
{ "index": 1, "name": "Brand Resolution & FSSAI Licence Mapping",
"rows_done": 400, "rows_total": 400,
"started_at": 1756612800.1, "finished_at": 1756612801.4 },
{ "index": 6, "name": "Image Search & Contamination Filtering",
"rows_done": 120, "rows_total": 400,
"started_at": 1756612809.7, "finished_at": null }
]
}
]
}
```
Timestamps are epoch seconds as floats. Stage 6 (image search) and stage 8 (barcode
enrichment) reach the network and are by far the slowest; a file sitting on either for
minutes is normal, which is what `rows_done` is for.
Poll every few seconds. Stop when `status` leaves `queued`/`running` — see the table
below.
### Status values
@@ -294,7 +379,119 @@ Drops nobody acts on are deleted after `BATCH_RETENTION_DAYS` (7).
---
## 2. Operator imports
## 2. Orchestration (admin)
§1 is the sender's half: drop a file, poll an id. This is the other half — seeing what
is waiting, deciding what runs, and choosing who runs it. Every route here is
`admin`-only.
### Getting a token
```bash
curl -s -X POST https://mcp.nearle.ai.in/api/auth/login \
-H 'Content-Type: application/json' \
-d '{"username":"admin","password":"<password>"}'
# -> {"access_token":"eyJ...","token_type":"bearer", ...}
```
Then `Authorization: Bearer <access_token>` on everything below. The token lasts 12
hours.
> **Before you hand this to anyone: there is exactly one admin account.** The second
> interactive account is disabled by design (its password hash is deliberately unset),
> so there is no way to issue a scoped orchestration login. Sharing this credential
> shares full administrative access to the whole application — catalogue writes,
> cancel, resume, the training endpoints — not just the routes in this section. If
> that is not what you want, keep the orchestration in-house and give the sender only
> §1, which needs no credential at all.
### The routes
| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/api/admin/catalog-batch/inbox` | Everything awaiting review, grouped by sender |
| `POST` | `/api/admin/catalog-batch/from-inbox` | Tick files and run them as one batch |
| `GET` | `/api/admin/catalog-batch/batches?limit=20` | Recent batches, slim (no per-product manifest) |
| `GET` | `/api/admin/catalog-batch/batches/{batch_id}` | One batch in full, with the stage timeline |
| `POST` | `/api/admin/catalog-batch/batches/{batch_id}/resume` | Restart a `queued` or `interrupted` batch |
| `POST` | `/api/admin/catalog-batch/batches/{batch_id}/cancel` | Stop before the remaining files begin |
| `POST` | `/api/admin/catalog-batch/inbox/dismiss` | Decline files without running them |
| `POST` | `/api/admin/catalog-batch/preview` | Parse sheets and report what was read — stores nothing |
| `POST` | `/api/admin/catalog-batch/ingest` | Upload and run directly, skipping the inbox |
The two `batches` reads return the same shape as §1's poll, so
[Stage-by-stage progress](#stage-by-stage-progress) applies unchanged. Use `preview`
before `ingest` when a sheet's headers are in doubt: it answers the column-mapping
question without writing anything.
### Starting a batch
```bash
# 1. See what is waiting
curl -s https://mcp.nearle.ai.in/api/admin/catalog-batch/inbox \
-H "Authorization: Bearer $TOKEN"
```
```jsonc
{ "pending_count": 2,
"submissions": [
{ "submission_id": "49a82536866a...", "submitted_by": "priya",
"created_at": 1756370000.0,
"files": [ { "file_id": "49a82536866a...:0", "filename": "catalog.csv",
"rows_total": 412, "size_bytes": 57000 } ] }
] }
```
```bash
# 2. Start the ones you want, as one batch
curl -X POST https://mcp.nearle.ai.in/api/admin/catalog-batch/from-inbox \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"file_ids":["49a82536866a...:0"],
"runner":"inprocess","use_llm":false,"fetch_images":true}'
# -> 202, and a NEW batch_id. Poll that one for the stages.
```
**`file_ids` are compound** — `"{batch_id}:{index}"` — and you take them verbatim from
`file_id` in the inbox response. Never assemble one by hand; a bare index is not unique
across two submissions.
**A partial start is a normal outcome, not an error.** Malformed or stale ids are
dropped silently rather than failing the request, because the admin UI polls every five
seconds and a tick can legitimately refer to a file another tab started a moment ago.
So check the returned batch's `files` for what was *actually* taken. If nothing was,
you get a `409` telling you to refresh.
The files you started leave the inbox, and the original drop's per-file `released_to`
is set to the new batch id — which is how the sender in §1 gets from the id they hold
to the run that carries their results.
### Choosing a runner
| `runner` | Who executes it |
| --- | --- |
| `"inprocess"` *(default)* | This container's worker thread. One batch at a time behind a queue. |
| `"dagster"` | Nobody, unless a Dagster instance is running to claim it. |
> ⚠️ **`runner: "dagster"` does not work in production, and fails silently.** Dagster
> is a local development orchestrator: it is absent from `requirements.txt`, from the
> production compose file and from the deployed image, which never even copies the
> `orchestration/` directory. A batch staged for it is deliberately handed to no one —
> it waits for `dagster dev` to pick it up, which in production is never. You get a
> batch parked at `queued` with the detail *"Waiting for the Dagster orchestrator to
> pick this batch up"*, indefinitely, which reads exactly like a hang.
>
> **Against `https://mcp.nearle.ai.in`, always send `"inprocess"`** — or omit the
> field, which means the same thing. To rescue one already stuck, `POST
> /api/admin/catalog-batch/batches/{batch_id}/resume` hands it to the in-process
> worker.
`use_llm` and `fetch_images` are chosen here rather than by the sender, because they
commit the host to outbound work. `fetch_images: true` makes stage 6 substantially
slower.
---
## 3. Operator imports
Three endpoints that write straight to the store, sales and nutrition tables — no pipeline, no review, no polling. Synchronous, with a per-row report. All three share one response shape. **Form field name is `file`** (singular).
@@ -389,7 +586,7 @@ Upserts use `COALESCE` throughout, so a later, thinner sheet can never blank a v
---
## 3. Errors
## 4. Errors
| Code | Cause | What to do |
| --- | --- | --- |
@@ -417,7 +614,7 @@ Sample sheets with the correct headers: `GET /api/upload/template/stores`, `/ana
---
## 4. If you already integrated
## 5. If you already integrated
| Was | Is now |
| --- | --- |
@@ -425,6 +622,7 @@ Sample sheets with the correct headers: `GET /api/upload/template/stores`, `/ana
| Files ran on arrival | Files wait for review. Expect `pending`, not `queued` |
| The drop id 404'd once an admin started it | It stays valid. The file reads `released` and carries `released_to` |
| The result was counts only | It also lists `products` with `image_id` / `product_sku` / `disposition` |
| Progress was one `stage_index` scalar | Each file also carries a `stages[]` timeline, and the batch carries `stage_names` and `runner` — see [Stage-by-stage progress](#stage-by-stage-progress) |
| `?use_llm` / `?fetch_images` on the drop | Ignored. The admin chooses at Start |
| `429` meant the worker queue was full | `429` now means the review inbox is full |
| `200` with `rows_imported: 0` | `422` with per-row reasons. Handle as a client error, not a server one |
@@ -434,7 +632,7 @@ Sample sheets with the correct headers: `GET /api/upload/template/stores`, `/ana
---
## 5. API keys are now optional
## 6. API keys are now optional
Nobody needs a key to send catalogue spreadsheets. Issue one only for a machine client
that wants the credentialed reads (`GET /api/uploads/catalog`) or the operator imports.
@@ -469,7 +667,7 @@ Constraints enforced at boot, before any request is served:
---
## 6. What an open drop costs
## 7. What an open drop costs
Disk, and nothing else, until somebody looks at it. The endpoint queues nothing, so it
cannot occupy the ingestion worker and cannot reach the catalogue on its own — the
@@ -486,7 +684,7 @@ from filling the inbox with sheets an admin then has to decline.
---
## 7. Checking what is deployed
## 8. Checking what is deployed
`GET /api/health` is public and answers `200` even when dependencies are down.
@@ -495,7 +693,18 @@ curl -s https://mcp.nearle.ai.in/api/health
```
- **`POST /api/uploads/catalog` with no credential and no file returns `400`/`422`** →
the open drop is live. A `401` means the old, credentialed build is still running.
the open drop is live. This is what production answers today; a `401` would mean the
old, credentialed build had been rolled back.
- **`GET /api/uploads/catalog/<32 random hex chars>` returns `404`, not `401`** → the
anonymous read is live and the id really is the credential.
- **The served schema carries the stage timeline** → `stages[]` and `stage_names` are
there, so a client can render the eleven stages:
```bash
curl -s https://mcp.nearle.ai.in/openapi.json \
| python -c "import json,sys; s=json.load(sys.stdin)['components']['schemas']; \
print('stage_names' in s['BatchOut']['properties'], 'stages' in s['BatchFileOut']['properties'])"
# -> True True
```
- **`auth.api_keys_count` / `api_keys` / `api_keys_source` present** → the build
includes the auth diagnostics. Absent → the deployment predates them.
- **`auth.api_keys[].fingerprint`** answers *"is my key on this deployment?"* without