diff --git a/docs/INGESTION_API.md b/docs/INGESTION_API.md index dd3375b..a16d983 100644 --- a/docs/INGESTION_API.md +++ b/docs/INGESTION_API.md @@ -6,10 +6,16 @@ Four endpoints accept a spreadsheet. One is **open — no credential needed** stages files for review; three are operator imports that require a credential and write directly to the store, sales and nutrition tables. -> **Deployment status.** The review-inbox behaviour in §1 is committed but **not yet -> deployed**. Until it is, `POST /api/uploads/catalog` still answers `401` without a -> credential. Confirm before integrating — see -> [§7 Checking what is deployed](#7-checking-what-is-deployed). +If you only need to **send sheets and watch them run**, §1 is the whole story and you +need no credential. §2 covers the admin side — the review inbox, starting a batch, and +picking who executes it. + +> **Deployment status.** Everything in §1 and §2 is **live on +> `https://mcp.nearle.ai.in`** — verified 2026-08-31 by probing the deployment +> itself, not by reading the source. The open drop takes no credential, and the +> stage timeline in [Stage-by-stage progress](#stage-by-stage-progress) is +> present in the served schema. If you want to re-confirm before integrating, +> [§8 Checking what is deployed](#8-checking-what-is-deployed) is a few curls. --- @@ -32,8 +38,8 @@ send it a spreadsheet. That is only safe because nothing sent there runs on arri files wait in an admin review inbox, so the cost of an unwanted drop is disk until somebody declines it — never products in the live catalogue. -The three operator imports (§2) write straight to the database and stay credentialed. -Two credential types are accepted there: +The orchestration routes (§2) and the three operator imports (§3) write straight to +the database and stay credentialed. Two credential types are accepted there: ``` X-API-Key: # machine clients @@ -49,6 +55,7 @@ wrong credential. | `POST /api/uploads/catalog` | **none** | anyone | | `GET /api/uploads/catalog/{batch_id}` | **none** — the id is the credential | anyone holding that id | | `GET /api/uploads/catalog` (list) | `upload_catalog` | `uploader`, `admin` | +| `/api/admin/catalog-batch/*` (§2) | `admin` | `admin` only | | `/api/upload/stores` | `upload_store_inventory` | `user`, `admin` | | `/api/upload/analytics` | `manage_analytics` | `admin` | | `/api/upload/nutrition` | `manage_nutrition` | `admin` | @@ -182,7 +189,85 @@ see filenames batched alongside your own. `GET /api/uploads/catalog?limit=20` — your recent submissions, newest first, under `{"batches": [...]}`. This one **does** need a credential: letting an anonymous caller enumerate every sender's drops is a different thing from letting one check the id they -hold. +hold. Note that production currently has **no API keys configured** at all +(`auth.api_keys_count` is `0`), so today this list is reachable only with an admin +bearer token. Ask for a key to be issued if you need it; the two endpoints above +need none. + +#### Poll anonymously, even if you hold a credential + +Ownership is matched on the sender name, exactly. A run an admin assembled from +several drops carries a joined list — `"alice, bob"` — so a caller presenting a +credential as `alice` gets a **`404` on a run containing their own file**, while the +identical request *without* the credential returns `200`. That is not a bug you can +work around from the outside; it is the ownership filter doing what it says. Hold the +id, send no credential. + +### Stage-by-stage progress + +The poll body carries enough to draw the whole pipeline, not just a percentage. + +**On the batch:** + +| Field | Meaning | +| --- | --- | +| `stage_names` | All eleven names, in order — see [The eleven stages](#the-eleven-stages) | +| `runner` | `"inprocess"` or `"dagster"` — who is executing it (§2) | + +`stage_names` is served rather than left for you to hardcode, deliberately: draw the +pipeline from it and your client cannot drift out of step when a stage is added or +renamed. Read it once per poll; it is eleven short strings. + +**On each file:** + +| Field | Meaning | +| --- | --- | +| `stage_index` / `stage_name` | Where this file is **right now** (`0` = not started) | +| `total_stages` | `11`, so a progress bar needs no constant | +| `rows_done` / `rows_total` | Progress *within* the current stage | +| `stages[]` | The timeline — one entry per stage this file has entered | + +Each `stages[]` entry is `{index, name, rows_done, rows_total, started_at, +finished_at}`. **`finished_at: null` is the stage running now.** The array persists +after the run ends, so a finished file can still show its full timeline — the scalars +above only ever describe the present moment, which is why they are not enough on their +own. + +```jsonc +// GET /api/uploads/catalog/{released_to} — mid-run +{ + "batch_id": "8dcef8a2ad94...", + "status": "running", + "runner": "inprocess", + "stage_names": ["Brand Resolution & FSSAI Licence Mapping", "Row Intake & Normalisation", "..."], + "files": [ + { + "filename": "catalog.csv", + "status": "running", + "stage_index": 6, + "stage_name": "Image Search & Contamination Filtering", + "total_stages": 11, + "rows_done": 120, + "rows_total": 400, + "stages": [ + { "index": 1, "name": "Brand Resolution & FSSAI Licence Mapping", + "rows_done": 400, "rows_total": 400, + "started_at": 1756612800.1, "finished_at": 1756612801.4 }, + { "index": 6, "name": "Image Search & Contamination Filtering", + "rows_done": 120, "rows_total": 400, + "started_at": 1756612809.7, "finished_at": null } + ] + } + ] +} +``` + +Timestamps are epoch seconds as floats. Stage 6 (image search) and stage 8 (barcode +enrichment) reach the network and are by far the slowest; a file sitting on either for +minutes is normal, which is what `rows_done` is for. + +Poll every few seconds. Stop when `status` leaves `queued`/`running` — see the table +below. ### Status values @@ -294,7 +379,119 @@ Drops nobody acts on are deleted after `BATCH_RETENTION_DAYS` (7). --- -## 2. Operator imports +## 2. Orchestration (admin) + +§1 is the sender's half: drop a file, poll an id. This is the other half — seeing what +is waiting, deciding what runs, and choosing who runs it. Every route here is +`admin`-only. + +### Getting a token + +```bash +curl -s -X POST https://mcp.nearle.ai.in/api/auth/login \ + -H 'Content-Type: application/json' \ + -d '{"username":"admin","password":""}' +# -> {"access_token":"eyJ...","token_type":"bearer", ...} +``` + +Then `Authorization: Bearer ` on everything below. The token lasts 12 +hours. + +> **Before you hand this to anyone: there is exactly one admin account.** The second +> interactive account is disabled by design (its password hash is deliberately unset), +> so there is no way to issue a scoped orchestration login. Sharing this credential +> shares full administrative access to the whole application — catalogue writes, +> cancel, resume, the training endpoints — not just the routes in this section. If +> that is not what you want, keep the orchestration in-house and give the sender only +> §1, which needs no credential at all. + +### The routes + +| Method | Path | Purpose | +| --- | --- | --- | +| `GET` | `/api/admin/catalog-batch/inbox` | Everything awaiting review, grouped by sender | +| `POST` | `/api/admin/catalog-batch/from-inbox` | Tick files and run them as one batch | +| `GET` | `/api/admin/catalog-batch/batches?limit=20` | Recent batches, slim (no per-product manifest) | +| `GET` | `/api/admin/catalog-batch/batches/{batch_id}` | One batch in full, with the stage timeline | +| `POST` | `/api/admin/catalog-batch/batches/{batch_id}/resume` | Restart a `queued` or `interrupted` batch | +| `POST` | `/api/admin/catalog-batch/batches/{batch_id}/cancel` | Stop before the remaining files begin | +| `POST` | `/api/admin/catalog-batch/inbox/dismiss` | Decline files without running them | +| `POST` | `/api/admin/catalog-batch/preview` | Parse sheets and report what was read — stores nothing | +| `POST` | `/api/admin/catalog-batch/ingest` | Upload and run directly, skipping the inbox | + +The two `batches` reads return the same shape as §1's poll, so +[Stage-by-stage progress](#stage-by-stage-progress) applies unchanged. Use `preview` +before `ingest` when a sheet's headers are in doubt: it answers the column-mapping +question without writing anything. + +### Starting a batch + +```bash +# 1. See what is waiting +curl -s https://mcp.nearle.ai.in/api/admin/catalog-batch/inbox \ + -H "Authorization: Bearer $TOKEN" +``` + +```jsonc +{ "pending_count": 2, + "submissions": [ + { "submission_id": "49a82536866a...", "submitted_by": "priya", + "created_at": 1756370000.0, + "files": [ { "file_id": "49a82536866a...:0", "filename": "catalog.csv", + "rows_total": 412, "size_bytes": 57000 } ] } + ] } +``` + +```bash +# 2. Start the ones you want, as one batch +curl -X POST https://mcp.nearle.ai.in/api/admin/catalog-batch/from-inbox \ + -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \ + -d '{"file_ids":["49a82536866a...:0"], + "runner":"inprocess","use_llm":false,"fetch_images":true}' +# -> 202, and a NEW batch_id. Poll that one for the stages. +``` + +**`file_ids` are compound** — `"{batch_id}:{index}"` — and you take them verbatim from +`file_id` in the inbox response. Never assemble one by hand; a bare index is not unique +across two submissions. + +**A partial start is a normal outcome, not an error.** Malformed or stale ids are +dropped silently rather than failing the request, because the admin UI polls every five +seconds and a tick can legitimately refer to a file another tab started a moment ago. +So check the returned batch's `files` for what was *actually* taken. If nothing was, +you get a `409` telling you to refresh. + +The files you started leave the inbox, and the original drop's per-file `released_to` +is set to the new batch id — which is how the sender in §1 gets from the id they hold +to the run that carries their results. + +### Choosing a runner + +| `runner` | Who executes it | +| --- | --- | +| `"inprocess"` *(default)* | This container's worker thread. One batch at a time behind a queue. | +| `"dagster"` | Nobody, unless a Dagster instance is running to claim it. | + +> ⚠️ **`runner: "dagster"` does not work in production, and fails silently.** Dagster +> is a local development orchestrator: it is absent from `requirements.txt`, from the +> production compose file and from the deployed image, which never even copies the +> `orchestration/` directory. A batch staged for it is deliberately handed to no one — +> it waits for `dagster dev` to pick it up, which in production is never. You get a +> batch parked at `queued` with the detail *"Waiting for the Dagster orchestrator to +> pick this batch up"*, indefinitely, which reads exactly like a hang. +> +> **Against `https://mcp.nearle.ai.in`, always send `"inprocess"`** — or omit the +> field, which means the same thing. To rescue one already stuck, `POST +> /api/admin/catalog-batch/batches/{batch_id}/resume` hands it to the in-process +> worker. + +`use_llm` and `fetch_images` are chosen here rather than by the sender, because they +commit the host to outbound work. `fetch_images: true` makes stage 6 substantially +slower. + +--- + +## 3. Operator imports Three endpoints that write straight to the store, sales and nutrition tables — no pipeline, no review, no polling. Synchronous, with a per-row report. All three share one response shape. **Form field name is `file`** (singular). @@ -389,7 +586,7 @@ Upserts use `COALESCE` throughout, so a later, thinner sheet can never blank a v --- -## 3. Errors +## 4. Errors | Code | Cause | What to do | | --- | --- | --- | @@ -417,7 +614,7 @@ Sample sheets with the correct headers: `GET /api/upload/template/stores`, `/ana --- -## 4. If you already integrated +## 5. If you already integrated | Was | Is now | | --- | --- | @@ -425,6 +622,7 @@ Sample sheets with the correct headers: `GET /api/upload/template/stores`, `/ana | Files ran on arrival | Files wait for review. Expect `pending`, not `queued` | | The drop id 404'd once an admin started it | It stays valid. The file reads `released` and carries `released_to` | | The result was counts only | It also lists `products` with `image_id` / `product_sku` / `disposition` | +| Progress was one `stage_index` scalar | Each file also carries a `stages[]` timeline, and the batch carries `stage_names` and `runner` — see [Stage-by-stage progress](#stage-by-stage-progress) | | `?use_llm` / `?fetch_images` on the drop | Ignored. The admin chooses at Start | | `429` meant the worker queue was full | `429` now means the review inbox is full | | `200` with `rows_imported: 0` | `422` with per-row reasons. Handle as a client error, not a server one | @@ -434,7 +632,7 @@ Sample sheets with the correct headers: `GET /api/upload/template/stores`, `/ana --- -## 5. API keys are now optional +## 6. API keys are now optional Nobody needs a key to send catalogue spreadsheets. Issue one only for a machine client that wants the credentialed reads (`GET /api/uploads/catalog`) or the operator imports. @@ -469,7 +667,7 @@ Constraints enforced at boot, before any request is served: --- -## 6. What an open drop costs +## 7. What an open drop costs Disk, and nothing else, until somebody looks at it. The endpoint queues nothing, so it cannot occupy the ingestion worker and cannot reach the catalogue on its own — the @@ -486,7 +684,7 @@ from filling the inbox with sheets an admin then has to decline. --- -## 7. Checking what is deployed +## 8. Checking what is deployed `GET /api/health` is public and answers `200` even when dependencies are down. @@ -495,7 +693,18 @@ curl -s https://mcp.nearle.ai.in/api/health ``` - **`POST /api/uploads/catalog` with no credential and no file returns `400`/`422`** → - the open drop is live. A `401` means the old, credentialed build is still running. + the open drop is live. This is what production answers today; a `401` would mean the + old, credentialed build had been rolled back. +- **`GET /api/uploads/catalog/<32 random hex chars>` returns `404`, not `401`** → the + anonymous read is live and the id really is the credential. +- **The served schema carries the stage timeline** → `stages[]` and `stage_names` are + there, so a client can render the eleven stages: + ```bash + curl -s https://mcp.nearle.ai.in/openapi.json \ + | python -c "import json,sys; s=json.load(sys.stdin)['components']['schemas']; \ +print('stage_names' in s['BatchOut']['properties'], 'stages' in s['BatchFileOut']['properties'])" + # -> True True + ``` - **`auth.api_keys_count` / `api_keys` / `api_keys_source` present** → the build includes the auth diagnostics. Absent → the deployment predates them. - **`auth.api_keys[].fingerprint`** answers *"is my key on this deployment?"* without