nearle_daily/catalogue_backend
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

sheet upload fix

Browse Source
This commit is contained in:
Suriyakumarvijayanayagam
2026-08-28 11:22:53 +05:30
parent 0e75d32f61
commit 52f5d3be1d
41 changed files with 2056 additions and 1507 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
orchestration/README.md
View File

@@ -251,9 +251,10 @@ allocates the whole 8GB VPS (ollama 3G, backend 2560M, postgres 1G), so there
is no headroom for a webserver plus daemon. Orchestration is a development
concern here.
**This includes `batch_ingestion_job`.** The Batch Catalog Ingestion screen on
the site does not talk to Dagster and does not need it running. It stages the
uploaded files, then executes the same functions this job wraps
**This includes `batch_ingestion_job`.** The catalog ingestion endpoints
(`POST /api/admin/catalog-batch/ingest` and `POST /api/uploads/catalog`) do not
talk to Dagster and do not need it running. They stage the uploaded files, then
execute the same functions this job wraps
(`app/core/batch_ingest.py`) on a single bounded worker thread inside the API
process - see `app/core/batch_worker.py` for why one worker rather than a
thread per upload. The job here exists so the graph is inspectable and a batch

7
orchestration/assets/batch_catalog.py
View File

@@ -73,9 +73,10 @@ def _pick_batch_id(config: BatchConfig) -> str:
raise Failure(
description=(
"No batch_id given and no staged batch is waiting to run.\n\n"
"Upload one through Admin -> Batch Catalog Ingestion on the site, "
"or pass {\"batch_id\": \"...\"} in the Launchpad. Staged batches "
"live under BATCH_UPLOAD_DIR ({}).".format(batch_ingest.batch_root())
"Stage one through POST /api/admin/catalog-batch/ingest (or "
"POST /api/uploads/catalog), or pass {\"batch_id\": \"...\"} in "
"the Launchpad. Staged batches live under BATCH_UPLOAD_DIR "
f"({batch_ingest.batch_root()})."
),
metadata={"staged_batches": len(batch_ingest.list_manifests())},
)