nearle_daily/catalogue_backend
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Dagster Orchestration

Browse Source
This commit is contained in:
sriram
2026-08-29 14:49:45 +05:30
parent 27d53fa957
commit 998df898db
28 changed files with 1455 additions and 76 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
orchestration/README.md
View File

@@ -260,8 +260,16 @@ process - see `app/core/batch_worker.py` for why one worker rather than a
thread per upload. The job here exists so the graph is inspectable and a batch
can be re-run from the Launchpad on a development machine.
Do not turn `batch_upload_sensor` on next to a running API: both would pick up
the same staged batch. That is why it ships STOPPED like the rest.
`batch_upload_sensor` is safe to run next to a live API. Each batch carries a
`runner` field naming its owner, and the sensor (and `_pick_batch_id`) claim
only `runner == "dagster"` — the batches an admin sent here from **Admin →
Dagster Orchestration**. Anything staged for the API's own worker is left
alone, so the two no longer race for the same files. It still ships STOPPED
like the rest: a development machine should not begin ingesting just because a
directory has something in it.
Passing an explicit `{"batch_id": "..."}` in the Launchpad bypasses the runner
filter — that is a person naming a batch, not a poll.
Port 3030, not Dagster's default 3000 - `serve.py` binds `PORTS=3000,8000` and
the frontend nginx also listens on 3000.