Backend- file ingestion API Updates

This commit is contained in:
sriram
2026-08-28 07:49:56 +05:30
parent f698720ee2
commit a54bd43f8b
17 changed files with 2306 additions and 138 deletions

View File

@@ -76,9 +76,31 @@ ml_training_job = define_asset_job(
),
)
# Uploaded spreadsheets -> the same 11 stages -> brand tables.
#
# Separate from catalog_ingestion_job because the source is different, not
# because the work is: that job reads seed JSON for a brand, this one reads a
# batch of files somebody uploaded. They share every stage in between, and
# keeping them apart means a failure here does not read as the brand pipeline
# being broken.
batch_ingestion_job = define_asset_job(
name="batch_ingestion_job",
description=(
"Ingest a batch of uploaded store spreadsheets: resolve the batch, "
"parse each file, run the 11 stages, and report what landed."
),
selection=AssetSelection.assets(
"batch_manifest",
"batch_parsed_files",
"batch_catalog_rows",
"batch_ingest_report",
),
)
ALL_JOBS = [
catalog_ingestion_job,
embedding_refresh_job,
nutrition_enrichment_job,
ml_training_job,
batch_ingestion_job,
]