Backend- file ingestion API Updates
This commit is contained in:
@@ -76,9 +76,31 @@ ml_training_job = define_asset_job(
|
||||
),
|
||||
)
|
||||
|
||||
# Uploaded spreadsheets -> the same 11 stages -> brand tables.
|
||||
#
|
||||
# Separate from catalog_ingestion_job because the source is different, not
|
||||
# because the work is: that job reads seed JSON for a brand, this one reads a
|
||||
# batch of files somebody uploaded. They share every stage in between, and
|
||||
# keeping them apart means a failure here does not read as the brand pipeline
|
||||
# being broken.
|
||||
batch_ingestion_job = define_asset_job(
|
||||
name="batch_ingestion_job",
|
||||
description=(
|
||||
"Ingest a batch of uploaded store spreadsheets: resolve the batch, "
|
||||
"parse each file, run the 11 stages, and report what landed."
|
||||
),
|
||||
selection=AssetSelection.assets(
|
||||
"batch_manifest",
|
||||
"batch_parsed_files",
|
||||
"batch_catalog_rows",
|
||||
"batch_ingest_report",
|
||||
),
|
||||
)
|
||||
|
||||
ALL_JOBS = [
|
||||
catalog_ingestion_job,
|
||||
embedding_refresh_job,
|
||||
nutrition_enrichment_job,
|
||||
ml_training_job,
|
||||
batch_ingestion_job,
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user