IngestionJob

class nominal.core.IngestionJob(rid: str, status: IngestionJobStatus, ingest_type: IngestType, dataset_rid: str | None, origin_files: Sequence[str], produced_file_count: int | None, created_by_rid: str | None, created_at: IntegralNanosecondsUTC | None, start_time: IntegralNanosecondsUTC | None, end_time: IntegralNanosecondsUTC | None)

Bases: HasRid, RefreshableConjureMixin[IngestJob]

A trackable record of one ingestion request moving through the async pipeline.

rid: str

status: IngestionJobStatus

ingest_type: IngestType

dataset_rid: str | None

origin_files: Sequence[str]

produced_file_count: int | None

created_by_rid: str | None

created_at: IntegralNanosecondsUTC | None

start_time: IntegralNanosecondsUTC | None

end_time: IntegralNanosecondsUTC | None

property nominal_url: str

Returns a link to the page for this ingest job in the Nominal app.

as_files_ingested(*, poll_interval: timedelta = datetime.timedelta(seconds=1), timeout: timedelta | None = None) → Iterable[DatasetFile]

Yield this job’s dataset files as each completes ingestion.

Waits out both stages of an ingest: the job producing its files, then those files finishing ingestion. The file list is re-read on each poll while the job is still running, so files that do not exist yet when this is called are still picked up — a containerized extractor registers its outputs only once its container has exited. list(job.as_files_ingested()) blocks until the job is terminal and every file it produced has finished ingesting.

Yielded files may have failed: a job can complete with some of its files FAILED, so check ingest_status on each yielded file if that matters. A job that was cancelled yields whatever did ingest and logs a warning, since a cancelled job is the caller getting what they asked for. Jobs with status DELETION_IN_PROGRESS or DELETED also stop job polling, yield any listed files as they finish ingesting, and log a warning; this does not wait for deletion.

Unlike the free function nominal.core.as_files_ingested, which only ever reports per-file status, this raises when the job itself fails — a failed job can have no files to report the failure through. Iterating this rather than wrapping it in list() keeps the files yielded before that point. For timeout / return-when control over a fixed set of files, use nominal.core.wait_for_files_to_ingest(job.dataset_files(), ...).

Parameters:

• poll_interval – Interval to sleep between polls of this job and of its pending files.

• timeout – Give up after this long and raise NominalIngestTimeout; None waits indefinitely. Never sleeps past the deadline.

Yields:

Dataset files as they finish ingesting. Due to the polling mechanics, the files are not yielded in strictly sorted order based on their ingestion completion time.

Raises:

• NominalIngestFailed – The job itself finished FAILED. Files that did ingest are named in the error, and were yielded before it was raised.

• NominalIngestTimeout – timeout elapsed while the job or one of its files was still in progress.

cancel() → Self

Cancel this ingest job.

SUBMITTED/QUEUED jobs transition directly to CANCELLED; IN_PROGRESS jobs have their underlying workflow cancelled. Cancelling a job already in a terminal state raises a conjure IngestJobNotCancellable (CONFLICT) error from the server.

dataset_files() → Sequence[DatasetFile]

Return the dataset files this ingest job has produced so far.

This is a point-in-time snapshot: a job that is not yet terminal can still produce more files, and one that produces its files in bulk has none to report until its work is done. To wait for the complete set, use as_files_ingested().

refresh() → Self