wait_for_files_to_ingest

nominal.core.wait_for_files_to_ingest(files: Sequence[DatasetFile], *, poll_interval: timedelta = datetime.timedelta(seconds=1), timeout: timedelta | None = None, return_when: IngestWaitType = IngestWaitType.ALL_COMPLETED) → tuple[Sequence[DatasetFile], Sequence[DatasetFile]]

Blocks until all of the dataset files have completed their ingestion (or other specified conditions) in a similar fashion to concurrent.futures.wait.

Any files that are already ingested (successfully or with errors) will be returned as “done”, whereas any files still ingesting by the time of this function’s exit will be returned as “not done”.

Parameters:

• files – Dataset files to monitor for ingestion completion.

• poll_interval – Interval to sleep between polling the remaining files under watch.

• timeout – If given, the maximum time to wait before returning. Never sleeps past the deadline.

• return_when – Condition for this function to exit. By default, this function will block until all files have completed their ingestion (successfully or unsuccessfully), but this can be changed to return upon the first completed or first failing ingest. This behavior mirrors that of concurrent.futures.wait.

Returns:

Returns a tuple of (done, not done) dataset files.