DatasetFile#
- class nominal.core.DatasetFile(id: str, dataset_rid: str, name: str, bounds: Bounds | None, uploaded_at: IntegralNanosecondsUTC, ingested_at: IntegralNanosecondsUTC | None, deleted_at: IntegralNanosecondsUTC | None, ingest_status: IngestStatus, timestamp_channel: str | None, timestamp_type: TypedTimestampType | None, file_tags: Mapping[str, str] | None, tag_columns: Mapping[str, str] | None)#
Bases:
RefreshableConjureMixin[DatasetFile]An uploaded dataset file and its ingestion status and time bounds.
- uploaded_at: IntegralNanosecondsUTC#
- ingested_at: IntegralNanosecondsUTC | None#
- deleted_at: IntegralNanosecondsUTC | None#
- ingest_status: IngestStatus#
- timestamp_type: TypedTimestampType | None#
- delete() None#
Deletes the dataset file, removing its data permanently from Nominal.
Note
This cannot be undone outside of fully re-ingesting the file into Nominal.
- download(output_directory: Path | str, *, part_size: int = DEFAULT_CHUNK_SIZE, num_retries: int = 3) Path#
Download the dataset file to a destination on local disk.
- Parameters:
output_directory – Download file to the given directory
part_size – Size (in bytes) of chunks to use when downloading file.
num_retries – Number of retries to perform per part download if any exception occurs
- Returns:
Path that the file was downloaded to
- Raises:
FileNotFoundError – Output directory doesn’t exist
FileExistsError – File already exists at destination
RuntimeError – Error downloading file
- download_original_files(output_directory: Path | str, *, part_size: int = DEFAULT_CHUNK_SIZE, num_retries: int = 3) Sequence[Path]#
Download the input file(s) for a containerized extractor to a destination on local disk.
- Parameters:
output_directory – Download file(s) to the given directory
part_size – Size (in bytes) of chunks to use when downloading files.
num_retries – Number of retries to perform per part download if any exception occurs
- Returns:
Path(s) that the file(s) were downloaded to
- Raises:
NotADirectoryError – Output directory exists, but is not a directory
FileNotFoundError – Output directory doesn’t exist
FileExistsError – File already exists at destination
RuntimeError – Failed to determine metadata about files to download
Note
Any file that fails to download will result in an error log and will not be returned as an output path
- get_file_size() int | None#
Retrieves the size of the file in bytes, or None if it could not be determined
Note
This has only been extensively tested on AWS-based environments, and may fail in some self-hosted environments– a RuntimeError will be thrown in this case.
- get_ingest_error() NominalIngestError | None#
Returns the ingest exception if the file FAILED, or None if there is no error.
Refreshes the file from the API before checking.