Skip to content

Containerized extractors¤

Declare files and settings with decorators, receive them as callback arguments, and use the context to declare outputs. The same declarations describe the image for registration.

The package walkthrough covers authoring, local execution, outputs, and publishing. The modules below provide the API reference. Their public objects are also available directly from nominal.experimental.extractor.

Decorators¤

decorators ¤

Declare an extractor's inputs, settings, failures, and output contract.

Place @manifest_extractor (or @single_file_extractor) outermost. Add @input, @parameter, and @error beneath it; the runner uses those declarations both to invoke your callback and to describe the image for registration.

error ¤

error(
    exception_type: type[Exception],
    *,
    code: str,
    exit_code: int,
    retryable: bool = False,
    message: str | None = None,
) -> _ErrorMapping

Declare a structured failure handled by the extractor runner.

Place below @manifest_extractor or @single_file_extractor, alongside @input and @parameter. Stack declarations for different Exception subclasses; the closest class in the raised exception's MRO wins, regardless of decorator order. Declaring the same class twice is an error. This decorator preserves the callback's signature.

code is a nonempty catalog error code; exit_code is an integer from 1 through 255. retryable describes whether another attempt may succeed without changing the input. Codes must fit the 4,096-byte termination JSON envelope. Long messages are shortened; mapped failures retain their full traceback on stderr.

Mappings cover startup, argument binding, extraction, and output finalization. They configure runtime reporting. message supplies static fallback text required by both :meth:Extractor.registration_kwargs and :meth:Extractor.catalog_manifest. Runtime messages come from the exception. Both exports validate and combine exit-code fallbacks. run(exit=False) and direct callback invocation propagate errors without reporting.

input ¤

input(
    argument: str,
    *,
    envvar: str | None = None,
    name: str | None = None,
    description: str | None = None,
    file_suffixes: Sequence[str] = (),
    default: None | _Missing = _MISSING,
) -> _Input

Declare a file your callback needs and receive its mounted path.

For example, @input("recording", envvar="SOURCE", file_suffixes=["csv"]) supplies the callback's recording: Path argument and describes that input for registration. Use default=None and Path | None for an optional file.

Place below the outer extractor decorator. The named callback argument must accept a keyword value and have no signature default. Annotations describe the received value; they do not control binding. Registered metadata is authoritative when present; otherwise the path comes from the declared environment variable. Lookup matches environment variables exactly, not display names.

Parameters:

  • argument ¤

    (str) –

    Name of the callback argument to populate.

  • envvar ¤

    (str | None, default: None ) –

    Input environment variable; defaults to the uppercase argument name.

  • name ¤

    (str | None, default: None ) –

    Registration display name; defaults to the argument name.

  • description ¤

    (str | None, default: None ) –

    Optional registration description.

  • file_suffixes ¤

    (Sequence[str], default: () ) –

    Registration filters such as ["csv", "mcap"]; empty accepts any suffix. These do not validate the local file's extension.

  • default ¤

    (None | _Missing, default: _MISSING ) –

    Omit to require the input, or use None to allow an absent file. Other defaults are unsupported.

Raises:

  • ValueError –

    A name is invalid or reserved, or the default is unsupported.

  • ExtractorError –

    At binding time, a required input is absent or a supplied path does not exist as a file. All inputs resolve before the callback runs.

manifest_extractor ¤

manifest_extractor(
    fn: Callable[..., None],
    *,
    default_timestamp_column: str | None = None,
    default_timestamp_type: _AnyTimestampType | None = None,
) -> Extractor[ManifestExtractorContext]

manifest_extractor(
    *,
    default_timestamp_column: str | None = None,
    default_timestamp_type: _AnyTimestampType | None = None,
) -> Callable[
    [Callable[..., None]], Extractor[ManifestExtractorContext]
]

manifest_extractor(
    fn: Callable[..., None] | None = None,
    *,
    default_timestamp_column: str | None = None,
    default_timestamp_type: _AnyTimestampType | None = None,
) -> (
    Extractor[ManifestExtractorContext]
    | Callable[
        [Callable[..., None]], Extractor[ManifestExtractorContext]
    ]
)

Turn a parsing function into a manifest extractor, recommended for new images.

Declare the callback's files and settings with @input and @parameter below this decorator. The framework resolves them before invoking the callback; ctx supplies output paths and job metadata. Add @error for expected failure mappings.

Write files under ctx.output_dir and declare each with :meth:ManifestExtractorContext.add_tabular, :meth:~ManifestExtractorContext.add_avro_stream, :meth:~ManifestExtractorContext.add_journal_json, or :meth:~ManifestExtractorContext.add_video. One run can produce several files with different timestamps, tags, and channel prefixes. :meth:Extractor.run writes manifest.json after the callback returns.

Supply both timestamp defaults here to reuse the declarations through :meth:Extractor.registration_kwargs or :meth:Extractor.catalog_manifest. They describe image defaults; per-output timestamp settings belong on the context's output methods. Register the image with MANIFEST; the runner rejects a different injected output format at startup.

Example::

from pathlib import Path

from nominal.experimental.extractor import ManifestExtractorContext, input, manifest_extractor, parameter

@manifest_extractor(
    default_timestamp_column="time_us",
    default_timestamp_type="epoch_microseconds",
)
@input("source")
@parameter("parts", type=int, default=2)
def split(ctx: ManifestExtractorContext, *, source: Path, parts: int) -> None:
    table = read_parquet(source)
    for i, chunk in enumerate(chunks_of(table, parts)):
        out = ctx.output_dir / f"part_{i}.parquet"
        write_parquet(chunk, out)
        ctx.add_tabular(out, timestamp_column="time_us", timestamp_type="epoch_microseconds")

    footage = ctx.output_dir / "camera.mp4"
    write_video(footage)
    ctx.add_video(footage, channel="camera/front", start=recording_started_at)

if __name__ == "__main__":
    split.run()

parameter ¤

parameter(
    argument: str,
    *,
    envvar: str | None = None,
    name: str | None = None,
    description: str | None = None,
    type: Callable[[str], object] | _Missing = _MISSING,
    default: object = _MISSING,
) -> _Parameter

Declare a setting so your callback receives a usable Python value.

For example, @parameter("parts", type=IntRange(min=1), default=2) supplies the callback's parts: int argument. Omission gives 2; a supplied value is converted and checked before extraction starts. This also declares the setting for registration.

Place below the outer extractor decorator, with no default on the callback argument. Binding happens before extraction and uses the runner's normal error mappings. Registered parameter metadata is authoritative: an unknown parameter fails even if it has a default. Registration stores names, descriptions, environment variables, and requiredness; converters and default values remain runtime settings.

Parameters:

  • argument ¤

    (str) –

    Name of the explicit callback argument to populate.

  • envvar ¤

    (str | None, default: None ) –

    Parameter environment variable; defaults to the uppercase argument name.

  • name ¤

    (str | None, default: None ) –

    Registration display name; defaults to the argument name.

  • description ¤

    (str | None, default: None ) –

    Optional registration description.

  • type ¤

    (Callable[[str], object] | _Missing, default: _MISSING ) –

    Converter for supplied strings. When omitted, infer str/int/float/bool from a concrete default, otherwise use str. Explicit converters take precedence; annotations do not select conversion. Boolean conversion accepts true/false, yes/no, on/off, and 1/0 case-insensitively. Float values must be finite.

  • default ¤

    (object, default: _MISSING ) –

    Already-converted Python value used when absent, including None; omit to require the parameter. Built-in types and constraints validate concrete defaults. Custom converters are never invoked on defaults. An empty supplied string is a value, not a request for the default.

Raises:

  • ValueError –

    A name is invalid or reserved, or a constrained default is invalid.

  • TypeError –

    The converter is not callable or a default has an incompatible type.

  • ExtractorError –

    At binding time, a parameter is missing or unregistered, or conversion raises ValueError or TypeError. Those converter messages and values are suppressed, except BadParameter, whose author-controlled diagnostic is included.

Other converter exceptions propagate unchanged to the runner, including their messages in tracebacks and mapped termination reports. Wrap third-party failures in ValueError or TypeError to sanitize them, or BadParameter with a message safe to display.

single_file_extractor ¤

single_file_extractor(
    fn: Callable[..., None],
    *,
    default_timestamp_column: str | None = None,
    default_timestamp_type: _AnyTimestampType | None = None,
    output_format: FileOutputFormat | None = None,
) -> Extractor[SingleFileExtractorContext]

single_file_extractor(
    *,
    default_timestamp_column: str | None = None,
    default_timestamp_type: _AnyTimestampType | None = None,
    output_format: FileOutputFormat | None = None,
) -> Callable[
    [Callable[..., None]], Extractor[SingleFileExtractorContext]
]

single_file_extractor(
    fn: Callable[..., None] | None = None,
    *,
    default_timestamp_column: str | None = None,
    default_timestamp_type: _AnyTimestampType | None = None,
    output_format: FileOutputFormat | None = None,
) -> (
    Extractor[SingleFileExtractorContext]
    | Callable[
        [Callable[..., None]], Extractor[SingleFileExtractorContext]
    ]
)

Declare a single-file extractor with ctx followed by injected arguments.

Use @error(ExceptionType, code="...", exit_code=64) below this decorator to declare structured failures handled automatically by :meth:Extractor.run. Declare inputs and parameters below this decorator with @input and @parameter; they are passed as keyword arguments after ctx. Declare default_timestamp_column and default_timestamp_type to generate :meth:Extractor.registration_kwargs.

For images registered with a single-file output format (PARQUET, CSV, ...): the ingest pipeline ingests exactly one output file, parsed per the registered format. Declare it with :meth:SingleFileExtractorContext.set_output. If the image's registered format turns out to be MANIFEST, :meth:Extractor.run fails at startup with a clear error.

Set output_format explicitly to generate registration metadata; it must then match the registered format exactly. Omitting it preserves legacy runtime behavior.

This is the original output contract, kept for images already registered against it. New extractors should use :func:manifest_extractor, a strict superset: this mode has no per-output timestamp, tag-column, or channel-prefix control, and changing an image's output format later requires registering a new image.

Example::

from pathlib import Path

from nominal.core.container_image import FileOutputFormat
from nominal.experimental.extractor import SingleFileExtractorContext, input, single_file_extractor

@single_file_extractor(
    output_format=FileOutputFormat.PARQUET,
    default_timestamp_column="time_us",
    default_timestamp_type="epoch_microseconds",
)
@input("source")
def convert(ctx: SingleFileExtractorContext, *, source: Path) -> None:
    table = read_input(source)
    out = ctx.output_dir / "converted.parquet"
    write_parquet(table, out)
    ctx.set_output(out)

if __name__ == "__main__":
    convert.run()

Output contexts¤

ManifestExtractorContext.add_tabular and add_avro_stream accept units= maps from channel names to unit symbols. Each declaration copies its map into that output's manifest entry; units are output metadata, not image-registration settings. For direct file uploads, use the ingestion builder.

context ¤

The execution contexts an extractor function receives.

logger module-attribute ¤

logger = getLogger(__name__)

ExtractorContext dataclass ¤

ExtractorContext(
    output_dir: Path,
    _env: Mapping[str, str],
    _input_dir: Path,
    _input_specs: list[_InputSpec] | None = None,
    _param_specs: list[_ParamSpec] | None = None,
    _declared: set[str] = set(),
)

The execution context handed to an extractor function.

Use this for output paths, output declarations, and job metadata. Declare inputs and parameters with @input and @parameter so the runtime passes them as callback arguments. Context lookup methods remain available for migration and warn once per run. Authors do not construct this directly; :meth:Extractor.run builds a :class:SingleFileExtractorContext or :class:ManifestExtractorContext.

additional_tags property ¤

additional_tags: dict[str, str]

Tags the ingest request applies to all data from this run; empty when Nominal didn't inject them.

dataset_rid property ¤

dataset_rid: str | None

RID of the dataset this run ingests into.

None when Nominal injected no value, empty included; see :attr:ingest_job_rid.

ingest_job_rid property ¤

ingest_job_rid: str | None

RID of the ingest job running this extractor.

None when Nominal injected no value, empty included -- an empty environment variable means the same thing as an absent one here, as it does everywhere else in this contract.

inputs property ¤

inputs: list[Path]

All input files Nominal mounted for this run.

Legacy authoring API: prefer a separate @input declaration per registered input. Access logs one migration warning per run; it contributes no registration metadata. Taken from the registered _NOMINAL_INPUTS metadata when present, in the order Nominal serializes them; otherwise discovered by listing the input mount, sorted by name.

job_timestamp_metadata property ¤

job_timestamp_metadata: TimestampMetadata | None

The job-level timestamp metadata this run's outputs default to.

This is the metadata the pipeline resolved for the whole job (the ingest request's override when given, else the image's registered default) -- the value a manifest output falls back to when it declares no per-output timestamp metadata of its own. None when Nominal didn't inject it.

output_dir instance-attribute ¤

output_dir: Path

get_param ¤

get_param(name: str, default: None = None) -> str | None

get_param(name: str, default: str) -> str

get_param(name: str, default: str | None = None) -> str | None

Read an optional parameter from the environment, or default when unset.

Legacy authoring API: prefer @parameter("parts", type=int, default=2) and a parts callback argument. This method logs one migration warning per run and contributes no registration metadata. Its returned values remain strings (or None). Name resolution matches :meth:param. With registered contract metadata present, an unregistered name raises :class:ExtractorError.

input ¤

input(name: str | None = None) -> Path

Resolve an input file.

Legacy authoring API: prefer @input("source", envvar="SOURCE") and a source callback argument. This method logs one migration warning per run and contributes no registration metadata.

With name -- the input's registered display name or its environment variable -- returns that input's path. Without it, returns the sole mounted input file, raising if there is not exactly one.

param ¤

param(name: str) -> str

Read a required parameter from the environment.

Legacy authoring API: prefer @parameter("parts", envvar="PARTS", type=int) and a parts callback argument. This method logs one migration warning per run and contributes no registration metadata.

name -- the parameter's registered display name or its environment variable -- is resolved against _NOMINAL_PARAMETERS when Nominal injected it; otherwise it is treated directly as the environment variable. Raises :class:ExtractorError when the parameter is not set. Values returned by this legacy method remain strings. With registered contract metadata present, an unregistered name raises :class:ExtractorError.

ManifestExtractorContext dataclass ¤

ManifestExtractorContext(
    output_dir: Path,
    _env: Mapping[str, str],
    _input_dir: Path,
    _input_specs: list[_InputSpec] | None = None,
    _param_specs: list[_ParamSpec] | None = None,
    _declared: set[str] = set(),
    _outputs: list[ManifestOutput] = list(),
    _video_outputs: list[ManifestVideoOutput] = list(),
)

Bases: ExtractorContext

Context for :func:manifest_extractor functions.

One declaration method per output format: :meth:add_tabular, :meth:add_avro_stream, :meth:add_journal_json, :meth:add_video. Each exposes only the options its format actually uses.

additional_tags property ¤

additional_tags: dict[str, str]

Tags the ingest request applies to all data from this run; empty when Nominal didn't inject them.

dataset_rid property ¤

dataset_rid: str | None

RID of the dataset this run ingests into.

None when Nominal injected no value, empty included; see :attr:ingest_job_rid.

ingest_job_rid property ¤

ingest_job_rid: str | None

RID of the ingest job running this extractor.

None when Nominal injected no value, empty included -- an empty environment variable means the same thing as an absent one here, as it does everywhere else in this contract.

inputs property ¤

inputs: list[Path]

All input files Nominal mounted for this run.

Legacy authoring API: prefer a separate @input declaration per registered input. Access logs one migration warning per run; it contributes no registration metadata. Taken from the registered _NOMINAL_INPUTS metadata when present, in the order Nominal serializes them; otherwise discovered by listing the input mount, sorted by name.

job_timestamp_metadata property ¤

job_timestamp_metadata: TimestampMetadata | None

The job-level timestamp metadata this run's outputs default to.

This is the metadata the pipeline resolved for the whole job (the ingest request's override when given, else the image's registered default) -- the value a manifest output falls back to when it declares no per-output timestamp metadata of its own. None when Nominal didn't inject it.

output_dir instance-attribute ¤

output_dir: Path

add_avro_stream ¤

add_avro_stream(
    path: str | PathLike[str],
    *,
    channel_prefix: str | None = None,
    units: Mapping[str, str] | None = None,
    timestamp_type: _AnyNumericTimestampType | None = None,
) -> Path

Declare an avro-stream file you wrote (.avro or .avro.gz).

Avro records carry their own channel, values, and tags, so this takes no tag columns and no timestamp column. The schema fixes which field holds the timestamps; timestamp_type says how to read the numbers in it, overriding the job-level timestamp metadata for this output. Only numeric types work here: absolute epochs (:class:ts.Epoch) or offsets from a starting time (:class:ts.Relative).

Omit timestamp_type to inherit the job-level metadata. That is only correct when the job-level type is numeric too, since avro timestamps are integers and a string format cannot read them.

channel_prefix is prepended to every channel from this file. units maps channel names to unit symbols for this output. The mapping is copied when declared, and symbols are passed through unchanged.

add_journal_json ¤

add_journal_json(path: str | PathLike[str]) -> Path

add_journal_json(
    path: str | PathLike[str],
    *,
    timestamp_column: str,
    timestamp_type: _AnyNumericTimestampType,
) -> Path

add_journal_json(
    path: str | PathLike[str],
    *,
    timestamp_column: str | None = None,
    timestamp_type: _AnyNumericTimestampType | None = None,
) -> Path

Declare a journal JSONL file you wrote (.jsonl or .jsonl.gz); it is ingested as logs.

Each line must carry a MESSAGE field and a timestamp field; lines missing either are skipped. Every other top-level field becomes a log arg, with its value converted to a string. timestamp_column/timestamp_type (provided together) name the top-level JSON field holding each line's timestamp, overriding the job-level metadata; the same numeric-only restriction as :meth:add_tabular applies.

Log samples carry no tags and every log point lands on one channel, so this takes neither tag columns nor a channel prefix -- the ingest pipeline ignores both for log outputs. The overloads make passing only one of the pair a type error.

add_tabular ¤

add_tabular(
    path: str | PathLike[str],
    *,
    tag_columns: Mapping[str, str] | None = ...,
    channel_prefix: str | None = ...,
    units: Mapping[str, str] | None = ...,
) -> Path

add_tabular(
    path: str | PathLike[str],
    *,
    tag_columns: Mapping[str, str] | None = ...,
    channel_prefix: str | None = ...,
    units: Mapping[str, str] | None = ...,
    timestamp_column: str,
    timestamp_type: _AnyNumericTimestampType,
) -> Path

add_tabular(
    path: str | PathLike[str],
    *,
    tag_columns: Mapping[str, str] | None = None,
    channel_prefix: str | None = None,
    units: Mapping[str, str] | None = None,
    timestamp_column: str | None = None,
    timestamp_type: _AnyNumericTimestampType | None = None,
) -> Path

Declare a CSV or Parquet file you wrote; its columns become channels.

tag_columns maps tag names to the columns carrying their values. channel_prefix is prepended to every channel from this file. timestamp_column/timestamp_type (provided together) override the job-level timestamp metadata for this output, so each file can carry its own timestamp column; only numeric types work here -- absolute epochs (:class:ts.Epoch) or offsets from a starting time (:class:ts.Relative). Outputs needing ISO 8601 or custom formats omit the pair and inherit the job-level metadata, which supports the full range. The overloads make passing only one of the pair a type error.

units maps channel names to unit symbols for this output. The mapping is copied when declared, and symbols are passed through unchanged.

add_video ¤

add_video(
    path: str | PathLike[str],
    *,
    channel: str,
    start: _InferrableTimestampType,
    ending_timestamp: _InferrableTimestampType | None = ...,
    true_frame_rate: float | None = ...,
    scale_factor: float | None = ...,
) -> Path

add_video(
    path: str | PathLike[str],
    *,
    channel: str,
    frame_timestamps: Sequence[IntegralNanosecondsUTC],
) -> Path

add_video(
    path: str | PathLike[str],
    *,
    channel: str,
    start: _InferrableTimestampType | None = None,
    frame_timestamps: Sequence[IntegralNanosecondsUTC] | None = None,
    ending_timestamp: _InferrableTimestampType | None = None,
    true_frame_rate: float | None = None,
    scale_factor: float | None = None,
) -> Path

Declare a video you wrote to the output directory; it becomes one video manifest entry.

The video is ingested as channel on the dataset this run writes to, alongside any telemetry outputs. Exactly one of start or frame_timestamps is required, and they fix the video's absolute time two different ways:

  • start -- the video's absolute starting timestamp; each frame's time comes from the video's own encoded presentation timestamps, offset from that start. Pass one of ending_timestamp, true_frame_rate, or scale_factor (at most one) when the media plays at a different rate than the camera recorded at.
  • frame_timestamps -- one absolute nanosecond timestamp per frame, when precise per-frame metadata is available. Unlike every other declaration method, this one writes: the runtime serializes the timestamps to a sidecar beside the video (cam.mp4 gets cam.mp4.timestamps.json) and declares it for you, so authors never have to reproduce that file's format.

The video file itself must already exist under output_dir and carry a supported video extension.

Rejections come back as :class:ExtractorError when they are about the extractor's own contract -- a reserved file name, a file outside the output directory -- and as the argument errors the rest of the client raises (all :class:ValueError subclasses) when the arguments themselves are malformed, which is what :meth:Dataset.add_video does too.

NOTE: video outputs require a recent version of the Nominal platform. An older ingest pipeline ignores them, and rejects a manifest whose only outputs are videos.

build_manifest ¤

build_manifest() -> dict[str, Any]

Build the manifest document from the declared outputs, exactly as it is written to disk.

get_param ¤

get_param(name: str, default: None = None) -> str | None

get_param(name: str, default: str) -> str

get_param(name: str, default: str | None = None) -> str | None

Read an optional parameter from the environment, or default when unset.

Legacy authoring API: prefer @parameter("parts", type=int, default=2) and a parts callback argument. This method logs one migration warning per run and contributes no registration metadata. Its returned values remain strings (or None). Name resolution matches :meth:param. With registered contract metadata present, an unregistered name raises :class:ExtractorError.

input ¤

input(name: str | None = None) -> Path

Resolve an input file.

Legacy authoring API: prefer @input("source", envvar="SOURCE") and a source callback argument. This method logs one migration warning per run and contributes no registration metadata.

With name -- the input's registered display name or its environment variable -- returns that input's path. Without it, returns the sole mounted input file, raising if there is not exactly one.

param ¤

param(name: str) -> str

Read a required parameter from the environment.

Legacy authoring API: prefer @parameter("parts", envvar="PARTS", type=int) and a parts callback argument. This method logs one migration warning per run and contributes no registration metadata.

name -- the parameter's registered display name or its environment variable -- is resolved against _NOMINAL_PARAMETERS when Nominal injected it; otherwise it is treated directly as the environment variable. Raises :class:ExtractorError when the parameter is not set. Values returned by this legacy method remain strings. With registered contract metadata present, an unregistered name raises :class:ExtractorError.

SingleFileExtractorContext dataclass ¤

SingleFileExtractorContext(
    output_dir: Path,
    _env: Mapping[str, str],
    _input_dir: Path,
    _input_specs: list[_InputSpec] | None = None,
    _param_specs: list[_ParamSpec] | None = None,
    _declared: set[str] = set(),
    _output_relative: str | None = None,
)

Bases: ExtractorContext

Context for :func:single_file_extractor functions: declare the one output via :meth:set_output.

additional_tags property ¤

additional_tags: dict[str, str]

Tags the ingest request applies to all data from this run; empty when Nominal didn't inject them.

dataset_rid property ¤

dataset_rid: str | None

RID of the dataset this run ingests into.

None when Nominal injected no value, empty included; see :attr:ingest_job_rid.

ingest_job_rid property ¤

ingest_job_rid: str | None

RID of the ingest job running this extractor.

None when Nominal injected no value, empty included -- an empty environment variable means the same thing as an absent one here, as it does everywhere else in this contract.

inputs property ¤

inputs: list[Path]

All input files Nominal mounted for this run.

Legacy authoring API: prefer a separate @input declaration per registered input. Access logs one migration warning per run; it contributes no registration metadata. Taken from the registered _NOMINAL_INPUTS metadata when present, in the order Nominal serializes them; otherwise discovered by listing the input mount, sorted by name.

job_timestamp_metadata property ¤

job_timestamp_metadata: TimestampMetadata | None

The job-level timestamp metadata this run's outputs default to.

This is the metadata the pipeline resolved for the whole job (the ingest request's override when given, else the image's registered default) -- the value a manifest output falls back to when it declares no per-output timestamp metadata of its own. None when Nominal didn't inject it.

output_dir instance-attribute ¤

output_dir: Path

get_param ¤

get_param(name: str, default: None = None) -> str | None

get_param(name: str, default: str) -> str

get_param(name: str, default: str | None = None) -> str | None

Read an optional parameter from the environment, or default when unset.

Legacy authoring API: prefer @parameter("parts", type=int, default=2) and a parts callback argument. This method logs one migration warning per run and contributes no registration metadata. Its returned values remain strings (or None). Name resolution matches :meth:param. With registered contract metadata present, an unregistered name raises :class:ExtractorError.

input ¤

input(name: str | None = None) -> Path

Resolve an input file.

Legacy authoring API: prefer @input("source", envvar="SOURCE") and a source callback argument. This method logs one migration warning per run and contributes no registration metadata.

With name -- the input's registered display name or its environment variable -- returns that input's path. Without it, returns the sole mounted input file, raising if there is not exactly one.

param ¤

param(name: str) -> str

Read a required parameter from the environment.

Legacy authoring API: prefer @parameter("parts", envvar="PARTS", type=int) and a parts callback argument. This method logs one migration warning per run and contributes no registration metadata.

name -- the parameter's registered display name or its environment variable -- is resolved against _NOMINAL_PARAMETERS when Nominal injected it; otherwise it is treated directly as the environment variable. Raises :class:ExtractorError when the parameter is not set. Values returned by this legacy method remain strings. With registered contract metadata present, an unregistered name raises :class:ExtractorError.

set_output ¤

set_output(path: str | PathLike[str]) -> Path

Declare the single file you wrote to the output directory.

Records the file (it must already exist under output_dir); it does not write anything itself. A single-file extractor produces exactly one output, so a second call raises. Use :func:manifest_extractor (with the image registered under the MANIFEST output format) to emit multiple files.

Parameter types¤

types ¤

Immutable parameter converters with safe, author-controlled diagnostics.

BadParameter ¤

Bases: ValueError

Reject a parameter with a message safe to include in extractor diagnostics.

The message is displayed to the user. Do not include raw parameter values or secrets.

Choice dataclass ¤

Choice(choices: Iterable[str])

Accept an exact, case-sensitive member of a nonempty collection of unique strings.

choices instance-attribute ¤

choices: tuple[str, ...]

FloatRange dataclass ¤

FloatRange(min: float | None = None, max: float | None = None)

Convert a finite number with optional inclusive minimum and maximum bounds.

max class-attribute instance-attribute ¤

max: float | None = None

min class-attribute instance-attribute ¤

min: float | None = None

IntRange dataclass ¤

IntRange(min: int | None = None, max: int | None = None)

Convert an integer with optional inclusive minimum and maximum bounds.

max class-attribute instance-attribute ¤

max: int | None = None

min class-attribute instance-attribute ¤

min: int | None = None

Execution and registration¤

runner ¤

The container entrypoint that drives an extractor function from the environment.

logger module-attribute ¤

logger = getLogger(__name__)

Extractor ¤

Extractor(
    _fn: Callable[..., None],
    _context_cls: type[_CtxT],
    _output_format: FileOutputFormat | None = None,
    _timestamp_column: str | None = None,
    _timestamp_type: _AnyTimestampType | None = None,
)

Bases: Generic[_CtxT]

A containerized-extractor entrypoint produced by :func:single_file_extractor or :func:manifest_extractor.

Declare callback inputs and parameters with @input and @parameter beneath the outer extractor decorator. They bind as keyword arguments after the output context. Use :meth:registration_kwargs to export the same contract for image registration. Call :meth:run as the container's entrypoint to drive it from the environment. In tests, drive it with :meth:run (env=..., exit=False) rather than constructing a context by hand. Carries the wrapped function's metadata (__name__, __doc__, ...) like any well-behaved decorator.

catalog_manifest ¤

catalog_manifest(
    *, id: str, version: str, display_name: str, description: str
) -> dict[str, Any]

Export a first-party catalog manifest from declarations and release identity.

Returns fresh JSON/YAML-serializable data for extractor.yaml, including error fallbacks. Does not execute extraction, read the environment, build an image, write a file, or publish a release. The publisher owns versioning and artifact identity.

Catalog export requires at least one input with suffix filters, uppercase environment variables, epoch or ISO 8601 timestamp defaults, and a message on each error declaration. Error codes must be non-reserved catalog identifiers. Identical error fallbacks sharing an exit code are combined; conflicting fallbacks are rejected. Input display metadata, converters and defaults have no catalog schema fields. Incomplete or unrepresentable metadata raises ValueError before publication.

registration_kwargs ¤

registration_kwargs() -> _RegistrationKwargs

Build image registration arguments exclusively from decorator declarations.

Pass the result as **entrypoint.registration_kwargs() to ContainerizedExtractor.register_image(tarball, tag=..., ...). Returns fresh lists of FileExtractionInput, FileExtractionParameter, and ExitCodeMapping objects, the output format, and the two default timestamp settings. List order follows the decorators from top to bottom.

Declare default_timestamp_column and default_timestamp_type on the outer decorator. Manifest output format is automatic; single-file extractors also need an explicit output_format. Missing required registration settings raise ValueError. Timestamp settings describe image defaults; they do not populate local job metadata.

Does not execute the callback or converters, inspect its body, read the environment, upload an image, or activate it. Parameter types/defaults remain runtime-only. Error mappings require a static message and non-reserved code; identical exit-code fallbacks are combined, while conflicting fallbacks raise ValueError. Legacy lookups add no metadata: no argument declarations yields empty inputs/parameters. Keep complete manual registration metadata for partially migrated extractors with additional undeclared dependencies.

run ¤

run(
    *,
    env: Mapping[str, str] | None = None,
    exit: bool = True,
    termination_log_path: str | Path = _DEFAULT_TERMINATION_LOG_PATH,
) -> _CtxT

Run the extractor against the environment and finalize its outputs.

Intended as the container entrypoint (if __name__ == "__main__": my_extractor.run()). Builds a context, resolves all declared arguments, invokes the callback, and finalizes its outputs. env replaces the process environment when supplied; required values must be present there, while optional parameters use their declared defaults. Output directories must already exist. Binding failures prevent callback execution and use the same error handling as startup, extraction, and finalization failures.

On success returns the context. With exit=True (the default), mapped exceptions write bounded structured JSON to the termination log and stderr, then print the full traceback to stderr before exiting with their mapped status. The closest mapped class in the exception's MRO wins. Other failures print a traceback and exit 1. Pass exit=False to re-raise the original exception without reporting -- useful in tests. termination_log_path is a trusted explicit output path (default /dev/termination-log), never read from the environment. Use a temporary path when testing mapped exits locally.