Catalog & specs¶

The catalog is the set of component definitions a deployment ships; specs are the serialized form a component travels in.

Catalog ¶

Bases: pydantic.main.BaseModel

Catalog of all component definitions.

model_config ¶

model_config = {}

dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)

discover ¶

discover() -> Catalog

Load the declared universe from interloper.components entry points.

Cheap to call repeatedly: entry points cannot change at runtime, so the scan and the class imports behind it are cached for the process and only the mapping is rebuilt.

Returns:

Type	Description
`Catalog`	Catalog of every component declared by every installed package.

dump ¶

dump() -> dict[str, dict[str, Any]]

Serialize all definitions to JSON-compatible dicts.

Returns:

Type	Description
`dict[str, dict[str, Any]]`	Mapping from component key to serialized definition.

from_assets ¶

from_assets(
    sources_or_assets: Iterable[type[Component]],
) -> Catalog

Build a catalog enabling the given component classes.

Parameters:

Name	Type	Description	Default
`sources_or_assets`	`Iterable[type[Component]]`	The enabled component classes, typically sources and assets, but any component kind is accepted.	required

Returns:

Type	Description
`Catalog`	Catalog of the given components, their dependencies and the
`Catalog`	framework's own components.

from_paths ¶

from_paths(paths: list[str]) -> Catalog

Build a catalog enabling the components at paths.

Parameters:

Name	Type	Description	Default
`paths`	`list[str]`	Fully qualified import paths to the enabled component classes: a deployment's `catalog` setting, or a `to_paths()` list crossing a process boundary. Paths that fail to import are skipped with a warning.	required

Returns:

Type	Description
`Catalog`	Catalog of the listed components, their dependencies and the
`Catalog`	framework's own components.

from_settings ¶

from_settings() -> Catalog

Load the catalog a deployment enables.

AppSettings.catalog lists the enabled import paths; when it is empty, the catalog is the whole declared universe.

Returns:

Type	Description
`Catalog`	Catalog of all enabled component definitions.

get ¶

get(key: str) -> ComponentDefinition | None

Look up a component definition by key.

A bare key names a catalog entry. A source's assets are not entries: they are declared inside their source, so one is named by its qualified key (source.asset, the form Component.qualified_key produces) and resolves only through that source's declaration, to the concrete AssetDefinition carrying the composite import path, the partitioning and the relations.

Parameters:

Name	Type	Description	Default
`key`	`str`	The bare or qualified component key.	required

Returns:

Type	Description
`ComponentDefinition \| None`	The component definition, or `None` when the key does not
`ComponentDefinition \| None`	resolve, including a qualified key whose source does not resolve
`ComponentDefinition \| None`	or does not declare the asset.

to_paths ¶

to_paths() -> list[str]

Extract the import paths of all components in the catalog.

Useful for passing the catalog across process boundaries (e.g. to a Docker container via env var) without serializing the full definitions.

Returns:

Type	Description
`list[str]`	Sorted list of fully qualified import paths.

vocabulary ¶

vocabulary(kind: str, key: str) -> dict[str, Relation]

The relation vocabulary governing a persisted component row.

The class definition is authoritative; a concrete class may extend its anchor's vocabulary (TriggerHook adds targets). The kind's anchor is the fallback when the key does not resolve to a matching definition (drifted keys), keeping validation fail-closed on the kind's shared minimum.

Parameters:

Name	Type	Description	Default
`kind`	`str`	The row's component kind.	required
`key`	`str`	The row's bare or qualified catalog key; a source-owned asset's declaration carries its relations.	required

Returns:

Type	Description
`dict[str, Relation]`	The declared relations keyed by name.

Serializable ¶

Serializable(**data: Any)

Bases: pydantic.main.BaseModel

A class-identified, serializable configuration object.

Anything whose instances are "a class plus its configuration" extends Serializable: it round-trips through Spec (to_spec() / from_spec()), resolves back from its import path, exposes a JSON Schema of its user-configurable fields, and rejects unknown constructor kwargs loudly. Runners, schemas and normalizers are Serializable; catalog citizens extend Component, which adds kind, identity and relations on top.

Validate kwargs strictly against the model's fields.

Unknown kwargs are a loud error rather than pydantic's silent extra="ignore" drop: a misnamed field would otherwise vanish.

Parameters:

Name	Type	Description	Default
`**data`	`Any`	Field values, keyed by field name. Every key must name a field declared on the model.	required

Raises:

Type	Description
`TypeError`	If a kwarg matches no field.

internal_fields ¶

internal_fields = frozenset()

frozenset() -> empty frozenset object frozenset(iterable) -> frozenset object

Build an immutable unordered collection of unique elements.

key ¶

key = ''

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to 'strict'.

model_config ¶

model_config = {'arbitrary_types_allowed': True, 'ignored_types': (<class 'interloper.serializable.base.IgnoredDescriptor'>,)}

dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)

name ¶

name = ''

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to 'strict'.

body_from_function ¶

body_from_function(fn: FunctionType) -> ClassBody

Read a decorated function into a class body; a kind decorating functions overrides it.

Parameters:

Name	Type	Description	Default
`fn`	`FunctionType`	The decorated function.	required

Raises:

Type	Description
`TypeError`	Always: this kind decorates a class.

build_class ¶

build_class(body: ClassBody, **settings: Any) -> type[Self]

Build a new class from a decorated target's body and a decorator's settings.

The one construction every decorator goes through: decorators build classes, they never modify one. Everything goes into the namespace of one new class, so the metaclass sees field defaults as fields and __init_subclass__ collects relations and validates settings where they are written. A target already extending this class is the base of the new one, which keeps its key and kind (rebuild).

Parameters:

Name	Type	Description	Default
`body`	`ClassBody`	What the decorated target contributes.	required
`**settings`	`Any`	The decorator's settings (see `_sort_settings`).	required

Returns:

Type	Description
`type[Self]`	The new class.

classpath ¶

classpath() -> str

Fully qualified import path for this class.

Returns:

Type	Description
`str`	Dotted path like `"module.submodule.ClassName"`.

config_schema ¶

config_schema() -> dict[str, Any]

JSON Schema of the class's user-configurable fields.

Properties are listed most specific first: the class's own fields, then each ancestor's, closest first, each in declaration order. Pydantic lists inherited fields first, which would open every form with what a whole kind shares (a source's dataset, a job's retry) rather than with what defines the component; forms render in property order.

Returns:

Type	Description
`dict[str, Any]`	The stripped schema, or `{}` when no configurable field remains,
`dict[str, Any]`	as a copy the caller is free to adjust.

from_init ¶

from_init(
    init: dict[str, Any], context: SerializationContext
) -> Self

Build an instance from a spec's loaded init, as one step of reconstructing a document.

Parameters:

Name	Type	Description	Default
`init`	`dict[str, Any]`	Constructor keyword arguments, nested specs already built.	required
`context`	`SerializationContext`	The document being read; a plain serializable needs nothing from it.	required

Returns:

Type	Description
`Self`	The instance.

from_spec ¶

from_spec(
    spec: Spec | dict[str, Any],
    catalog: Catalog | None = None,
    *,
    resolve: Callable[[str], Component] | None = None,
) -> Self

Reconstruct an instance from a spec.

Called on a subclass, the reconstructed instance must be of that subclass: Source.from_spec(spec).

Parameters:

Name	Type	Description	Default
`spec`	`Spec \| dict[str, Any]`	The spec (or its dict payload) to reconstruct.	required
`catalog`	`Catalog \| None`	Catalog used to resolve `key` references. Defaults to the settings-configured catalog, built lazily.	`None`
`resolve`	`Callable[[str], Component] \| None`	Called with the id of a reference the spec itself does not carry; `None` makes such a reference an error.	`None`

Returns:

Type	Description
`Self`	The reconstructed instance.

Raises:

Type	Description
`TypeError`	If the spec reconstructs to an instance that is not of the receiving class.

from_spec_file ¶

from_spec_file(
    path: str | Path,
    catalog: Catalog | None = None,
    *,
    resolve: Callable[[str], Component] | None = None,
) -> Self

Reconstruct an instance from a spec file.

Loads a Spec document from YAML (with ${VAR} env interpolation) and reconstructs it, with the same subclass-scoped check as from_spec: invalid documents surface as SpecError, mismatched kinds as TypeError.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Filesystem path to the YAML spec document, as a string or `Path`.	required
`catalog`	`Catalog \| None`	Catalog used to resolve `key` references. Defaults to the settings-configured catalog, built lazily.	`None`
`resolve`	`Callable[[str], Component] \| None`	Called with the id of a reference the document itself does not carry; `None` makes such a reference an error.	`None`

Returns:

Type	Description
`Self`	The reconstructed instance.

path ¶

path() -> str

Fully qualified import path for this instance's class.

Returns:

Type	Description
`str`	The class's `classpath`.

resolve_path ¶

resolve_path(path: str) -> type[Self]

Resolve an import path to a class of this (sub)class.

Accepts dotted and composite paths (module.Class, module:Source.Asset). Called on a subclass, the resolved class must be of that subclass; anything else raises TypeError.

Parameters:

Name	Type	Description	Default
`path`	`str`	Dotted or composite import path to resolve.	required

Returns:

Type	Description
`type[Self]`	The resolved class.

to_spec ¶

to_spec() -> Spec

Serialize this instance to a reconstructible spec.

Returns:

Type	Description
`Spec`	A Spec capturing this instance's state.

Spec ¶

Bases: pydantic.main.BaseModel

Serialized representation of a Component instance.

A component is referenced by exactly one of two names, each keeping its own meaning: path is a fully qualified import path (what to_spec() emits), key is a catalog key, so hand-authored specs may reference components the way the catalog names them.

REFERENCE_KEY ¶

REFERENCE_KEY = 'ref'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to 'strict'.

model_config ¶

model_config = {}

dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)

all_from_file ¶

all_from_file(path: str | Path) -> list[Spec]

Load every YAML document of a file as a spec, interpolating ${VAR} placeholders.

${VAR} placeholders in any string value are replaced from the process environment at load time, so credentials never need to live in the file. Unresolved variables are a hard error, reported for the whole file at once.

Several documents are how a graph of roots travels in one file: they share one id space, so a {"ref": id} in one document may name a component another document carries.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Filesystem path to the YAML spec document(s), as a string or `Path`.	required

Returns:

Type	Description
`list[Spec]`	The validated specs, in document order.

Raises:

Type	Description
`SpecError`	If the file is missing, unparsable, holds a document that is not a mapping, references undefined environment variables, or holds an invalid spec.

dump_value ¶

dump_value(value: Any) -> Any

Serialize a component field value for a Spec init payload.

The wire format is uniform: anything with class identity is Serializable and serializes via its own spec; lists and dicts are walked; everything else must be a JSON-able scalar.

Parameters:

Name	Type	Description	Default
`value`	`Any`	The field value to serialize: a `Serializable`, a list or dict of values to walk, or a JSON-able scalar.	required

Returns:

Type	Description
`Any`	A JSON-able value understood by `Spec.reconstruct`.

from_file ¶

from_file(path: str | Path) -> Spec

Load the single spec a YAML file holds, interpolating ${VAR} placeholders.

Delegates to all_from_file, which documents the interpolation and the SpecError cases a malformed file raises.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Filesystem path to the YAML spec document, as a string or `Path`.	required

Returns:

Type	Description
`Spec`	The validated spec.

Raises:

Type	Description
`SpecError`	If the file holds anything other than exactly one document.

is_reference ¶

is_reference(value: Any) -> bool

Whether a loaded init value is a reference rather than a component.

Parameters:

Name	Type	Description	Default
`value`	`Any`	The value to inspect, as the document carries it.	required

Returns:

Type	Description
`bool`	True for a mapping whose only key is `ref`.

reconstruct ¶

reconstruct(
    catalog: Catalog | None = None,
    *,
    resolve: Callable[[str], Component] | None = None,
    context: SerializationContext | None = None,
) -> Serializable

Import the class and rebuild the instance, walking nested specs.

key references resolve through the catalog and must name components; path references import directly and accept any Serializable class (a normalizer nested in an asset's config, for example).

Reconstruction runs in two passes over one SerializationContext. This method is the first: every inline component is built and added to the context, and a {"ref": id} relation value is held back rather than constructed with, so a component is only ever built from the targets the document carries inline. The second pass, SerializationContext.bind, binds those references and validates what they complete. Without a context this call owns one and runs both passes; with one it contributes to a document the caller finishes.

Parameters:

Name	Type	Description	Default
`catalog`	`Catalog \| None`	Catalog used to resolve `key` references, passed down to nested specs. Defaults to the settings-configured catalog, built lazily when a key is first encountered.	`None`
`resolve`	`Callable[[str], Component] \| None`	Called with the id of a reference the document itself does not carry, to reach a component that lives outside it. `None` makes such a reference an error. Only read when this call starts the context; a given context carries its own.	`None`
`context`	`SerializationContext \| None`	The state shared by every spec of a multi-root document. `None` starts one for this spec alone and binds its references before returning.	`None`

Returns:

Type	Description
`Serializable`	The reconstructed instance.

reference ¶

reference(component_id: str) -> dict[str, str]

Build the reference value that stands in for an already-emitted component.

Parameters:

Name	Type	Description	Default
`component_id`	`str`	Id of the component the reference names.	required

Returns:

Type	Description
`dict[str, str]`	The reference mapping, `{"ref": id}`.