Component model¶

Everything a developer defines in Interloper is a component: assets, sources, destinations, connections, configs, jobs, hooks. Components share one base with two layers. Serializable is anything that is "a class plus its configuration"; Component adds kind, identity and relations and makes the object a catalog citizen. This page is for people writing new component classes or new kinds. The reference comes first; Writing a kind at the end walks through it in the order you would do it.

Serializable¶

A pydantic model with:

• key: class-level, snake_cased from the class name unless declared.
• Strict construction: unknown keyword arguments raise TypeError.
• Specs: to_spec(), from_spec(), from_spec_file(), classpath(), resolve_path(). See Specs and serialization. Several roots written or read as one document share a SerializationContext, which holds the one rule for inlining, referencing and dropping a bound target; every public entry point starts its own, so only a multi-root container such as the DAG ever passes one.
• config_schema(): the JSON Schema of user-facing fields, with framework fields and the class's internal_fields stripped.
• build_class(body, **settings): the one construction behind every decorator. A decorator reads its target into a ClassBody (a class as written, or a function read by the kind's body_from_function) and gets one new class back, with every setting in its namespace: public ClassVars as values, field defaults as fields, relations in the relations dict. A class that already extends the kind becomes the base of the new one and keeps its key and kind. Decorators build classes; they never modify one.

Runners, normalizers and schemas are Serializable without being components.

Component¶

On top of Serializable:

definition() returns a ComponentDefinition (kind, key, path, name, icon, description, tags, config_schema, state_schema, relations); it is the one place a class is described, and every FetchField provider is checked against the relations there. A kind whose definition carries more builds its own model from the base one, SourceDefinition(**dict(super().definition()), assets=...): SourceDefinition adds assets, AssetDefinition adds asset_schema and partitioning, ResourceDefinition adds provider, checkable and renewable. A destination describes nothing beyond the base, so Destination.definition() returns a plain ComponentDefinition.

Kinds and anchors¶

Each kind has an anchor: the base-most class declaring it (Connection for every connection). Component.anchor() resolves it, and the KINDS registry maps kind names to anchors. Anchors are declared through the interloper.kinds entry-point group; the core declares source, asset, destination, resource, connection, config, job and hook. A new kind is a new anchor class and one entry-point line:

class Report(il.Component):
    """A rendered document built from assets."""

    inputs: list[il.Asset] = il.Relation("asset", many=True, optional=True)

[project.entry-points."interloper.kinds"]
report = "my_package.report:Report"

A catalog containing a component of an unregistered kind raises ConfigError.

Relations¶

A relation is one declared link from a component to the components that may fill it. One class, il.Relation, declares every link on every kind: a connection an asset injects, the destinations a source writes to, a job's targets, an upstream asset.

Two attributes are derived, not declared: name is stamped from the attribute the relation is declared under, and target is the class the il.Relation(cls) shorthand was written with, which is what a fallback is built from. Neither is a constructor argument.

il.Relation(PostgresConnection) is shorthand for il.Relation(kind="connection", key="postgres_connection") with the class kept as target. accepts(kind, identity, owner=...) is the one place a candidate is judged against a declaration, and ComponentIdentity.satisfies the one place a declared key is compared to a concrete component. An asset's key is source-local, so a bare key is scoped to the owner's source; every other kind is keyed globally by its catalog key.

Three declaration forms, in increasing precedence, all merged at class creation:

class Widget(il.Component):
    connection: PostgresConnection                                   # an annotation naming a class
    config: WidgetConfig = il.Relation(WidgetConfig, optional=True)  # a typed Relation attribute
    relations = {"cache": il.Relation(Cache)}                        # what the decorators emit

The map merges over every base's, so a subclass entry replaces the inherited one of the same name and nothing an ancestor declared is lost. Each entry is copied, stamped with its name and installed under it: a Relation is its own descriptor, so Widget.connection is the declaration and widget.connection what is bound to it. An annotated relation is dropped from the class's annotations before pydantic collects its fields, so a relation is never also a field. The two collectors read the annotations against different namespaces (the declaring module here, the full defining scope in pydantic), so an annotation naming a component class that ends up a plain field, and one that resolves nowhere at all, are both TypeError at class definition rather than a component silently missing a relation.

Operations, all of them on Component:

Inherited settings: a field declared with inherit=True on a field factory (or json_schema_extra={"x-inherit": True} on a plain Field) stores only the component's own value, None while unset, and reads through its owner on every read. The owner decides what its child reads in one method, inherited(child, name, own): the base returns own when set, else the owner's own value under that name. Source overrides it to fill a dataset's {field} template and to disable its assets while it is disabled. Specs and configs carry stored values only.

Asset kinds vary through three hooks, and override nothing else of the pipeline:

A relation name is also a constructor keyword and an assignable attribute; assignment goes through Relation.__set__, which shares bind's single write path: the replacement is checked before the existing binding is touched (so a rejected assignment leaves the previous one exactly as it was), duplicates collapse, and clearing a non-optional relation raises ConfigError.

definition().relations exports each relation for the catalog and the UI: kind, key, many, optional, on_delete.

Discriminator¶

One configuration field may carry discriminator=True. discriminator and instance_name() expose its value, discriminator_field() its name for callers holding a config without an instance; sources use it for per-instance table names. Two marked fields raise TypeError.

Writing a kind¶

A kind is an anchor class plus one entry-point line. Everything above applies to it; this is the order in which it comes up.

1. Subclass the anchor you extend, or il.Component for a new kind. kind derives from the class name for a direct child of Component; a subclass of an existing kind inherits its. Give the class a docstring: it becomes the description users see.

   class Report(il.Component):
       """A rendered document built from assets."""
   

2. Declare fields for what the user configures. Ordinary pydantic fields, with il.InputField, il.SelectField, il.SecretField and friends where the UI needs to know how to render them. Fields the framework fills belong in internal_fields. One field may carry discriminator=True; it names instances.

3. Declare relations for what fills the component. An annotation naming a component class is the shorthand; an il.Relation value is where anything else is said (a key filter, many, optional, on_delete). The relation name is the constructor keyword, the attribute, and what persistence and the UI call the link.

   class Report(il.Component):
       """A rendered document built from assets."""
   
       title: str = il.InputField()
       inputs: list[il.Asset] = il.Relation("asset", many=True, optional=True)
       storage: il.Destination
   

4. Register the anchor. One line in pyproject.toml; the catalog raises ConfigError for a component whose kind it does not know.

   [project.entry-points."interloper.kinds"]
   report = "my_package.report:Report"
   

5. Check definition(). It is what the catalog, the API and the app read: config_schema from the fields, relations from the declarations, state_schema from state_model if the kind has machine-owned state. If the definition says what you meant, the kind is done.

Behaviour comes last and is the kind's own: a Workload exposes operations(), an Operation exposes run(). See Operations.

Writing a decorator¶

A kind that wants @il.report(...) sugar makes one Decorator for it, and documents it with an attribute docstring (the reference renders it):

from interloper.component import Decorator

report = Decorator(Report)
"""Create a Report subclass from a decorated class."""

The decorator works bare (@report) or with settings (@report(name="Weekly"), which returns a decorator carrying them). What it accepts is what the kind declares: a public ClassVar becomes a class attribute, a pydantic field a default, relations= declares relations (an il.Relation, a component class, or a list of classes narrowing a relation the kind declares), and anything else is a TypeError naming what is accepted. Nothing is hand-listed, so a new ClassVar on Report is a new decorator option with no other change. A kind that decorates functions, as Asset and Source do, overrides body_from_function to say what a function becomes.

Definitions in the catalog¶

Every class reachable through the interloper.components entry point becomes a catalog entry through its definition(). Nothing is inferred or registered at import time: installation is registration, and the catalog contains exactly what was declared. See Catalog and Entry points.