Models#

Models are the fundamental building blocks of Interlace pipelines. A model is a transformation that produces one table, written either as a SQL file or as a Python function.

SQL Models#

Any .sql file under a model path (default: models/) is a model. The file path becomes the model name and the name’s dotted prefix becomes its schema:

Configuration lives in a leading block comment containing YAML under an interlace: key:

/* interlace:
  strategy: merge
  key: order_id
  columns: {order_id: BIGINT, customer_id: BIGINT, amount: DOUBLE}
  checks:
    - not_null: order_id
*/
SELECT order_id, customer_id, amount
FROM raw_orders

The header is optional — a bare SELECT is a valid model (materialised as a virtual snapshot with a full refresh). The rest of the file must be exactly one SQL statement.

Header Options#

Unknown keys are ignored, and only the first block comment in the file is parsed — keep the config block above any other /* ... */ comment.

Python Models#

Python models use the @model decorator and exchange Apache Arrow data:

from interlace import model
import pyarrow as pa

@model(depends_on=["orders", "customers"], strategy="merge", key="order_id")
def enriched_orders(orders, customers) -> pa.Table:
    o, c = orders.table(), customers.table()
    return o.join(c, keys="customer_id")

The decorator registers the model and returns the function unchanged, so it stays an ordinary, unit-testable Python function. The name defaults to the function name.

Decorator Options#

@model accepts the same options as the SQL header (materialise, strategy, key, dialect, engine, depends_on, interval, time_column, tags, owner, description, columns, schedule, checks), plus one Python-only option:

Python models are always virtual (an owned snapshot): view and ephemeral are SQL-only, and the terminal table/file planes need a SQL model (write one over the Python model’s output). incremental works with a key (the staged output is upserted window by window); without one it is refused, because a Python function has already run in full before the window could narrow it — use cursor with merge there.

Inputs: RelationHandle#

Each dependency named as a function parameter arrives as a RelationHandle — a single-pass, streaming view of the upstream table:

A handle can be consumed once — call .table() or .reader(), not both. Convert at the edges if you prefer another library: handle.table().to_pandas(), polars.from_arrow(handle.table()).

Outputs#

A Python model returns Arrow data:

Both def and async def are supported — sync functions run in a thread.

Reserved Parameters#

Two parameter names are reserved and injected by Interlace rather than mapped to dependencies:

• cursor — the max value of the declared cursor column from this model’s previous build (None on the first run). The classic incremental-extraction pattern:

@model(strategy="merge", key="id", cursor="updated_at")
def events(cursor):
    return fetch_rows(since=cursor)   # cursor is None on the first run

• this — a RelationHandle over this model’s previous materialisation (None on the first run), for anti-join and self-referential patterns.

Dynamic / Programmatic Models#

A model .py file is imported and its top-level code runs every time the project loads (discovery executes each module), and a model registers the instant its declaration runs. So an ordinary Python loop is the mechanism for generating many models from data — the same logic per tenant, region, or source, each with its own filter. There is no separate templating DSL; it’s just Python.

Per-tenant SQL models — register a ModelDef directly for each item in a list:

# models/per_tenant.py
from interlace.dsl.decorators import REGISTRY, ModelDef

def get_tenants():                      # any Python: a DB query, a file, an env var…
    return ["acme", "globex"]

for tenant in get_tenants():
    REGISTRY.register_model(ModelDef(
        name=f"orders_{tenant}",
        sql=f"SELECT order_id, amount FROM raw WHERE tenant_id = '{tenant}'",
        strategy="merge", key=("order_id",),
    ))

This produces one snapshot table and environment view per tenant (orders_acme, orders_globex, …), each with an independent fingerprint, plan/apply, checks, and incremental ledger — full isolation between tenants.

Per-tenant Python models — use a factory so each closure captures its own value (the one thing to get right):

from interlace import model
import pyarrow.compute as pc

def make(tenant):
    @model(name=f"orders_{tenant}", depends_on=("raw",), strategy="merge", key=("order_id",))
    def _orders(raw, tenant=tenant):          # bind tenant HERE, not via the loop variable
        t = raw.table()
        return t.filter(pc.equal(t["tenant_id"], tenant))
    return _orders

for t in get_tenants():
    make(t)

Things to get right:

• Names must be unique — register_model raises on a duplicate, so put the tenant in the name.
• Closure late-binding — the classic Python trap; bind the loop variable via a factory or a default argument (as above), or every generated function filters on the last value.
• depends_on for Python models — a function’s parameters must each be a declared dependency (SQL models auto-discover deps from their table references; Python models don’t).
• The generator runs on every command — get_tenants() is called each time interlace loads the project (plan, apply, models, serve). Keep it fast and deterministic; if it queries a database, every CLI call pays that cost. interlace serve compiles once at startup, so a tenant added while the daemon is running only appears after it re-compiles/restarts.
• Quote interpolated values — for a trusted internal list, string interpolation into SQL is fine; for untrusted input, quote via sqlglot or parameterise.

If instead you want a single model carrying a tenant column (no per-tenant tables), that’s just an ordinary model — but for the same logic applied per tenant with isolation, the loop above is the right shape.

Macros#

A macro is a named SQL expression, written once and called from any model. Definitions live in macros/*.sql (configurable with macro_paths), in the engine’s own CREATE MACRO form:

-- macros/money.sql
CREATE MACRO cents_to_dollars(amount) AS (amount / 100)::numeric(16, 2);

-- models/stg_orders.sql
SELECT order_id, cents_to_dollars(subtotal) AS subtotal FROM raw_orders

The call is expanded into the model’s AST while it compiles — before the fingerprint, before lineage, before transpilation. That ordering is the design, and it buys three things.

A macro edit rebuilds its callers. A model’s fingerprint is its canonical SQL, and the expansion is part of it, so changing cents_to_dollars re-plans every model that calls it and everything downstream. A macro registered in the warehouse instead — CREATE MACRO against the engine — would be invisible to the fingerprint: every caller’s SQL stays byte-identical, so nothing rebuilds and the tables quietly stop matching the definition.

One definition per dialect, not one per adapter. dbt writes default__cents_to_dollars, postgres__cents_to_dollars, bigquery__cents_to_dollars and a dispatcher, because Jinja renders text and the text has to differ. Expansion happens in dialect-agnostic AST, so the transpiler does it:

Postgres would do integer division on /, and BigQuery spells the type differently. Neither needs a second macro.

Lineage sees through it. Column lineage reads the AST, and by then there is no opaque function call in it.

Details worth knowing:

• Scalar expressions only. A table macro (AS TABLE SELECT ...) has no call site to expand into — that is a model.
• Macros may call macros, to a depth of 10; recursion is a compile error, not a hang.
• A macro body may reference a model, and that becomes a dependency of every model calling it, because expansion runs before dependency resolution.
• The macro does not exist in the warehouse. This is the real cost: someone querying the built tables by hand cannot call it. It is a build-time abstraction.

Column Contracts#

The columns option declares the model’s output contract, in either form:

columns: [order_id, customer_id, amount] # names only
columns: { order_id: BIGINT, amount: DOUBLE } # names + engine types

After every build, before the snapshot is recorded: a missing contracted column or a type mismatch (compared case-insensitively against the engine’s reported types) fails the build. Extra columns beyond the contract are allowed — contracts guarantee a floor, not a ceiling.

Fingerprints and Rebuild-Skip#

Every model gets a data fingerprint — a hash of its canonical SQL (or Python source), its strategy config, and the sorted fingerprints of its upstreams. Any change that could affect output changes the fingerprint, which changes the physical snapshot table name. This is how plan knows what to rebuild; it classifies each changed model:

• breaking — output data may differ → rebuild; downstream inherits breaking.
• additive — only new columns appeared → rebuild; downstream stays non-breaking.
• clean — output provably identical → not rebuilt; the new snapshot reuses the previous physical table and the environment view just repoints.

Column pruning extends clean to semantic upstream changes: if a change provably touched only certain columns and a downstream provably consumes none of them, the downstream is clean too. Both proofs are conservative — any ambiguity falls back to “rebuild”.

Two tools help you reason about this:

• interlace impact <model>.<column> — the column-level blast radius: every downstream column derived from that one, plus models that consume it wholesale (Python models or * projections).
• --select state:modified — target exactly the models whose fingerprint differs from what the target environment has promoted (add + for their descendants: state:modified+).

Next Steps#

• Dependencies — how models connect
• Strategies — how tables are updated
• Python models guide — streaming, cursors, patterns