Extract dbt {% macro %} definitions from raw .sql models

[alexisperinger-ux]

What problem does this solve?

dbt .sql models are Jinja-templated ({{ ref() }}, {{ source() }}, {% macro %}), which the SQL grammar cannot parse, and dbt's manifest only exists after dbt compile. So in a raw checkout, models and their dependencies are invisible: a model file is only a generic Module, and a referenced model like stg_users is not even a node, so ref() lineage cannot form.

Public test bed: dbt-labs/jaffle_shop (index the raw models/ without compiling).

Proposed solution

Run an additive tree-sitter-jinja2 pass on dbt-templated .sql (files containing {{ / {%):

• {{ ref('m') }} / {{ source('s','t') }} become USAGE lineage edges.
• A dbt model (a .sql file with no macro defs) becomes a Model node keyed by file stem, so cross-file {{ ref('that_model') }} resolves into model-to-model lineage.
• {% macro name(...) %} becomes a Macro.

Zero schema change (freeform labels + existing USAGE edge). Model is emitted only on the .sql path, so a plain .jinja / .j2 template is not treated as a model. These source-level Model nodes coexist with the manifest path (#576) without conflict.

Caveat: the vendored tree-sitter-jinja2 grammar models only {{ }} expressions (so ref() / source() are parsed from the AST). It has no rule for {% %} statements, so {% macro %} names are recovered with a small text scan until a statement-aware grammar is vendored.

Alternatives considered

• Reusing the embedded-language (<script>-in-HTML) re-parse path: rejected because dbt Jinja is interleaved throughout the file, not a delimited sub-region; a full-file second parse is the right shape.
• Extending the grammar to parse {% %} statements: larger and touches grammar vendoring; deferred.

Confirmations

• I searched existing issues and this is not a duplicate.

[DeusData]

Thanks — the raw-checkout gap (Jinja-templated models being invisible before dbt compile) is real, and the additive tree-sitter-jinja2 pass emitting ref() / source() as USAGE edges and file-stem Model nodes is a clean approach that coexists with the manifest path (#576).

Implemented in your PR #584 — continuing there. Labeling enhancement / parsing/quality.

[DeusData]

The lineage half of this is implemented in #1756 (co-authored with @alexisperinger-ux): {{ ref('model') }} and {{ source('group','table') }} are read out of raw .sql models during ordinary indexing, producing a Model node per file and model -USAGE-> relation lineage edges. Model is a relation label alongside the Table/View nodes added in #1749, so dbt lineage and plain-SQL lineage form a single graph — a model's source() resolves onto a table declared in a DDL migration in the same repo.

Leaving this issue open, narrowed to macros, because that half is genuinely not done and the reason is worth recording:

The vendored tree-sitter-jinja2 grammar has node types for {{ ... }} expressions only — jinja_expression, fn_call, fn_name, lit_string, argument_list — and none at all for {% ... %} statements. There is no macro node, no endmacro, no if/for. So {% macro my_macro(x) %} cannot be recovered through the parser; it needs a hand-written scanner.

The approach in #584 was a plain text scan, which inherits two limits that matter for graph quality: it cannot distinguish a commented-out macro from a live one, and it has nothing to close against, so every macro is reported as a single-line span regardless of body length.

Doing it properly means one of:

1. a small purpose-built scanner that tracks Jinja comment state ({# ... #}) and pairs {% macro %} with {% endmacro %} for a correct end line; or
2. extending the vendored grammar to cover Jinja statements — larger, but it would also unlock {% if %}/{% for %} structure inside models.

Worth noting for whoever picks this up: dbt macros are callable, so they should almost certainly be Function-labelled rather than Macro. Macro nodes are deliberately excluded from the registry and the delta proxy set (they are never lookup targets — C macros dominate that label at millions of nodes), which would leave {{ my_macro() }} call sites unable to resolve to the definition.

[DeusData]

The lineage half landed on main — commit 43ecc09 (merge 34d18ae), via #1756.

{{ ref('model') }} and {{ source('group','table') }} in raw .sql models now produce a Model node per file plus model -USAGE-> relation lineage edges, resolved during ordinary indexing. Because Model is a relation label alongside the Table/View nodes from #1749, a model's source() also resolves onto a table declared in a plain DDL migration in the same repo — dbt lineage and SQL lineage are one graph.

This issue stays open for the macro half only (see the previous comment for why the vendored jinja2 grammar can't reach {% macro %} and what a real fix would take).