Skip to content

fix(dbt): write column marker to config.meta so dbt Fusion (v2) accepts the YAML - #1633

Open
FredrikBakken wants to merge 2 commits into
datacontract:mainfrom
FredrikBakken:fix/dbt-fusion-column-config-meta
Open

FredrikBakken wants to merge 2 commits into
datacontract:mainfrom
FredrikBakken:fix/dbt-fusion-column-config-meta

Conversation

@FredrikBakken

Copy link
Copy Markdown
Contributor

dbt sync stamps every provenance block under config.meta.datacontract_cli - model entries, test entries and version bullets - except the per-column "generated" marker, which was written to a top-level meta: on the column.

dbt-core (v1) accepts that (deprecated), but dbt Fusion (v2) validates model YAML against a closed schema and rejects it outright:

Property meta is not allowed. (yaml-schema: dbt YAML files 513)

The marker is only stamped on columns the CLI creates, so the failure was invisible when the model YAML already existed and only surfaced when dbt sync generated the file from scratch - the two merge routines had silently diverged.

config.meta validates under both engines, so write there instead:

  • _mark_managed_column writes config.meta.datacontract_cli.generated.
  • _is_managed_column reads config.meta and falls back to the legacy top-level location, so projects synced by an older CLI stay recognized (otherwise cleanup/--prune would orphan the columns it used to own).
  • _migrate_legacy_column_meta relocates legacy blocks on the next sync, wired into both _merge_model_entry and _merge_versioned_model_entry. User keys sitting alongside ours under meta: are left untouched.

Verified end to end against a dbt project: dbt-fusion dbt parse went from 10 errors to clean, dbt-core 1.12.4 dbt parse stays clean, and dbt ls --select config.meta.datacontract_cli.include_in_tests:true still matches every managed test under v1 (dbt-core back-populates columns.*.meta from config.meta, so nothing downstream regresses).

  • Tests pass (uv run pytest)
  • Code formatted (uv run ruff check --fix && uv run ruff format)
  • Docs updated (if relevant)
  • CHANGELOG.md entry added

…ts the YAML

`dbt sync` stamps every provenance block under `config.meta.datacontract_cli` —
model entries, test entries and version bullets — except the per-column
"generated" marker, which was written to a top-level `meta:` on the column.

dbt-core (v1) accepts that (deprecated), but dbt Fusion (v2) validates model YAML
against a closed schema and rejects it outright:

    Property meta is not allowed. (yaml-schema: dbt YAML files 513)

The marker is only stamped on columns the CLI creates, so the failure was
invisible when the model YAML already existed and only surfaced when `dbt sync`
generated the file from scratch — the two merge routines had silently diverged.

`config.meta` validates under both engines, so write there instead:

- `_mark_managed_column` writes `config.meta.datacontract_cli.generated`.
- `_is_managed_column` reads `config.meta` and falls back to the legacy
  top-level location, so projects synced by an older CLI stay recognized
  (otherwise cleanup/`--prune` would orphan the columns it used to own).
- `_migrate_legacy_column_meta` relocates legacy blocks on the next sync, wired
  into both `_merge_model_entry` and `_merge_versioned_model_entry`. User keys
  sitting alongside ours under `meta:` are left untouched.

Verified end to end against a dbt project: dbt-fusion `dbt parse` went from 10
errors to clean, dbt-core 1.12.4 `dbt parse` stays clean, and
`dbt ls --select config.meta.datacontract_cli.include_in_tests:true` still
matches every managed test under v1 (dbt-core back-populates `columns.*.meta`
from `config.meta`, so nothing downstream regresses).

Tests cover the fresh-generation path, the merge-into-existing path, an explicit
assertion that the two agree on marker location, the versioned path, legacy
migration, legacy-read tolerance, and that retirement of a dropped column still
works.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant