fix(dbt): write column marker to config.meta so dbt Fusion (v2) accepts the YAML - #1633
Open
FredrikBakken wants to merge 2 commits into
Open
FredrikBakken wants to merge 2 commits into
FredrikBakken wants to merge 2 commits into
Conversation
…ts the YAML
`dbt sync` stamps every provenance block under `config.meta.datacontract_cli` —
model entries, test entries and version bullets — except the per-column
"generated" marker, which was written to a top-level `meta:` on the column.
dbt-core (v1) accepts that (deprecated), but dbt Fusion (v2) validates model YAML
against a closed schema and rejects it outright:
Property meta is not allowed. (yaml-schema: dbt YAML files 513)
The marker is only stamped on columns the CLI creates, so the failure was
invisible when the model YAML already existed and only surfaced when `dbt sync`
generated the file from scratch — the two merge routines had silently diverged.
`config.meta` validates under both engines, so write there instead:
- `_mark_managed_column` writes `config.meta.datacontract_cli.generated`.
- `_is_managed_column` reads `config.meta` and falls back to the legacy
top-level location, so projects synced by an older CLI stay recognized
(otherwise cleanup/`--prune` would orphan the columns it used to own).
- `_migrate_legacy_column_meta` relocates legacy blocks on the next sync, wired
into both `_merge_model_entry` and `_merge_versioned_model_entry`. User keys
sitting alongside ours under `meta:` are left untouched.
Verified end to end against a dbt project: dbt-fusion `dbt parse` went from 10
errors to clean, dbt-core 1.12.4 `dbt parse` stays clean, and
`dbt ls --select config.meta.datacontract_cli.include_in_tests:true` still
matches every managed test under v1 (dbt-core back-populates `columns.*.meta`
from `config.meta`, so nothing downstream regresses).
Tests cover the fresh-generation path, the merge-into-existing path, an explicit
assertion that the two agree on marker location, the versioned path, legacy
migration, legacy-read tolerance, and that retirement of a dropped column still
works.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dbt syncstamps every provenance block underconfig.meta.datacontract_cli- model entries, test entries and version bullets - except the per-column "generated" marker, which was written to a top-levelmeta:on the column.dbt-core (v1) accepts that (deprecated), but dbt Fusion (v2) validates model YAML against a closed schema and rejects it outright:
The marker is only stamped on columns the CLI creates, so the failure was invisible when the model YAML already existed and only surfaced when
dbt syncgenerated the file from scratch - the two merge routines had silently diverged.config.metavalidates under both engines, so write there instead:_mark_managed_columnwritesconfig.meta.datacontract_cli.generated._is_managed_columnreadsconfig.metaand falls back to the legacy top-level location, so projects synced by an older CLI stay recognized (otherwise cleanup/--prunewould orphan the columns it used to own)._migrate_legacy_column_metarelocates legacy blocks on the next sync, wired into both_merge_model_entryand_merge_versioned_model_entry. User keys sitting alongside ours undermeta:are left untouched.Verified end to end against a dbt project: dbt-fusion
dbt parsewent from 10 errors to clean, dbt-core 1.12.4dbt parsestays clean, anddbt ls --select config.meta.datacontract_cli.include_in_tests:truestill matches every managed test under v1 (dbt-core back-populatescolumns.*.metafromconfig.meta, so nothing downstream regresses).uv run pytest)uv run ruff check --fix && uv run ruff format)