Skip to content

Allow upgrades with NUL in serialized Dags - #71442

Open
dhkim1920 wants to merge 1 commit into
apache:mainfrom
dhkim1920:agent/fix-serialized-dag-nul-migration
Open

Allow upgrades with NUL in serialized Dags#71442
dhkim1920 wants to merge 1 commit into
apache:mainfrom
dhkim1920:agent/fix-serialized-dag-nul-migration

Conversation

@dhkim1920

@dhkim1920 dhkim1920 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

PostgreSQL accepts U+0000 escapes in JSON, but rejects them when migration 0089 casts serialized_dag.data to JSONB. This blocks upgrades from Airflow 3.1 when a serialized Dag contains a NUL character.

This draft applies the sanitization approach established by #69064 to migration 0089:

  • protect escaped backslashes so JSON strings containing the literal escape spelling are preserved;
  • strip active U+0000 escapes before the JSONB cast;
  • add a PostgreSQL regression test covering both cases.

Decision needed

#69064 strips U+0000 from historical XCom and DagRun data because PostgreSQL JSONB cannot represent it. Applying the same policy to serialized_dag is semantically different: a delimiter such as "@@\0@@" becomes "@@@@", changing the Dag definition.

This draft asks whether lossy sanitization is acceptable for this migration or whether serialized Dag data needs a different policy. Migration-only sanitization may also not address a later reserialization of the same Dag into JSONB, so the final scope depends on that policy decision.

closes: #65379

Tests

  • breeze run --backend postgres pytest airflow-core/tests/unit/migrations/test_0089_change_serialized_dag_data_column_to_jsonb.py -xvs
  • prek run --from-ref main --stage pre-commit
  • prek run --from-ref main --stage manual --skip compile-ui-assets-dev --skip view-skill-eval
  • breeze testing core-tests --run-in-parallel

Was generative AI tooling used to co-author this PR?
  • Yes — Codex (GPT-5.6)

Generated-by: Codex (GPT-5.6) following the guidelines

PostgreSQL rejects U+0000 while converting serialized Dag JSON to JSONB, blocking upgrades from Airflow 3.1.
@boring-cyborg boring-cyborg Bot added the area:db-migrations PRs with DB migration label Aug 11, 2026
@dhkim1920
dhkim1920 marked this pull request as ready for review August 11, 2026 15:02
@dhkim1920
dhkim1920 marked this pull request as draft August 11, 2026 15:03
@dhkim1920
dhkim1920 marked this pull request as ready for review August 14, 2026 11:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:db-migrations PRs with DB migration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Airflow 3.2 unable to handle \0 in dag files

1 participant