Skip to content

Show each Dag only once in airflow dags list - #70934

Merged
henry3260 merged 1 commit into
apache:mainfrom
Eason09053360:fix-dags-list-duplicate-serialized-dags
Aug 12, 2026
Merged

Show each Dag only once in airflow dags list#70934
henry3260 merged 1 commit into
apache:mainfrom
Eason09053360:fix-dags-list-duplicate-serialized-dags

Conversation

@Eason09053360

Copy link
Copy Markdown
Contributor

Airflow 3 stores one serialized_dag row per Dag version, but airflow dags list
selected every row without filtering by version, so a Dag was printed once per
version it had.

Use the existing SerializedDagModel.read_all_dags() helper, which selects the
row with the highest version_number per dag_id.

Note that #60868 / #61077 fixed one source of spurious version growth, but
legitimate versions still accumulate whenever a Dag is edited, so the duplicate
rows remain reproducible.

closes: #60594

This supersedes #60637, which reached the same diagnosis but was closed by the
stale bot without review.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Airflow 3 stores one serialized_dag row per Dag version, so every Dag with
more than one version was listed once per version.
@Eason09053360
Eason09053360 marked this pull request as ready for review August 2, 2026 05:06
@Eason09053360
Eason09053360 force-pushed the fix-dags-list-duplicate-serialized-dags branch from 90f4247 to 0ec3d95 Compare August 2, 2026 17:27
@henry3260 henry3260 added the backport-to-v3-3-test Backport to v3-3-test label Aug 12, 2026

@henry3260 henry3260 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well done!

While reviewing this I noticed DBDagBag.iter_all_latest_version_dags()
(airflow-core/src/airflow/models/dagbag.py) has the same issue — the name says
latest-version, but the body is an unfiltered select(SerializedDagModel). Same
origin as this one (#53918 dropped the read_all_dags() call in both places).
Out of scope here. would you mind fixing it in a follow-up PR?

@henry3260
henry3260 merged commit b9ff1d7 into apache:main Aug 12, 2026
127 of 133 checks passed
@github-actions github-actions Bot added this to the Airflow 3.3.2 milestone Aug 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Hi maintainer, this PR was merged without a milestone set.
We've automatically set the milestone to Airflow 3.3.2 based on: backport label targeting v3-3-test
If this milestone is not correct, please update it to the appropriate milestone.

This comment was generated by Milestone Tag Assistant.

@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-3-test PR Link

github-actions Bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Aug 12, 2026
Airflow 3 stores one serialized_dag row per Dag version, so every Dag with
more than one version was listed once per version.
(cherry picked from commit b9ff1d7)

Co-authored-by: Y-C <easoneason0905@gmail.com>
Co-authored-by: Eason09053360 <185830721+Eason09053360@users.noreply.github.com>
aws-airflow-bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Aug 12, 2026
Airflow 3 stores one serialized_dag row per Dag version, so every Dag with
more than one version was listed once per version.
(cherry picked from commit b9ff1d7)

Co-authored-by: Y-C <easoneason0905@gmail.com>
Co-authored-by: Eason09053360 <185830721+Eason09053360@users.noreply.github.com>
henry3260 pushed a commit that referenced this pull request Aug 12, 2026
…1481)

Airflow 3 stores one serialized_dag row per Dag version, so every Dag with
more than one version was listed once per version.
(cherry picked from commit b9ff1d7)

Co-authored-by: Y-C <easoneason0905@gmail.com>
Co-authored-by: Eason09053360 <185830721+Eason09053360@users.noreply.github.com>
@Eason09053360

Copy link
Copy Markdown
Contributor Author

@henry3260 Thanks for reviewing !
Sure I will take a look on it and push a follow up pr :)

vatsrahul1001 pushed a commit that referenced this pull request Aug 13, 2026
…1481)

Airflow 3 stores one serialized_dag row per Dag version, so every Dag with
more than one version was listed once per version.
(cherry picked from commit b9ff1d7)

Co-authored-by: Y-C <easoneason0905@gmail.com>
Co-authored-by: Eason09053360 <185830721+Eason09053360@users.noreply.github.com>
dabla pushed a commit to dabla/airflow that referenced this pull request Aug 14, 2026
Airflow 3 stores one serialized_dag row per Dag version, so every Dag with
more than one version was listed once per version.

Co-authored-by: Eason09053360 <185830721+Eason09053360@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

airflow dags list shows multiple copies of the same dag

2 participants