Resolve VariableInterval deadlines safely at DagRun creation by seanghaeli · Pull Request #68917 · apache/airflow

seanghaeli · 2026-06-23T20:17:17Z

Why

This re-introduces the VariableInterval resolution portion of #66608.

A DeadlineAlert configured with a VariableInterval is resolved when the scheduler creates a DagRun, inside DAG._process_dagrun_deadline_alerts (which runs under the prohibit_commit guard). Two problems are fixed:

Resolution goes through the full secrets chain (env vars, configured secrets backends, then the metadata DB) via a dedicated _resolve_variable_interval helper, rather than Variable.get / begin_nested. Variable.get and a SAVEPOINT release both commit on the scheduler's session, tripping prohibit_commit (UNEXPECTED COMMIT) and silently dropping deadlines for every scheduled DagRun. The helper passes the scheduler session through to the metastore backend so the DB read does not commit, and reading via the secrets chain (not the variable table directly) means AIRFLOW_VAR_* env vars and secrets-backend-backed Variables resolve too.
Each deadline alert is isolated with a plain try/except (deliberately not begin_nested, which would commit a SAVEPOINT and trip the same guard). Creating a deadline is auxiliary to creating the DagRun; a single bad alert — a missing/invalid backing Variable, or an undecodable serialized blob — must never abort the DagRun and stop the DAG from scheduling.

VariableInterval.resolve is split into resolve + coerce_to_timedelta so the scheduler-side reader reuses the exact same validation (including the OverflowError -> ValueError translation) without going through Variable.get.

Tests

airflow-core/tests/unit/models/test_dagrun.py: VariableInterval resolves from a real Variable row and from an AIRFLOW_VAR_* env var; a missing Variable and an undecodable alert are isolated (DagRun still created, no Deadline row, error logged).
task-sdk/tests/task_sdk/definitions/test_deadline.py: coerce_to_timedelta validation (non-integer, <= 0, overflow).

Verified locally in Breeze: 8 passed (dagrun deadline/variable) + 14 passed (SDK TestVariableInterval).

Generated-by: Claude Code (Opus via Claude Code) on behalf of Sean Ghaeli

SameerMesiah97

I did not manage to get to the tests but if I have to be honest, I think this needs a round of polish before a maintainer can review it. Conceptually, it looks correct but there are classic smells of unvetted AI-generated content such as messy code, long commnets/docstrings etc. I would convert it to draft and clean it up before requesting review.

I have left some comments.

seanghaeli · 2026-07-29T00:56:32Z

@SameerMesiah97 could I get your inputs on the updated version?

SameerMesiah97

Looks good to me. I just left a few more comments.

A ``DeadlineAlert`` configured with a ``VariableInterval`` is resolved when the scheduler creates a DagRun, inside ``DAG._process_dagrun_deadline_alerts`` (which runs under the ``prohibit_commit`` guard). Two problems are fixed: 1. Resolution now goes through the full secrets chain (env vars, configured secrets backends, then the metadata DB) via a dedicated ``_resolve_variable_interval`` helper, rather than ``Variable.get`` / ``begin_nested``. ``Variable.get`` and a SAVEPOINT release both commit on the scheduler's session, tripping ``prohibit_commit`` ("UNEXPECTED COMMIT") and silently dropping deadlines for every scheduled DagRun. The helper passes the scheduler session through to the metastore backend so the DB read happens without committing, and reading via the secrets chain (not the variable table directly) means ``AIRFLOW_VAR_*`` env vars and secrets backends resolve too. 2. Each deadline alert is isolated with a plain ``try``/``except`` (NOT ``begin_nested``). Creating a deadline is auxiliary to creating the DagRun; a single bad alert -- a missing/invalid backing Variable, or an undecodable serialized blob -- must never abort the DagRun and stop the DAG scheduling. ``VariableInterval.resolve`` is split into ``resolve`` + ``coerce_to_timedelta`` so the scheduler-side reader reuses the exact same validation (including the OverflowError -> ValueError translation) without going through ``Variable.get``. Generated-by: Claude Code (Sonnet/Opus via Claude Code) on behalf of Sean Ghaeli

…trings

ferruzzi

Apparently I reviewed this last week and forgot to submit. Sorry for the delay

ferruzzi · 2026-08-13T18:05:56Z

+                **({"session": session} if isinstance(backend, MetastoreBackend) else {}),
+            )
+            if value is not None:
+                return interval.coerce_to_timedelta(value)


We're using the secrets backend, do we need to mask or unmask here? In theory it's retrieving a number and that number itself may not be a secret, but I'm not positive if everything in there gets masked by default, etc.

Nothing fetched here would be masked, so I don't believe there's any need to consider that here.

ferruzzi · 2026-08-13T18:07:23Z

+        :return: The resolved ``timedelta``.
+        :raises ValueError: If the Variable cannot be resolved or converted to a valid ``timedelta``.
+        """
+        for backend in ensure_secrets_loaded():


The core version of this (Variable.get_variable_from_secrets) has more logic like a tyr/except block and caching the returned value. Is there a reason we don't need any of that in here?

I believe there's no notion of caching in the scheduler. The analogous try/except blocks, I agree, we can add that. Added in the latest commit.

ferruzzi · 2026-08-13T18:08:34Z

+        ],
+    )
    @mock.patch.object(Deadline, "prune_deadlines")
-    def test_dagrun_deadline_variable_interval_stable(self, _, mock_get, session, deadline_test_dag):


Looks like a dropped test, is this covered by another or am I just misreading the diff and it's just a rename?

This PR rewrites the path that's tested here so it forced a decision about what to do with the test: delete it or update it. Upon further inspecting the test, I think it never asserted anything useful about deadlines to begin with, so I decided to delete it:

when the test does dag_run.update_state(session) it doesn't actually update the deadline. Therefore, the test can never possibly fail to begin with.

airflow/airflow-core/src/airflow/models/dagrun.py

Lines 1371 to 1381 in c74cfcb

if dag.deadline:

# The dagrun has succeeded. If there were any Deadlines for it which were not breached, they are no longer needed.

deadline_alerts = [

DeadlineAlertModel.get_by_id(alert_id, session=session) for alert_id in dag.deadline

]

if any(

deadline_alert.reference_class in SerializedReferenceModels.TYPES.DAGRUN

for deadline_alert in deadline_alerts

):

Deadline.prune_deadlines(session=session, conditions={DagRun.id: self.id})

In the source code for update_state the only logic for deadlines is pruning on success. Perhaps this points to a gap in update_state, but that's something we can address in a separate issue.

seanghaeli requested review from amoghrajesh, ashb, bolkedebruin and kaxil as code owners June 23, 2026 20:17

boring-cyborg Bot added area:DAG-processing area:deadline-alerts AIP-86 (former AIP-57) area:task-sdk labels Jun 23, 2026

seanghaeli force-pushed the feature/variable-interval-resolution branch 2 times, most recently from 7aee60c to 94825ec Compare June 23, 2026 20:54

seanghaeli requested a review from potiuk as a code owner June 23, 2026 20:54

seanghaeli marked this pull request as draft June 23, 2026 20:57

seanghaeli force-pushed the feature/variable-interval-resolution branch from 94825ec to 2342c03 Compare June 23, 2026 21:31

seanghaeli marked this pull request as ready for review June 23, 2026 23:00

potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jun 25, 2026

SameerMesiah97 reviewed Jul 1, 2026

View reviewed changes

seanghaeli force-pushed the feature/variable-interval-resolution branch 2 times, most recently from 56b2666 to 3e4399b Compare July 8, 2026 06:46

seanghaeli force-pushed the feature/variable-interval-resolution branch 2 times, most recently from 69649b1 to 4754b7c Compare July 28, 2026 22:33

SameerMesiah97 reviewed Aug 1, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/serialization/definitions/dag.py

Comment thread airflow-core/tests/unit/models/test_dagrun.py Outdated

Comment thread airflow-core/tests/unit/models/test_dagrun.py Outdated

Comment thread airflow-core/tests/unit/models/test_dagrun.py Outdated

seanghaeli added 3 commits August 5, 2026 23:05

Re-trigger CI

c6d243e

Address review feedback: cover evaluate_with failures, trim test docs…

e1a907b

…trings

seanghaeli force-pushed the feature/variable-interval-resolution branch from c176287 to e1a907b Compare August 5, 2026 23:14

ferruzzi reviewed Aug 6, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/serialization/definitions/dag.py Outdated

Scope VariableInterval Variable lookup to the DagRun's team

3cb692a

ferruzzi reviewed Aug 13, 2026

View reviewed changes

seanghaeli added 2 commits August 15, 2026 00:16

Test VariableInterval resolution under prohibit_commit

e9d840f

Keep SDK docstring free of core internals

e65ee16

Skip a failing secrets backend when resolving VariableInterval deadlines

f27bb1f

	if dag.deadline:
	# The dagrun has succeeded. If there were any Deadlines for it which were not breached, they are no longer needed.
	deadline_alerts = [
	DeadlineAlertModel.get_by_id(alert_id, session=session) for alert_id in dag.deadline
	]

	if any(
	deadline_alert.reference_class in SerializedReferenceModels.TYPES.DAGRUN
	for deadline_alert in deadline_alerts
	):
	Deadline.prune_deadlines(session=session, conditions={DagRun.id: self.id})

Uh oh!

Conversation

seanghaeli commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

Tests

Uh oh!

SameerMesiah97 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

seanghaeli commented Jul 29, 2026

Uh oh!

SameerMesiah97 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ferruzzi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ferruzzi Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

seanghaeli Aug 15, 2026

Choose a reason for hiding this comment

Uh oh!

ferruzzi Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

seanghaeli Aug 15, 2026

Choose a reason for hiding this comment

Uh oh!

ferruzzi Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

seanghaeli Aug 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

seanghaeli commented Jun 23, 2026 •

edited

Loading

SameerMesiah97 left a comment •

edited

Loading