Pass Dag runs directly to Deadline references - #70714
Open
imrichardwu wants to merge 8 commits into
Open
Conversation
imrichardwu
requested review from
XD-DENG,
amoghrajesh,
ashb,
bolkedebruin and
kaxil
as code owners
July 30, 2026 05:27
imrichardwu
marked this pull request as draft
July 30, 2026 05:29
imrichardwu
force-pushed
the
fix/deadline-reference-context
branch
2 times, most recently
from
July 30, 2026 05:33
e9c48fd to
3b99b93
Compare
imrichardwu
marked this pull request as ready for review
July 30, 2026 05:33
Contributor
Author
|
@ferruzzi can I get a review? |
Preserve the Dag run evaluation context for custom references that accept additional keyword arguments, so they continue to evaluate correctly.
imrichardwu
force-pushed
the
fix/deadline-reference-context
branch
from
July 31, 2026 01:39
3b99b93 to
db15212
Compare
ferruzzi
reviewed
Aug 6, 2026
|
|
||
| @abstractmethod | ||
| def _evaluate_with(self, *, session: Session, **kwargs: Any) -> datetime | None: | ||
| def _evaluate_with(self, *, session: Session, dagrun: Any) -> datetime | None: |
Contributor
There was a problem hiding this comment.
Rather than using Any, I think a cleaner solution might be to type dagrun as a DagRunProtocol and add queued_at to that protocol.
Contributor
There was a problem hiding this comment.
It doesn't look like this was actually addressed, was it? You would at east need to add queued_at to DagRunProtocol for that to work and it's not showing in the diff.
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
The custom deadline reference examples returned the dagrun attribute directly, which read as if the attribute was the required return value rather than an input available to the author's business logic. Type the evaluation dagrun with the shared DagRunProtocol so authors get a concrete contract for what is available, and show the attribute feeding into business logic in the examples.
A custom deadline reference whose _evaluate_with declares **kwargs is expected to receive the full evaluation context, matching the standard Python "accept anything" convention rather than having the context filtered away. Align the test with that behaviour.
ferruzzi
reviewed
Aug 10, 2026
| def _evaluate_with(self, *, session: Session, **kwargs) -> datetime: | ||
| # Put your business logic here | ||
| return some_datetime | ||
| # Optionally, you can specify when you want it calculated by providing a DeadlineReference.TYPES |
Contributor
There was a problem hiding this comment.
This comment duplicates the one just above the decorator. The one outside feels like a better place.
| _datetime: datetime | ||
|
|
||
| def _evaluate_with(self, *, session: Session, **kwargs: Any) -> datetime | None: | ||
| def _evaluate_with(self, *, session: Session, dagrun: Any) -> datetime | None: |
Contributor
There was a problem hiding this comment.
Same as before; dagrun is never used, so replacing it with **kwargs is a bit cleaner.
| and give every field a default value. | ||
| * **Plugin Placement**: One convenient place for custom references is in the plugins directory. | ||
| * **API Server Restart**: Restart the Airflow API Server after adding or modifying custom references. | ||
| * **Required Parameters**: ``required_kwargs`` declares which Dag run context values Airflow should |
Contributor
There was a problem hiding this comment.
Since Required Parameters is still available until 4.0, we should keep it here and just add a note that it is deprecated
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Preserve the Dag run evaluation context for custom references that accept additional keyword arguments, so they continue to evaluate correctly.
Pass the in-memory Dag run to Deadline references so built-in references avoid redundant database reads and custom references can evaluate the current state.
Deprecate
required_kwargswhile preserving it for existing custom references. Use the shared evaluator when recalculating queued-at deadlines.closes: #70710
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.