Skip to content

Allow PythonOperator to override task interpreter mode #70280

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fat-catTW wants to merge 12 commits into
base: main
Choose a base branch
from
Open

Allow PythonOperator to override task interpreter mode #70280

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
4e92962
Allow PythonOperator to override task interpreter mode
fat-catTW Jul 23, 2026
8ae6cdb
Fix PythonOperator docs heading underline
fat-catTW Jul 23, 2026
fd23831
Fix PythonOperator serialized task roundtrip
fat-catTW Jul 23, 2026
203a657
Use compat serialization helper in PythonOperator test
fat-catTW Jul 23, 2026
82514c1
Update Edge worker API schema for task interpreter mode
fat-catTW Jul 24, 2026
781dade
Add newsfragment for task interpreter config fix
fat-catTW Aug 3, 2026
e92c54f
Trigger CI for task interpreter PR
fat-catTW Aug 4, 2026
ef9b5f2
Guard task interpreter serialization for compat
fat-catTW Aug 5, 2026
59688f9
Handle older serialized operator compatibility
fat-catTW Aug 5, 2026
202a7f2
Read task interpreter override from serialized task
fat-catTW Aug 7, 2026
a37ce04
Trigger CI for task interpreter override PR
fat-catTW Aug 7, 2026
e3cb594
Restore task interpreter config fallback
fat-catTW Aug 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions airflow-core/newsfragments/70280.bugfix.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Restored support for ``[core] execute_tasks_new_python_interpreter`` when running tasks with LocalExecutor and CeleryExecutor.
6 changes: 4 additions & 2 deletions task-sdk/src/airflow/sdk/execution_time/supervisor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -515,8 +515,10 @@ def exit(n: int) -> NoReturn:


def _should_use_exec() -> bool:
"""Whether forked children should ``exec`` a fresh interpreter on this platform."""
return sys.platform in _FORK_EXEC_PLATFORMS
"""Whether forked children should ``exec`` a fresh interpreter."""
return sys.platform in _FORK_EXEC_PLATFORMS or conf.getboolean(
"core", "execute_tasks_new_python_interpreter", fallback=False
)


def _resolve_child_target(dotted: str) -> Callable[[], None]:
Expand Down
11 changes: 11 additions & 0 deletions task-sdk/tests/task_sdk/execution_time/test_supervisor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,17 @@
TI_ID = uuid7()


@pytest.mark.parametrize(
("platform", "config_value", "expected"),
[("darwin", False, True), ("linux", True, True), ("linux", False, False)],
)
def test_should_use_exec_honors_platform_and_config(monkeypatch, platform, config_value, expected):
monkeypatch.setattr(supervisor.sys, "platform", platform)
monkeypatch.setattr(supervisor.conf, "getboolean", lambda *args, **kwargs: config_value)

assert supervisor._should_use_exec() is expected


def lineno():
"""Returns the current line number in our program."""
return inspect.currentframe().f_back.f_lineno
Expand Down