Fix task logs unavailable while waiting for retry

[MonsterChenzhuo]

Symptom Explanation

1. Task failure (Try 12) : The task attempt fails and the task instance transitions to UP_FOR_RETRY (waiting for the next retry).

2. Logs disappear : While the task instance is in UP_FOR_RETRY , Airflow treats it as neither “running” ( RUNNING ) nor “finished” (terminal states).

   • Airflow’s log-reading logic only proactively fetches served logs from the worker log server when the task is RUNNING or DEFERRED .
   • For UP_FOR_RETRY , the task is not running but also not considered finished, so Airflow skips fetching served logs from the worker, resulting in an empty/placeholder log response.

Brief description of the changes.

• Fix an issue where task instance logs could become temporarily unavailable after a failed try when the task transitions to UP_FOR_RETRY (or UP_FOR_RESCHEDULE ).

• In this window, the log endpoint could return only the “Log message source details” header without actual log content.

• Update FileTaskHandler to also attempt fetching served logs from the worker log server for UP_FOR_RETRY / UP_FOR_RESCHEDULE states when no local or remote logs are found.
  Why this change is needed.

• UP_FOR_RETRY is treated as an “unfinished” state, so the existing logic did not fetch served logs (which was only done for RUNNING / DEFERRED or finished tasks), causing a gap where logs from the failed attempt were not accessible until the next try started.
  What was changed.

• In airflow-core/src/airflow/utils/log/file_task_handler.py , extend the condition that triggers reading served logs to include TaskInstanceState.UP_FOR_RETRY and TaskInstanceState.UP_FOR_RESCHEDULE .

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[MonsterChenzhuo]

The problem has been fixed.(#54547)