Skip to content

fix(ci): raise nightly Windows build timeout to 180m (nightly red since 26.7.49) - #645

Merged
logbie merged 1 commit into
mainfrom
warden/nightly-timeout-release-test-binaries
Jul 26, 2026
Merged

fix(ci): raise nightly Windows build timeout to 180m (nightly red since 26.7.49)#645
logbie merged 1 commit into
mainfrom
warden/nightly-timeout-release-test-binaries

Conversation

@logbie

@logbie logbie commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

What was broken

The Nightly Build is red. Run 30190411454 (2026-07-26 06:03 UTC, sha 8d6aaed2 = v26.7.49) was cancelled, not failed: the Build WFL for Windows job hit its timeout-minutes: 90 cap at 07:34:05 and GitHub killed it. Run tests was cancelled mid-step; Run LSP Tests, the VS Code extension build, the WiX MSI and the whole release job were skipped, so no nightly release was published for 2026-07-26.

This was the first full nightly after PRs #641 and #643 merged. The 07-22 → 07-25 nightlies were all no-change skips on 9a8e0cca, so the cost increase was invisible until today.

Root cause

The Run tests step runs cargo test --release --locked, which compiles every integration test binary in tests/ against the full dependency graph — in release mode, and with [profile.release] debug = true, so each binary also links a large PDB on x86_64-pc-windows-msvc.

PRs #641/#643 added 26 new test files (tests/ is now 112 targets) plus new deps (tokio-tungstenite 0.30, rcgen, yasna, pem). Measured against the last full nightly:

run 29805777509 (v26.7.47, 07-21) run 30190411454 (v26.7.49, 07-26)
Run tests 06:15:05 → 06:42:14 = 27 min, success 06:23:57 → cancelled 07:34 = >70 min, never reached Finished
job total ~47 min 90 min (capped)

Nothing is hung. The final log line is an ordinary Compiling tokio-tungstenite v0.30.0 at 06:26:00, and no Finished/Running line for the test profile ever appears — the step was still in codegen/link when the cap fired.

Worth flagging: the failure is self-perpetuating. A cancelled job does not run its cache-save post step (Post Cache Cargo registry + target dir is skipped in this run), so the enlarged release test binaries never entered the Swatinem/rust-cache entry. Every subsequent nightly restarts from the same pre-#641 cache and times out at the same point. The nightly will stay red until the cap is raised once and the cache repopulates.

The fix

One line plus a rationale comment: timeout-minutes: 90180 on the build job. This mirrors PR #618, which raised the same cap 60 → 90 for the same class of growth.

180 min gives the first post-#641 nightly room to finish and save its cache; warm runs should return to well under an hour once it does.

Verification

  • YAML parses and the value is applied (jobs.build.timeout-minutes == 180).
  • No source, dependency, Cargo.lock, or .build_meta.json change — CI-config only, so there is no language-behavior surface and no TestPrograms/ impact. Backward compatibility is untouched.
  • Honest limitation: I could not reproduce locally. The failure is specific to release-mode MSVC codegen on windows-latest, and this triage environment is aarch64 Linux with no Rust toolchain. The diagnosis rests on the run logs and the step-level timing comparison above, which are unambiguous about where the time went. ci.yml will not validate this change either — its integration lanes compile the test binaries in debug (14 min on Windows today), which is exactly why ci.yml stayed green while the nightly went red.
  • Real validation is the next scheduled nightly (or a workflow_dispatch run of nightly.yml from this branch) after merge.

Recommended follow-ups (not in this PR)

  1. Structural: 112 separate release test binaries is the actual cost driver, and it will grow again. Consolidating related integration tests into fewer harnesses, or splitting Run tests into its own parallel job, would cut wall-clock more durably than raising the cap a third time.
  2. The nightly-only lint/test gap is still open — ci.yml never exercises release-mode Windows test compilation, so this class of regression can only ever surface at 06:00 UTC.

Automated triage PR from the WFL repo warden — opened for a human to review and merge; I have not merged it.


Open in Devin Review

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Summary by CodeRabbit

  • Chores
    • Increased the Windows nightly build time limit to reduce failures during longer test compilation and cache recovery.
    • Added workflow documentation explaining the extended timeout.

The nightly build job was cancelled mid-compile at the 90-minute cap in run
30190411454 (2026-07-26, sha 8d6aaed, v26.7.49) — the first full nightly after
PRs #641 and #643 merged.

Root cause: the "Run tests" step (cargo test --release --locked) must compile
every integration test binary in tests/ in release mode with debug info. #641 and
#643 added 26 new test files (112 total), and the step went from ~27 min in run
29805777509 (v26.7.47) to over 70 min without reaching "Finished". Nothing is
hung — the last log line is a normal Compiling entry and no test ever ran.

The failure is self-perpetuating: a cancelled job skips its cache-save post step,
so the enlarged release test binaries never entered the rust-cache entry and each
following nightly restarts cold and times out again.

This raises only the job timeout, mirroring the 60 -> 90 bump in PR #618. No
source, dependency, or language behavior changes, so no TestPrograms impact.
Copilot AI review requested due to automatic review settings July 26, 2026 09:06
@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e94f2d02-455d-4215-9381-a695fac80607

📥 Commits

Reviewing files that changed from the base of the PR and between 8d6aaed and 7376669.

📒 Files selected for processing (1)
  • .github/workflows/nightly.yml

📝 Walkthrough

Walkthrough

The Windows nightly build job timeout increases from 90 to 180 minutes, with comments explaining test compilation growth and cache repopulation after cancellations.

Changes

Nightly workflow configuration

Layer / File(s) Summary
Windows build timeout
.github/workflows/nightly.yml
The Windows build job timeout is increased to 180 minutes, with explanatory comments added.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

Suggested reviewers: copilot

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: increasing the nightly Windows build timeout to 180 minutes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch warden/nightly-timeout-release-test-binaries

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Windows-only nightly GitHub Actions workflow to prevent the nightly pipeline from being cancelled due to the job-level timeout during long release-mode test compilation on windows-latest.

Changes:

  • Increased the Build WFL for Windows job timeout from 90 minutes to 180 minutes.
  • Added an in-file rationale comment explaining why the higher timeout is needed (and why it should help the cache warm back up).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +126 to +137
# `cargo test --release` here compiles EVERY integration test binary in
# tests/ (112 of them as of 26.7.49) against the full dependency graph, in
# release mode with `[profile.release] debug = true` — so each one also links
# a large PDB on the MSVC target. PRs #641/#643 added 26 new test files,
# which pushed this single step from ~27 min (26.7.47, run 29805777509) to
# over 70 min without finishing, and the job hit the previous 90-minute cap
# (run 30190411454, cancelled mid-compile). Because a cancelled job never
# runs its cache-save post step, the enlarged test binaries never landed in
# the Swatinem/rust-cache entry, so every following nightly restarted from
# the same cold state and timed out again. 180 min gives the first post-#641
# nightly room to finish and repopulate the cache; warm runs stay far below
# it. Previous bump was 60 -> 90 in PR #618 for the same class of growth.
@logbie
logbie merged commit bfd6dbd into main Jul 26, 2026
19 checks passed
@logbie
logbie deleted the warden/nightly-timeout-release-test-binaries branch July 26, 2026 10:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants