emrg: fix Windows cmd-window storm + daemon spawn throttle (v0.2.15 regression hotfix) - #592
Conversation
…egression hotfix)
Rant 2026-08-09T13:16:36 (emergency, highest priority): Windows v0.2.15
host observed continuous cmd popups (had to reboot) + daemon startup
failure leaving GUI/scheduler unable to connect.
Root causes:
1. Zero CREATE_NO_WINDOW anywhere in the codebase — every subprocess
spawn (git/gh/bash tool/scheduler/daemon spawn) popped a console
window on Windows.
2. GUI reconnect loop respawned emrgd.cmd every ~5s forever; each
spawn raced the previous daemon's startup (G43 stale-port unlink
deleted a healthy daemon's port file → its scheduler logged
'cannot connect' 93 times while the GUI kept spawning).
3. Scheduler had no connect-failure backoff.
Fixes:
- NEW emrg/_win.py: win32_no_window_kwargs() = {creationflags:
CREATE_NO_WINDOW} on Windows, {} elsewhere. Splatted into all 34
subprocess call sites (bash_tool, scheduler x12, daemon x5,
git_utils, daemon_manager, installer, __main__ x3, client/app x9).
- GUI daemon_client.js: spawn throttle (max 3 attempts per connect
lifecycle, then throw with emrgd.log tail instead of respawning);
reset counter on successful auth. Both spawn-timeout errors now
surface the real emrgd.log tail (readLogTail).
- GUI main.js: reconnect exponential backoff 1s→2s→4s→…cap 60s;
daemon_stopped status surfaces the real failure to the renderer
(zh/en i18n) instead of infinite 5s respawns.
- scheduler.py: connect-failure exponential backoff
max(30s, interval*2^n) capped at 10 min — no more per-tick retry
storm while the daemon is down.
Tests: +6 Python (win32_no_window_kwargs POSIX/Windows/splat; backoff
zero/exp/cap/floor; 641→647), +2 GUI (spawn throttle + counter reset;
96→98). Docs synced (#511 guard). macOS/Linux unaffected (win32 branch
is a no-op empty dict).
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle 2026-08-09T16:49 (Round 186)
Reviewed the full diff on the PR branch:
emrg/_win.pyis a clean cross-platform design:win32_no_window_kwargs()returns{creationflags: CREATE_NO_WINDOW}on Windows and{}on POSIX (verified via unit tests incl. the splat-compat test).- All 34 Python subprocess call sites carry the splat (script-verified, zero gaps).
- Scheduler exponential backoff
max(30s, interval*2^n)cap 10 min — covered by 3 new tests (interval/no-failure, growth+capping, 30s floor). - GUI spawn throttle (3 attempts per lifecycle, reset on auth) + reconnect backoff (1s→…→60s) — covered by 2 new tests.
_readLogTailon both spawn-timeout paths surfaces the real emrgd.log failure reason (rant acceptance item ②).
Verified on the PR branch: uv run pytest tests/ → 647 passed; npm test → 98 passed; import/--help OK. Doc counts synced (#511 guard). POSIX no-op confirmed — zero macOS/Linux behavior change.
|
Tested this PR on Windows (win32, Python 3.13.4) — the hotfix behaves as designed: |
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle 2026-08-09T17:11 (Round 189)
Second independent review of the storm-fix foundation PR (re-verified fresh this cycle):
emrg/_win.pywin32_no_window_kwargs()— clean cross-platform design (CREATE_NO_WINDOW on Windows, empty dict on POSIX;getattrfallback keeps POSIX importable/testable).- All 34 Python subprocess call sites carry the splat (script-verified, zero gaps across bash_tool/scheduler/daemon/git_utils/daemon_manager/installer/main/client).
- Scheduler exponential backoff
max(30s, interval*2^n)cap 10min — covered by 3 tests (interval/no-failure, growth+capping, 30s floor). - GUI spawn throttle (3 attempts/lifecycle, reset on auth) + reconnect backoff (1s→…→60s) +
_readLogTailon both spawn-timeout paths — 2 new tests. - POSIX no-op confirmed (empty-dict splat changes nothing on macOS/Linux).
Fresh verification on the PR branch: uv run pytest tests/ → 647 passed; npm test → 98 passed; import + --help OK. Doc counts synced (#511).
… 2026-08-09T13:16:36) After the spawn throttle exhausts, scheduleReconnect keeps retrying with backoff capped at 60s — each retry hit the throttle and re-sent daemon_stopped status, so the renderer appended a duplicate 'run emrg server' system message every minute forever. Add daemonStoppedNotified: the warning is sent once per connect lifecycle (reset on successful connect), symmetric with the TUI _throttle_warned guard (PR #594).
|
Re-verified the delta commit 4cb0fe8 (one-time daemonStoppedNotified guard) on this Windows host: |
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle 2026-08-09T17:25 (Round 192)
Third independent review (3/3 consecutive LGTMs: C186, C189, C192). Fresh verification on the PR branch:
- uv run pytest tests/ → 647 passed
- npm test (emrg/gui) → 98 passed
- import + --help OK
Final diff audit (2 commits):
- 37acdc0 — storm containment: emrg/_win.py win32_no_window_kwargs() splatted into all 34 subprocess call sites (script-verified zero gaps); scheduler connect backoff max(30s, interval*2^n) cap 10min; GUI spawn throttle (3/lifecycle, reset on auth) + reconnect backoff 1s→…→60s + real emrgd.log tail in spawn errors.
- 4cb0fe8 — daemon_stopped one-time per connect lifecycle (symmetric with TUI _throttle_warned, PR #594), prevents duplicate system messages every backoff retry.
POSIX no-op confirmed (empty-dict splat). Doc counts synced (#511). Merge-ready.
…ate (Windows v0.2.15 root cause) (#593) Rant 2026-08-09T13:16:36 root-cause follow-up: PR #592 contains the window-storm (CREATE_NO_WINDOW + spawn throttle + backoff), but the underlying deletion race remains: G43 stale-port logic deletes a healthy daemon's emrgd.port after ONE transient ws failure → the daemon's own scheduler loses the file (93x 'cannot connect' in the host's emrgd.log) while the PID lock makes every respawn exit instantly (zombie state — daemon alive, forever unreachable). Two complementary fixes: 1. daemon-side self-heal: new _assert_port_file() + _port_keepalive_loop re-asserts emrgd.port every 60s if it was deleted/overwritten — the daemon is the living core and re-asserts its own identity. 2. GUI-side guard: _daemonProcessAlive() checks emrgd.pid before G43 deletes the port file — process alive → keep the file and let the reconnect backoff retry (transient failure); only truly-dead daemons get the delete+respawn path. Tests: +3 Python (test_daemon.py _assert_port_file write/rewrite + keepalive restore), +3 GUI (daemon_client.test.js G43 guard alive/dead/ no-pid). 641→644 py, 96→99 gui, docs synced (#511).
* emrg: TUI daemon spawn throttle — complete the anti-storm fix (rant 2026-08-09T13:16:36) PRs #592 (CREATE_NO_WINDOW + GUI spawn throttle + scheduler backoff) and #593 (port-file self-heal + G43 PID guard) contain the Windows v0.2.15 storm fix, but the TUI client had the same storm pattern the GUI had: app.py _reconnect() loops every 1s calling ensure_connected() → with a down daemon each iteration calls start_daemon() → spawns a NEW daemon process every second. On Windows each spawn was a cmd-window source (pre-#592); even windowless it is process churn + PID-lock exit races post-#592. This completes rant acceptance item ① ("启动 GUI/TUI 后零 cmd 窗口弹出"): - daemon_manager.start_daemon: spawn throttle — max 3 attempts per connect lifecycle, then raise with a clear 'run emrg server manually' message instead of spawning forever; counter resets on successful connect in ensure_connected (mirrors GUI daemon_client.js MAX_SPAWN_ATTEMPTS). - app.py _reconnect: on throttle, surface a one-time system message + status hint so the host knows to start the daemon manually (recovery path intact: host starts daemon → is_running True → connect succeeds → counter resets). Tests: +2 (start_daemon throttles after 3 attempts, no 4th spawn; counter resets on success). 641→643 py, doc counts synced (#511). * emrg: sync doc test counts to 652 (post-#593 merge + #594 tests) --------- Co-authored-by: EMRG Evolution <emrg@argszero.dev>
…T13:16:36) (#596) Carries the emergency Windows v0.2.15 regression hotfix (host observed cmd-window storm + daemon startup failure, had to reboot): - #592: CREATE_NO_WINDOW across all 34 subprocess sites (emrg/_win.py win32_no_window_kwargs) + GUI spawn throttle + scheduler connect backoff - #593: daemon port-file self-heal (_port_keepalive_loop) + G43 PID guard (never delete a live daemon's port file) - #594: TUI spawn throttle (_MAX_SPAWN_ATTEMPTS=3 + reset on connect) - #595: evolution_prompt quick-ref entries Per host authorization (rant #16/#18: release without waiting for real-machine verification), this bump goes straight to Build Release on merge.
Summary
Emergency hotfix for the v0.2.15 Windows regression reported in rant 2026-08-09T13:16:36 (highest priority — host observed continuous cmd popups and had to reboot; daemon startup failure left GUI/scheduler unable to connect).
Root causes
grep -rn "CREATE_NO_WINDOW|creationflags" emrg/had zero hits. Everysubprocess.Popen/create_subprocess_*call (git, gh, bash tool, scheduler, daemon spawn) popped a console window on Windows.ensureConnectedrespawned emrgd.cmd every ~5s forever on failure. Each spawn raced the previous daemon's startup; the G43 stale-port unlink deleted a healthy daemon's port file → its scheduler loggedcannot connect93× while the GUI kept spawning (emrgd.log evidence: 92 prompt built / 93 cannot connect / 0 connected).Changes
Python (all 34 subprocess call sites)
emrg/_win.py:win32_no_window_kwargs()returns{"creationflags": subprocess.CREATE_NO_WINDOW}on Windows,{}elsewhere (no-op on macOS/Linux).bash_tool.py,scheduler.py(12 sites),daemon.py(5),git_utils.py,client/daemon_manager.py,skills/installer.py,__main__.py(3),client/app.py(9).Scheduler backoff (rant item ③)
max(30s, interval × 2^n)capped at 10 min — retry storm eliminated while daemon is down.GUI anti-storm master switch (rant item ⑤)
daemon_client.js: spawn throttle — max 3 spawn attempts per connect lifecycle, then throw with the real emrgd.log tail instead of respawning; counter resets on successful auth.daemon_client.js: both spawn-timeout errors now append the emrgd.log tail (_readLogTail, mirrors daemon_manager R124) — host sees the real reason, notfailed to start within timeout(rant item ②).main.js: reconnect exponential backoff 1s→2s→4s→…cap 60s; throttled failure surfaces asdaemon_stoppedstatus with the real error + log tail (zh/en i18n).Tests
win32_no_window_kwargs(POSIX no-op / Windows CREATE_NO_WINDOW / splat-compat) + scheduler backoff (no-failure interval / exponential growth / 10-min cap / 30s floor).Verification
uv run pytest tests/: 647 passeduv run python -c "from emrg.client.app import run_client": OKuv run python -m emrg --help: OKnpm test(emrg/gui): 98 passedPer rant: this is a hotfix — once merged, the v0.2.16 release flow should follow (host #16/#18 pre-authorized release without waiting for real-machine verification).