Wire disk-pressure self-heal into the Tentacle host lifecycle#435
Merged
Conversation
The self-heal scaffolding (SelfHealController, DiskPressureHealAction,
DefaultWorkspaceCleanupPolicy, and their tests) was built but never
scheduled — TentacleFlavor returned BackgroundTasks = [], so a disk-full
agent failed every deployment via LocalScriptService's admission gate
with no auto-reclaim of completed-script workspaces.
Wire it:
- WorkspaceProbe enumerates {tempRoot}/squid-tentacle-{ticketId} dirs and
classifies each from its persisted ScriptState — Complete+0 => Succeeded,
Complete+nonzero => Failed, started-not-complete => Active, no state =>
Unknown — plus recursive size + last-write, so the cleanup policy can pick
oldest-evictable workspaces while protecting a retention window.
- SelfHealBackgroundTask adapts SelfHealController to the host's
ITentacleBackgroundTask lifecycle (Start on launch, drain on shutdown);
ForLocalWorkspaces builds the disk-pressure action over the temp root with
the running script backend as the live-workspace veto.
- TentacleFlavor now schedules it (BackgroundTasks = [selfHeal]).
The running-script reporter veto means an in-flight deployment's workspace
is never swept; only completed/orphaned workspaces are reclaimed, and only
under disk pressure.
Tests:
- WorkspaceProbeTests: real temp dirs + real ScriptState files — status
classification matrix, prefix filtering, size, missing-root
- SelfHealBackgroundTaskTests: RunAsync starts the heal loop + drains on cancel
- TentacleFlavorSelfHealWiringTests: the flavor schedules a SelfHeal task
(the dead-code-is-now-live regression guard)
…upt-state reclaim, backoff
The wiring that takes the disk self-heal live (the 30s DiskPressureHealAction over
{tempRoot}/squid-tentacle-*) is a second sweeper over the same directories the
24h orphan sweep owns, and the probe/policy had several robustness gaps. Address
them before the sweep runs in production:
- Honour the operator's orphan-workspace TTL as a retention floor: under mild
(non-critical) pressure the new sweep no longer evicts inside the post-mortem
window the existing age-based orphan sweep promises; under critical pressure the
floor is bypassed so reclaiming disk still wins.
- Protect a just-created workspace via a 60s fresh-grace floor (even under
critical pressure), closing the StartScript TOCTOU gap between CreateDirectory
and the first state Save / running-script registration.
- Classify a present-but-corrupt state file as Unknown instead of skipping the
workspace forever — otherwise a dead-but-corrupt workspace is permanently
un-reclaimable, exactly when disk pressure most needs the space.
- Skip reparse points in the recursive size walk so a directory symlink (extracted
package or runtime ln -s) cannot loop the enumeration or mis-attribute a
symlink target's size to the workspace.
- Warn once + back off exponentially when a sweep reclaims everything it is
allowed to but the disk stays under pressure (non-workspace usage), instead of
churning every 30s.
- Expose the retention counts + low-disk trigger behind env vars with pinned
const names (SelfHealOptions), mirroring the orphan-TTL env-var pattern.
All floors default off in DefaultWorkspaceCleanupPolicy/DiskPressure so the
parameterless constructors behave exactly as before; only the live
ForLocalWorkspaces wiring injects them — non-breaking.
Tests: SelfHealOptions env-var/default pins; corrupt-state, recursive nested-size
and symlink-safety probe cases; non-critical-TTL + critical-bypass + fresh-grace
policy cases; backoff growth/cap/reset; and a high-fidelity wired-chain E2E
(real probe + policy + ScriptStateStore + Directory.Delete) proving real reclaim
while a Running workspace and the in-TTL window survive.
…tage parse Make MinKeepCount/MaxKeepCount public (mirroring LocalScriptService's orphan-TTL bounds) and extract the env-value parsing into internal testable helpers so the reject/fallback branches are directly pinned (Rule 9 branch coverage). Parse the free-percentage with invariant culture so an operator's '0.20' reads the same on a comma-decimal locale server.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SelfHealController,DiskPressureHealAction,DefaultWorkspaceCleanupPolicy— all with tests) was built but never scheduled:TentacleFlavorreturnedBackgroundTasks = [], so a disk-full agent failed every deployment viaLocalScriptService's admission gate with no auto-reclaim of completed-script workspaces (SOTA-audit P2 'wired-but-dead code').WorkspaceProbe: enumerates{tempRoot}/squid-tentacle-{ticketId}dirs and classifies each from its persistedScriptState(Complete+0 → Succeeded, Complete+nonzero → Failed, started-not-complete → Active, none → Unknown) + recursive size + last-write.SelfHealBackgroundTask: adaptsSelfHealControllerto the hostITentacleBackgroundTasklifecycle (start on launch, drain on shutdown);ForLocalWorkspacesbuilds the disk-pressure action over the temp root with the running script backend as the live-workspace veto (an in-flight deployment's workspace is never swept).TentacleFlavornow schedules it. Reclaim happens only under disk pressure and only on completed/orphaned workspaces, keeping a per-status retention window for post-mortem.Test plan
WorkspaceProbeTests(real temp dirs + realScriptStatefiles — status matrix, prefix filter, size, missing-root);SelfHealBackgroundTaskTests(RunAsync starts the loop + drains on cancel);TentacleFlavorSelfHealWiringTests(the flavor schedules a SelfHeal task — dead-code-is-live guard) — all greenLocalScriptServiceTestscan't run on the author's local machine (disk 98% full trips LocalScriptService's own admission gate — proven pre-existing by stashing this change); CI runs them on free disk