From ead859a9a94c50d3c46ca9cf0bba7559d3b2b4c3 Mon Sep 17 00:00:00 2001 From: Simon Cropp Date: Tue, 21 Jul 2026 08:19:14 +1000 Subject: [PATCH 1/6] Automatically reclaim orphaned instances and residue The per run cleanup only sees instances that still have a data directory under the data root. Once that directory is gone, cleared with the temp directory or by the cleanup itself, nothing points at the instance and the system databases, logs and traces LocalDB keeps for it are never reclaimed. On top of that, deleting an instance leaves its logs and event files behind as an abandoned directory. Both accumulate indefinitely. The directory LocalDB keeps for each instance is now swept at startup: - A directory with no registered instance behind it is residue from an already deleted instance and is removed. - A registered instance with no data directory, and not running, is an orphan and is stopped, deleted, and its directory removed. Only directories untouched for InstanceCleanupThreshold, defaulting to 30 days, are processed. This keeps the startup scan cheap and avoids racing an instance being created or in use. Configurable via the LocalDBInstanceCleanupDays environment variable; zero disables it. Attribution is by state, not authorship: residue has no live instance so it is safe to remove whatever created it, and only instances with no data directory are treated as orphans, so an instance in active use is left alone. The default instances LocalDB manages itself are always skipped. The test suite sets the threshold to zero in its module initializer, so running the tests does not sweep the real instance root as a side effect. --- pages/directory-and-name-resolution.md | 2 + .../directory-and-name-resolution.source.md | 2 + src/LocalDb.Tests/InstanceRootCleanerTests.cs | 161 ++++++++++++++++++ src/LocalDb.Tests/LocalDbSettingsTests.cs | 17 ++ src/LocalDb.Tests/ModuleInitializer.cs | 4 + src/LocalDb/DirectoryCleaner.cs | 102 +++++++++++ src/LocalDb/DirectoryFinder.cs | 3 +- src/LocalDb/LocalDbCleanup.cs | 10 +- src/LocalDb/Settings.cs | 25 +++ 9 files changed, 320 insertions(+), 6 deletions(-) create mode 100644 src/LocalDb.Tests/InstanceRootCleanerTests.cs diff --git a/pages/directory-and-name-resolution.md b/pages/directory-and-name-resolution.md index 8c3250a3..d0edeeb8 100644 --- a/pages/directory-and-name-resolution.md +++ b/pages/directory-and-name-resolution.md @@ -67,6 +67,8 @@ That location is owned by LocalDB and cannot be changed. It is derived from the Deleting an instance reclaims the system databases but leaves the logs and event files behind, so purging an instance removes both the instance and this directory. +The per run purge above only sees instances that still have a data directory. Once that directory is gone, cleared with the temp directory or by the purge itself, nothing under the data root points at the instance and it can no longer be found that way. To catch these, this directory is also swept: any instance directory untouched for a threshold, defaulting to 30 days, is removed, along with directories left behind by instances that were already deleted. Instances that LocalDB creates and manages itself, such as `MSSQLLocalDB`, are never touched. The threshold can be configured via the `LocalDBInstanceCleanupDays` environment variable, or `LocalDbSettings.InstanceCleanupThreshold`; set it to zero to disable. To reclaim these immediately rather than waiting for the threshold, call `LocalDbCleanup.DeleteOrphanInstances`. + ## Virus scanning exclusions diff --git a/pages/mdsource/directory-and-name-resolution.source.md b/pages/mdsource/directory-and-name-resolution.source.md index eba2dc26..6119217c 100644 --- a/pages/mdsource/directory-and-name-resolution.source.md +++ b/pages/mdsource/directory-and-name-resolution.source.md @@ -32,6 +32,8 @@ That location is owned by LocalDB and cannot be changed. It is derived from the Deleting an instance reclaims the system databases but leaves the logs and event files behind, so purging an instance removes both the instance and this directory. +The per run purge above only sees instances that still have a data directory. Once that directory is gone, cleared with the temp directory or by the purge itself, nothing under the data root points at the instance and it can no longer be found that way. To catch these, this directory is also swept: any instance directory untouched for a threshold, defaulting to 30 days, is removed, along with directories left behind by instances that were already deleted. Instances that LocalDB creates and manages itself, such as `MSSQLLocalDB`, are never touched. The threshold can be configured via the `LocalDBInstanceCleanupDays` environment variable, or `LocalDbSettings.InstanceCleanupThreshold`; set it to zero to disable. To reclaim these immediately rather than waiting for the threshold, call `LocalDbCleanup.DeleteOrphanInstances`. + ## Virus scanning exclusions diff --git a/src/LocalDb.Tests/InstanceRootCleanerTests.cs b/src/LocalDb.Tests/InstanceRootCleanerTests.cs new file mode 100644 index 00000000..710b0c4e --- /dev/null +++ b/src/LocalDb.Tests/InstanceRootCleanerTests.cs @@ -0,0 +1,161 @@ +[TestFixture] +public class InstanceRootCleanerTests +{ + static readonly TimeSpan threshold = TimeSpan.FromHours(6); + + static string MakeStaleDir(string root, string name, params string[] files) + { + var dir = Path.Combine(root, name); + Directory.CreateDirectory(dir); + var old = DateTime.Now.AddDays(-3); + foreach (var file in files) + { + var path = Path.Combine(dir, file); + File.WriteAllText(path, "x"); + File.SetLastWriteTime(path, old); + } + + Directory.SetCreationTime(dir, old); + return dir; + } + + [Test] + public void RemovesResidueDirectoryWithNoInstance() + { + using var instanceRoot = new TempDirectory(); + using var dataRoot = new TempDirectory(); + // a name that is not a registered instance, holding only leftover diagnostic files + var dir = MakeStaleDir(instanceRoot, "ResidueNoInstanceTest", "error.log", "log_1.trc"); + + DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold); + + False(Directory.Exists(dir)); + } + + [Test] + public void LeavesRecentDirectory() + { + using var instanceRoot = new TempDirectory(); + using var dataRoot = new TempDirectory(); + var dir = Path.Combine(instanceRoot, "RecentResidueTest"); + Directory.CreateDirectory(dir); + File.WriteAllText(Path.Combine(dir, "error.log"), "x"); + + DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold); + + True(Directory.Exists(dir)); + } + + [Test] + public void LeavesDefaultInstanceDirectory() + { + using var instanceRoot = new TempDirectory(); + using var dataRoot = new TempDirectory(); + var dir = MakeStaleDir(instanceRoot, "MSSQLLocalDB", "error.log"); + + DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold); + + True(Directory.Exists(dir)); + } + + [Test] + public void LeavesEverythingWhenDisabled() + { + using var instanceRoot = new TempDirectory(); + using var dataRoot = new TempDirectory(); + var dir = MakeStaleDir(instanceRoot, "DisabledCleanupTest", "error.log"); + + DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, TimeSpan.Zero); + + True(Directory.Exists(dir)); + } + + // The routing for a real instance is tested per directory rather than through + // CleanInstanceRoot, so the sweep is never run against the real instance root where it + // would process every instance on the machine. A cutoff in the future makes every + // directory count as stale, isolating the branch under test from timing. + static HashSet Registered() => + new(LocalDbApi.GetInstanceNames(), StringComparer.OrdinalIgnoreCase); + + [Test] + public async Task RemovesStaleOrphanedInstance() + { + var name = "OrphanInstanceRootTest"; + LocalDbApi.StopAndDelete(name); + + using var dataRoot = new TempDirectory(); + var directory = Path.Combine(dataRoot, name); + using var wrapper = new Wrapper(name, directory); + wrapper.Start(new(2000, 1, 1), TestDbBuilder.CreateTable); + await wrapper.AwaitStart(); + + var instanceDir = DirectoryFinder.FindInstance(name); + True(LocalDbApi.GetInstance(name).Exists); + True(Directory.Exists(instanceDir)); + + // orphan it: stop and remove the data directory + LocalDbApi.StopInstance(name, ShutdownMode.KillProcess); + Directory.Delete(directory, true); + + DirectoryCleaner.CleanInstanceDirectory(instanceDir, dataRoot, Registered(), DateTime.Now.AddMinutes(5)); + + False(LocalDbApi.GetInstance(name).Exists); + False(Directory.Exists(instanceDir)); + } + + [Test] + public async Task LeavesRunningInstance() + { + var name = "RunningInstanceRootTest"; + LocalDbApi.StopAndDelete(name); + + using var dataRoot = new TempDirectory(); + var directory = Path.Combine(dataRoot, name); + using var wrapper = new Wrapper(name, directory); + wrapper.Start(new(2000, 1, 1), TestDbBuilder.CreateTable); + await wrapper.AwaitStart(); + + var instanceDir = DirectoryFinder.FindInstance(name); + + // no data directory and treated as stale, but running: must be left alone + Directory.Delete(directory, true); + + try + { + DirectoryCleaner.CleanInstanceDirectory(instanceDir, dataRoot, Registered(), DateTime.Now.AddMinutes(5)); + + True(LocalDbApi.GetInstance(name).Exists); + } + finally + { + DirectoryCleaner.RemoveInstance(name); + } + } + + [Test] + public void LeavesTrackedInstanceWithDataDirectory() + { + var name = "TrackedInstanceRootTest"; + LocalDbApi.StopAndDelete(name); + LocalDbApi.CreateInstance(name); + + using var dataRoot = new TempDirectory(); + // an existing data directory means CleanRoot governs it, so the sweep leaves it + Directory.CreateDirectory(Path.Combine(dataRoot, name)); + + try + { + DirectoryCleaner.CleanInstanceDirectory( + DirectoryFinder.FindInstance(name), + dataRoot, + Registered(), + DateTime.Now.AddMinutes(5)); + + True(LocalDbApi.GetInstance(name).Exists); + } + finally + { + DirectoryCleaner.RemoveInstance(name); + } + } +} diff --git a/src/LocalDb.Tests/LocalDbSettingsTests.cs b/src/LocalDb.Tests/LocalDbSettingsTests.cs index eb938da2..59e1eb70 100644 --- a/src/LocalDb.Tests/LocalDbSettingsTests.cs +++ b/src/LocalDb.Tests/LocalDbSettingsTests.cs @@ -82,4 +82,21 @@ public void DBAutoOffline_CanBeSetProgrammatically() LocalDbSettings.DBAutoOffline = original; } } + + // the default cannot be asserted here: the module initializer sets this to zero so the + // suite does not sweep the real instance root + [Test] + public void InstanceCleanupThreshold_CanBeSetProgrammatically() + { + var original = LocalDbSettings.InstanceCleanupThreshold; + try + { + LocalDbSettings.InstanceCleanupThreshold = TimeSpan.FromDays(7); + That(LocalDbSettings.InstanceCleanupThreshold, Is.EqualTo(TimeSpan.FromDays(7))); + } + finally + { + LocalDbSettings.InstanceCleanupThreshold = original; + } + } } diff --git a/src/LocalDb.Tests/ModuleInitializer.cs b/src/LocalDb.Tests/ModuleInitializer.cs index 3d5c6288..ae4359f1 100644 --- a/src/LocalDb.Tests/ModuleInitializer.cs +++ b/src/LocalDb.Tests/ModuleInitializer.cs @@ -5,6 +5,10 @@ public static void Initialize() { VerifierSettings.InitializePlugins(); LocalDbLogging.EnableVerbose(); + // do not let the test suite sweep the real instance root as a side effect: it would + // delete instances belonging to other projects on the machine. The sweep is tested + // directly instead. Must be set before DirectoryFinder initializes + LocalDbSettings.InstanceCleanupThreshold = TimeSpan.Zero; LocalDbSettings.ConnectionBuilder(_ => _.ConnectTimeout = 300); VerifierSettings.ScrubLinesContaining("filename = '"); VerifierSettings.IgnoreMember(_ => _.OwnerSID); diff --git a/src/LocalDb/DirectoryCleaner.cs b/src/LocalDb/DirectoryCleaner.cs index 2582f6a6..6d3c1e5d 100644 --- a/src/LocalDb/DirectoryCleaner.cs +++ b/src/LocalDb/DirectoryCleaner.cs @@ -78,6 +78,108 @@ static void Clean(string directory, bool deleteInstance) } } + /// + /// Reclaims instances and residue from the directory LocalDB keeps for each instance. + /// + /// Covers what cannot see: instances whose data directory is gone, so + /// nothing under the data root points at them, and directories left behind by instances that + /// were already deleted, since LocalDB does not remove the logs and event files on delete. + /// + /// + /// Only directories untouched for are processed, both to keep the + /// startup scan cheap and to avoid racing an instance that is being created or is in use. + /// + /// + public static void CleanInstanceRoot(string instanceRoot, string dataRoot, TimeSpan threshold) + { + if (threshold <= TimeSpan.Zero || + !Directory.Exists(instanceRoot)) + { + return; + } + + var cutoff = DateTime.Now - threshold; + + HashSet registered; + try + { + registered = new(LocalDbApi.GetInstanceNames(), StringComparer.OrdinalIgnoreCase); + } + catch (Exception exception) + { + LocalDbLogging.LogIfVerbose($"Failed to enumerate instances, skipping instance root cleanup. {exception.Message}"); + return; + } + + foreach (var directory in Directory.EnumerateDirectories(instanceRoot)) + { + try + { + CleanInstanceDirectory(directory, dataRoot, registered, cutoff); + } + catch (Exception exception) + { + // cleanup must never break the test run that triggered it + LocalDbLogging.LogIfVerbose($"Failed to clean instance directory: {directory}. {exception.Message}"); + } + } + } + + internal static void CleanInstanceDirectory(string directory, string dataRoot, HashSet registered, DateTime cutoff) + { + var name = Path.GetFileName(directory); + + // instances LocalDB creates and manages, never owned by this library + if (LocalDbCleanup.IsDefaultInstance(name)) + { + return; + } + + // recently touched, so either in use or too new to be sure it is abandoned + if (NewestWrite(directory) >= cutoff) + { + return; + } + + if (!registered.Contains(name)) + { + // no instance backs this directory: the instance was already deleted and only the + // logs and event files remain. Safe to remove whatever created it + Directory.Delete(directory, true); + return; + } + + // likely in use by another process + if (LocalDbApi.GetInstance(name).IsRunning) + { + return; + } + + // still tracked by a data directory, so CleanRoot governs when it is reclaimed + if (Directory.Exists(Path.Combine(dataRoot, name))) + { + return; + } + + LocalDbLogging.LogIfVerbose($"Removing orphaned instance: {name}"); + RemoveInstance(name); + } + + static DateTime NewestWrite(string directory) + { + var newest = Directory.GetCreationTime(directory); + foreach (var file in Directory.EnumerateFiles(directory)) + { + var write = File.GetLastWriteTime(file); + if (write > newest) + { + newest = write; + } + } + + return newest; + } + /// /// Stops and deletes an instance, then removes the directory LocalDB keeps for it. Deleting /// the instance only reclaims the system databases: the logs and traces are left behind. diff --git a/src/LocalDb/DirectoryFinder.cs b/src/LocalDb/DirectoryFinder.cs index 20aab3c5..b4c1c975 100644 --- a/src/LocalDb/DirectoryFinder.cs +++ b/src/LocalDb/DirectoryFinder.cs @@ -7,6 +7,7 @@ static DirectoryFinder() dataRoot = FindDataRoot(); var root = dataRoot; DirectoryCleaner.CleanRoot(root); + DirectoryCleaner.CleanInstanceRoot(instanceRoot, root, LocalDbSettings.InstanceCleanupThreshold); } public static string Find(string instanceName) => Path.Combine(dataRoot, instanceName); @@ -25,7 +26,7 @@ public static void Delete(string instanceName) /// logs and traces of each instance. The location is owned by LocalDB and cannot be /// configured: it is always derived from the local application data folder. /// - static string instanceRoot = Path.Combine( + public static string instanceRoot = Path.Combine( Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), "Microsoft", "Microsoft SQL Server Local DB", diff --git a/src/LocalDb/LocalDbCleanup.cs b/src/LocalDb/LocalDbCleanup.cs index 20f56b0c..300a1a4b 100644 --- a/src/LocalDb/LocalDbCleanup.cs +++ b/src/LocalDb/LocalDbCleanup.cs @@ -5,11 +5,11 @@ namespace LocalDb; #endif /// -/// Removes LocalDB instances left behind by previous runs. +/// Immediately removes LocalDB instances left behind by previous runs. /// -/// The automatic cleanup only knows about instances that still have a data directory. Once that -/// directory has been removed, for example by the temp directory being cleared, the instance is -/// orphaned and the system databases, logs and traces LocalDB keeps for it are never reclaimed. +/// The same instances are removed automatically once they have been untouched for +/// . This is the forced version, for when +/// they should be reclaimed now rather than waiting for that threshold. /// /// public static class LocalDbCleanup @@ -17,7 +17,7 @@ public static class LocalDbCleanup /// /// Instances that LocalDB creates and manages itself, and that are never owned by this library. /// - static bool IsDefaultInstance(string name) => + internal static bool IsDefaultInstance(string name) => name == "MSSQLLocalDB" || // automatic instances, eg "v11.0" name.Length > 1 && name[0] == 'v' && char.IsDigit(name[1]); diff --git a/src/LocalDb/Settings.cs b/src/LocalDb/Settings.cs index d1e18da5..26375b3c 100644 --- a/src/LocalDb/Settings.cs +++ b/src/LocalDb/Settings.cs @@ -37,6 +37,15 @@ internal static string BuildConnectionString(string instance, string database, b /// public static bool? DBAutoOffline { get; set; } = ResolveDBAutoOffline(); + /// + /// How long an instance directory must be untouched before automatic cleanup removes the + /// instance and the directory LocalDB keeps for it. This reclaims instances whose data + /// directory is gone, which the per run cleanup can no longer see. + /// Can be configured via the LocalDBInstanceCleanupDays environment variable. + /// Defaults to 30 days. Set to to disable. + /// + public static TimeSpan InstanceCleanupThreshold { get; set; } = ResolveInstanceCleanupThreshold(); + static ushort ResolveShutdownTimeout() { var envValue = Environment.GetEnvironmentVariable("LocalDBShutdownTimeout"); @@ -63,4 +72,20 @@ static ushort ResolveShutdownTimeout() _ => null }; } + + static TimeSpan ResolveInstanceCleanupThreshold() + { + var envValue = Environment.GetEnvironmentVariable("LocalDBInstanceCleanupDays"); + if (envValue is null) + { + return TimeSpan.FromDays(30); + } + + if (ushort.TryParse(envValue, out var days)) + { + return TimeSpan.FromDays(days); + } + + throw new ArgumentException($"Failed to parse LocalDBInstanceCleanupDays environment variable: {envValue}"); + } } From fdb6c708a85ff0eb6c1038ef77887600faa8fdb6 Mon Sep 17 00:00:00 2001 From: Simon Cropp Date: Tue, 21 Jul 2026 10:10:55 +1000 Subject: [PATCH 2/6] Cap instances removed per cleanup run A first run against a large backlog would otherwise stop and delete every orphaned instance and remove every residue directory in one startup, which is disk bound and adds a noticeable delay to that run. At most InstanceCleanupLimit removals happen per run, defaulting to 20, so the backlog drains over several runs. Skips are cheap and do not count against the limit, so recently touched or in use directories never crowd out the stale ones. Configurable via the LocalDBInstanceCleanupLimit environment variable; zero removes the limit. DeleteOrphanInstances remains the way to reclaim everything at once. --- pages/directory-and-name-resolution.md | 4 ++- .../directory-and-name-resolution.source.md | 4 ++- src/LocalDb.Tests/InstanceRootCleanerTests.cs | 34 +++++++++++++++++++ src/LocalDb.Tests/LocalDbSettingsTests.cs | 25 ++++++++++++++ src/LocalDb/DirectoryCleaner.cs | 29 +++++++++++----- src/LocalDb/DirectoryFinder.cs | 2 +- src/LocalDb/Settings.cs | 24 +++++++++++++ 7 files changed, 111 insertions(+), 11 deletions(-) diff --git a/pages/directory-and-name-resolution.md b/pages/directory-and-name-resolution.md index d0edeeb8..d3b9631a 100644 --- a/pages/directory-and-name-resolution.md +++ b/pages/directory-and-name-resolution.md @@ -67,7 +67,9 @@ That location is owned by LocalDB and cannot be changed. It is derived from the Deleting an instance reclaims the system databases but leaves the logs and event files behind, so purging an instance removes both the instance and this directory. -The per run purge above only sees instances that still have a data directory. Once that directory is gone, cleared with the temp directory or by the purge itself, nothing under the data root points at the instance and it can no longer be found that way. To catch these, this directory is also swept: any instance directory untouched for a threshold, defaulting to 30 days, is removed, along with directories left behind by instances that were already deleted. Instances that LocalDB creates and manages itself, such as `MSSQLLocalDB`, are never touched. The threshold can be configured via the `LocalDBInstanceCleanupDays` environment variable, or `LocalDbSettings.InstanceCleanupThreshold`; set it to zero to disable. To reclaim these immediately rather than waiting for the threshold, call `LocalDbCleanup.DeleteOrphanInstances`. +The per run purge above only sees instances that still have a data directory. Once that directory is gone, cleared with the temp directory or by the purge itself, nothing under the data root points at the instance and it can no longer be found that way. To catch these, this directory is also swept: any instance directory untouched for a threshold, defaulting to 30 days, is removed, along with directories left behind by instances that were already deleted. Instances that LocalDB creates and manages itself, such as `MSSQLLocalDB`, are never touched. The threshold can be configured via the `LocalDBInstanceCleanupDays` environment variable, or `LocalDbSettings.InstanceCleanupThreshold`; set it to zero to disable. + +To keep the sweep off the startup path, at most `LocalDbSettings.InstanceCleanupLimit` instances are removed per run, defaulting to 20, so a large backlog is drained over several runs rather than in a single startup. Configurable via the `LocalDBInstanceCleanupLimit` environment variable; set it to zero for no limit. To reclaim everything immediately rather than waiting, call `LocalDbCleanup.DeleteOrphanInstances`. ## Virus scanning exclusions diff --git a/pages/mdsource/directory-and-name-resolution.source.md b/pages/mdsource/directory-and-name-resolution.source.md index 6119217c..ebe4b344 100644 --- a/pages/mdsource/directory-and-name-resolution.source.md +++ b/pages/mdsource/directory-and-name-resolution.source.md @@ -32,7 +32,9 @@ That location is owned by LocalDB and cannot be changed. It is derived from the Deleting an instance reclaims the system databases but leaves the logs and event files behind, so purging an instance removes both the instance and this directory. -The per run purge above only sees instances that still have a data directory. Once that directory is gone, cleared with the temp directory or by the purge itself, nothing under the data root points at the instance and it can no longer be found that way. To catch these, this directory is also swept: any instance directory untouched for a threshold, defaulting to 30 days, is removed, along with directories left behind by instances that were already deleted. Instances that LocalDB creates and manages itself, such as `MSSQLLocalDB`, are never touched. The threshold can be configured via the `LocalDBInstanceCleanupDays` environment variable, or `LocalDbSettings.InstanceCleanupThreshold`; set it to zero to disable. To reclaim these immediately rather than waiting for the threshold, call `LocalDbCleanup.DeleteOrphanInstances`. +The per run purge above only sees instances that still have a data directory. Once that directory is gone, cleared with the temp directory or by the purge itself, nothing under the data root points at the instance and it can no longer be found that way. To catch these, this directory is also swept: any instance directory untouched for a threshold, defaulting to 30 days, is removed, along with directories left behind by instances that were already deleted. Instances that LocalDB creates and manages itself, such as `MSSQLLocalDB`, are never touched. The threshold can be configured via the `LocalDBInstanceCleanupDays` environment variable, or `LocalDbSettings.InstanceCleanupThreshold`; set it to zero to disable. + +To keep the sweep off the startup path, at most `LocalDbSettings.InstanceCleanupLimit` instances are removed per run, defaulting to 20, so a large backlog is drained over several runs rather than in a single startup. Configurable via the `LocalDBInstanceCleanupLimit` environment variable; set it to zero for no limit. To reclaim everything immediately rather than waiting, call `LocalDbCleanup.DeleteOrphanInstances`. ## Virus scanning exclusions diff --git a/src/LocalDb.Tests/InstanceRootCleanerTests.cs b/src/LocalDb.Tests/InstanceRootCleanerTests.cs index 710b0c4e..dec3a7ba 100644 --- a/src/LocalDb.Tests/InstanceRootCleanerTests.cs +++ b/src/LocalDb.Tests/InstanceRootCleanerTests.cs @@ -58,6 +58,40 @@ public void LeavesDefaultInstanceDirectory() True(Directory.Exists(dir)); } + [Test] + public void RemovesAtMostLimitPerRun() + { + using var instanceRoot = new TempDirectory(); + using var dataRoot = new TempDirectory(); + MakeStaleDir(instanceRoot, "LimitResidueA", "error.log"); + MakeStaleDir(instanceRoot, "LimitResidueB", "error.log"); + MakeStaleDir(instanceRoot, "LimitResidueC", "error.log"); + + DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold, limit: 2); + AreEqual(1, Directory.EnumerateDirectories(instanceRoot).Count()); + + // the remainder is drained on the next run + DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold, limit: 2); + AreEqual(0, Directory.EnumerateDirectories(instanceRoot).Count()); + } + + [Test] + public void SkipsDoNotCountAgainstLimit() + { + using var instanceRoot = new TempDirectory(); + using var dataRoot = new TempDirectory(); + // a recent directory is skipped and must not consume the limit that the stale one needs + var recent = Path.Combine(instanceRoot, "LimitRecent"); + Directory.CreateDirectory(recent); + File.WriteAllText(Path.Combine(recent, "error.log"), "x"); + var stale = MakeStaleDir(instanceRoot, "LimitStale", "error.log"); + + DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold, limit: 1); + + False(Directory.Exists(stale)); + True(Directory.Exists(recent)); + } + [Test] public void LeavesEverythingWhenDisabled() { diff --git a/src/LocalDb.Tests/LocalDbSettingsTests.cs b/src/LocalDb.Tests/LocalDbSettingsTests.cs index 59e1eb70..46f804f9 100644 --- a/src/LocalDb.Tests/LocalDbSettingsTests.cs +++ b/src/LocalDb.Tests/LocalDbSettingsTests.cs @@ -99,4 +99,29 @@ public void InstanceCleanupThreshold_CanBeSetProgrammatically() LocalDbSettings.InstanceCleanupThreshold = original; } } + + [Test] + public void InstanceCleanupLimit_DefaultsTo20WhenEnvVarNotSet() + { + var envValue = Environment.GetEnvironmentVariable("LocalDBInstanceCleanupLimit"); + if (envValue is null) + { + That(LocalDbSettings.InstanceCleanupLimit, Is.EqualTo(20)); + } + } + + [Test] + public void InstanceCleanupLimit_CanBeSetProgrammatically() + { + var original = LocalDbSettings.InstanceCleanupLimit; + try + { + LocalDbSettings.InstanceCleanupLimit = 5; + That(LocalDbSettings.InstanceCleanupLimit, Is.EqualTo(5)); + } + finally + { + LocalDbSettings.InstanceCleanupLimit = original; + } + } } diff --git a/src/LocalDb/DirectoryCleaner.cs b/src/LocalDb/DirectoryCleaner.cs index 6d3c1e5d..3497c720 100644 --- a/src/LocalDb/DirectoryCleaner.cs +++ b/src/LocalDb/DirectoryCleaner.cs @@ -90,7 +90,7 @@ static void Clean(string directory, bool deleteInstance) /// startup scan cheap and to avoid racing an instance that is being created or is in use. /// /// - public static void CleanInstanceRoot(string instanceRoot, string dataRoot, TimeSpan threshold) + public static void CleanInstanceRoot(string instanceRoot, string dataRoot, TimeSpan threshold, int limit = 0) { if (threshold <= TimeSpan.Zero || !Directory.Exists(instanceRoot)) @@ -111,11 +111,23 @@ public static void CleanInstanceRoot(string instanceRoot, string dataRoot, TimeS return; } + // each removal is disk work, so a backlog is drained over several runs rather than in + // one startup. Skips are cheap and do not count against the limit + var removed = 0; foreach (var directory in Directory.EnumerateDirectories(instanceRoot)) { + if (limit > 0 && + removed >= limit) + { + break; + } + try { - CleanInstanceDirectory(directory, dataRoot, registered, cutoff); + if (CleanInstanceDirectory(directory, dataRoot, registered, cutoff)) + { + removed++; + } } catch (Exception exception) { @@ -125,20 +137,20 @@ public static void CleanInstanceRoot(string instanceRoot, string dataRoot, TimeS } } - internal static void CleanInstanceDirectory(string directory, string dataRoot, HashSet registered, DateTime cutoff) + internal static bool CleanInstanceDirectory(string directory, string dataRoot, HashSet registered, DateTime cutoff) { var name = Path.GetFileName(directory); // instances LocalDB creates and manages, never owned by this library if (LocalDbCleanup.IsDefaultInstance(name)) { - return; + return false; } // recently touched, so either in use or too new to be sure it is abandoned if (NewestWrite(directory) >= cutoff) { - return; + return false; } if (!registered.Contains(name)) @@ -146,23 +158,24 @@ internal static void CleanInstanceDirectory(string directory, string dataRoot, H // no instance backs this directory: the instance was already deleted and only the // logs and event files remain. Safe to remove whatever created it Directory.Delete(directory, true); - return; + return true; } // likely in use by another process if (LocalDbApi.GetInstance(name).IsRunning) { - return; + return false; } // still tracked by a data directory, so CleanRoot governs when it is reclaimed if (Directory.Exists(Path.Combine(dataRoot, name))) { - return; + return false; } LocalDbLogging.LogIfVerbose($"Removing orphaned instance: {name}"); RemoveInstance(name); + return true; } static DateTime NewestWrite(string directory) diff --git a/src/LocalDb/DirectoryFinder.cs b/src/LocalDb/DirectoryFinder.cs index b4c1c975..84c388f9 100644 --- a/src/LocalDb/DirectoryFinder.cs +++ b/src/LocalDb/DirectoryFinder.cs @@ -7,7 +7,7 @@ static DirectoryFinder() dataRoot = FindDataRoot(); var root = dataRoot; DirectoryCleaner.CleanRoot(root); - DirectoryCleaner.CleanInstanceRoot(instanceRoot, root, LocalDbSettings.InstanceCleanupThreshold); + DirectoryCleaner.CleanInstanceRoot(instanceRoot, root, LocalDbSettings.InstanceCleanupThreshold, LocalDbSettings.InstanceCleanupLimit); } public static string Find(string instanceName) => Path.Combine(dataRoot, instanceName); diff --git a/src/LocalDb/Settings.cs b/src/LocalDb/Settings.cs index 26375b3c..3c5c76f0 100644 --- a/src/LocalDb/Settings.cs +++ b/src/LocalDb/Settings.cs @@ -46,6 +46,14 @@ internal static string BuildConnectionString(string instance, string database, b /// public static TimeSpan InstanceCleanupThreshold { get; set; } = ResolveInstanceCleanupThreshold(); + /// + /// The maximum number of instances and directories the automatic cleanup removes per run, + /// so a large backlog is drained over several runs rather than in a single startup. + /// Can be configured via the LocalDBInstanceCleanupLimit environment variable. + /// Defaults to 20. Set to 0 for no limit. + /// + public static ushort InstanceCleanupLimit { get; set; } = ResolveInstanceCleanupLimit(); + static ushort ResolveShutdownTimeout() { var envValue = Environment.GetEnvironmentVariable("LocalDBShutdownTimeout"); @@ -88,4 +96,20 @@ static TimeSpan ResolveInstanceCleanupThreshold() throw new ArgumentException($"Failed to parse LocalDBInstanceCleanupDays environment variable: {envValue}"); } + + static ushort ResolveInstanceCleanupLimit() + { + var envValue = Environment.GetEnvironmentVariable("LocalDBInstanceCleanupLimit"); + if (envValue is null) + { + return 20; + } + + if (ushort.TryParse(envValue, out var limit)) + { + return limit; + } + + throw new ArgumentException($"Failed to parse LocalDBInstanceCleanupLimit environment variable: {envValue}"); + } } From 90f2850f2b95c9711ba3339d40dbbebe5aed6343 Mon Sep 17 00:00:00 2001 From: Simon Cropp Date: Tue, 21 Jul 2026 11:38:17 +1000 Subject: [PATCH 3/6] Only reclaim instances marked as created by this library LocalDB instances are shared by everything on the machine that uses LocalDB, and nothing in an instance records what created it. Reclaiming one therefore meant guessing, and the sweep could remove an instance belonging to something else entirely. A marker file is now written into the directory LocalDB keeps for each instance this library starts, and the ongoing sweep only reclaims marked instances. Marking on start as well as on create means instances that predate marking are picked up the next time they are used. Marking is best effort: an instance that cannot be marked is simply never reclaimed automatically. Instances that predate marking cannot be told apart from instances belonging to anything else, so they get a single best effort pass, recorded under %LocalAppData%\LocalDb so it happens once per machine. That pass removes directories left behind by already deleted instances, which have no instance to harm, and orphans that still carry the shrunk model.mdf this library leaves behind. Nothing else is touched, so it can never remove a foreign instance. The pass is only recorded as done when the sweep actually ran, so a sweep that could not start does not consume the one chance the backlog gets. This replaces the per run cap. The ongoing sweep now only ever considers instances known to be ours, which is a much smaller set. --- pages/directory-and-name-resolution.md | 6 +- .../directory-and-name-resolution.source.md | 6 +- src/EfClassicLocalDb/EfClassicLocalDb.csproj | 1 + src/EfLocalDb/EfLocalDb.csproj | 1 + src/LocalDb.Tests/InstanceRootCleanerTests.cs | 200 ++++++++++++++---- src/LocalDb.Tests/LocalDbSettingsTests.cs | 24 --- src/LocalDb.Tests/ModuleInitializer.cs | 5 +- src/LocalDb/DirectoryCleaner.cs | 73 ++++--- src/LocalDb/DirectoryFinder.cs | 39 +++- src/LocalDb/InstanceMarker.cs | 71 +++++++ src/LocalDb/LocalDbApi.cs | 8 +- src/LocalDb/Settings.cs | 24 --- 12 files changed, 331 insertions(+), 127 deletions(-) create mode 100644 src/LocalDb/InstanceMarker.cs diff --git a/pages/directory-and-name-resolution.md b/pages/directory-and-name-resolution.md index d3b9631a..dee2973f 100644 --- a/pages/directory-and-name-resolution.md +++ b/pages/directory-and-name-resolution.md @@ -67,9 +67,11 @@ That location is owned by LocalDB and cannot be changed. It is derived from the Deleting an instance reclaims the system databases but leaves the logs and event files behind, so purging an instance removes both the instance and this directory. -The per run purge above only sees instances that still have a data directory. Once that directory is gone, cleared with the temp directory or by the purge itself, nothing under the data root points at the instance and it can no longer be found that way. To catch these, this directory is also swept: any instance directory untouched for a threshold, defaulting to 30 days, is removed, along with directories left behind by instances that were already deleted. Instances that LocalDB creates and manages itself, such as `MSSQLLocalDB`, are never touched. The threshold can be configured via the `LocalDBInstanceCleanupDays` environment variable, or `LocalDbSettings.InstanceCleanupThreshold`; set it to zero to disable. +The per run purge above only sees instances that still have a data directory. Once that directory is gone, cleared with the temp directory or by the purge itself, nothing under the data root points at the instance and it can no longer be found that way. To catch these, this directory is also swept. -To keep the sweep off the startup path, at most `LocalDbSettings.InstanceCleanupLimit` instances are removed per run, defaulting to 20, so a large backlog is drained over several runs rather than in a single startup. Configurable via the `LocalDBInstanceCleanupLimit` environment variable; set it to zero for no limit. To reclaim everything immediately rather than waiting, call `LocalDbCleanup.DeleteOrphanInstances`. +LocalDB instances are shared by everything on the machine that uses LocalDB, and nothing in an instance records what created it. So a marker file is written into the directory of each instance this library starts, and the sweep only ever reclaims marked instances. An instance created by anything else has no marker and is never removed. A marked instance is reclaimed once it has no data directory, is not running, and has been untouched for a threshold defaulting to 30 days. Configurable via the `LocalDBInstanceCleanupDays` environment variable, or `LocalDbSettings.InstanceCleanupThreshold`; set it to zero to disable the sweep. + +Instances that predate marking cannot be told apart from instances belonging to anything else. They get a single best effort pass, once per machine, which removes directories left behind by already deleted instances and orphaned instances that still carry the shrunk `model.mdf` this library leaves behind. Anything else is left alone. That the pass has run is recorded under `%LocalAppData%\LocalDb`. To reclaim the remainder, call `LocalDbCleanup.DeleteOrphanInstances`. ## Virus scanning exclusions diff --git a/pages/mdsource/directory-and-name-resolution.source.md b/pages/mdsource/directory-and-name-resolution.source.md index ebe4b344..1bcaa171 100644 --- a/pages/mdsource/directory-and-name-resolution.source.md +++ b/pages/mdsource/directory-and-name-resolution.source.md @@ -32,9 +32,11 @@ That location is owned by LocalDB and cannot be changed. It is derived from the Deleting an instance reclaims the system databases but leaves the logs and event files behind, so purging an instance removes both the instance and this directory. -The per run purge above only sees instances that still have a data directory. Once that directory is gone, cleared with the temp directory or by the purge itself, nothing under the data root points at the instance and it can no longer be found that way. To catch these, this directory is also swept: any instance directory untouched for a threshold, defaulting to 30 days, is removed, along with directories left behind by instances that were already deleted. Instances that LocalDB creates and manages itself, such as `MSSQLLocalDB`, are never touched. The threshold can be configured via the `LocalDBInstanceCleanupDays` environment variable, or `LocalDbSettings.InstanceCleanupThreshold`; set it to zero to disable. +The per run purge above only sees instances that still have a data directory. Once that directory is gone, cleared with the temp directory or by the purge itself, nothing under the data root points at the instance and it can no longer be found that way. To catch these, this directory is also swept. -To keep the sweep off the startup path, at most `LocalDbSettings.InstanceCleanupLimit` instances are removed per run, defaulting to 20, so a large backlog is drained over several runs rather than in a single startup. Configurable via the `LocalDBInstanceCleanupLimit` environment variable; set it to zero for no limit. To reclaim everything immediately rather than waiting, call `LocalDbCleanup.DeleteOrphanInstances`. +LocalDB instances are shared by everything on the machine that uses LocalDB, and nothing in an instance records what created it. So a marker file is written into the directory of each instance this library starts, and the sweep only ever reclaims marked instances. An instance created by anything else has no marker and is never removed. A marked instance is reclaimed once it has no data directory, is not running, and has been untouched for a threshold defaulting to 30 days. Configurable via the `LocalDBInstanceCleanupDays` environment variable, or `LocalDbSettings.InstanceCleanupThreshold`; set it to zero to disable the sweep. + +Instances that predate marking cannot be told apart from instances belonging to anything else. They get a single best effort pass, once per machine, which removes directories left behind by already deleted instances and orphaned instances that still carry the shrunk `model.mdf` this library leaves behind. Anything else is left alone. That the pass has run is recorded under `%LocalAppData%\LocalDb`. To reclaim the remainder, call `LocalDbCleanup.DeleteOrphanInstances`. ## Virus scanning exclusions diff --git a/src/EfClassicLocalDb/EfClassicLocalDb.csproj b/src/EfClassicLocalDb/EfClassicLocalDb.csproj index 46639ff7..6cf60179 100644 --- a/src/EfClassicLocalDb/EfClassicLocalDb.csproj +++ b/src/EfClassicLocalDb/EfClassicLocalDb.csproj @@ -18,6 +18,7 @@ + diff --git a/src/EfLocalDb/EfLocalDb.csproj b/src/EfLocalDb/EfLocalDb.csproj index c2e0ce74..e70ceb69 100644 --- a/src/EfLocalDb/EfLocalDb.csproj +++ b/src/EfLocalDb/EfLocalDb.csproj @@ -21,6 +21,7 @@ + diff --git a/src/LocalDb.Tests/InstanceRootCleanerTests.cs b/src/LocalDb.Tests/InstanceRootCleanerTests.cs index dec3a7ba..0ef50776 100644 --- a/src/LocalDb.Tests/InstanceRootCleanerTests.cs +++ b/src/LocalDb.Tests/InstanceRootCleanerTests.cs @@ -3,12 +3,18 @@ public class InstanceRootCleanerTests { static readonly TimeSpan threshold = TimeSpan.FromHours(6); - static string MakeStaleDir(string root, string name, params string[] files) + static string MakeStaleDir(string root, string name, bool marked, params string[] files) { var dir = Path.Combine(root, name); Directory.CreateDirectory(dir); + var all = files.ToList(); + if (marked) + { + all.Add(".localdbwrapper"); + } + var old = DateTime.Now.AddDays(-3); - foreach (var file in files) + foreach (var file in all) { var path = Path.Combine(dir, file); File.WriteAllText(path, "x"); @@ -20,76 +26,65 @@ static string MakeStaleDir(string root, string name, params string[] files) } [Test] - public void RemovesResidueDirectoryWithNoInstance() + public void RemovesMarkedResidue() { using var instanceRoot = new TempDirectory(); using var dataRoot = new TempDirectory(); - // a name that is not a registered instance, holding only leftover diagnostic files - var dir = MakeStaleDir(instanceRoot, "ResidueNoInstanceTest", "error.log", "log_1.trc"); + var dir = MakeStaleDir(instanceRoot, "MarkedResidueTest", marked: true, "error.log"); - DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold); + DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold, backlogPass: false); False(Directory.Exists(dir)); } [Test] - public void LeavesRecentDirectory() + public void LeavesUnmarkedResidueOutsideBacklogPass() { using var instanceRoot = new TempDirectory(); using var dataRoot = new TempDirectory(); - var dir = Path.Combine(instanceRoot, "RecentResidueTest"); - Directory.CreateDirectory(dir); - File.WriteAllText(Path.Combine(dir, "error.log"), "x"); + var dir = MakeStaleDir(instanceRoot, "UnmarkedResidueTest", marked: false, "error.log"); - DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold); + DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold, backlogPass: false); True(Directory.Exists(dir)); } [Test] - public void LeavesDefaultInstanceDirectory() + public void RemovesUnmarkedResidueInBacklogPass() { using var instanceRoot = new TempDirectory(); using var dataRoot = new TempDirectory(); - var dir = MakeStaleDir(instanceRoot, "MSSQLLocalDB", "error.log"); + var dir = MakeStaleDir(instanceRoot, "BacklogResidueTest", marked: false, "error.log"); - DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold); + DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold, backlogPass: true); - True(Directory.Exists(dir)); + False(Directory.Exists(dir)); } [Test] - public void RemovesAtMostLimitPerRun() + public void LeavesRecentDirectory() { using var instanceRoot = new TempDirectory(); using var dataRoot = new TempDirectory(); - MakeStaleDir(instanceRoot, "LimitResidueA", "error.log"); - MakeStaleDir(instanceRoot, "LimitResidueB", "error.log"); - MakeStaleDir(instanceRoot, "LimitResidueC", "error.log"); + var dir = Path.Combine(instanceRoot, "RecentResidueTest"); + Directory.CreateDirectory(dir); + File.WriteAllText(Path.Combine(dir, ".localdbwrapper"), ""); - DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold, limit: 2); - AreEqual(1, Directory.EnumerateDirectories(instanceRoot).Count()); + DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold, backlogPass: true); - // the remainder is drained on the next run - DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold, limit: 2); - AreEqual(0, Directory.EnumerateDirectories(instanceRoot).Count()); + True(Directory.Exists(dir)); } [Test] - public void SkipsDoNotCountAgainstLimit() + public void LeavesDefaultInstanceDirectory() { using var instanceRoot = new TempDirectory(); using var dataRoot = new TempDirectory(); - // a recent directory is skipped and must not consume the limit that the stale one needs - var recent = Path.Combine(instanceRoot, "LimitRecent"); - Directory.CreateDirectory(recent); - File.WriteAllText(Path.Combine(recent, "error.log"), "x"); - var stale = MakeStaleDir(instanceRoot, "LimitStale", "error.log"); + var dir = MakeStaleDir(instanceRoot, "MSSQLLocalDB", marked: true, "error.log"); - DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold, limit: 1); + DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold, backlogPass: true); - False(Directory.Exists(stale)); - True(Directory.Exists(recent)); + True(Directory.Exists(dir)); } [Test] @@ -97,9 +92,9 @@ public void LeavesEverythingWhenDisabled() { using var instanceRoot = new TempDirectory(); using var dataRoot = new TempDirectory(); - var dir = MakeStaleDir(instanceRoot, "DisabledCleanupTest", "error.log"); + var dir = MakeStaleDir(instanceRoot, "DisabledCleanupTest", marked: true, "error.log"); - DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, TimeSpan.Zero); + DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, TimeSpan.Zero, backlogPass: true); True(Directory.Exists(dir)); } @@ -111,10 +106,39 @@ public void LeavesEverythingWhenDisabled() static HashSet Registered() => new(LocalDbApi.GetInstanceNames(), StringComparer.OrdinalIgnoreCase); + static void Clean(string directory, string dataRoot, bool backlogPass = false) => + DirectoryCleaner.CleanInstanceDirectory( + directory, + dataRoot, + Registered(), + DateTime.Now.AddMinutes(5), + backlogPass); + [Test] - public async Task RemovesStaleOrphanedInstance() + public async Task MarksInstanceItCreates() { - var name = "OrphanInstanceRootTest"; + var name = "MarkedOnCreateTest"; + LocalDbApi.StopAndDelete(name); + + using var dataRoot = new TempDirectory(); + using var wrapper = new Wrapper(name, Path.Combine(dataRoot, name)); + wrapper.Start(new(2000, 1, 1), TestDbBuilder.CreateTable); + await wrapper.AwaitStart(); + + try + { + True(InstanceMarker.IsMarked(DirectoryFinder.FindInstance(name))); + } + finally + { + DirectoryCleaner.RemoveInstance(name); + } + } + + [Test] + public async Task RemovesMarkedOrphanedInstance() + { + var name = "MarkedOrphanTest"; LocalDbApi.StopAndDelete(name); using var dataRoot = new TempDirectory(); @@ -125,22 +149,53 @@ public async Task RemovesStaleOrphanedInstance() var instanceDir = DirectoryFinder.FindInstance(name); True(LocalDbApi.GetInstance(name).Exists); - True(Directory.Exists(instanceDir)); // orphan it: stop and remove the data directory LocalDbApi.StopInstance(name, ShutdownMode.KillProcess); Directory.Delete(directory, true); - DirectoryCleaner.CleanInstanceDirectory(instanceDir, dataRoot, Registered(), DateTime.Now.AddMinutes(5)); + // marked, so reclaimed without needing the backlog pass + Clean(instanceDir, dataRoot); False(LocalDbApi.GetInstance(name).Exists); False(Directory.Exists(instanceDir)); } [Test] - public async Task LeavesRunningInstance() + public async Task LeavesUnmarkedOrphanedInstanceOutsideBacklogPass() { - var name = "RunningInstanceRootTest"; + var name = "UnmarkedOrphanTest"; + LocalDbApi.StopAndDelete(name); + + using var dataRoot = new TempDirectory(); + var directory = Path.Combine(dataRoot, name); + using var wrapper = new Wrapper(name, directory); + wrapper.Start(new(2000, 1, 1), TestDbBuilder.CreateTable); + await wrapper.AwaitStart(); + + var instanceDir = DirectoryFinder.FindInstance(name); + LocalDbApi.StopInstance(name, ShutdownMode.KillProcess); + Directory.Delete(directory, true); + + // an instance that predates marking + File.Delete(Path.Combine(instanceDir, ".localdbwrapper")); + + try + { + Clean(instanceDir, dataRoot); + + True(LocalDbApi.GetInstance(name).Exists); + } + finally + { + DirectoryCleaner.RemoveInstance(name); + } + } + + [Test] + public async Task RemovesUnmarkedOrphanWithShrunkModelInBacklogPass() + { + var name = "BacklogOrphanTest"; LocalDbApi.StopAndDelete(name); using var dataRoot = new TempDirectory(); @@ -150,13 +205,68 @@ public async Task LeavesRunningInstance() await wrapper.AwaitStart(); var instanceDir = DirectoryFinder.FindInstance(name); + LocalDbApi.StopInstance(name, ShutdownMode.KillProcess); + Directory.Delete(directory, true); + File.Delete(Path.Combine(instanceDir, ".localdbwrapper")); + + // this library shrinks model when it creates an instance, so the fingerprint holds + True(InstanceMarker.HasShrunkModel(instanceDir)); + + Clean(instanceDir, dataRoot, backlogPass: true); + + False(LocalDbApi.GetInstance(name).Exists); + False(Directory.Exists(instanceDir)); + } + + [Test] + public void LeavesUnmarkedOrphanWithoutShrunkModelInBacklogPass() + { + var name = "ForeignOrphanTest"; + LocalDbApi.StopAndDelete(name); + + // created through the API alone rather than through a Wrapper, so model is never + // shrunk: this is what an instance belonging to anything else looks like + LocalDbApi.CreateInstance(name); + LocalDbApi.StartInstance(name); + LocalDbApi.StopInstance(name, ShutdownMode.KillProcess); + + var instanceDir = DirectoryFinder.FindInstance(name); + File.Delete(Path.Combine(instanceDir, ".localdbwrapper")); + False(InstanceMarker.HasShrunkModel(instanceDir)); + + using var dataRoot = new TempDirectory(); + try + { + // orphaned, stale, and the backlog pass is running, but it is not ours + Clean(instanceDir, dataRoot, backlogPass: true); + + True(LocalDbApi.GetInstance(name).Exists); + True(Directory.Exists(instanceDir)); + } + finally + { + DirectoryCleaner.RemoveInstance(name); + } + } + + [Test] + public async Task LeavesRunningInstance() + { + var name = "RunningInstanceRootTest"; + LocalDbApi.StopAndDelete(name); + + using var dataRoot = new TempDirectory(); + var directory = Path.Combine(dataRoot, name); + using var wrapper = new Wrapper(name, directory); + wrapper.Start(new(2000, 1, 1), TestDbBuilder.CreateTable); + await wrapper.AwaitStart(); // no data directory and treated as stale, but running: must be left alone Directory.Delete(directory, true); try { - DirectoryCleaner.CleanInstanceDirectory(instanceDir, dataRoot, Registered(), DateTime.Now.AddMinutes(5)); + Clean(DirectoryFinder.FindInstance(name), dataRoot, backlogPass: true); True(LocalDbApi.GetInstance(name).Exists); } @@ -179,11 +289,7 @@ public void LeavesTrackedInstanceWithDataDirectory() try { - DirectoryCleaner.CleanInstanceDirectory( - DirectoryFinder.FindInstance(name), - dataRoot, - Registered(), - DateTime.Now.AddMinutes(5)); + Clean(DirectoryFinder.FindInstance(name), dataRoot, backlogPass: true); True(LocalDbApi.GetInstance(name).Exists); } diff --git a/src/LocalDb.Tests/LocalDbSettingsTests.cs b/src/LocalDb.Tests/LocalDbSettingsTests.cs index 46f804f9..ae0171e8 100644 --- a/src/LocalDb.Tests/LocalDbSettingsTests.cs +++ b/src/LocalDb.Tests/LocalDbSettingsTests.cs @@ -100,28 +100,4 @@ public void InstanceCleanupThreshold_CanBeSetProgrammatically() } } - [Test] - public void InstanceCleanupLimit_DefaultsTo20WhenEnvVarNotSet() - { - var envValue = Environment.GetEnvironmentVariable("LocalDBInstanceCleanupLimit"); - if (envValue is null) - { - That(LocalDbSettings.InstanceCleanupLimit, Is.EqualTo(20)); - } - } - - [Test] - public void InstanceCleanupLimit_CanBeSetProgrammatically() - { - var original = LocalDbSettings.InstanceCleanupLimit; - try - { - LocalDbSettings.InstanceCleanupLimit = 5; - That(LocalDbSettings.InstanceCleanupLimit, Is.EqualTo(5)); - } - finally - { - LocalDbSettings.InstanceCleanupLimit = original; - } - } } diff --git a/src/LocalDb.Tests/ModuleInitializer.cs b/src/LocalDb.Tests/ModuleInitializer.cs index ae4359f1..0a6ee6f7 100644 --- a/src/LocalDb.Tests/ModuleInitializer.cs +++ b/src/LocalDb.Tests/ModuleInitializer.cs @@ -7,7 +7,10 @@ public static void Initialize() LocalDbLogging.EnableVerbose(); // do not let the test suite sweep the real instance root as a side effect: it would // delete instances belonging to other projects on the machine. The sweep is tested - // directly instead. Must be set before DirectoryFinder initializes + // directly instead. Must be set before DirectoryFinder initializes. + // Set as an environment variable as well as a setting, since the multi process tests + // spawn a helper exe that runs none of this and would otherwise sweep with the defaults + Environment.SetEnvironmentVariable("LocalDBInstanceCleanupDays", "0"); LocalDbSettings.InstanceCleanupThreshold = TimeSpan.Zero; LocalDbSettings.ConnectionBuilder(_ => _.ConnectTimeout = 300); VerifierSettings.ScrubLinesContaining("filename = '"); diff --git a/src/LocalDb/DirectoryCleaner.cs b/src/LocalDb/DirectoryCleaner.cs index 3497c720..58dc42f6 100644 --- a/src/LocalDb/DirectoryCleaner.cs +++ b/src/LocalDb/DirectoryCleaner.cs @@ -86,16 +86,25 @@ static void Clean(string directory, bool deleteInstance) /// were already deleted, since LocalDB does not remove the logs and event files on delete. /// /// + /// Only instances marked as created by this library are reclaimed, so an instance belonging to + /// anything else on the machine is never removed. additionally + /// reclaims unmarked directories that predate marking, and is run once per machine. + /// + /// /// Only directories untouched for are processed, both to keep the /// startup scan cheap and to avoid racing an instance that is being created or is in use. /// /// - public static void CleanInstanceRoot(string instanceRoot, string dataRoot, TimeSpan threshold, int limit = 0) + /// + /// Whether the sweep ran to completion. False when it was disabled or could not start, so a + /// caller recording that the backlog pass has happened does not do so for a pass that never ran. + /// + public static bool CleanInstanceRoot(string instanceRoot, string dataRoot, TimeSpan threshold, bool backlogPass) { if (threshold <= TimeSpan.Zero || !Directory.Exists(instanceRoot)) { - return; + return false; } var cutoff = DateTime.Now - threshold; @@ -108,26 +117,14 @@ public static void CleanInstanceRoot(string instanceRoot, string dataRoot, TimeS catch (Exception exception) { LocalDbLogging.LogIfVerbose($"Failed to enumerate instances, skipping instance root cleanup. {exception.Message}"); - return; + return false; } - // each removal is disk work, so a backlog is drained over several runs rather than in - // one startup. Skips are cheap and do not count against the limit - var removed = 0; foreach (var directory in Directory.EnumerateDirectories(instanceRoot)) { - if (limit > 0 && - removed >= limit) - { - break; - } - try { - if (CleanInstanceDirectory(directory, dataRoot, registered, cutoff)) - { - removed++; - } + CleanInstanceDirectory(directory, dataRoot, registered, cutoff, backlogPass); } catch (Exception exception) { @@ -135,47 +132,73 @@ public static void CleanInstanceRoot(string instanceRoot, string dataRoot, TimeS LocalDbLogging.LogIfVerbose($"Failed to clean instance directory: {directory}. {exception.Message}"); } } + + return true; } - internal static bool CleanInstanceDirectory(string directory, string dataRoot, HashSet registered, DateTime cutoff) + internal static void CleanInstanceDirectory( + string directory, + string dataRoot, + HashSet registered, + DateTime cutoff, + bool backlogPass) { var name = Path.GetFileName(directory); // instances LocalDB creates and manages, never owned by this library if (LocalDbCleanup.IsDefaultInstance(name)) { - return false; + return; } // recently touched, so either in use or too new to be sure it is abandoned if (NewestWrite(directory) >= cutoff) { - return false; + return; } + var marked = InstanceMarker.IsMarked(directory); + if (!registered.Contains(name)) { // no instance backs this directory: the instance was already deleted and only the - // logs and event files remain. Safe to remove whatever created it - Directory.Delete(directory, true); - return true; + // logs and event files remain. Nothing can be harmed by removing them, but an + // unmarked one is only taken in the backlog pass since it was not necessarily ours + if (marked || backlogPass) + { + LocalDbLogging.LogIfVerbose($"Removing residue directory: {name}"); + Directory.Delete(directory, true); + } + + return; } // likely in use by another process if (LocalDbApi.GetInstance(name).IsRunning) { - return false; + return; } // still tracked by a data directory, so CleanRoot governs when it is reclaimed if (Directory.Exists(Path.Combine(dataRoot, name))) { - return false; + return; + } + + if (!marked) + { + // predates marking. Only reclaimed in the backlog pass, and only when it still + // carries the shrunk model this library leaves behind, so an instance created by + // anything else is never removed + if (!backlogPass || + !InstanceMarker.HasShrunkModel(directory)) + { + return; + } } LocalDbLogging.LogIfVerbose($"Removing orphaned instance: {name}"); RemoveInstance(name); - return true; } static DateTime NewestWrite(string directory) diff --git a/src/LocalDb/DirectoryFinder.cs b/src/LocalDb/DirectoryFinder.cs index 84c388f9..bb9af310 100644 --- a/src/LocalDb/DirectoryFinder.cs +++ b/src/LocalDb/DirectoryFinder.cs @@ -7,7 +7,44 @@ static DirectoryFinder() dataRoot = FindDataRoot(); var root = dataRoot; DirectoryCleaner.CleanRoot(root); - DirectoryCleaner.CleanInstanceRoot(instanceRoot, root, LocalDbSettings.InstanceCleanupThreshold, LocalDbSettings.InstanceCleanupLimit); + + var threshold = LocalDbSettings.InstanceCleanupThreshold; + if (threshold > TimeSpan.Zero) + { + // instances that predate marking cannot be told apart from instances belonging to + // anything else, so they get one best effort pass and are then left alone + var backlogPass = !File.Exists(backlogFlag); + var swept = DirectoryCleaner.CleanInstanceRoot(instanceRoot, root, threshold, backlogPass); + // only recorded when the pass actually ran, so a sweep that could not start does not + // consume the one chance the unattributable backlog gets + if (backlogPass && swept) + { + WriteBacklogFlag(); + } + } + } + + /// + /// Records that the one time pass over instances predating marking has been run. Persistent, + /// unlike the data directory, so the pass happens once rather than after every temp clear. + /// + static string backlogFlag = Path.Combine( + Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), + "LocalDb", + "backlog-cleaned"); + + static void WriteBacklogFlag() + { + try + { + Directory.CreateDirectory(Path.GetDirectoryName(backlogFlag)!); + File.WriteAllText(backlogFlag, string.Empty); + } + catch (Exception exception) + { + // only costs a repeat of the pass on the next run + LocalDbLogging.LogIfVerbose($"Failed to write backlog flag. {exception.Message}"); + } } public static string Find(string instanceName) => Path.Combine(dataRoot, instanceName); diff --git a/src/LocalDb/InstanceMarker.cs b/src/LocalDb/InstanceMarker.cs new file mode 100644 index 00000000..80f612a1 --- /dev/null +++ b/src/LocalDb/InstanceMarker.cs @@ -0,0 +1,71 @@ +/// +/// Records which LocalDB instances were created by this library. +/// +/// LocalDB instances are shared across everything on the machine that uses LocalDB, and nothing +/// in an instance says what created it. Without a record, reclaiming an instance means guessing, +/// so a marker is written into the directory LocalDB keeps for each instance this library starts. +/// The ongoing cleanup only reclaims marked instances, and so can never remove one belonging to +/// anything else. +/// +/// +/// The marker lives in the instance directory rather than a registry of its own, so it is removed +/// with the instance and cannot drift out of sync. It survives the temp directory being cleared, +/// which is what orphans an instance in the first place. +/// +/// +static class InstanceMarker +{ + const string fileName = ".localdbwrapper"; + + /// + /// Records an instance as created by this library. Best effort: a machine where the marker + /// cannot be written still works, those instances are just never reclaimed automatically. + /// + public static void Mark(string instanceName) + { + try + { + var directory = DirectoryFinder.FindInstance(instanceName); + // LocalDB owns this directory and creates it with the instance + if (!Directory.Exists(directory)) + { + return; + } + + var marker = Path.Combine(directory, fileName); + if (!File.Exists(marker)) + { + File.WriteAllText(marker, string.Empty); + } + } + catch (Exception exception) + { + LocalDbLogging.LogIfVerbose($"Failed to mark instance: {instanceName}. {exception.Message}"); + } + } + + public static bool IsMarked(string instanceDirectory) => + File.Exists(Path.Combine(instanceDirectory, fileName)); + + /// + /// Whether model.mdf has been shrunk below the size a fresh instance has. + /// + /// Only used for the one time pass over instances that predate marking. This library shrinks + /// model when it creates an instance, and nothing else does, so a smaller model.mdf means the + /// instance was created by it. Instances left at the default size are not matched, so the + /// pass can never remove an instance belonging to anything else. + /// + /// + public static bool HasShrunkModel(string instanceDirectory) + { + var model = Path.Combine(instanceDirectory, "model.mdf"); + if (!File.Exists(model)) + { + return false; + } + + var length = new FileInfo(model).Length; + return length > 0 && + length < 8 * 1024 * 1024; + } +} diff --git a/src/LocalDb/LocalDbApi.cs b/src/LocalDb/LocalDbApi.cs index 3edce438..297377d5 100644 --- a/src/LocalDb/LocalDbApi.cs +++ b/src/LocalDb/LocalDbApi.cs @@ -164,8 +164,11 @@ public static void StopAndDelete(string instanceName, ShutdownMode mode, TimeSpa DeleteInstance(instanceName); } - public static void CreateInstance(string instanceName) => + public static void CreateInstance(string instanceName) + { createInstance(ApiVersion, instanceName, 0); + InstanceMarker.Mark(instanceName); + } public static State CreateAndStart(string instance) { @@ -192,6 +195,9 @@ public static void StartInstance(string instanceName) var size = connection.Capacity; startInstance(instanceName, 0, connection, ref size); + // also marked here, not just on create, so instances created before marking existed + // are picked up the next time they are used + InstanceMarker.Mark(instanceName); } public static void StopInstance(string instanceName, ShutdownMode mode = ShutdownMode.UseSqlShutdown) => diff --git a/src/LocalDb/Settings.cs b/src/LocalDb/Settings.cs index 3c5c76f0..26375b3c 100644 --- a/src/LocalDb/Settings.cs +++ b/src/LocalDb/Settings.cs @@ -46,14 +46,6 @@ internal static string BuildConnectionString(string instance, string database, b /// public static TimeSpan InstanceCleanupThreshold { get; set; } = ResolveInstanceCleanupThreshold(); - /// - /// The maximum number of instances and directories the automatic cleanup removes per run, - /// so a large backlog is drained over several runs rather than in a single startup. - /// Can be configured via the LocalDBInstanceCleanupLimit environment variable. - /// Defaults to 20. Set to 0 for no limit. - /// - public static ushort InstanceCleanupLimit { get; set; } = ResolveInstanceCleanupLimit(); - static ushort ResolveShutdownTimeout() { var envValue = Environment.GetEnvironmentVariable("LocalDBShutdownTimeout"); @@ -96,20 +88,4 @@ static TimeSpan ResolveInstanceCleanupThreshold() throw new ArgumentException($"Failed to parse LocalDBInstanceCleanupDays environment variable: {envValue}"); } - - static ushort ResolveInstanceCleanupLimit() - { - var envValue = Environment.GetEnvironmentVariable("LocalDBInstanceCleanupLimit"); - if (envValue is null) - { - return 20; - } - - if (ushort.TryParse(envValue, out var limit)) - { - return limit; - } - - throw new ArgumentException($"Failed to parse LocalDBInstanceCleanupLimit environment variable: {envValue}"); - } } From 6bf5243fd492a681ae142a02545b90d34bd9e6a5 Mon Sep 17 00:00:00 2001 From: Simon Cropp Date: Tue, 21 Jul 2026 12:06:43 +1000 Subject: [PATCH 4/6] Remove LocalDbCleanup now that cleanup is automatic The only instances DeleteOrphanInstances reclaimed that the automatic sweep does not are the unmarked ones that carry no sign of having been created here. Removing those is exactly what the marker exists to prevent, so the API contradicted the model it now sits behind. It was also never able to clean up after itself: it enumerated registered instances, so directories left behind by already deleted instances were invisible to it, and those are the bulk of what accumulates. Anything the one time pass deliberately leaves is now left for SqlLocalDB.exe delete. Residue gets a short threshold of its own rather than the instance threshold. It has no instance to race, only the window where LocalDB has created the directory but not yet registered the instance, and the full threshold would strand any residue not yet old enough when the backlog pass runs, since that pass only happens once. IsDefaultInstance moves to DirectoryCleaner, its only remaining caller. --- pages/directory-and-name-resolution.md | 2 +- .../directory-and-name-resolution.source.md | 2 +- src/EfClassicLocalDb/EfClassicLocalDb.csproj | 1 - src/EfLocalDb/EfLocalDb.csproj | 1 - src/LocalDb.Tests/InstanceRootCleanerTests.cs | 24 ++++- src/LocalDb.Tests/LocalDbCleanupTests.cs | 88 ------------------- src/LocalDb/DirectoryCleaner.cs | 37 ++++++-- src/LocalDb/LocalDbCleanup.cs | 84 ------------------ 8 files changed, 53 insertions(+), 186 deletions(-) delete mode 100644 src/LocalDb.Tests/LocalDbCleanupTests.cs delete mode 100644 src/LocalDb/LocalDbCleanup.cs diff --git a/pages/directory-and-name-resolution.md b/pages/directory-and-name-resolution.md index dee2973f..eee58713 100644 --- a/pages/directory-and-name-resolution.md +++ b/pages/directory-and-name-resolution.md @@ -71,7 +71,7 @@ The per run purge above only sees instances that still have a data directory. On LocalDB instances are shared by everything on the machine that uses LocalDB, and nothing in an instance records what created it. So a marker file is written into the directory of each instance this library starts, and the sweep only ever reclaims marked instances. An instance created by anything else has no marker and is never removed. A marked instance is reclaimed once it has no data directory, is not running, and has been untouched for a threshold defaulting to 30 days. Configurable via the `LocalDBInstanceCleanupDays` environment variable, or `LocalDbSettings.InstanceCleanupThreshold`; set it to zero to disable the sweep. -Instances that predate marking cannot be told apart from instances belonging to anything else. They get a single best effort pass, once per machine, which removes directories left behind by already deleted instances and orphaned instances that still carry the shrunk `model.mdf` this library leaves behind. Anything else is left alone. That the pass has run is recorded under `%LocalAppData%\LocalDb`. To reclaim the remainder, call `LocalDbCleanup.DeleteOrphanInstances`. +Instances that predate marking cannot be told apart from instances belonging to anything else. They get a single best effort pass, once per machine, which removes directories left behind by already deleted instances and orphaned instances that still carry the shrunk `model.mdf` this library leaves behind. Anything else is left alone, since there is no way to tell it apart from an instance belonging to something else, and is left for `SqlLocalDB.exe delete` to remove by hand. That the pass has run is recorded under `%LocalAppData%\LocalDb`. ## Virus scanning exclusions diff --git a/pages/mdsource/directory-and-name-resolution.source.md b/pages/mdsource/directory-and-name-resolution.source.md index 1bcaa171..949fac67 100644 --- a/pages/mdsource/directory-and-name-resolution.source.md +++ b/pages/mdsource/directory-and-name-resolution.source.md @@ -36,7 +36,7 @@ The per run purge above only sees instances that still have a data directory. On LocalDB instances are shared by everything on the machine that uses LocalDB, and nothing in an instance records what created it. So a marker file is written into the directory of each instance this library starts, and the sweep only ever reclaims marked instances. An instance created by anything else has no marker and is never removed. A marked instance is reclaimed once it has no data directory, is not running, and has been untouched for a threshold defaulting to 30 days. Configurable via the `LocalDBInstanceCleanupDays` environment variable, or `LocalDbSettings.InstanceCleanupThreshold`; set it to zero to disable the sweep. -Instances that predate marking cannot be told apart from instances belonging to anything else. They get a single best effort pass, once per machine, which removes directories left behind by already deleted instances and orphaned instances that still carry the shrunk `model.mdf` this library leaves behind. Anything else is left alone. That the pass has run is recorded under `%LocalAppData%\LocalDb`. To reclaim the remainder, call `LocalDbCleanup.DeleteOrphanInstances`. +Instances that predate marking cannot be told apart from instances belonging to anything else. They get a single best effort pass, once per machine, which removes directories left behind by already deleted instances and orphaned instances that still carry the shrunk `model.mdf` this library leaves behind. Anything else is left alone, since there is no way to tell it apart from an instance belonging to something else, and is left for `SqlLocalDB.exe delete` to remove by hand. That the pass has run is recorded under `%LocalAppData%\LocalDb`. ## Virus scanning exclusions diff --git a/src/EfClassicLocalDb/EfClassicLocalDb.csproj b/src/EfClassicLocalDb/EfClassicLocalDb.csproj index 6cf60179..9f671cee 100644 --- a/src/EfClassicLocalDb/EfClassicLocalDb.csproj +++ b/src/EfClassicLocalDb/EfClassicLocalDb.csproj @@ -17,7 +17,6 @@ - diff --git a/src/EfLocalDb/EfLocalDb.csproj b/src/EfLocalDb/EfLocalDb.csproj index e70ceb69..fabb6320 100644 --- a/src/EfLocalDb/EfLocalDb.csproj +++ b/src/EfLocalDb/EfLocalDb.csproj @@ -20,7 +20,6 @@ - diff --git a/src/LocalDb.Tests/InstanceRootCleanerTests.cs b/src/LocalDb.Tests/InstanceRootCleanerTests.cs index 0ef50776..b6266a52 100644 --- a/src/LocalDb.Tests/InstanceRootCleanerTests.cs +++ b/src/LocalDb.Tests/InstanceRootCleanerTests.cs @@ -62,10 +62,12 @@ public void RemovesUnmarkedResidueInBacklogPass() } [Test] - public void LeavesRecentDirectory() + public void LeavesJustWrittenResidue() { using var instanceRoot = new TempDirectory(); using var dataRoot = new TempDirectory(); + // guards the window where LocalDB has created the directory but not yet registered + // the instance, so it would look like residue var dir = Path.Combine(instanceRoot, "RecentResidueTest"); Directory.CreateDirectory(dir); File.WriteAllText(Path.Combine(dir, ".localdbwrapper"), ""); @@ -75,6 +77,26 @@ public void LeavesRecentDirectory() True(Directory.Exists(dir)); } + [Test] + public void RemovesResidueYoungerThanTheInstanceThreshold() + { + using var instanceRoot = new TempDirectory(); + using var dataRoot = new TempDirectory(); + // an hour old: well inside the instance threshold, but residue has no instance to + // race, so it must not be stranded by the backlog pass only running once + var dir = Path.Combine(instanceRoot, "YoungResidueTest"); + Directory.CreateDirectory(dir); + var file = Path.Combine(dir, "error.log"); + File.WriteAllText(file, "x"); + var hourAgo = DateTime.Now.AddHours(-1); + File.SetLastWriteTime(file, hourAgo); + Directory.SetCreationTime(dir, hourAgo); + + DirectoryCleaner.CleanInstanceRoot(instanceRoot, dataRoot, threshold, backlogPass: true); + + False(Directory.Exists(dir)); + } + [Test] public void LeavesDefaultInstanceDirectory() { diff --git a/src/LocalDb.Tests/LocalDbCleanupTests.cs b/src/LocalDb.Tests/LocalDbCleanupTests.cs deleted file mode 100644 index 08d7d2e7..00000000 --- a/src/LocalDb.Tests/LocalDbCleanupTests.cs +++ /dev/null @@ -1,88 +0,0 @@ -[TestFixture] -public class LocalDbCleanupTests -{ - static void CreateStoppedInstance(string name) - { - LocalDbApi.StopAndDelete(name); - DirectoryFinder.Delete(name); - - LocalDbApi.CreateInstance(name); - // starting materializes the directory LocalDB keeps for the instance - LocalDbApi.StartInstance(name); - LocalDbApi.StopInstance(name, ShutdownMode.KillProcess); - } - - [Test] - public void FindsInstanceWithNoDataDirectory() - { - var name = "OrphanCleanupTest"; - CreateStoppedInstance(name); - - try - { - True(LocalDbCleanup.FindOrphanInstances().Contains(name)); - } - finally - { - DirectoryCleaner.RemoveInstance(name); - } - } - - [Test] - public void SkipsInstanceWithDataDirectory() - { - var name = "OrphanCleanupWithDirectoryTest"; - CreateStoppedInstance(name); - Directory.CreateDirectory(DirectoryFinder.Find(name)); - - try - { - False(LocalDbCleanup.FindOrphanInstances().Contains(name)); - } - finally - { - DirectoryFinder.Delete(name); - DirectoryCleaner.RemoveInstance(name); - } - } - - [Test] - public void SkipsDefaultInstances() => - False(LocalDbCleanup.FindOrphanInstances().Contains("MSSQLLocalDB")); - - [Test] - public void DeletesInstanceAndDirectory() - { - var name = "OrphanCleanupDeleteTest"; - CreateStoppedInstance(name); - - var instanceDirectory = DirectoryFinder.FindInstance(name); - True(Directory.Exists(instanceDirectory)); - - // the filter keeps the sweep away from instances this test does not own - var deleted = LocalDbCleanup.DeleteOrphanInstances(_ => _ == name); - - True(deleted.Contains(name)); - False(LocalDbApi.GetInstance(name).Exists); - False(Directory.Exists(instanceDirectory)); - } - - [Test] - public void FilterKeepsUnmatchedInstances() - { - var name = "OrphanCleanupFilterTest"; - CreateStoppedInstance(name); - - try - { - var deleted = LocalDbCleanup.DeleteOrphanInstances(_ => false); - - False(deleted.Any()); - True(LocalDbApi.GetInstance(name).Exists); - } - finally - { - DirectoryCleaner.RemoveInstance(name); - } - } -} diff --git a/src/LocalDb/DirectoryCleaner.cs b/src/LocalDb/DirectoryCleaner.cs index 58dc42f6..304289d5 100644 --- a/src/LocalDb/DirectoryCleaner.cs +++ b/src/LocalDb/DirectoryCleaner.cs @@ -91,8 +91,9 @@ static void Clean(string directory, bool deleteInstance) /// reclaims unmarked directories that predate marking, and is run once per machine. /// /// - /// Only directories untouched for are processed, both to keep the + /// Only instances untouched for are processed, both to keep the /// startup scan cheap and to avoid racing an instance that is being created or is in use. + /// Residue has no instance to race, so it uses a short threshold of its own. /// /// /// @@ -136,6 +137,22 @@ public static bool CleanInstanceRoot(string instanceRoot, string dataRoot, TimeS return true; } + /// + /// Instances that LocalDB creates and manages itself, and that are never owned by this library. + /// + static bool IsDefaultInstance(string name) => + name == "MSSQLLocalDB" || + // automatic instances, eg "v11.0" + name.Length > 1 && name[0] == 'v' && char.IsDigit(name[1]); + + /// + /// Residue is only guarded against the window where LocalDB has created the directory for an + /// instance but not yet registered it, so it uses a short threshold of its own. Applying the + /// full threshold would strand any residue not yet old enough when the backlog pass runs, + /// since that pass only happens once. + /// + static readonly TimeSpan residueThreshold = TimeSpan.FromMinutes(5); + internal static void CleanInstanceDirectory( string directory, string dataRoot, @@ -146,17 +163,12 @@ internal static void CleanInstanceDirectory( var name = Path.GetFileName(directory); // instances LocalDB creates and manages, never owned by this library - if (LocalDbCleanup.IsDefaultInstance(name)) - { - return; - } - - // recently touched, so either in use or too new to be sure it is abandoned - if (NewestWrite(directory) >= cutoff) + if (IsDefaultInstance(name)) { return; } + var newestWrite = NewestWrite(directory); var marked = InstanceMarker.IsMarked(directory); if (!registered.Contains(name)) @@ -164,7 +176,8 @@ internal static void CleanInstanceDirectory( // no instance backs this directory: the instance was already deleted and only the // logs and event files remain. Nothing can be harmed by removing them, but an // unmarked one is only taken in the backlog pass since it was not necessarily ours - if (marked || backlogPass) + if ((marked || backlogPass) && + newestWrite < DateTime.Now - residueThreshold) { LocalDbLogging.LogIfVerbose($"Removing residue directory: {name}"); Directory.Delete(directory, true); @@ -173,6 +186,12 @@ internal static void CleanInstanceDirectory( return; } + // recently touched, so either in use or too new to be sure it is abandoned + if (newestWrite >= cutoff) + { + return; + } + // likely in use by another process if (LocalDbApi.GetInstance(name).IsRunning) { diff --git a/src/LocalDb/LocalDbCleanup.cs b/src/LocalDb/LocalDbCleanup.cs deleted file mode 100644 index 300a1a4b..00000000 --- a/src/LocalDb/LocalDbCleanup.cs +++ /dev/null @@ -1,84 +0,0 @@ -#if EF -namespace EfLocalDb; -#else -namespace LocalDb; -#endif - -/// -/// Immediately removes LocalDB instances left behind by previous runs. -/// -/// The same instances are removed automatically once they have been untouched for -/// . This is the forced version, for when -/// they should be reclaimed now rather than waiting for that threshold. -/// -/// -public static class LocalDbCleanup -{ - /// - /// Instances that LocalDB creates and manages itself, and that are never owned by this library. - /// - internal static bool IsDefaultInstance(string name) => - name == "MSSQLLocalDB" || - // automatic instances, eg "v11.0" - name.Length > 1 && name[0] == 'v' && char.IsDigit(name[1]); - - /// - /// Finds instances that have no corresponding data directory. Running instances and the - /// LocalDB managed default instances are never treated as orphans. - /// - /// - /// Applied to each candidate name. Return false to keep an instance. Use this when the machine - /// has LocalDB instances that were not created by this library. - /// - public static IReadOnlyList FindOrphanInstances(Func? filter = null) - { - var orphans = new List(); - foreach (var name in LocalDbApi.GetInstanceNames()) - { - if (IsDefaultInstance(name)) - { - continue; - } - - if (Directory.Exists(DirectoryFinder.Find(name))) - { - continue; - } - - // an instance that is currently running is likely in use by another process - if (LocalDbApi.GetInstance(name).IsRunning) - { - continue; - } - - if (filter != null && !filter(name)) - { - continue; - } - - orphans.Add(name); - } - - return orphans; - } - - /// - /// Deletes the instances returned by and the directories - /// LocalDB keeps for them. Returns the names of the deleted instances. - /// - /// - /// Applied to each candidate name. Return false to keep an instance. Use this when the machine - /// has LocalDB instances that were not created by this library. - /// - public static IReadOnlyList DeleteOrphanInstances(Func? filter = null) - { - var orphans = FindOrphanInstances(filter); - foreach (var orphan in orphans) - { - LocalDbLogging.LogIfVerbose($"Deleting orphan instance: {orphan}"); - DirectoryCleaner.RemoveInstance(orphan); - } - - return orphans; - } -} From 4c0851e9334adc14c72808cafeee1e4bca09be65 Mon Sep 17 00:00:00 2001 From: Simon Cropp Date: Tue, 21 Jul 2026 12:20:47 +1000 Subject: [PATCH 5/6] Document the instance root cleanup The readme described the data root cleanup as stopping a running instance before deleting its files. It has stopped, deleted, and removed the directory LocalDB keeps for the instance since that cleanup was changed, so both the prose and the flowchart node understated what it does. The sweep of the directory LocalDB keeps for each instance was not documented at all, though it is where the growth being reclaimed actually accumulates. Adds a flowchart for the decision it makes per directory: whether the directory is LocalDB managed, backed by a registered instance or only residue, marked as created here, still tracked by a data directory, and whether the one time backlog pass applies. --- readme.md | 87 ++++++++++++++++++++++++++++++++++++++++++++++-- readme.source.md | 86 +++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 167 insertions(+), 6 deletions(-) diff --git a/readme.md b/readme.md index e5053088..c316559c 100644 --- a/readme.md +++ b/readme.md @@ -59,6 +59,7 @@ Provides a wrapper around [SqlLocalDB](https://docs.microsoft.com/en-us/sql/data * [Inputs](#inputs) * [SqlInstance Startup Flow](#sqlinstance-startup-flow) * [Clean Directory Flow](#clean-directory-flow) + * [Clean Instance Root Flow](#clean-instance-root-flow) * [Create SqlDatabase Flow](#create-sqldatabase-flow) * [Performance](#performance) * [Hardware](#hardware) @@ -353,7 +354,7 @@ flowchart TD ### Clean Directory Flow -On first access, the library scans the data root directory (`%TEMP%\LocalDb` by default) and cleans up stale database files. This prevents unbounded disk growth from old test runs. +On first access, the library scans the data root directory (`%TEMP%\LocalDb` by default) and cleans up stale database files. This prevents unbounded disk growth from old test runs. The directory LocalDB keeps for each instance is swept separately, see [Clean Instance Root Flow](#clean-instance-root-flow). ```mermaid flowchart TD @@ -366,7 +367,7 @@ flowchart TD deleteDirEmpty[Delete Directory] findNewest[Find Newest
File Write Time] checkNewest{Newest File
Older Than
6 Hours?} - stopInstance[Stop LocalDB
Instance If Running] + stopInstance[Remove Instance:
Stop, Delete, and Remove
Its LocalDB Directory] deleteAllFiles[Delete All Files] checkEmptyAfter{Directory
Empty?} deleteDirAfter[Delete Directory] @@ -397,9 +398,89 @@ Key behaviors: * **Triggered once** during `DirectoryFinder` static initialization, before any `SqlInstance` starts. * **All-or-nothing cleanup**: The newest file write time in a directory determines whether the entire instance is stale. If any file is recent, nothing is touched. This avoids partially cleaning an active instance. * **6-hour cutoff**: An instance directory is only cleaned if all its files have a last-write time older than 6 hours. - * **Stops running instances**: If stale files belong to a running LocalDB instance, the instance is stopped (via `KillProcess`) before deletion. This prevents `UnauthorizedAccessException` from locked `.mdf`/`.ldf` files. + * **Removes the instance, not only the files**: If stale files belong to a LocalDB instance, the instance is stopped (via `KillProcess`), deleted, and the directory LocalDB keeps for it removed. Stopping first prevents `UnauthorizedAccessException` from locked `.mdf`/`.ldf` files. Removing the instance along with its data means it never becomes an orphan that nothing under the data root points at. * **Empty directory cleanup**: Empty directories older than 6 hours are removed. * The instance name is derived from the directory name (e.g. `%TEMP%\LocalDb\MyTestInstance` → instance name `MyTestInstance`). + * The instance that is about to be started is treated differently: only its stale data files are removed, and the instance is left in place to be reused, since recreating one is significantly slower than restarting it. + + +### Clean Instance Root Flow + +LocalDB keeps a directory of its own for every instance at `%LocalAppData%\Microsoft\Microsoft SQL Server Local DB\Instances\InstanceName`, holding the system databases (`master`, `model`, `msdb`, `tempdb`), the error logs, and the extended event files. That location is owned by LocalDB and cannot be moved. + +The flow above only sees instances that still have a data directory. Once that directory is gone, cleared with the temp directory or by the flow above, nothing under the data root points at the instance and it is invisible there. Those orphans, and the directories left behind by instances that were already deleted, are reclaimed by this second sweep. + +LocalDB instances are shared by everything on the machine that uses LocalDB, and nothing in an instance records what created it. So a marker file is written into the directory of each instance this library starts, and the sweep only reclaims instances carrying that marker. + +```mermaid +flowchart TD + start[Start: CleanInstanceRoot] + checkEnabled{Cleanup Enabled?
threshold not zero} + checkRootExists{Instance Root
Exists?} + iterateDirs[Iterate Instance
Directories] + checkDefault{LocalDB Managed?
MSSQLLocalDB, v11.0} + checkRegistered{Backed By A
Registered Instance?} + + checkResidueOwn{Marked, Or
Backlog Pass?} + checkResidueAge{Untouched For
5 Minutes?} + deleteResidue[Delete Directory] + + checkAge{Untouched For
Threshold?
default 30 days} + checkRunning{Running?} + checkDataDir{Has A Data
Directory?} + checkMarked{Marked As Created
By This Library?} + checkBacklog{Backlog Pass?} + checkModel{model.mdf
Shrunk?} + removeInstance[Remove Instance:
Stop, Delete, and Remove
Its LocalDB Directory] + + leave[Leave Alone] + done[Done] + + start --> checkEnabled + checkEnabled -->|No| done + checkEnabled -->|Yes| checkRootExists + checkRootExists -->|No| done + checkRootExists -->|Yes| iterateDirs + iterateDirs --> checkDefault + checkDefault -->|Yes| leave + checkDefault -->|No| checkRegistered + + checkRegistered -->|No: residue| checkResidueOwn + checkResidueOwn -->|No| leave + checkResidueOwn -->|Yes| checkResidueAge + checkResidueAge -->|No| leave + checkResidueAge -->|Yes| deleteResidue + deleteResidue --> done + + checkRegistered -->|Yes| checkAge + checkAge -->|No| leave + checkAge -->|Yes| checkRunning + checkRunning -->|Yes| leave + checkRunning -->|No| checkDataDir + checkDataDir -->|Yes: handled above| leave + checkDataDir -->|No: orphan| checkMarked + checkMarked -->|Yes| removeInstance + checkMarked -->|No| checkBacklog + checkBacklog -->|No| leave + checkBacklog -->|Yes| checkModel + checkModel -->|No| leave + checkModel -->|Yes| removeInstance + + removeInstance --> done + leave --> done +``` + +Key behaviors: + + * **Triggered once** during `DirectoryFinder` static initialization, immediately after the data root cleanup above. + * **Only reclaims what it created**: an instance with no marker is never removed, so an instance belonging to anything else on the machine is left alone. The default instances LocalDB manages itself are always skipped. + * **Marked on start as well as on create**: instances that predate marking are picked up the next time they are used. + * **An instance is only an orphan if it has no data directory**: while one exists, the data root cleanup above governs when it is reclaimed, so the two never contend. + * **Running instances are skipped**: they are likely in use by another process. + * **30-day threshold** for instances, configurable via the `LocalDBInstanceCleanupDays` environment variable or `LocalDbSettings.InstanceCleanupThreshold`. Set it to zero to disable the sweep. + * **Residue uses a 5-minute threshold** instead. A directory with no registered instance behind it has no instance to race, only the window where LocalDB has created the directory but not yet registered the instance. Using the full threshold would strand any residue not old enough when the backlog pass runs, since that pass only happens once. + * **The backlog pass runs once per machine**: instances that predate marking cannot be told apart from instances belonging to anything else, so they get a single best effort pass which takes unmarked residue and unmarked orphans that still carry the shrunk `model.mdf` this library leaves behind. A fresh instance leaves `model.mdf` at 8MB, so an instance created by anything else is never matched. That the pass has run is recorded under `%LocalAppData%\LocalDb`, and only once a sweep actually completes. + * **Never fails a test run**: every directory is handled independently and any failure is logged and skipped. ### Create SqlDatabase Flow diff --git a/readme.source.md b/readme.source.md index c7938148..05a7cb7e 100644 --- a/readme.source.md +++ b/readme.source.md @@ -307,7 +307,7 @@ flowchart TD ### Clean Directory Flow -On first access, the library scans the data root directory (`%TEMP%\LocalDb` by default) and cleans up stale database files. This prevents unbounded disk growth from old test runs. +On first access, the library scans the data root directory (`%TEMP%\LocalDb` by default) and cleans up stale database files. This prevents unbounded disk growth from old test runs. The directory LocalDB keeps for each instance is swept separately, see [Clean Instance Root Flow](#clean-instance-root-flow). ```mermaid flowchart TD @@ -320,7 +320,7 @@ flowchart TD deleteDirEmpty[Delete Directory] findNewest[Find Newest
File Write Time] checkNewest{Newest File
Older Than
6 Hours?} - stopInstance[Stop LocalDB
Instance If Running] + stopInstance[Remove Instance:
Stop, Delete, and Remove
Its LocalDB Directory] deleteAllFiles[Delete All Files] checkEmptyAfter{Directory
Empty?} deleteDirAfter[Delete Directory] @@ -351,9 +351,89 @@ Key behaviors: * **Triggered once** during `DirectoryFinder` static initialization, before any `SqlInstance` starts. * **All-or-nothing cleanup**: The newest file write time in a directory determines whether the entire instance is stale. If any file is recent, nothing is touched. This avoids partially cleaning an active instance. * **6-hour cutoff**: An instance directory is only cleaned if all its files have a last-write time older than 6 hours. - * **Stops running instances**: If stale files belong to a running LocalDB instance, the instance is stopped (via `KillProcess`) before deletion. This prevents `UnauthorizedAccessException` from locked `.mdf`/`.ldf` files. + * **Removes the instance, not only the files**: If stale files belong to a LocalDB instance, the instance is stopped (via `KillProcess`), deleted, and the directory LocalDB keeps for it removed. Stopping first prevents `UnauthorizedAccessException` from locked `.mdf`/`.ldf` files. Removing the instance along with its data means it never becomes an orphan that nothing under the data root points at. * **Empty directory cleanup**: Empty directories older than 6 hours are removed. * The instance name is derived from the directory name (e.g. `%TEMP%\LocalDb\MyTestInstance` → instance name `MyTestInstance`). + * The instance that is about to be started is treated differently: only its stale data files are removed, and the instance is left in place to be reused, since recreating one is significantly slower than restarting it. + + +### Clean Instance Root Flow + +LocalDB keeps a directory of its own for every instance at `%LocalAppData%\Microsoft\Microsoft SQL Server Local DB\Instances\InstanceName`, holding the system databases (`master`, `model`, `msdb`, `tempdb`), the error logs, and the extended event files. That location is owned by LocalDB and cannot be moved. + +The flow above only sees instances that still have a data directory. Once that directory is gone, cleared with the temp directory or by the flow above, nothing under the data root points at the instance and it is invisible there. Those orphans, and the directories left behind by instances that were already deleted, are reclaimed by this second sweep. + +LocalDB instances are shared by everything on the machine that uses LocalDB, and nothing in an instance records what created it. So a marker file is written into the directory of each instance this library starts, and the sweep only reclaims instances carrying that marker. + +```mermaid +flowchart TD + start[Start: CleanInstanceRoot] + checkEnabled{Cleanup Enabled?
threshold not zero} + checkRootExists{Instance Root
Exists?} + iterateDirs[Iterate Instance
Directories] + checkDefault{LocalDB Managed?
MSSQLLocalDB, v11.0} + checkRegistered{Backed By A
Registered Instance?} + + checkResidueOwn{Marked, Or
Backlog Pass?} + checkResidueAge{Untouched For
5 Minutes?} + deleteResidue[Delete Directory] + + checkAge{Untouched For
Threshold?
default 30 days} + checkRunning{Running?} + checkDataDir{Has A Data
Directory?} + checkMarked{Marked As Created
By This Library?} + checkBacklog{Backlog Pass?} + checkModel{model.mdf
Shrunk?} + removeInstance[Remove Instance:
Stop, Delete, and Remove
Its LocalDB Directory] + + leave[Leave Alone] + done[Done] + + start --> checkEnabled + checkEnabled -->|No| done + checkEnabled -->|Yes| checkRootExists + checkRootExists -->|No| done + checkRootExists -->|Yes| iterateDirs + iterateDirs --> checkDefault + checkDefault -->|Yes| leave + checkDefault -->|No| checkRegistered + + checkRegistered -->|No: residue| checkResidueOwn + checkResidueOwn -->|No| leave + checkResidueOwn -->|Yes| checkResidueAge + checkResidueAge -->|No| leave + checkResidueAge -->|Yes| deleteResidue + deleteResidue --> done + + checkRegistered -->|Yes| checkAge + checkAge -->|No| leave + checkAge -->|Yes| checkRunning + checkRunning -->|Yes| leave + checkRunning -->|No| checkDataDir + checkDataDir -->|Yes: handled above| leave + checkDataDir -->|No: orphan| checkMarked + checkMarked -->|Yes| removeInstance + checkMarked -->|No| checkBacklog + checkBacklog -->|No| leave + checkBacklog -->|Yes| checkModel + checkModel -->|No| leave + checkModel -->|Yes| removeInstance + + removeInstance --> done + leave --> done +``` + +Key behaviors: + + * **Triggered once** during `DirectoryFinder` static initialization, immediately after the data root cleanup above. + * **Only reclaims what it created**: an instance with no marker is never removed, so an instance belonging to anything else on the machine is left alone. The default instances LocalDB manages itself are always skipped. + * **Marked on start as well as on create**: instances that predate marking are picked up the next time they are used. + * **An instance is only an orphan if it has no data directory**: while one exists, the data root cleanup above governs when it is reclaimed, so the two never contend. + * **Running instances are skipped**: they are likely in use by another process. + * **30-day threshold** for instances, configurable via the `LocalDBInstanceCleanupDays` environment variable or `LocalDbSettings.InstanceCleanupThreshold`. Set it to zero to disable the sweep. + * **Residue uses a 5-minute threshold** instead. A directory with no registered instance behind it has no instance to race, only the window where LocalDB has created the directory but not yet registered the instance. Using the full threshold would strand any residue not old enough when the backlog pass runs, since that pass only happens once. + * **The backlog pass runs once per machine**: instances that predate marking cannot be told apart from instances belonging to anything else, so they get a single best effort pass which takes unmarked residue and unmarked orphans that still carry the shrunk `model.mdf` this library leaves behind. A fresh instance leaves `model.mdf` at 8MB, so an instance created by anything else is never matched. That the pass has run is recorded under `%LocalAppData%\LocalDb`, and only once a sweep actually completes. + * **Never fails a test run**: every directory is handled independently and any failure is logged and skipped. ### Create SqlDatabase Flow From c4a3fef53e43c64b9393988d221072c2a2c121e5 Mon Sep 17 00:00:00 2001 From: Simon Cropp Date: Tue, 21 Jul 2026 12:34:31 +1000 Subject: [PATCH 6/6] . --- src/Directory.Build.props | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/Directory.Build.props b/src/Directory.Build.props index 2565135a..ceca605d 100644 --- a/src/Directory.Build.props +++ b/src/Directory.Build.props @@ -2,7 +2,7 @@ CS1591;CA1416;CS8632;NU1608;NU1109 - 24.1.5 + 25.0.0 preview 1.0.0 false