You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Lite periodic archive: a kill between the parquet promote and the DELETE archives the same rows twice #4720
Lite's periodic archive exports a table's old rows to parquet, promotes the file, and then deletes those rows from DuckDB. If the process is killed between the promote and the DELETE, the next run exports the same rows again. The archive then holds them twice, permanently, and the archive views count them twice.
Where: Lite/Services/ArchiveService.cs, the periodic archive path (:148-170 at 1248a67).
Expected: the export is idempotent. Record a per-table "archived through" cutoff after the promote, and clear it after the DELETE. The next run first deletes rows older than a recorded cutoff, then exports.
Test that must fail first: a kill between the promote and the DELETE leaves no duplicate rows after the next run.
These are small gaps next to #4718's archive fixes, in the same file:
A process killed during a COPY leaves its partial .tmp file in the archive folder, and nothing removes it. Remove stale .tmp files from the archive folder at startup, before any archive work runs.
If CompactParquetFiles throws, its callers skip the archive view rebuild, so a view can keep a glob that matches nothing until the next hourly run. Rebuild the views in a finally.
Add a test that the background service's retention step rebuilds the views after it deletes files.
Compaction swaps the merged file in and then rebuilds the views, without the write lock. A reader in between can hit a glob that matches nothing. Hold the write lock around the swap and the rebuild, but not the merge. Readers then wait milliseconds, not the whole compaction.
Lite's periodic archive exports a table's old rows to parquet, promotes the file, and then deletes those rows from DuckDB. If the process is killed between the promote and the DELETE, the next run exports the same rows again. The archive then holds them twice, permanently, and the archive views count them twice.
Lite/Services/ArchiveService.cs, the periodic archive path (:148-170at 1248a67).Also in the same change
These are small gaps next to #4718's archive fixes, in the same file:
.tmpfile in the archive folder, and nothing removes it. Remove stale.tmpfiles from the archive folder at startup, before any archive work runs.CompactParquetFilesthrows, its callers skip the archive view rebuild, so a view can keep a glob that matches nothing until the next hourly run. Rebuild the views in afinally.