Skip to content

Lite periodic archive: a kill between the parquet promote and the DELETE archives the same rows twice #4720

Description

@erikdarlingdata

Lite's periodic archive exports a table's old rows to parquet, promotes the file, and then deletes those rows from DuckDB. If the process is killed between the promote and the DELETE, the next run exports the same rows again. The archive then holds them twice, permanently, and the archive views count them twice.

  • Where: Lite/Services/ArchiveService.cs, the periodic archive path (:148-170 at 1248a67).
  • Expected: the export is idempotent. Record a per-table "archived through" cutoff after the promote, and clear it after the DELETE. The next run first deletes rows older than a recorded cutoff, then exports.
  • Test that must fail first: a kill between the promote and the DELETE leaves no duplicate rows after the next run.
  • Related: Lite archive: keep every row through compaction, the size reset and tab Refresh #4718 made compaction, the size reset and Refresh keep every row. This path is separate from those.

Also in the same change

These are small gaps next to #4718's archive fixes, in the same file:

  • A process killed during a COPY leaves its partial .tmp file in the archive folder, and nothing removes it. Remove stale .tmp files from the archive folder at startup, before any archive work runs.
  • If CompactParquetFiles throws, its callers skip the archive view rebuild, so a view can keep a glob that matches nothing until the next hourly run. Rebuild the views in a finally.
  • Add a test that the background service's retention step rebuilds the views after it deletes files.
  • Compaction swaps the merged file in and then rebuilds the views, without the write lock. A reader in between can hit a glob that matches nothing. Hold the write lock around the swap and the rebuild, but not the merge. Readers then wait milliseconds, not the whole compaction.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions