txlog: address correctness, efficiency, and design review findings - #197
Open
schenksj wants to merge 1 commit into
Open
txlog: address correctness, efficiency, and design review findings#197schenksj wants to merge 1 commit into
schenksj wants to merge 1 commit into
Conversation
Implements the fixes from TXLOG_MODULE_REVIEW.md plus a follow-up multi-agent review pass. Correctness: - read_versions_concurrent propagates non-NotFound I/O errors instead of silently dropping versions; auto-checkpoint aborts rather than persisting a lossy checkpoint (C3). - Data-skipping typed comparison returns Option<Ordering>; an unknown comparison (bare-datetime filter, no session tz) no longer prunes live files (C4). - JNI copies the full config map so session.timezone.offset.seconds flows through (C5). - read_previous_checkpoint merges the schema registry into metadata (C6). - checkpoint_interval honored as a real interval (version % N == 0), not a boolean flag (C7). - Truncation guards added to Eq/Gt data skipping (C9); assorted edge fixes (negative epoch micros, purge age default, probe gap fallback, UTF-8 slice, phantom v0 checkpoint) (C10). Efficiency: - Post-checkpoint version actions parsed for the snapshot are reused by list_files, eliminating a 2x GET per query (E1). - Selective compaction collapsed into full compaction (was output-identical with an extra manifest fan-out) (E3). - Retry jitter mixes in PID; bounded cache registry; purge cursor removes O(N+R) instead of O(N*R) (E4). Design: - Structural TxLogError::NotFound variant (D4); deleted dead garbage_collection.rs and metrics.rs (D2/D5); shared datetime parser and skip-action encoder; ISO-8601 time-range values exported to Arrow instead of dropping to null (D6). Follow-up review fixes: gate the never-re-added diagnostic behind the debug flag, delete the orphaned selective-compaction helpers, de-duplicate the timestamp parse chain (now also handles date-only), and correct stale comments. The @order(36) integration test uses checkpoint_interval=1 on its post-purge append so the carry-forward assertion holds under interval semantics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses the findings in
TXLOG_MODULE_REVIEW.mdfor thenative/src/txlog/module, plus a follow-up multi-agent review pass over the changes themselves.Correctness
Option<Ordering>; an unknown comparison (bare-datetime filter with no session timezone) no longer prunes live files (C4). Regression tests added.session.timezone.offset.secondsreaches the skipping layer (C5).version % N == 0), not a boolean on/off flag (C7). Unit tests added.Eq/Gtdata skipping (C9); assorted edge fixes: negative epoch micros, purge age default, probe gap → LIST fallback, UTF-8-safe debug slice, phantom v0 checkpoint refused (C10).Efficiency
list_files, eliminating a 2× GET per query on tables with recent commits (E1).Design
TxLogError::NotFoundvariant instead of substring matching (D4).garbage_collection.rsandmetrics.rs(D2/D5).Follow-up review fixes
Gated the never-re-added diagnostic behind the debug flag, deleted the orphaned selective-compaction helpers, de-duplicated the timestamp parse chain (now also handles date-only), and corrected stale comments.
checkpoint_intervalsemantics were confirmed as modulo (matching Delta/Quickwit convention and the@Order(30)test's own comments); the@Order(36)test usescheckpoint_interval=1on its post-purge append so the carry-forward assertion holds under interval semantics.Testing
TransactionLogIntegrationTest(42) andArrowFfiListFilesIntegrationTest(7): BUILD SUCCESS.🤖 Generated with Claude Code