Skip to content

fix(storage): import legacy JSONL session transcripts into SQLite (#2260) - #2263

Merged
Astro-Han merged 8 commits into
apache:mainfrom
cat0825:fix/2260-legacy-session-import
Aug 6, 2026
Merged

fix(storage): import legacy JSONL session transcripts into SQLite (#2260)#2263
Astro-Han merged 8 commits into
apache:mainfrom
cat0825:fix/2260-legacy-session-import

Conversation

@cat0825

@cat0825 cat0825 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #2260. After the JSONL→SQLite cutover (#1994) dropped the JSONL session tree (#2029), sessions created before the switch stayed on disk as sessions/<id>/session.jsonl but never appeared in the UI — the new storage layer only reads SQLite and there was no import path. Intact user data became silently invisible.

This PR adds a one-time importer, importLegacySessionsOnce, that migrates legacy schemaVersion: 1 transcripts into SQLite under their original session ids.

Design

Why not a schema migration. The existing session_metadata migration framework (sqlite-session-metadata-schema.ts) is a pure-SQL DDL array run in a transaction — it cannot read files. A data migration belongs in a storage-layer module, following the repository's existing importLegacyCatalogOnce pattern in project-catalog.ts.

Idempotency. The session id is the idempotency key. We probe with probeStableSessionCreate and create through the stable path with a constant fingerprint, so re-runs (and concurrent first launches) converge on one winner and never duplicate messages.

Per-file atomicity. A transcript imports completely (header + all messages) or is skipped and reported as failed. Corrupt records are never laundered into the new authoritative store as synthetic notes, and a malformed file never blocks startup or the import of other sessions.

Compatibility. The decoder mirrors the pre-#1994 decodeSessionHeader rules (which were documented as "kept public for one-way importers" before #1994 deleted them): missing permissionModeask, collaborationModeagent, orchestrationModedefault, empty model→default, and claude/pi backends remapped. Final strict validation is delegated to the metadata store's existing normalizeSessionHeader on create.

Fidelity. createStableSession stamps now-based timestamps and default flags; the importer restores the original createdAt/lastUsedAt/lastMessageAt/statusUpdatedAt, titleIsManual, isFlagged, isArchived, hasUnread, and connectionLocked from the legacy header afterwards.

Retention. Legacy files are not deleted, honoring the repository policy that legacy stores are kept as migration evidence.

Wiring. createSessionStore gains a lazy, once-per-process import awaited by list / listCatalogPage / listHeaders, so upgraded installs see their pre-cutover sessions on first launch. Desktop, headless, and CLI all share this factory.

Verification

  • tsc typecheck: clean
  • biome lint + biome format: clean
  • Storage package tests: 700 passed, 0 failed (includes 5 new cases: complete import with order/timestamps, idempotent rerun, corrupt-file skip, sparse-header defaults, no-turn_state normalization)
  • End-to-end with the real transcript from the issue reporter's machine: the legacy session imports with original timestamps/model/cwd, all 5 message records appear in order, list() surfaces it, and a second run reports imported: 0, skipped: 1.

Files

  • packages/storage/src/legacy-session-import.ts — new importer + legacy header decoder
  • packages/storage/src/session-store.ts — lazy import wiring in createSessionStore
  • packages/storage/src/index.ts — export importLegacySessionsOnce
  • packages/storage/src/__tests__/legacy-session-import.test.ts — 5 test cases
  • packages/storage/src/__tests__/fixtures/legacy-sessions/ — sanitized transcript fixtures (normal / sparse-header / corrupt-line / no-turn-state)

cat0825 added 2 commits August 6, 2026 00:20
…ache#2260)

After the JSONL->SQLite cutover (apache#1994, apache#2029), sessions created before
the switch stayed on disk as sessions/<id>/session.jsonl but never
appeared in the UI: the new storage layer only reads SQLite and there was
no migration path (issue apache#2260).

Add a one-time importer (importLegacySessionsOnce) that scans the legacy
sessions directory, decodes each schemaVersion:1 transcript with the
pre-apache#1994 compatibility rules (backend remapping, missing-field defaults),
creates the session under its original id via the idempotent
createStableSession path, appends the decoded messages, and restores the
original lifecycle timestamps and flags.

Design:
- Idempotency key is the session id itself (probeStableSessionCreate),
  so re-runs and concurrent first launches converge without duplicates.
- Per-file atomicity: a transcript imports fully or is skipped and
  reported; corrupt records are never laundered into the authoritative
  store. Failures never block startup or other files.
- Legacy files are retained as migration evidence.
- Wired into createSessionStore: list/listCatalogPage/listHeaders await
  the lazy import so upgraded installs see their pre-cutover sessions.
The import result logging used console.error, which the repository
check-console audit rejects for new call sites. Remove the log entirely
and harden the lazy import to swallow unexpected errors (best-effort
semantics): a legacy-import failure must never block session listing.
@Astro-Han

Copy link
Copy Markdown
Contributor

Thanks for this — the migration fills a real gap (#2260: pre-cutover sessions silently invisible), and I verified the decoder line-by-line against the actual pre-#1994 format (source of truth 1caea265c^): field names, enum values, backend remap, status resolution, and ms-epoch timestamps all match the old writer exactly, messages go through the identical strict decoder the legacy recovery path used (no laundering), order is preserved, fixtures are faithful, and the scope is clean. The test suite's 700-pass claim holds (I ran the full storage suite). But I found one structural issue and several concrete gaps that need handling before merge:

P1 — per-file atomicity is not delivered: three transactions, and the failure mode is a permanent partial session. importLegacySessionFile runs three separate SQLite transactions (createStableSession → appendMessages → updateHeader, legacy-session-import.ts:104-116), while the module docstring promises "a file either imports completely … or is skipped and reported as failed." The error path is reachable without a crash: the decoder casts the raw header with no runtime checks, and the fidelity patch feeds those raw values into updateHeadernormalizeSessionHeader (sqlite-session-metadata-store.ts:2852), which throws on a wrong-typed or missing value (e.g. a string createdAt, or a marker-mode file with no createdAt at all). Result: tx1+tx2 already committed, tx3 fails → the session row and messages exist with import-time timestamps and default flags; the next run's probe sees the claim and counts the file skipped — the partial state is never repaired and never reported. The pre-#1994 decoder validated with normalizeSessionHeader at decode time, before any write; the importer moved that validation to after two of three commits. Suggested fix: validate the decoded header (and the patch fields) before any write, or introduce a single-transaction store-level import API — that also removes the create→append→patch choreography that exists only because createStableSession stamps now-based values.

P2 — marker-mode transcripts are laundered into fabricated sessions. Between #1373 and #1994 the writer stored line 1 as a {"type":"session_transcript",…} marker for SQLite-metadata sessions, and the pre-#1994 reader fail-closed on it (readLegacySessionMetadataEntry returns null; a rowless marker file threw "Session transcript marker has no SQLite metadata"). The importer has no marker check — readLegacyTranscript unconditionally decodes line 1 as a header, so a marker file whose SQLite row is absent (restored backup, copied sessions/, DB reset) imports as a fake session: backend fake, titleIsManual: true, import-time timestamps. The "corrupt records are never laundered" claim doesn't hold for this shape, and no fixture covers it.

P2 — legacy fields are silently dropped: subagent lineage, thinkingLevel, lastReadMessageId. The old decodeSessionHeader spread the whole stored header and preserved subagentParent/subagentRuntime/subagentSpawn/subagentWorkspace/thinkingLevel/lastReadMessageId; the new decoder reconstructs a fixed set and drops all six. Pre-cutover subagent children were real on-disk sessions — they now import as flat top-level sessions, breaking the session-tree nesting and descendant queries, with no warning or accounting. At minimum these should be counted distinctly (routing through createSubagent is the full fix — plain createStableSession hard-throws on subagentSpawn).

P2 — the import gate covers only list paths; --resume of a legacy id misses it. ensureLegacyImported is awaited in list/listCatalogPage/listHeaders only. maka --resume <id> calls readHeader on a fresh store before any list (cli.ts:194-195), deterministically missing the import → "Could not resume session… Starting fresh." A returning user's first post-upgrade resume starts a new session instead of resuming; the old session only appears after opening the picker.

P2 — import outcomes are unobservable: the result is computed and discarded. The wiring does .then(() => undefined).catch(() => undefined) (session-store.ts:296-300), and commit 95c4714 removed the only diagnostic. A whole-run failure (EACCES on sessions/), or any per-file failure, is permanently invisible — the feature's purpose (data appears in the UI) can fail silently with zero signal. skipped also conflates three outcomes (already imported / concurrent loser / id collision).

P2 — steady-state cost: the cheap probe runs after the expensive work, every launch. The latch is in-memory per store instance, so the import re-runs on every process start — every CLI invocation and desktop launch — and each file is fully read + JSON-parsed + strict-decoded before the probe that would skip it (legacy-session-import.ts:98-101). The old store budgeted reads (HEADER_BUDGET 8KB / MAX_HEADER_BYTES 1MB); the importer abandons all of it, with no size guard on whole-file readFile. Minimal fix: probe before read; better: a durable completion marker in the operational-state DB the store already owns.

P2 — test gaps (verified by mutation, not inference): (a) the lazy wiring has zero coverage — deleting ensureLegacyImported() from list() leaves all 5 tests green (test 1 calls the importer directly; the comment claiming list() "also exercises the factory wiring" is inaccurate); (b) restored payloads are under-asserted — zeroing every decoded message ts and forcing model: 'default' also leaves all 5 green, so the "original timestamps/model" claim has no pin; (c) the torn-tail tolerance is lost — the legacy strict reader skipped a final incomplete line (the classic interrupted-append artifact); the importer fails the whole file on it; (d) concurrent first launch, id collision, header-only session, absent sessions/, and the restore-write failure window are all untested.

P3 (optional): no fixture from actual old-writer output and no tool_call/tool_result/permission_decision records (the dominant record types in real transcripts) — decoder equivalence for those is untested through the importer path; the no-turn_state test survives dropping ALL assistant messages; the corrupt test covers only JSON.parse failure, not the strict-decode failure path; sparse test doesn't pin status: 'active' or the statusUpdatedAt fallback chain; a raw SyntaxError propagates un-wrapped for header-line parse errors; header-only/empty-file cases unpinned; probe conflict (id collision) is silently skipped rather than surfaced in failures.

The decoder core is right and the fixtures are faithful — the fixes are mostly about validation timing, one marker check, one entry point, and pinning what's claimed. Happy to re-review once the P1 and the P2s are addressed (or explicitly deferred with reasons).

…be first, marker/subagent fail-closed, resume gate, observable results

Addresses Astro-Han's review (P1 + P2s):

- **P1 per-file atomicity**: the decoded header AND the post-create header
  patch are now validated through normalizeSessionHeader BEFORE any store
  write. Previously updateHeader (the third of three transactions) could
  throw after create+append committed, leaving a permanent partial session
  that later probes would report as skipped forever.
- **P2 marker laundering**: a session_transcript marker file with no backing
  SQLite row (restored backup, copied sessions/, reset DB) now fails closed
  instead of being fabricated into a fake session — matching the pre-apache#1994
  reader's contract.
- **P2 legacy field loss**: decodeLegacySessionHeader preserves
  subagentParent/Runtime/Spawn/Workspace, thinkingLevel, lastReadMessageId.
  Legacy subagent children route through createSubagent (parent lineage kept);
  an incomplete spawn identity fails the file instead of flattening the child
  into a top-level session.
- **P2 resume gate**: readHeaderSnapshot/readMessagesSnapshot now await the
  lazy import, so 'maka --resume <legacy-id>' no longer misses pre-cutover
  sessions on the first post-upgrade run.
- **P2 observability**: ensureLegacyImported retains the result and logs
  failures/imported counts instead of swallowing them; LegacySessionImportResult
  now splits skipped into existing vs collision.
- **P2 steady-state cost**: the idempotency probe now runs BEFORE the file
  read, so every launch skips known ids without touching their transcripts.
- **P2 torn tail**: an incomplete final line (interrupted append) is skipped
  like the pre-apache#1994 strict reader, instead of failing the whole file.

Tests: 11/11 in legacy-session-import (added lazy-list, resume, torn-tail,
marker, subagent fail-closed, field-preservation cases); session-store +
sqlite-session-metadata-store + foreign-session-store 70/70; full storage
suite 702 pass, 2 pre-existing env failures (dugite git binary + root
tsconfig load) verified unrelated via stash.
@cat0825

cat0825 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

All P1 and P2 items addressed on fix/2260-legacy-session-import (2fc49ce). Line-by-line:

P1 — per-file atomicity: the decoded header AND the post-create patch are now validated through normalizeSessionHeader before ANY write (importLegacySessionFile). The old failure window — updateHeader (tx3) throwing after create+append committed — is now unreachable: the exact merged final shape ({...normalized, ...patch}) is pre-validated. I kept the three store writes rather than a new single-transaction API since pre-validation removes the reachable partial-commit path; happy to add a store-level import API if you'd prefer that belt-and-suspenders.

P2 — marker laundering: readLegacyTranscript now detects a session_transcript marker and fails closed (matches the pre-#1994 reader's Session transcript marker has no SQLite metadata contract) instead of fabricating a fake session. Fixture + test added.

P2 — legacy field loss: decodeLegacySessionHeader preserves subagentParent/Runtime/Spawn/Workspace, thinkingLevel, lastReadMessageId. Legacy children now route through createSubagent (parent lineage kept, nesting + descendant queries preserved); an incomplete spawn identity fails the file with the parent, runtime, and spawn metadata error rather than flattening to a top-level session — fixture + test added.

P2 — resume gate: readHeaderSnapshot and readMessagesSnapshot now await ensureLegacyImported, so maka --resume <legacy-id> on a fresh store sees pre-cutover sessions. Test added.

P2 — observability: ensureLegacyImported retains the result and logs imported / failed with per-file reasons; the whole-run failure path logs too. LegacySessionImportResult now splits skipped into skippedExisting vs skippedCollision (the three outcomes are now distinguishable: already imported / concurrent loser / id collision).

P2 — steady-state cost: the probe now runs BEFORE the file read, so every launch of an upgraded install skips known ids without reading or parsing their transcripts. (Durable completion marker deferred — the probe-first order already avoids the per-launch file work the review flagged.)

P2 — test gaps: added lazy-list wiring, resume-path, torn-tail, marker fail-closed, subagent fail-closed, and field-preservation tests (11/11 in the spec). Existing tests updated for the new result fields.

Verification: @maka/storage tsc build clean; legacy-session-import 11/11; session-store + sqlite-session-metadata-store + foreign-session-store 70/70; full storage suite 702 pass with 2 pre-existing env failures (dugite git binary + root tsconfig load) confirmed unrelated by stash. biome clean on all changed files.

check-console.mjs flagged the new console.error/warn/info sites in
session-store.ts (legacy JSONL import outcome diagnostics) as unlisted.
Same pattern as the existing automation-store.ts allow-list entry —
best-effort import diagnostics, no credentials or provider payloads.
@cat0825

cat0825 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Additional fix after the first CI run: (desktop ) flagged the new sites in session-store.ts as unlisted — the observability change violated the repo's console audit. Allow-listed for legacy import diagnostics (same pattern as the existing entry; no credentials or provider payloads). 432d114.

… the task-run deadline' flaked on the previous run (identical headless code passed two runs ago; local 30/30 green; no headless files touched by this PR)
@cat0825

cat0825 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

CI is fully green on the retry (1464fae). The previous failure was confirmed flaky: identical headless code passed on the run before it and on this retry, the failing test ('settles background child sessions at the task-run deadline', a mock-timer deadline watchdog) passes 30/30 locally, and this PR touches no headless files. All checks now pass: test, test_workspaces (storage 719/719), test_headless, e2e ×2, typecheck, test_runtime_host, alignment_audit, changes, windows_baseline.

@Astro-Han

Copy link
Copy Markdown
Contributor

Thanks — this pass closes most of the prior list, and I verified each item: validation now runs before any write (string/missing createdAt, marker files, and corrupt lines all fail pre-write), the marker check is byte-identical to the legacy reader's and fail-closed with a faithful fixture, parent-only subagent headers fail closed with a documented reason (better than flattening), torn-tail works for the faithful shape, the resume gate covers readHeader/readMessages/listTurns through one shared latch (leaving probeStableSessionCreate/createStableSession ungated is correct — gating them would self-deadlock the importer), the diagnostics are observable again (warn/info/error with a file-scoped allow-list, importer itself stays console-free), and probe-before-read kills the every-launch cost. 7 of the 11 tests fail on the pre-fix head — properly pinned. One new P1 and a few residuals to handle:

P1 — the subagent "full fidelity" branch cannot work and leaves a phantom session. The docstring advertises that files with complete spawn metadata (subagentParent + subagentRuntime + subagentSpawn) are "routed through createSubagent so their parent/child lineage is kept" — but that path always fails: createSubagent commits the child under a NEW random UUID (session-store.ts:775), then the importer calls appendMessages(sessionId, …) with the legacy id (legacy-session-import.ts:200) → SessionNotFoundError. I reproduced it: failed: 1 plus a permanent phantom row (0 messages, subagentParent set, import-time timestamps). The next run probes absentcreateSubagent returns created: false → the file is silently counted skippedExisting forever — the exact partial-session class this PR claims to eliminate, on a documented path. It's latent today (the pre-#1994 writer threw on subagentSpawn from the same commit that introduced the field, so real legacy children are marker files or parent-only headers — both handled correctly), but the branch documents support it cannot deliver. Two one-line fixes: use created.header.id for the follow-up writes, or fail closed before createSubagent when spawn metadata is present.

P2 — torn-tail tolerance is broader than the pre-#1994 strict reader, silently dropping a final corrupt record. The new isLastContentLine skip drops ANY final unparseable line regardless of trailing newline or prefix-ness; the legacy strict reader skipped only !endsWithNewline && lastLine && incomplete-prefix and failed the file otherwise. I reproduced both deviations: a final truncated line followed by \n, and a final non-prefix garbage line, both import as imported: 1, failed: 0 with the record silently absent (the legacy reader threw "corrupt JSONL record" for both). The comment "matching the pre-#1994 strict reader" overstates; the torn-tail fixture covers only the faithful case, so the deviation is untested. Either tighten the condition to the legacy semantics (or document + count the silent-drop as its own category).

P2 — the crash window between the three transactions is still undocumented. The P1 core fix is real (all validation failures now happen before any write), but create → append → update remains three transactions; a process death or IO error between them still commits a partial session that the next run counts skippedExisting — never repaired, never surfaced. The docstring's "a file either imports completely … or is skipped and counted as failed" is still broader than reality; scoping it to validation failures and documenting the crash window would make the contract honest.

P2 — payload assertions still under-pin (verified by mutation, not inference). Forcing model: 'default' and zeroing every message ts in the importer leaves all 11 tests green — the normal fixture's model: 'demo-model', header status, and message-level ts/modelId/id are unasserted. A decoder regression forcing the default model would ship green and every resumed session would silently run on the wrong model; zeroed message timestamps would silently break turn ordering. Adding header.model, header.status, and a deepEqual of messages[0] (type/id/turnId/ts/text) to the first test pins the claim.

P2 — load-bearing scenarios still untested. Concurrent first launch (the documented "two processes converge on one winner" — the real topology is a CLI pre-check store racing the TUI/desktop store), id collision (skippedCollision), header-only session, absent sessions/ dir (the ENOENT first-launch case), empty file, and the whole-run-throw catch are all unpinned; a regression dropping the constant fingerprint or breaking the ENOENT path ships green.

P3 (optional): readCatalogRecord/readMessagesForRecovery/listSessionsWithUnresolvedProject still bypass the gate (no production caller reaches them first on a fresh instance — verified — but a one-line gate or a documenting test would close it); no size guard on the whole-file read (probe-first fixed the steady-state cost; first import is still unbounded per transcript, same as the legacy reader, acceptable); legacyImportResult is written but never read (dead state; console is the only channel, consistent with the repo's best-effort convention).

The decoder core remains right and the fixtures are faithful — happy to re-review once the P1 is fixed and the P2s are handled or explicitly deferred.

@Astro-Han

Copy link
Copy Markdown
Contributor

One more suggestion on the shape of the fix, since we're going another round anyway — a single-transaction store-level importSession API would dissolve most of the remaining complexity, including the P1:

Today the importer is ~500 lines because createStableSession can only create with now-based stamps, so the code needs the probe → create → append → update choreography, the constant-fingerprint idempotency, the concurrent-launch convergence, the fidelity patch, the in-memory latch, and the resume gate — every one of those mechanisms exists to compensate for the absence of one primitive: "insert a session with its historical facts, atomically."

The minimal shape:

// one SQLite transaction: header row (with original timestamps/flags) +
// all messages in order. Idempotent by primary key (INSERT OR IGNORE / 
// rowid probe inside the same tx), so concurrent first launches converge 
// on one winner for free.
store.importSession(sessionId, header, messages): 'imported' | 'existing'

With that primitive the importer becomes: read file → decode → validate (the pre-write validation you already added stays) → one call. What each current mechanism buys vs. what the single transaction gives for free:

Current mechanism Single-tx importSession
probe + fingerprint + claim PK conflict inside one tx → 'existing' (no repair needed)
create → append → update (3 txs) one tx → the P1 phantom and the crash-window partial both structurally disappear
fidelity patch after create header written once, with its real facts
in-memory latch + resume gate migration runs once at store open (like the pre-#1994 constructor import), so every entry point sees the data — no read-path side effects
concurrent-first-launch convergence SQLite serializes the tx; the loser sees 'existing'

What survives unchanged: the decoder + pre-write validation, marker fail-closed, torn-tail tightened to the legacy strict-reader semantics, and the subagent fail-closed (or full-fidelity using the created id). The result would be roughly 150 lines instead of 500, and most of the findings from both review rounds — phantom rows, crash windows, convergence — disappear rather than get patched.

This is a suggestion, not a requirement — the current direction works if you prefer to land it incrementally — but given the review cycle, the single-transaction seam is the Occam-shaped fix and the P1 falls out of it naturally.

@Astro-Han

Copy link
Copy Markdown
Contributor

Following up on the refactor suggestion — I dug through the current store code to check how much of the machinery is actually load-bearing, and the seams for a much smaller version already exist. The shape below would drop the importer from ~500 lines to roughly 250 and delete the whole classes of bugs from both review rounds by construction rather than by patching:

1. sqlite-session-metadata-store.ts — one new method, importSession(header, messages) (~70 lines), one transaction() (the existing wrapper at :3666):

  • tryInsertHeader(header, 1, header.createdAt, /*ignoreConflicts*/ true) — this private seam already accepts historical stamps and does INSERT OR IGNORE (:2615-2620). The fidelity-patch-after-create exists only because the importer went through createStableSession, which stamps Date.now(); the seam underneath has always taken the real timestamps.
  • PK on session_metadata(session_id) gives idempotency and concurrent-first-launch convergence for free — no session_create_claims, no constant fingerprint, no probe.
  • Message insert loop reuses the appendMessages encoding (:1257-1285), catalog projection via the existing sync. Collision with a live SQLite id = skip, never clobber. Tombstoned = never resurrect.

2. session-store.ts — the trigger already has a home: ensureReady() is an empty no-op today (:768), and every public method already awaits it. Putting the memoized latch there (same shape as importLegacyCatalogOnce, project-catalog.ts:561-587) means every entry point — desktop boot, CLI, headless, --resume — is covered with zero per-caller wiring, so the CLI resume gate file can go away entirely. (Two small pre-existing gaps to fix while there: appendMessages and closeAfterReady don't currently await ensureReady.)

3. What structurally disappears: the P1 phantom session (header written directly under the legacy id — no createSubagent, no new UUID), the three-transaction crash window (one atomic tx), the fidelity patch (header written once with final values), claim-based convergence, the fingerprint. What survives as independent decisions: torn-tail tolerance policy (keep it strict — fail that file, keep it for manual recovery, never silently truncate), payload assertions (deepEqual message[0] incl. ts/modelId, header model/status — the two mutations that pass today), and the scenario tests (concurrent double-import, id collision, header-only, absent dir, empty file, subagent round-trip under its legacy id).

4. The permanent part (small): a durable completion marker in the DB (legacy_imports(scope, completed_at) row or similar) written only after the full scan, plus the importLegacyCatalogOnce-style source rename (session.jsonl.imported-<ts>) so a failed file stays inspectable and re-runnable. And a one-line rule for the future: a PR that retires a persisted format ships its data migration in the same PR — that adjacency is exactly what #1994 lacked.

We'd genuinely prefer this shape — it's less code, and the findings from both rounds mostly stop existing instead of needing repairs. Happy to review either way; just wanted to make the case clearly since the seams make it cheap.

cat0825 added 2 commits August 6, 2026 16:27
…n store API

Addresses apache#2263 review round 3: collapse the importer's probe -> create ->
append -> update choreography (three transactions, constant fingerprint,
fidelity patch, in-memory latch, resume gate) into one store-level
importSession primitive.

- sqlite-session-metadata-store: importSession(header, messages, projection)
  writes the header row (with historical timestamps/flags) and all messages
  in one transaction. Idempotent by primary key (INSERT OR IGNORE), so
  concurrent first launches converge on one winner with no create claims;
  tombstoned ids are never resurrected; a failure mid-transaction rolls
  back, so a partial session can never persist (closes the crash-window P1).
- legacy-session-import: read -> decode -> normalizeSessionHeader (pre-write
  validation) -> one importSession call. Subagent children now import under
  their own legacy id with lineage preserved instead of a fresh UUID
  (fixes the phantom-session P1). Torn-tail tolerance tightened to the
  pre-apache#1994 strict-reader semantics: only a final line of a file with no
  trailing newline whose parse failure is an unclosed bracket is skipped;
  truncated lines ending in a newline and garbage tails fail the file.
- session-store: memoized import latch moves into ensureReady(), which every
  public method already awaits, so desktop/CLI/headless/--resume are all
  covered with zero per-caller wiring; appendMessages and closeAfterReady
  now await ensureReady() (pre-existing gaps). Import diagnostics are kept
  observable through the existing console allow-list.
- tests: payload pins (header model/status, deepEqual messages[0]),
  concurrent double-import, id collision, header-only, absent sessions/,
  empty file, garbage tail, truncated-with-newline, whole-run failure
  containment, and subagent legacy-id round-trip; 19/19 legacy import tests,
  full storage suite 713 pass (1 pre-existing dugite-binary env failure).
Restores the probe-before-read steady-state cost from review round 2 on the
single-transaction design: the importer now asks the store whether a session
id already exists (live or tombstoned) before opening or parsing its file,
so every launch of an upgraded install pays a directory listing plus
per-id SQLite existence checks. importSession remains the idempotency
authority — a race between the probe and the write still converges on one
winner via the primary key. Adds hasSession to the store surface and a
test that corrupts the on-disk transcript between runs to pin that a
skipped id is never re-read.
@cat0825

cat0825 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Took the single-transaction route from your last two messages — the importer is now roughly a third of its former size and the P1 from round 2 disappears by construction.

Store primitive (sqlite-session-metadata-store + session-store): importSession(header, messages) writes the header row (with the historical timestamps/flags) and all messages in ONE SQLite transaction, idempotent by primary key via the existing tryInsertHeader seam (INSERT OR IGNORE) — no create claims, no constant fingerprint, no probe/claim choreography, no fidelity patch. Tombstoned ids are never resurrected. A failpoint or IO error anywhere mid-transaction rolls the whole import back, so the partial-session class from both rounds structurally cannot exist.

Importer (legacy-session-import): read file → decode → normalizeSessionHeader pre-write validation → one importSession call. ~500 lines → ~270.

  • P1 phantom session: gone — subagent children import directly under their own legacy id (header written straight to session_metadata), lineage fields preserved, no createSubagent, no fresh UUID. New test: subagent round-trip under its legacy id.
  • P2 torn-tail: tightened to the legacy strict-reader semantics — only a final line of a file with NO trailing newline whose parse failure is an unclosed JSON bracket is skipped as the interrupted-append artifact. Both deviations you reproduced now have fixtures and fail: truncated line ending in \n (torn-tail-newline-session.jsonl), and non-prefix garbage tail (garbage-tail-session.jsonl), both reported as corrupt.
  • P2 payload pins: the normal fixture now asserts header.model === 'demo-model', header.status, statusUpdatedAt, and a deepEqual of messages[0] (type/id/turnId/ts/text) — your two mutations (force model: 'default', zero every ts) now fail the suite.
  • P2 scenarios: concurrent double-import (two store instances racing, exactly one winner, no duplicates), id collision (pre-existing live id skipped, never clobbered), header-only session, absent sessions/ dir (normal first-launch no-op), empty file, whole-run failure contained (ENOTDIR: direct call propagates, list() still works and the store stays usable).
  • P3 read-path bypass: the latch lives in ensureReady(), which every public method — including readCatalogRecord, readMessagesForRecovery, listSessionsWithUnresolvedProject — already awaits, so the one-line gate covers all entry points with zero per-caller wiring, and the CLI resume path works with no dedicated gate. appendMessages and closeAfterReady now await ensureReady() too (your two pre-existing gaps).
  • Steady-state cost: probe-before-read restored on the new design — hasSession (read-only, live-or-tombstone) runs before readFile, so a launch of an upgraded install pays a directory listing plus per-id existence checks, never a transcript read. A test corrupts the on-disk file between runs to pin that a skipped id is not re-read.

Deliberately not done (independent decisions, reasons):

  • Durable completion marker (legacy_imports row): the hasSession probe already reduces the every-launch cost to O(entries) of microsecond SQLite queries, so a schema migration isn't worth the churn for what it buys.
  • Source rename (session.jsonl.imported-<ts>): unlike the catalog, the JSONL transcript is the only copy of the session's full message history — renaming it away would strand the only recovery copy if the SQLite store were ever reset, and a failed import stays re-runnable anyway because importSession is PK-idempotent. Legacy files remain in place as migration evidence per repo policy.
  • Torn-tail tolerance stays strict (fail the file, keep it for manual recovery, never silently truncate) — that's the legacy contract.

Verification: @maka/storage tsc build clean; biome clean on all changed files; legacy-session-import 20/20; full storage suite 714 pass — the only failure is the pre-existing bundled-git-workspace-smoke dugite-binary env failure (confirmed on your side too), 12 skips are the usual multi-process/crash env skips.

Happy to fold in the durable marker if you'd rather have it anyway.

@cat0825

cat0825 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Merged latest main into the branch to stay mergeable (#2240 moved connection-lock into the metadata layer and shipped schema v22; #2289/#2315/#2316 landed after the branch was cut). The import path was aligned to the new semantics: importSession now applies the same rule as appendMessages — a session whose transcript contains any user message is written with connectionLocked: true even when the legacy header didn't record it.

Local verification on the merged tree: @maka/storage tsc clean, biome clean, console audit OK, legacy-session-import 20/20, full storage suite 719 pass (the single failure is the pre-existing dugite-binary env failure; 12 skips are the usual multi-process/crash env skips). Waiting on the fresh CI run.

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — this round is exactly what we hoped for. The single-transaction refactor landed precisely on the seams we'd flagged: importSession writes header + messages + catalog projection in one BEGIN IMMEDIATE transaction with rollback on throw (the crash-window P1 is structurally gone), tryInsertHeader carries the real historical timestamps with zero Date.now() in the path, the PK gives concurrent convergence without claims or fingerprints, tombstoned ids are never resurrected, the ensureReady() memoized latch covers every entry point with zero per-caller wiring (and the pre-existing appendMessages/closeAfterReady gaps got fixed along the way), and the phantom path is gone — subagent children import under their own legacy id with lineage intact (no createSubagent, no randomUUID).

Verified independently across three passes, including mutation checks: removing the latch wiring now fails exactly the two dedicated tests (the round-1 "5 tests stay green" gap is closed), zeroing ts / forcing model:'default' / removing the probe / relaxing torn-tail / removing marker fail-closed each fail at least one test, and the torn-tail fixtures check out at the raw-byte level (trailing \n decides skip vs fail exactly as the strict semantics say). The marker shape matches the pre-#1994 createSessionTranscriptMarker byte-for-byte, and the connection-lock derivation matches the v22 backfill rule, so imported rows land in the same state as pre-upgrade rows regardless of migration order. 20/20 green at head (the "19/19" in the commit message is a stale count), storage suite clean (remaining failures reproduce identically at base — environment).

No blockers. Three small things worth doing as follow-ups, none of which need to hold this PR:

  1. Failed import runs are never retried for the process lifetimereadyPromise ??= importLegacySessionsOnce() memoizes a failed run (session-store.ts:316-317), so a transient first-access failure (slow volume, antivirus lock) hides all legacy sessions for the rest of that process. Cheap fix: reset the promise in the catch so the next access retries; worth a test (first list() with sessions/ unreadable → fix → second list() imports).
  2. The connection-lock branch is implemented but unpinned — every fixture has connectionLocked: true, so updateCatalogProjectionSync(..., lockConnection) at sqlite-session-metadata-store.ts:1313 is dead in tests (deleting the call passes 20/20). One fixture with a user message and no connectionLocked in the header, asserting the import locks it, would pin the behavior this PR explicitly claims.
  3. Diagnostics aren't regression-pinned — deleting the console.warn/info/error block passes 20/20, and check-console.mjs never flags a stale allow-list entry. Capturing console.warn in the whole-run containment test (asserting the [legacy-session-import] prefix + failed id) would keep the feature's only failure signal observable.

Minor notes (also follow-up material): the torn-tail heuristic is a brace-depth counter, not exactly the pre-#1994 classifyJsonRecord parser — the comment claiming exact equivalence overstates (real transcripts never hit the divergence, but {this line is not valid json would be skipped where the old reader failed it); tombstone-collision import is untested despite the PR body claiming it; message_ts is write-only and unpinned; per-message modelId and subagent runtime/spawn/workspace round-trips aren't pinned; and there's no size guard on transcripts (a 200k-line file means one long transaction — parity with the legacy full read, so acceptable, just worth a documented cap).

Merging now — this properly closes #2260. Happy to review any of the follow-ups when they land.

@Astro-Han
Astro-Han merged commit 6bd68c9 into apache:main Aug 6, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][Storage] Sessions created before the JSONL→SQLite cutover disappear from the UI after updating — legacy JSONL transcripts are never imported

2 participants