refactor(core): decompose the V2 session projector

[kitlangton]

@kitlangton

Summary

Refactor the V2 SessionProjector into a small composition root backed by cohesive internal projection modules.

The current transactional model is sound and should remain unchanged:

• Durable projectors execute inside the durable event transaction.
• A projector failure rolls back the event and all projection writes.
• Cross-table session transitions are atomic.
• Derived UsageUpdated publication happens after commit.
• SessionMessageUpdater already provides a useful storage-independent transcript fold.

The problem is structural: packages/core/src/session/projector.ts currently combines legacy V1 compatibility, current session metadata, transcript folding, pending-input lifecycle, fork materialization, revert truncation, usage accounting, instruction-state coordination, registration, and post-commit subscriptions.

Goal

Make SessionProjector a deep, legible module whose top-level file shows the projection families and transaction wiring without containing every SQL implementation.

Target shape:

packages/core/src/session/
  projector.ts
  projection/
    transcript.ts
    transitions.ts
    fork.ts
    usage.ts
    legacy-v1.ts

Keep the existing substantive domain modules:

session/
  message-updater.ts
  pending.ts
  instruction-state.ts
  sql.ts

Do not create one file per event or one file per table. The seams should follow durable state transitions and projection invariants.

Proposed Design

projector.ts: composition root

Reduce the current file to dependency acquisition, projection-family registration, post-commit subscription startup, and the global node.

Conceptually:

const layer = Layer.effectDiscard(
  Effect.gen(function* () {
    const events = yield* EventV2.Service
    const db = (yield* Database.Service).db

    yield* LegacyV1Projection.register(events, db)
    yield* SessionTransitionProjection.register(events, db)
    yield* TranscriptProjection.register(events, db)
    yield* ForkProjection.register(events, db)
    yield* UsageProjection.register(events, db)
    yield* UsageProjection.subscribe(events, db)
  }),
)

This should remain one visible manifest of active projection families.

Do not introduce a new Effect service or Layer per projection. These are internal modules operating with the transaction-bound database supplied by EventV2; hiding that database behind additional service provisioning would obscure the transaction seam.

projection/transcript.ts: persistent transcript fold

Extract the database-backed SessionMessageUpdater.Adapter currently constructed inside the generic run function.

This module should own:

• Message schema encoding and decoding.
• Selecting the newest assistant by aggregate sequence.
• Refusing to fall back to an older incomplete assistant when a newer assistant exists.
• Session-scoped assistant lookup and updates.
• Latest-shell lookup by logical shell ID.
• Running-compaction lookup.
• Appending projected messages at the durable event sequence.
• Delegation to SessionMessageUpdater.update.

Prefer a narrow internal interface:

export function project(db: DatabaseService, event: MessageEvent): Effect.Effect<void>

export function insertPromoted(
  db: DatabaseService,
  event: DurableEvent,
  message: SessionMessage.Info,
): Effect.Effect<void>

Rename the current generic operations:

• run -> projectTranscriptEvent
• insertMessage -> appendAtEventSequence

Keep the SQL adapter private to this module.

projection/transitions.ts: cross-model atomic transitions

Own handlers whose reaction spans multiple read models:

• Input admission and promotion.
• Compaction admission, completion, and failure.
• Session movement.
• Agent and model selection.
• Revert staging, clearing, and committing.
• Instruction-state application, reset, and epoch advancement.

Each event should have one owning handler when ordering or atomicity matters. For example, input promotion must consume the pending row and insert the visible transcript message in one transaction.

Model selection must preserve its current ordering: project the transcript event before updating the selected model because transcript projection reads the previous model. Agent selection currently has different ordering; preserve intentional differences explicitly in the handler rather than distributing writes across independent registrations.

projection/fork.ts: fork materializer

Move projectFork, ForkBatchSize, title generation, and related helpers into one deep module with one principal operation:

export const project = Effect.fn("SessionForkProjection.project")(...)

The module should continue to:

• Resolve and validate the parent and exclusive fork boundary.
• Create the child session.
• Copy inherited transcript history in bounded batches.
• Generate fresh message IDs while retaining inherited sequence order.
• Remap pending references to the new message IDs.
• Exclude running compaction messages and compaction pending rows.
• Reserve the inherited aggregate sequence.
• Rebuild inherited instruction state.

Keep the copy sequential and inside the durable event transaction. Do not replace the keyset cursor with parallel effects or move materialization post-commit.

projection/usage.ts: usage materialization and notification

Move usage recognition, cumulative SQL updates, V2 step contributions, and post-commit usage publication into one module.

Suggested names:

• usage -> legacyPartUsage
• applyUsage -> adjustTotals
• publishSessionUsage -> publishSnapshot

Fold legacy removal/replacement contributions before updating SQL so each event performs one aggregate adjustment where possible.

Keep UsageUpdated publication visibly post-commit. It is a best-effort derived snapshot; SessionTable counters remain authoritative. Do not publish it recursively from a transactional projector.

projection/legacy-v1.ts: compatibility projection

Move all SessionV1.Event.* handling and conversion helpers here:

• sessionRow
• messageData
• partData
• V1 create/update/delete
• V1 message update/removal
• V1 part update/removal

The separation should make clear that V1 mutable snapshot projection is compatibility behavior, not the model for new V2 event projection. Delegate usage deltas to UsageProjection rather than duplicating counter SQL.

Effect Guidance

• Use Effect.fnUntraced for small transaction-local handlers and registration functions where a tracing span adds no value.
• Keep named Effect.fn boundaries for meaningful operations such as fork materialization and usage publication.
• Keep explicit Effect.gen sequencing where domain order matters.
• Do not introduce nested database transactions.
• Do not fork inside transactional projector handlers.
• Do not parallelize projection writes.
• Do not publish derived events from inside the durable projection transaction.
• Keep the transaction-bound database explicit rather than hiding it behind new services.

Durable Event Typing

The projector repeatedly verifies that event.durable exists even though these handlers are registered for durable definitions. One compaction handler currently accesses event.durable.seq before its defensive undefined check.

After the structural refactor is stable, improve this invariant in a separate step:

1. Prefer updating EventV2.project typing so a durable definition supplies a payload with a required durable envelope.
2. Otherwise centralize the runtime assertion at the registration seam.
3. Remove duplicated handler-level checks once the invariant is proven.

Do not mix a broad EventV2 interface redesign into the first mechanical extraction.

Invariants To Preserve

• Projector writes and durable event persistence remain one transaction.
• A projector defect/failure aborts all projection writes and event persistence.
• Transcript and pending rows use aggregate sequence as their shared ordering coordinate.
• Promotion consumes pending work and exposes its message atomically.
• Agent/model transition ordering remains unchanged.
• Revert deletes transcript rows and pending admissions from the same inclusive boundary.
• Move and committed revert reset instruction state.
• Completed compaction advances instruction state and settles manual compaction when appropriate.
• Fork boundaries remain exclusive.
• Forked messages receive new identities while retaining inherited ordering.
• Running/manual compaction state is not inherited by forks.
• Usage increments continue to rely on exactly-once projector dispatch.
• Transcript lookups and updates remain session-scoped.
• Derived usage publication remains post-commit and best-effort.

Implementation Sequence

1. Extract transcript persistence

• Add projection/transcript.ts.
• Move the SQL adapter currently inside run without changing queries.
• Move durable message insertion.
• Rename the transcript operation.
• Keep registrations and behavior unchanged.

2. Extract fork materialization

• Add projection/fork.ts.
• Move the existing cohesive fork operation mechanically.
• Preserve statement order, batching, and transaction scope.

3. Extract V1 compatibility

• Add projection/legacy-v1.ts.
• Move V1 helpers and registrations.
• Initially delegate to existing usage adjustment behavior.

4. Extract usage

• Add projection/usage.ts.
• Move cumulative adjustment and post-commit publication.
• Collapse multiple adjustments into one delta update where safe.

5. Extract cross-model transitions

• Add projection/transitions.ts.
• Move promotion, compaction, move, model/agent, revert, and instruction coordination.
• Keep every cross-table transition explicitly sequential.

6. Reduce projector.ts to wiring

• Retain the visible registration-family manifest.
• Retain the Layer and global node.

7. Improve durable projector typing

• Address required durable envelopes separately.
• Remove redundant runtime checks only after types enforce the invariant.

Each step should be independently reviewable and preserve behavior.

Testing

Keep the existing projector integration tests operating through events.publish; transaction behavior is part of the contract.

Add focused tests for the extracted modules using the real test database rather than mocks.

Transcript

• Select the newest incomplete assistant by sequence.
• Do not fall back to an older incomplete assistant after a newer completed assistant.
• Keep assistant, shell, and compaction lookup session-scoped.
• Prevent updates from affecting another session.

Fork

• Treat the boundary as exclusive.
• Generate fresh copied message IDs.
• Remap pending message references.
• Exclude running compaction and compaction pending rows.
• Reserve child aggregate sequence.
• Rebuild instruction lineage.
• Roll back the entire fork after any failure.

Transitions

• Promotion failure leaves pending input intact.
• Successful promotion consumes pending and inserts transcript atomically.
• Model selection records the previous model before replacing current state.
• Revert applies one inclusive boundary to messages and pending admissions.
• Revert failure restores all affected projections.
• Move and revert invalidate instruction state.

Usage

• V1 part replacement subtracts the old contribution and adds the new contribution once.
• Failed V2 steps contribute only when both cost and token data are available.
• Post-commit publication observes final aggregate totals.
• Failed projection emits no usage notification.

Non-Goals

• A generic projector DSL or public registration framework.
• One module per event or database table.
• A generic SQL mutation language.
• Pure reducers over the entire loaded session state.
• Parallel projector execution.
• Nested transaction ownership in projection modules.
• Moving fork materialization out of the durable transaction.
• Changing event delivery or replay semantics.
• Making UsageUpdated durable.

First Reviewable Slice

The first PR should extract only projection/transcript.ts.

That removes the densest SQL adapter plumbing from the composition root, creates a direct test seam for transcript persistence policy, and changes almost no behavior. The broader package decomposition can then proceed through small mechanical PRs rather than one large rewrite.

[HOYALIM]

I would like to take the first reviewable slice described here: extract transcript persistence without changing behavior or transaction ownership, with the existing projector integration tests kept intact. Is this slice still available, and should the PR target dev or v2?

[Dante-dan]

I’ll work on the transcript-only first slice on current v2: move the database-backed adapter and durable message insertion into session/projection/transcript.ts, preserving the queries, event registration order, and transaction boundary. I’ll retain integration coverage through events.publish and add focused database-backed checks for session isolation and newest-assistant selection. This follows the scope of the earlier, unmerged #36811, adapted to the current agent, model, and location lookups.