fix(persona): retain observations when a digest is truncated

[YellowSnnowmann]

Summary

Bulk coding-session ingest silently dropped observations on dense sessions. DIGEST_MAX_OUTPUT_TOKENS = 4096 truncated digest responses mid-JSON (EOF while parsing a list), the parse failure soft-failed to Ok(vec![]), and the caller committed the window cursor anyway — so the window was marked done with zero observations captured, unrecoverably.

This PR:

• Raises DIGEST_MAX_OUTPUT_TOKENS 4096 → 16384 (documented inline; dense Codex digests exceed 4096, matching the observed ~15–19k-column truncations).
• Introduces a DigestError so a truncated/unparseable digest returns Err instead of a committable empty result. The caller's existing hard-failure arm then skips the cursor commit, so a truncated window is retried on the next ingest run rather than lost. Genuinely-empty digests still commit as before.

Addresses tinyhumansai/openhuman#5510, and the digest half of tinyhumansai/openhuman#5509. The other half of #5509 (the OpenHuman-side ingest RPC timeout budget) is a separate OpenHuman PR that will also bump this submodule pointer once this merges.

API Or Behavior Changes

digest_window / digest_session now propagate a truncated/unparseable digest as Err (previously a silent empty Ok). Public return types are unchanged (anyhow::Result). A truncated window no longer commits its cursor.

Tests

• cargo fmt --check
• cargo clippy --all-targets -- -D warnings
• cargo build --all-targets
• cargo test

Updated distill_tests.rs (bad/truncated JSON now asserts a non-committable Err, not Ok+empty) and added pipeline_tests.rs coverage proving a truncated window does not commit its cursor and is re-processed on a later run.

Documentation

None needed — the token-cap rationale is documented inline at the constant.

Summary by CodeRabbit

• Bug Fixes
  • Prevented incomplete or unparseable memory digests from being saved as empty results.
  • Failed digest processing now remains eligible for retry instead of advancing progress.
  • Improved handling of provider failures and truncated responses.
  • Increased the maximum digest response size to support larger memory summaries.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8748d462-9b01-4228-9697-3649be182f0e

📥 Commits

Reviewing files that changed from the base of the PR and between 4f6517c and 5ffcfe0.

📒 Files selected for processing (4)

• src/memory/persona/distill.rs
• src/memory/persona/distill_tests.rs
• src/memory/persona/pipeline.rs
• src/memory/persona/pipeline_tests.rs

---

📝 Walkthrough

Walkthrough

The digest token limit increases to 16,384. Provider and parsing failures now remain retryable and prevent cursor commits. Valid empty digests still commit. Tests cover truncated responses and successful retries.

Changes

Persona digest retry handling

Layer / File(s)	Summary
Digest error contract src/memory/persona/distill.rs, src/memory/persona/distill_tests.rs	The digest limit increases to 16,384. Provider failures and unparseable or truncated responses return errors. Valid empty responses return empty digests.
Pipeline retry semantics src/memory/persona/pipeline.rs, src/memory/persona/pipeline_tests.rs	Failed digests do not commit cursors or watermarks. End-to-end tests verify that truncated responses are retried successfully.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Mergeability Score: 🔵 Low · up to 5ffcf

Malformed digests will now be retried instead of silently losing observations, but a persistently failing window could repeat provider work, cost, and transcript transmission across runs. The impact is bounded per run and the PR is mergeable with owner awareness of this follow-up risk.

Possibly related issues

• tinyhumansai/openhuman#5510 — The change addresses the same token-cap and cursor-commit behavior for truncated or unparseable digests.

Suggested labels: priority: p2

Poem

I’m a rabbit guarding each little byte,
No broken digest slips out of sight.
Empty is valid, failure must wait,
Cursors stay still while retries operate.
With longer tokens, the trail stays bright.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the primary fix: preventing observation loss when persona digest responses are truncated.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[tinysweeper]

tinysweeper found nothing blocking. Approving.

             $0.0287 · 32,798 in / 8,928 out · 27,931 cached (85%) · z-ai/glm-5.2
critique:    $0.0114 · 9,978 in  / 3,849 out · 8,564 cached (86%)  · z-ai/glm-5.2
security:    $0.0079 · 9,894 in  / 2,404 out · 8,494 cached (86%)  · z-ai/glm-5.2
tests:       $0.0055 · 5,825 in  / 1,737 out · 4,872 cached (84%)  · z-ai/glm-5.2
description: $0.0023 · 6,256 in  / 351 out   · 5,225 cached (84%)  · z-ai/glm-5.2