feat: Session/Memory 多后端回放一致性测试框架 (issue #89)#178
Open
wzxsdf wants to merge 1 commit into
Open
Conversation
- replay harness: 同一组轨迹驱动 InMemory/SQLite/Redis,比较事件/state/memory/summary - 归一化(占位符,保留字段存在性)+ JSONPath 精确 allowed_diff + 覆盖率治理 - summary: SDK 确定性模型 + 三分比较 + loss/overwrite/affiliation 检测 - 检出验证: 快照层注入 + 端到端后端注入(改 SQL 行 / Redis key 重读) - 实测发现 SQLite summary 持久化 drift,标 KNOWN_DRIFT 只报告不改 - 29 tests pass, 1 skipped (Redis 需 TRPC_REPLAY_REDIS_URL)
Author
|
I have read the CLA Document and I hereby sign the CLA |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #178 +/- ##
==========================================
Coverage ? 87.55867%
==========================================
Files ? 467
Lines ? 44103
Branches ? 0
==========================================
Hits ? 38616
Misses ? 5487
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概述
实现一个可复用的回放一致性(replay consistency)框架:用同一组标准化 Agent 轨迹驱动 InMemory / SQLite / Redis 三个后端,经规范化比较事件、state、memory 与 Session Summary,自动生成可定位到
session_id / event_index / summary_id / field_path的差异报告。它既是测试工具,也是后端实现质量的基准。Closes #89
主要改动
replay harness(
tests/sessions/replay/)harness.py:ReplayCase/ReplayOp/ReplayBackend/ReplaySnapshot数据模型 +replay_case()驱动 +_DeterministicSummarizer(覆写_compress_session_to_summary,跑 SDK 真实压缩流程、不调 LLM)normalizer.py:占位符归一化(保留字段存在性)、剥离temp:state、memory 确定性排序comparator.py:单一递归visit(),DiffEntry 内联定位字段allowed_diff.py:JSONPath 精确匹配(token 化,避开 fnmatch 字符集陷阱)+ 强制 reason + 覆盖率治理(条数 ≤ 8、占比 ≤ 10%)summary_checks.py:loss / overwrite / affiliation 三类专项 + 分词 Jaccard 语义相似度injectors.py:快照层注入 + 端到端后端注入report.py/backends.py:schema_version=3 报告 / 三后端实例化 + env 门控replay case(
replay_cases/cases.jsonl):10 条,覆盖 issue 全部 8 类场景(单轮/多轮/工具调用/state 覆写/memory 读写/summary 生成与更新/summary 截断/异常恢复)报告产物:
session_memory_summary_diff_report.json(schema_version=3)设计文档:
docs/superpowers/specs/2026-07-13-session-memory-replay-consistency-design.md后端与运行模式
:memory:)TRPC_REPLAY_SQL_URLTRPC_REPLAY_REDIS_URL,未设置则pytest.skip三后端显式传同一
SessionServiceConfig(store_historical_events=True)消除默认值差异;memory 三后端enabled=True;不要求本地安装真实 Redis/MySQL。归一化与比较策略
temp:state;memory entry timestamp 归一化 + 排序;long_running_tool_ids的 None / 空集合统一。检出验证(两层)
UPDATE events SET author=.../ 改 Redis key,重新get_session读出,验证 harness 对真实后端数据漂移的感知能力。验收映射
test_each_case_detects_injection+ 8 类 kind + 端到端 SQL/Redissession_id/event_index/summary_id/field_path/双后端值本地验证