Skip to content

feat(engine): community state as context, measured and on by default (JEV-30) - #12

Closed
ohernandezdev wants to merge 4 commits into
mainfrom
jev-30-community-state
Closed

ohernandezdev wants to merge 4 commits into
mainfrom
jev-30-community-state

Conversation

@ohernandezdev

Copy link
Copy Markdown
Owner

Experiment D (JEV-30): does a compact community-state line beside the conversation window change verdicts.

  • Pre-registered criterion in benchmark/community_state/run.py, pushed in 9948223 before the first paid request.
  • 178 constructed rows (spam, harassment, scam: paired incident/calm halves, bystanders during incidents, busy normal chat, controls), blind second labeller.
  • Result (benchmark/community_state/REPORT.md): the whole line passes, pooled gain +32 points (97.5% lower end +20), no bystander or busy-chat verdict moved, busy-chat drift +0.036 under the +0.10 guard; +41 billed tokens (1.019x). The gain is recall, mostly spam copies; it does not clear calm lines; harassment not shown.
  • Engine: Message.community (a CommunityState of counts) rendered by render_state into community_state, at most 50 estimated tokens, part of the cache key and the token estimate, behind JEVMOD_FULL_CONTEXT (default on). No caller fills it yet, so every request today is byte for byte unchanged. npm has no per-message context, no change.

$0.099 spent.

…nner (JEV-30)

The criterion is in benchmark/community_state/run.py and is fixed before the
first paid request. The engine carries the state line behind JEVMOD_FULL_CONTEXT
so the run goes through production's path; what ships is decided by the run.
… ships (JEV-30)

By the criterion pushed before the run, the state line with the message's own
part passes: pooled gain +32 points (97.5% lower end +20), no bystander or busy
chat verdict moved, +41 billed tokens. $0.099 spent.
@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Review summary

🔴 Someone should look at this before it is merged

feat(engine): community state as context, measured and on by default (JEV-30)
by @ohernandezdev · 14 files changed · 3099 lines changed

Why

  • It is large: 3099 lines changed, and the limit for merging without a person is 400
  • If this change is wrong the damage is medium: shared logic that several features rely on
  • It may change more than its title says (56%)
  • A mistake here might go unnoticed (51%)
  • The worst case cannot be ruled out (16%)
  • It may touch money

What the automatic review found

Question Answer
If this change is wrong, how bad is it? 🟠 Medium -- shared logic that several features rely on
Does the title describe the whole change? ⚠️ Not confident either way (does what it says 93%, extra changes 56%)
Does it touch money, accounts or personal data? ⚠️ Not confident either way (61%)
Could a mistake here break things quietly? ⚠️ Not confident either way (51%)
Does it include tests? ✅ Yes

Each percentage is how likely the review thinks that answer is -- not a grade for the code.
Nothing was merged automatically -- this review only leaves a comment.

Raw scores
dimension max across files
risk_level 1.99 / 3
worst_case_risk 0.16
diff_matches_title 0.93
hidden_scope 0.56
silent_failure 0.85
silent_failure_weighted 0.51
tests_expected 0.96
touches_money 0.61
touches_accounts 0.24
touches_personal_data 0.38
sensitive_area 0.61

114899 input tokens · cost of this run: less than a cent ($0.004826)

13 files reviewed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant