Repository navigation
feat(engine): community state as context, measured and on by default (JEV-30) - #12
Closed
ohernandezdev wants to merge 4 commits into
Closed
ohernandezdev wants to merge 4 commits into
ohernandezdev wants to merge 4 commits into
Conversation
…nner (JEV-30) The criterion is in benchmark/community_state/run.py and is fixed before the first paid request. The engine carries the state line behind JEVMOD_FULL_CONTEXT so the run goes through production's path; what ships is decided by the run.
… ships (JEV-30) By the criterion pushed before the run, the state line with the message's own part passes: pooled gain +32 points (97.5% lower end +20), no bystander or busy chat verdict moved, +41 billed tokens. $0.099 spent.
Review summary🔴 Someone should look at this before it is merged
Why
What the automatic review found
Each percentage is how likely the review thinks that answer is -- not a grade for the code. Raw scores
13 files reviewed. |
…e not numbers, canonical words (JEV-30)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment D (JEV-30): does a compact community-state line beside the conversation window change verdicts.
benchmark/community_state/run.py, pushed in 9948223 before the first paid request.benchmark/community_state/REPORT.md): the whole line passes, pooled gain +32 points (97.5% lower end +20), no bystander or busy-chat verdict moved, busy-chat drift +0.036 under the +0.10 guard; +41 billed tokens (1.019x). The gain is recall, mostly spam copies; it does not clear calm lines; harassment not shown.Message.community(aCommunityStateof counts) rendered byrender_stateintocommunity_state, at most 50 estimated tokens, part of the cache key and the token estimate, behindJEVMOD_FULL_CONTEXT(default on). No caller fills it yet, so every request today is byte for byte unchanged. npm has no per-message context, no change.$0.099 spent.