feat: TypeSafe System One (Jev) typed proposers + hermetic test suite - #146
Merged
Merged
Conversation
The suite inherited the developer's environment, so it was green in CI and red on any machine where forge was actually installed and enabled -- the two things a maintainer does. An exported FORGE_LLM=1 both flipped the "llm off by default" assertion in test/substrate.test.js and made the faculties fire real model calls; a real ~/.forge reached doctor()'s machine-scoped install check through test/doctor.test.js. 593s wall, two failures. test/_setup.js is preloaded via --import into every test process (all three invocation sites, including the Windows job that bypasses `npm test`). It scrubs FORGE_*/provider env by prefix denylist, sandboxes $HOME to a throwaway tmpdir, and sets FORGE_LLM_HTTP=1 to force the keyless HTTP runner rather than shelling out to a real `claude` binary. 0 failures in ~40s. test/hermetic.test.js pins the scrub list against envVarsRead() so the two cannot drift, and fails if anyone drops the --import wiring. Two assertions were wrong rather than merely leaky: - doctor asserted a global `failed === 0` to prove a local property about `na` rows, making it depend on unrelated machine state. - a substrate comment claimed no runner reaches the real CLI -- the opposite of the truth, and why that file spent 85s on live calls. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Where the LLM layer asked a text model for a judgment that is really a classification or a yes/no — route's complexity band and preflight's assumption gate — forge can now ask TypeSafe's Jev instead: typed choice/noul answers with probabilities and confidence in ~150ms, batched into one call, versus seconds of text generation plus JSON parsing. - src/jev.js: zero-dep client on the adjudicate contract — opt-in (FORGE_LLM=1 + TYPESAFE_API_KEY), fail-safe (null never moves a verdict), key via child env, secret-refusing on outgoing state, answers validated against the questions asked - route: Jev choice proposer preferred, text-LLM fallback, BAND_FLOOR reconcile untouched; --json gains llm.provider + confidence - preflight: all four rubric dimensions scored as one batched noul call; clarifying questions stay with the deterministic rubric - hermetic boundary: test/_setup.js scrubs TYPESAFE_*; docs check covers the new env surface both directions; 11 new tests
Windows CI caught it: a plain absolute path (D:\…) handed to --import parses as the URL scheme "d:" and dies with ERR_UNSUPPORTED_ESM_URL_SCHEME before the canary process can run. --import resolves module specifiers, so give it the URL href — valid on every platform.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Two changes, rebased onto latest master:
1.
fix(test): the test suite is hermetic. It inherited the developer's environment — green in CI, red on any machine where forge was installed and enabled. A newtest/_setup.js(preloaded via--import) scrubsFORGE_*/provider env by prefix, sandboxes$HOME, and forces the keyless HTTP runner: 0 failures in ~40s (was 593s with two failures).test/hermetic.test.jspins the scrub list againstenvVarsRead()so the two cannot drift.2.
feat: TypeSafe System One (Jev) as the fast typed proposer. Where the LLM layer asked a text model for a judgment that is really a classification or a yes/no —route's complexity band and preflight's assumption gate — forge now asks TypeSafe's Jev instead: typedchoice/noulanswers with probability distributions and confidence in ~150ms (batched into one call for the gate's four dimensions), versus seconds of text generation plus JSON parsing.The new
src/jev.jsclient keeps the existing proposer contract exactly:FORGE_LLM=1andTYPESAFE_API_KEY; without the key, behavior is byte-identicalBAND_FLOORand the ±band assumption reconcile are untouched_FORGE_JEV_KEY), never argv, never logged;hasSecretrefuses secret-shaped stateforge route --jsonnow reportsllm.provider(jev/text) with Jev's confidence; preflight recordsassumption.provenance.provider. Clarifying free-text questions stay with the deterministic rubric — a System One model judges, it does not author prose.Docs updated everywhere the proposer layer is documented:
CHANGELOG.md,docs/GUIDE.md(section + env table),ARCHITECTURE.md,mintlify/concepts/model-routing.mdx,mintlify/concepts/pre-action-gate.mdx.forge docs checkpasses, withTYPESAFE_*covered in both directions and_FORGE_JEV_KEYregistered as internal plumbing.Checklist
npm testpasses (1145 tests, 0 failures — 11 new for Jev)npm run checkpasses (Biome lint + format)test/jev.test.js, injected transport, no network)fix(test)+feat)CHANGELOG.mdupdated under## [Unreleased]Risk & rollback
featcommit; the hermeticfix(test)commit stands alone either wayExtra checks (tick if applicable)
npm run typecheckpasseshasSecret; metrics log no prompt text)Note: no API keys are committed anywhere in this PR. Live smoke test for a keyed machine: