Conversation
…ateway) (#1) * refactor: move TypeSafe API behind a provider class TypeSafeJevProvider now implements JevPort; createSdkAdapter and classifyError keep their names and behavior. The SDK client is injectable so provider tests need no network. * feat: add Vercel AI Gateway Jev provider and JEV_PROVIDER selection VercelJevProvider translates Jev questions/answers to the gateway evaluation API (typesafe-ai/jev) via experimental_evaluate from the ai package. JEV_PROVIDER=typesafe (default) | vercel, validated at startup; each provider owns its classifier. Confidence absent from gateway answers is derived from the distribution (selected-choice mass / max level mass). * test: cover Jev providers and provider selection 22 tests: provider selection and startup failure, TypeSafe request forwarding and error mapping, Vercel question/answer translation, confidence derivation, padding, model-id reporting, missing keys, CLI-level JEV_PROVIDER behavior, and provider-independent review behavior. All network calls mocked. Docs: provider configuration in README and architecture.md. * fix: enforce the executor timeout in the Vercel provider experimental_evaluate has no timeout parameter, so the provider wraps the call in its own AbortController: a timer aborts at timeoutMs and forwards an external run signal, surfacing as AbortError/TimeoutError which classifyVercelError maps to aborted/transient. * fix: preserve TypeSafe confidence, bind gateway credentials, and secure defaults - Read TypeSafe's separate confidence statistic from providerMetadata.typesafe.confidence[questionId] (Choice/Score); derive from the distribution only when that metadata is genuinely absent, documented as an approximation. - createVercelAdapter now binds the supplied AI_GATEWAY_API_KEY via createGateway({apiKey}).evaluationModel(id) instead of relying on the process-global gateway; process.env is never mutated. Proven by a local-server test asserting the Bearer header actually sent. - Model namespaces stay separate: unqualified TypeSafe ids (jev-1.13.0) select the configured gateway model (JEV_GATEWAY_MODEL, default typesafe-ai/jev); slash-qualified gateway ids pass through. - gateway.zeroDataRetention=true by default (evaluations contain source code); JEV_GATEWAY_ZERO_DATA_RETENTION=0 opts out. - Already-aborted external signals now propagate before any evaluation. - Optional live smoke: npm run smoke:vercel (JEV_SMOKE=1), skips cleanly. - README/help updated to match actual behavior. * fix: tighten Vercel provider bookkeeping and smoke gating - ask() computes the effective gateway model once and uses it for both the model factory and the response.model fallback. - smoke-env JEV_SMOKE regex anchored as a group; '10', 'yes1', 'true1' no longer enable live smoke tests (covered by tests). - createVercelAdapter accepts test-injection options (baseURL, evaluate) so the credential test drives the adapter itself end-to-end against a local server; the Bearer assertion is unchanged. * fix: address review findings on budget accounting, classifier coupling, retries - P1: missing gateway usage no longer zeroes the input-token budget; the envelope reports the same deterministic estimate the executor reserved, so --max-input-tokens keeps guarding Vercel runs. - P2: createWorkflowDependencies now accepts a provider name and maps it through classifierFor; jevFromEnvironment(env) callers can no longer pair a Vercel port with the TypeSafe classifier. The CLI passes configuredProvider(io.env) directly. - P2: malformed gateway answers (missing/wrong-typed) throw ValidationError instead of plain errors, so the executor routes them through its invalid-response retry path instead of failing the frame. - Test env-leak hardening: the credential test asserts process.env is unchanged across construction rather than assuming the var is unset. * fix: address round-2 review findings on retries, classifier derivation, aborts - P2: malformed gateway answers now return a valid envelope with an empty answer map instead of throwing: the executor's response- validation path (which retries) rejects it via the frame parser, rather than the transport-error path (unknown, never retried). Verified end-to-end: a frame with a validating parser gets its configured retries (3 attempts with retries: 2). - P2: createWorkflowDependencies now derives the default classifier from the port itself (VercelJevProvider.classifyError) instead of an independent provider-name default, closing the two-argument pairing gap; the CLI returns to the two-argument call. - P2: caller-supplied abort reasons are normalized to abort-typed errors (DOMException AbortError) when forwarded and when thrown for an already-aborted signal, so classifyVercelError reports aborted instead of unknown. * fix: address round-3 review findings on transient failures and answer keys - P2: classifyVercelError recognizes plain connection failures as transient — TypeError: fetch failed (with ECONNREFUSED/ENOTFOUND/ EAI_AGAIN/ECONNRESET/EPIPE on the cause), retryable-flagged AI SDK errors, and network error names. - P2: unexpected gateway answer keys are no longer silently dropped; translation rejects and the empty-answer envelope routes the frame into the invalid-response retry path. - P3: README now documents the conservative input-size estimate reported when gateway usage is absent. * fix: address round-4 review findings on key validation and classifier binding - P2: answer-key unexpectedness uses Object.hasOwn, so prototype-named keys (constructor, toString) count as unexpected instead of passing through the inherited-property 'in' check. - P2: extra probability labels reject instead of being silently dropped — choice answers with probabilities for unknown choices and score answers with out-of-range level keys both route into the invalid-response retry path. - P2: defaultClassifierFor binds the port's classifyError to the port, so method-reference invocation through WorkflowDependencies keeps the right 'this'. * fix: address round-5 findings — canonical score keys, bound classifier, README accuracy - P2 (real miss caught by review): the round-4 classifier bind never landed (heredoc failure silently dropped it from that commit). defaultClassifierFor now genuinely binds the port's classifyError to the port; regression test invokes it through the dependencies layer and asserts instance state stays reachable. - P2: score probability keys are compared against exact canonical level keys ('0'..'n-1'), so '1.5', '01', '1e0' reject instead of coercing into range and being silently dropped. - P2: privacy guidance now discloses that the Vercel provider routes through the AI Gateway, which processes requests even under zero-data-retention; users are pointed at both services' terms. - P3: provider section no longer promises identical results across providers (confidence fallback can differ); describes shared schema and capability instead. - P3: requirements/setup are provider-neutral (either TypeSafe key or gateway key per JEV_PROVIDER). * fix: address round-6 findings — choice synthesis, env redaction, smoke pinning, docs - P2: choice answers that omit probabilities (an allowed gateway shape) now synthesize a point mass on the selected choice instead of an all-zero distribution that readChoice would reject, burning every retry on otherwise valid answers. - P2: redaction is env-aware — createRedaction(env) unions the supplied and process environments' credential values, and createWorkflowDependencies threads io.env through, so a custom environment's gateway key is redacted from evidence and errors. - P2: smoke-real forces JEV_PROVIDER=typesafe so the TypeSafe smoke never silently exercises Vercel (or fails parsing its error packet) when a gateway env is present. - P3: the workflow overview is provider-neutral (TypeSafe key or gateway key per JEV_PROVIDER). * fix: synthesize score distributions when probabilities are omitted - Score answers without probabilities (an allowed gateway shape) now get a distribution consistent with the reported score: point mass for integral scores, linear interpolation between adjacent levels for fractional ones (readScore accepts and expects both). Previously the all-zero fallback was rejected by readScore and burned every retry on valid answers. - Out-of-range scores (beyond the rubric) still reject into the invalid-response retry path; bounds match readScore's tolerance. - Regression tests cover integral, fractional, and out-of-range cases. * fix: redact port-owned credentials in the two-argument composition - Ports expose credentialSecrets (TypeSafeJevProvider and VercelJevProvider both populate them); portSecrets(jev) reads them. - createWorkflowDependencies unions the port's credentials into the redaction dependency, so createWorkflowDependencies(root, createVercelAdapter(customEnv)) scrubs the custom-env key from evidence and errors even though no environment was passed. - createRedaction(env, extraSecrets) accepts the extra values. - Tested for both providers through the plain two-argument call.
The squash-merge of PR #1 left classifyVercelError, classifyError, configuredProvider, PROVIDER_ENV and RedactionPort imported but unused, which fails npm run lint (biome noUnusedImports) on merged main.
JEV_PROVIDER=cloudflare routes Jev through Cloudflare's native AI REST
run endpoint (POST /accounts/{id}/ai/run, model alias typesafe/jev) using
CLOUDFLARE_ACCOUNT_ID + CLOUDFLARE_API_TOKEN (JEV_CLOUDFLARE_API_TOKEN as
an application-specific alias that wins when both are set).
- CloudflareJevProvider behind the existing JevPort; direct fetch, no SDK
- native noul/choice/score contract, no boolean translation
- v4 envelope + execution-state double-nest unwrapping; failed execution
states are provider errors, never empty answer sets
- malformed payloads resolve to the empty-envelope invalid-response path
- classifyCloudflareError with the shared TransportFailure taxonomy
- credentials exposed through credentialSecrets/portSecrets and envSecrets
- JEV_CLOUDFLARE_MODEL (default typesafe/jev); TypeSafe-direct ids never
forwarded, only typesafe/-qualified slugs pass through
- usage passthrough with the deterministic input estimate fallback
- npm run smoke:cloudflare (JEV_SMOKE=1 gated, skips cleanly)
- README/docs/CLI help updated; default provider remains typesafe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
x