MiniJev, minus the ceiling: a Jev-style decision model on a Qwen3-0.6B backbone (LoRA, rank 16), fused with a full OpenAI-compatible generation server — one set of weights, two capabilities, zero-copy LoRA switching. Built and benchmarked end-to-end on Apple Silicon (MPS).
This is a learning project — the continuation of MiniJev. MiniJev proved the System One API shape works with a from-scratch 680K model, then face-planted on hand-written phrasing (58.8% holdout) because a word-level tokenizer cannot generalize. JevNext asks: how much of the gap does a pretrained backbone close, and what does it cost? It is not production software.
MiniJev's holdout failure decomposed into two root causes — token coverage (473-word vocab, 42% UNK on novel phrasing) and semantic generalization. Both are properties a small from-scratch model cannot acquire from synthetic data. JevNext keeps MiniJev's data, holdout, and evaluation protocol byte-for-byte identical and swaps only the model, so the comparison is a controlled experiment.
Result: same 102-question hand-written holdout, MiniJev 58.8% → JevNext 86–91% (see Evaluation).
(state, question, candidate) --Qwen BPE--> one full text path per candidate (no prefix sharing)
|
v
Qwen3-0.6B decoder body (bf16 on MPS, frozen; LoRA rank-16 injected into all 196 projections)
| last-token hidden state per path
v
scalar scoring head: LayerNorm + Linear(hidden -> 1)
set-attention head: candidates attend to each other (4 heads, dim 128, + log(k) feature, zero-init output)
|
v
candidates of the same question -> softmax -> noul / choice / score distributions
Same "everything is candidate scoring" philosophy as MiniJev/NanoJev: non-autoregressive, usage.output_tokens = 0 on the decision route.
pip install -r requirements.txt # torch (MPS build), transformers, safetensors, numpy
# Train the decision LoRA on the Instruct backbone (~50 min on M4 32GB, ~1.3GB RAM)
python -m jevnext.train \
--data ../MiniJev/data/decisions.jsonl \
--backbone Qwen/Qwen3-0.6B \
--mode lora --steps 240 --out checkpoints-lora-instruct
# Evaluate all splits + the 45-case holdout
python -m jevnext.evaluate \
--checkpoint checkpoints-lora-instruct \
--data ../MiniJev/data/decisions.jsonl
# Serve: decisions + generation on one port
python -m jevnext.server --checkpoint checkpoints-lora-instruct --port 8189A pre-trained adapter (checkpoints-lora-instruct/, ~40MB, seed 17) is included so you can skip training. The Qwen3-0.6B backbone (Apache-2.0) is downloaded automatically from Hugging Face on first run.
| Topic | Conclusion |
|---|---|
| GPU | PyTorch MPS (Metal) backend; --device auto |
| Full fine-tune | fits (8.4GB static) but swaps under memory pressure — not worth it |
| LoRA | backbone bf16 frozen (1.28GB, zero optimizer state), 10.3M trainable (1.7%), ~10s/step stable |
| MPI | not applicable — on a single SoC it only adds IPC overhead; MPS is the answer on one Mac |
| Route | Mode | LoRA state |
|---|---|---|
POST /v1/systemone |
decision (candidate scoring) | ON |
POST /v1/completions |
raw text generation | OFF (clean backbone) |
POST /v1/chat/completions |
chat with thinking | OFF |
FusedEngine loads one Qwen3-0.6B (with LM head) and wraps its decoder body in DecisionModel. Switching modes is a zero-copy flag flip on each LoRALinear (the LoRA is trained for scoring, not generation — leaving it on would pollute generation). A global lock serializes MPS inference across threads.
# Decision (System One)
curl -s --noproxy '*' -X POST http://127.0.0.1:8189/v1/systemone \
-H "Content-Type: application/json" \
-d '{"model":"jevnext-latest",
"state":"I was billed twice for my annual plan and need one charge reversed.",
"questions":{"route":{"type":"choice","instructions":"Which team?",
"criteria":{"billing":"payment or charge problems",
"technical":"application bugs"}}}}'
# Chat with visible thinking (reasoning_content, DeepSeek/vLLM style)
curl -s --noproxy '*' -X POST http://127.0.0.1:8189/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"jevnext-latest",
"messages":[{"role":"user","content":"9.11 and 9.8, which is larger?"}],
"max_tokens":512,"temperature":0.6}'
# Streaming (SSE, OpenAI chunk format)
curl -sN --noproxy '*' -X POST http://127.0.0.1:8189/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"jevnext-latest","messages":[{"role":"user","content":"tell me a joke"}],
"max_tokens":256,"stream":true,"stream_options":{"include_usage":true}}'- Token-level generation loop with manually managed KV cache — not fake chunking.
ThinkSplitter: an incremental state machine that splits<think>...</think>intoreasoning_contenteven when the tags are split across tokens.SafePieceDecoder: byte-level BPE can split one character (an emoji, a rare CJK glyph) into 2–3 tokens; naive per-token decoding emits U+FFFD. The decoder buffers until the UTF-8 sequence is complete.enable_thinking: falsepasses through to the Qwen3 chat template. Note: thinking and content share themax_tokensbudget.
temperature, top_p, top_k, min_p, repetition_penalty, presence_penalty, frequency_penalty, seed, stop, stop_token_ids, include_stop_str_in_output, ignore_eos, stream_options.include_usage — pipeline order matches vLLM: penalties -> temperature -> top_k -> softmax -> min_p -> top_p -> sample.
ThreadingHTTPServer accepts concurrent requests; MPS inference is serialized by a lock (GPU is physically single-stream). Verified: 6 concurrent requests (3 decisions + 3 generations) all succeed; a decision issued mid-stream returns identical bits to the idle-case result (no LoRA-state leakage).
Protocol identical to MiniJev (same splits, same 45-case hand-written holdout, fail-fast, ECE/Brier + jev-eval metrics). Length-bucketed batching (evaluate.py) gives +61% eval throughput on MPS.
| Holdout (102 questions) | MiniJev (from-scratch) | JevNext-Base (Qwen3-0.6B-Base + LoRA) | JevNext-Instruct (Qwen3-0.6B + LoRA, seed 17 / seed 42) |
|---|---|---|---|
| Accuracy | 58.8% | 91.2% (93) | 86.3% (88) / 87.3% (89) |
Notes on reading these numbers (the interesting part):
- The Base-vs-Instruct gap is not a systematic regression. Per-question paired analysis (McNemar): only 3 questions fail under both Instruct seeds while passing under Base (net systematic difference ≈ 1 question); the two Instruct seeds disagree with each other on 11 questions (seed noise σ ≈ ±3 on a 102-question set). p ≈ 0.27 — the two backbones are statistically indistinguishable for this task.
- Calibration direction is fixed vs. MiniJev: mean confidence on correct answers (0.92) exceeds that on errors (0.83).
- Latency trade-off is real: ~1.4s per decision case (0.6B × k candidate paths) vs. MiniJev's 1.6ms. Decision quality vs. speed is the actual engineering choice.
- Full numbers:
results/*.json, incl. per-question holdout rows and the paired-analysis inputs.
All decision data is synthetic, rule-generated, and shared verbatim with MiniJev (same files, same splits) so results are comparable across projects:
| Set | Size | Purpose |
|---|---|---|
train / dev / test / ood |
../MiniJev/data/decisions.jsonl |
see the MiniJev README for generation rules |
| holdout | 45 cases / 102 questions, defined in ../MiniJev/holdout_eval.py |
hand-written synonym paraphrases, novel markers, adversarial distractors, far-OOD (German); never in any training split |
# Syntax check of the full package
python -m py_compile jevnext/*.py *.py
# Unit-adjacent sanity: run the MiniJev holdout against a running server
python -m jevnext.server --checkpoint checkpoints-lora-instruct --port 8189 &
python ../MiniJev/holdout_eval.py --url http://127.0.0.1:8189/v1/systemone \
--model jevnext-latest --out results/holdout_report.jsonBenchmarks: python -m jevnext.bench (MPS vs CPU), batch_bench.py / bucket_ab.py (bucketing A/B), calibrate.py (temperature scaling — current conclusion: T≈0.95≈1, no scaling needed).
- Jev — System One models (TypeSafe) — the decision-model concept and API spec. Independent learning project; not affiliated with or endorsed by TypeSafe.
- MiniJev — the from-scratch predecessor whose data, holdout, and eval protocol this project reuses.
- NanoJev — the scoring-head + frozen-backbone design this architecture follows.
- Qwen3 (Apache-2.0) — the 0.6B backbone. Backbone weights are fetched from Hugging Face at runtime and are not redistributed here.
- vLLM — the sampling-parameter semantics this server aligns to.
- LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021).
reasoning_contentfield convention popularized by DeepSeek's API.
This software is provided "as is" for educational purposes only, without warranty of any kind. The decision dataset is synthetic; model behavior on real-world text is not guaranteed. Do not use JevNext to make or automate real-world decisions (routing, escalation, triage) without human oversight. Generation quality is bounded by a 0.6B model — treat its outputs (including its "thinking") as demonstrations, not authoritative reasoning. The authors accept no liability for any use of this software.
MIT. The included LoRA adapter is a derivative of Qwen3-0.6B (Apache-2.0); the Qwen3 license and notice apply to the backbone weights.