Gemma 4 reader cannot load mixed compressed-tensors NVFP4 checkpoints

[Poyo-Yon]

Before you start

• I have read the FAQ and my problem is not answered there.
• I have read the Roadmap and this is not already planned there.
• I have searched existing issues and found no duplicate.
• I am on the latest release, or on a freshly rebuilt main when building from source.

What happened

FreeToken 0.1.3 / current main cannot start unsloth/gemma-4-26B-A4B-it-NVFP4.

The checkpoint mixes compressed-tensors per-channel FP8 attention with compressed-tensors NVFP4 shared MLPs and routed experts, while the Gemma 4 reader currently understands the corresponding ModelOpt spellings only.

The 16.9 GB safetensors file is read, then model initialization stops at:

KeyError: 'model.layers.0.self_attn.qkv_proj.weight_scale'

The checkpoint stores attention weights as FP8 weight plus per-channel [out_features, 1] weight_scale tensors; shared MLPs and routed experts as weight_packed, weight_scale, weight_global_scale, and input_global_scale; and static k_scale / v_scale quantizer metadata.

Expected: these roles should map to the existing FP8 linear, NVFP4 linear, and NVFP4 expert-bank buffers so the checkpoint can serve like the supported ModelOpt Gemma 4 NVFP4 checkpoints.

How did you install FreeToken

Built from source

FreeToken version

cac247a (0.1.3)

OS

Windows 11

OS details

Windows 11 + Ubuntu 24.04 under WSL2; kernel 6.18.33.2-microsoft-standard-WSL2; Python 3.12

GPU and driver

NVIDIA GeForce RTX 5060 Ti 16 GB, driver 616.92; second GPU: NVIDIA GeForce RTX 5070 Ti 16 GB

CPU and system RAM

12th Gen Intel Core i7-12700F; 31 GiB RAM exposed to WSL

Checkpoint

unsloth/gemma-4-26B-A4B-it-NVFP4

Command

ft serve \
  --model /path/to/gemma-4-26B-A4B-it-NVFP4 \
  --served-model-name unsloth-gemma4-26b-a4b-nvfp4 \
  --gpu 0 \
  --host 127.0.0.1 --port 1919 \
  --moe-strategy offload --moe-cache-rate 0.4 --expert-load serial \
  --memory-ratio 0.8 --max-seq-len-override 4096 \
  --max-output-tokens 512 --max-prefill-length 512 \
  --max-running-requests 1 --cuda-graph-max-bs 1 --text-model-only

Full log

Model initialization reaches Gemma 4 weight loading and fails with:

KeyError: 'model.layers.0.self_attn.qkv_proj.weight_scale'

No earlier download or safetensors read error was reported.

Anything else

Checkpoint revision: 20df0542b1a86ce19f495ac2eca2c7c12bce82f9.

I have a small patch plus weight-free regression tests. On the real checkpoint, the patched build completes weight loading, expert-bank construction, KV-cache allocation, CUDA graph capture, and OpenAI-compatible Japanese chat generation.

Validation: text-only, 4096 context, NVFP4 expert offload, 40% expert cache; 52,597-token KV cache; post-warm-up output rates 34.48, 59.44, and 60.65 tok/s (median 59.44 tok/s).

AI assistance was used to prepare the patch and tests. I reviewed the behavior and ran the real-checkpoint validation described above.