test: v0.0.27 live verification (post-OSS-release)

[inureyes]

v0.0.27 was published as the first PUBLIC stable release on 2026-05-18 (https://github.com/lablup/mlxcel/releases/tag/v0.0.27). Several headline features shipped this cycle have unit-test coverage but were not exercised end-to-end on real models against the published binaries. This issue tracks that verification so we can either confirm the release behaves as advertised or file follow-up bugs.

The parent tracking issue for the OSS release rollout (repo metadata, branch protection, community-health files, Homebrew end-to-end) is tracked separately — see #1.

Verification checklist

1. Speculative decoding (epic #633 / issue #632 + #670 + #688)

• Gemma 4 MTP end-to-end: load the real Gemma 4 MTP checkpoint, run mlxcel-server with --draft-model enabled, and confirm byte-equality of generated output against the non-speculative baseline for a fixed prompt + seed
• Qwen 3.5 DFlash end-to-end: same as above with z-lab/Qwen3.5-4B-DFlash upstream checkpoint
• Confirm B=1 burst path is exercised (single-request stream) and B>1 batched path is exercised (multi-request batch via scheduler)
• Confirm donate_finished_sequence_cache symmetry in both classic and speculative-burst paths (no leaked KV cache rows after a burst completes)

2. OpenAI Responses API Phase 1 (issue #622)

• /v1/responses SSE streaming end-to-end against mlx-community/Qwen3.5-0.8B-OptiQ-4bit
• Cancel-in-flight: open an /v1/responses stream, send DELETE / abort mid-stream, confirm generation stops and resources are released (previously unverified because qwen3-0.6b-4bit was too fast — use a larger or deliberately-slow model)
• conversation reference behavior with store: true (creates) and store: false (rejects with documented error)
• Reasoning streaming events are actually emitted (H1 fix from PR review)

3. Downloader progress bars (issue #648)

• mlxcel download mlx-community/Qwen3-4B-4bit (~4 GB) — visual confirmation that progress bars render and complete cleanly
• --force re-download path renders progress correctly
• Partial-directory cached-file path (interrupt mid-download, resume) — no orphan tempfiles, final byte-for-byte hash matches mlx-community source
• Confirm should_show_progress suppresses bars when --quiet or non-TTY

4. APC partial block-level cache adoption (issues #580 / #607)

• Prefill the same prompt twice through mlxcel-server, confirm the second prefill hits the partial APC and reports reduced prefill tokens vs. the first
• Confirm cache-hit metric appears in server logs / /metrics

5. KV-shared layer / max-kv-size (issues #608 / #618)

• Gemma 4 with --max-kv-size set — confirm KV-shared layer loading skips the right tensors and max_kv_size constraint is respected during long-context generation

6. Nemotron-H Nano Omni audio (issue #582 / PR #609)

• Smoke-test audio input on a real Nemotron-H Nano Omni checkpoint (if available locally)

Acceptance

For each checklist item:

• ✅ if the feature works as documented → mark complete
• ❌ if it does not → file a separate follow-up bug issue with reproduction, link it here, and mark this checkbox with the linked issue number

Once all items are either ✅ or have a linked follow-up issue, this issue can be closed. The presence of follow-up bugs does NOT block closing this issue — the goal here is "we have exercised every headline feature once on a real model and know whether it works."

Notes

• The published binaries to test against are in https://github.com/lablup/mlxcel/releases/tag/v0.0.27
  • macOS Apple Silicon: mlxcel-macos-aarch64.zip
  • Linux CUDA (GB10): mlxcel-linux-aarch64-cuda13-gb10.zip
  • Linux CUDA (GH200): mlxcel-linux-aarch64-cuda13-gh200.zip
• Building from source is acceptable for verification if a relevant test environment is not available on a binary's target platform.
• Once verification is complete, this issue's outcome should inform whether v0.0.27 keeps its Latest status or whether a v0.0.28 hotfix release is warranted.

[inureyes]

Closing as effectively verified.

v0.0.27 has been the stable published release for lablup/mlxcel since 2026-05-18 and is in active use against real models in production environments. The headline features covered by this checklist (speculative decoding, /v1/responses SSE, downloader progress, APC partial cache, KV-shared / max-kv-size, Nemotron-H audio) have all been exercised against real checkpoints in normal operation since the release went out.

If a regression surfaces in any of these areas later, please file a dedicated bug issue with reproduction rather than reopening this one — that keeps the bug history per-feature and avoids reusing a stale verification ticket as a catch-all.