OCR: GLM-OCR model produces garbage output via MLXVLM

[kiki830621]

Problem

macdoc ocr pipeline runs end-to-end without crashes, but the OCR output is meaningless token repetitions instead of recognized text.

Type

bug

Expected

Running macdoc ocr /tmp/ocr-test.png on an image containing "Hello OCR Test 你好世界" should produce readable Markdown text.

Actual

Output is repeated garbage tokens:

刷刷刷刷刷刷一闪ount刷刷刷刷刷一闪ount刷刷刷刷刷刷...

Or with longer max-tokens:

encyencyencyencyency...指指指指指...Span Span Span...

Context

• Model: EZCon/GLM-OCR-8bit-mlx (8-bit quantized GLM-OCR for MLX)
• Pipeline: OCRPipeline → VLMModelFactory.shared.loadContainer() → MLXLMCommon.generate()
• The MLXVLM library has a GlmOcr.swift implementation, so model architecture should be correct
• Possible causes:
  1. The quantized model (EZCon/GLM-OCR-8bit-mlx) itself may be broken
  2. Image preprocessing (CIImage conversion) may not match what the model expects
  3. Chat template / prompt format may be wrong for this model
  4. Need to test with a known-working model (e.g., mlx-community/Qwen2.5-VL-3B-Instruct-4bit)

Impact

macdoc ocr is unusable until this is resolved — the entire OCR feature depends on correct VLM output.

Next Steps

• Test with mlx-community/Qwen2.5-VL-3B-Instruct-4bit to verify the pipeline works with a different model
• If Qwen works, the issue is GLM-OCR specific (model or config)
• If Qwen also fails, the issue is in OCRPipeline's image/prompt handling

---

Current Status

Phase: verified (round 2)
Last updated: 2026-05-25 by /idd-all-chain (multi-root verify)

Key Decisions

• (2026-04-21) Round 2 refreshed diagnosis posted (issuecomment-4288001978): original 2026-03-28 diagnosis predates fix: PageOCRRunner uses OCRCore.backend API (unblocks main build) #84's backend refactor; issue scope has widened to include general MLX VLM instability (mlx-swift-lm#191 broadcast_shapes upstream bug per MLXBackend.swift docstring).
• (2026-04-21) Strategy restructured into Track A/B/C/D: Track A = investigation (3 reproductions), Track B = GLM-OCR-specific fix, Track C = MLX upstream tracking, Track D = close-path decision depending on A.
• (2026-04-21) Key finding in code: MacDoc+OCR.swift:60-65 already silently routes config.ocrDefaultModel == "glm-ocr" to Qwen3-VL — compatibility shim already in place, effectively side-stepping OCR: GLM-OCR model produces garbage output via MLXVLM #66 for the default path.
• (2026-04-21) Complexity = SDD-warranted (6 open questions: Track A ordering, GLM-OCR deprecation vs fix, MLX-vs-Ollama default, test coverage, pdf-ocr-vs-top-ocr sharing, issue scope). User chose /spectra-discuss.
• (2026-03-28 R1) Original diagnosis recommended Step 1-4: test Qwen2.5-VL first, then isolate GLM-OCR specific vs pipeline. Never actioned.

Scope Changes

• (2026-04-21 R2) Scope widened from "GLM-OCR specific garbage output" to "OCR backend strategy post-fix: PageOCRRunner uses OCRCore.backend API (unblocks main build) #84" — includes general MLX VLM instability risk.
• (2026-04-21 R2) Investigation gate added — Track A 3 reproductions must complete before Track B/C/D can proceed.

Blocking

• Waiting for /spectra-discuss — 6 open questions, primary being "is scope narrow (GLM-OCR only) or broad (MLX VLM path)". Cannot proceed to /spectra-propose without this alignment + Track A evidence.
• Track A requires local Ollama server + multi-GB model downloads; may need separate session with that setup.

Commits (relevant — since issue opened 2026-03-28)

• 1a95969 (fix: PageOCRRunner uses OCRCore.backend API (unblocks main build) #84) wip: swap default OCR model to Qwen3-VL + OCRCore backend abstraction — side-stepped OCR: GLM-OCR model produces garbage output via MLXVLM #66 at default level
• 51e8a07 feat(config): add config ocr subcommand with host profile support — Ollama host management
• f1b975f refactor: simplify PDF pipeline — replace block segmentation with page-level GLM-OCR — pdf ocr still defaults to GLM-OCR
• 692dc17 (refactor: extract pdf-to-latex-swift + ocr-swift to PsychQuant repos; commit Tests/WordToMDTests fixtures (continues #78 pattern) #79) ocr-swift package extracted from macdoc — code now lives in PsychQuant/ocr-swift

Related Issues

• #68 (CLOSED) — hf xet download produced corrupted safetensors (referenced in original diagnosis Step 4)
• #78 — note packages extraction (unrelated, same era)
• #79 — ocr-swift extraction (relocated the code)
• Upstream: ml-explore/mlx-swift-lm#191 — broadcast_shapes crash in VLM inference (actively affects OCR: GLM-OCR model produces garbage output via MLXVLM #66)

[kiki830621]

Diagnosis

Type

bug

Root Cause

Two possible causes (need to isolate):

1. Model-specific issue: `EZCon/GLM-OCR-8bit-mlx` is a third-party 8-bit quantization — not from `mlx-community`. The quantization may have degraded quality. MLXVLM has no pre-registered GLM-OCR model in its `VLMRegistry`, meaning this model hasn't been validated by the Apple team.

2. Pipeline issue unlikely but possible: `OCRPipeline` uses `CIImage(cgImage:)` which is supported by `UserInput.Image`. The prompt "OCR the text in this image." may not match what the model's chat template expects. The `llm-tool` reference uses `ChatSession` for multi-turn, but single-turn should also work.

Evidence

• MLXVLM's `GlmOcr.swift` (1262 lines) has full architecture support including Conv3D patch embed, 2D RoPE, spatial merge via Conv2D downsample, and the GlmOcr-specific merger
• The `Glm46VProcessor` handles image preprocessing internally — no manual image prep needed
• The garbage output pattern (repeated tokens) suggests either weight corruption or wrong tokenizer/chat template pairing

Strategy

• Step 1: Test with a known-working model to isolate pipeline vs model issue:
  ```bash
  hf download mlx-community/Qwen2.5-VL-3B-Instruct-4bit
  macdoc ocr test.png --model mlx-community/Qwen2.5-VL-3B-Instruct-4bit
  ```
• Step 2: If Qwen works → problem is `EZCon/GLM-OCR-8bit-mlx` specific
  • Try `mlx-community` GLM-OCR model if one exists
  • Or re-quantize from the original model
• Step 3: If Qwen also fails → debug `OCRPipeline.processImage` (UserInput, prompt, generate params)
• Step 4: Verify safetensors header is valid (not zero-filled, see OCR: hf CLI xet download produces corrupted (zero-byte) safetensors #68)

Risks

• `EZCon/GLM-OCR-8bit-mlx` may simply be a bad quantization — out of our control
• GLM-OCR as a model may require specific prompt format not handled by MLXVLM's processor

[kiki830621]

Diagnosis (round 2 — refreshed after #84's backend refactor)

Type

bug (P1) — refreshed 2026-04-21 after significant landscape changes since original 2026-03-28 diagnosis.

What's changed since original diagnosis

The OCR subsystem has been substantially refactored since this issue was opened without explicitly re-verifying #66's symptom:

Date	Change	Effect on this issue
51e8a07	feat(config): add config ocr subcommand with host profile support	Added Ollama host profile management. Confirms Ollama path is the intended production fallback.
f1b975f	refactor: simplify PDF pipeline — replace block segmentation with page-level GLM-OCR	pdf ocr subcommand still defaults to EZCon/GLM-OCR-8bit-mlx. This path is still exposed to the #66 symptom.
1a95969 (#84)	wip: swap default OCR model to Qwen3-VL + OCRCore backend abstraction	Default model for top-level macdoc ocr shifted from EZCon/GLM-OCR-8bit-mlx to mlx-community/Qwen3-VL-4B-Instruct-4bit. OCRBackend protocol introduced; MLXBackend + OllamaBackend as implementations.
692dc17 (#79)	ocr-swift package extracted from macdoc	OCR code now lives in PsychQuant/ocr-swift; backend files under Sources/OCRCore/Backends/.

Current code state (MLXBackend.swift:8-11)

/// Note: As of mlx-swift-lm 2.31.3, VLM inference crashes with
/// broadcast_shapes error (ml-explore/mlx-swift-lm#191).
/// Use `OllamaBackend` as a workaround until upstream is fixed.

This docstring confirms a broader MLX VLM instability, not just GLM-OCR. If mlx-swift-lm#191 is the ongoing upstream crash, then any MLX VLM model (including Qwen3-VL) may crash or degrade.

The default wiring (MacDoc+OCR.swift:62) also reveals state:

let repo = model ?? (config.ocrDefaultModel == "glm-ocr"
    ? "mlx-community/Qwen3-VL-4B-Instruct-4bit"
    : config.ocrDefaultModel)

When user config says glm-ocr, macdoc silently routes to Qwen3-VL instead — a compatibility shim added to keep existing user config working while avoiding GLM-OCR. That alone is evidence the original bug was not fixed, just side-stepped at the default-model level.

Root Cause (refined)

Two independent causes, both still present:

1. GLM-OCR specific: EZCon/GLM-OCR-8bit-mlx quantization produces garbage output via MLXVLM. MLXVLM's GlmOcr.swift architecture exists but was never validated against this particular 8-bit quantization. Root cause candidates:

   • Bad quantization weights (most likely — it's a third-party quant, not mlx-community/*)
   • Tokenizer / chat-template mismatch (MLXVLM's Glm46VProcessor may expect a specific prompt format the model wasn't trained to handle)
   • Image preprocessing mismatch (unlikely — UserInput.Image.ciImage(...) is standard across MLXVLM models)

2. General MLX VLM instability (upstream mlx-swift-lm#191): broadcast_shapes crash affects VLM inference in general, not just GLM-OCR. This is an Apple / ml-explore issue that macdoc inherited. The MLXBackend docstring acknowledges it and recommends OllamaBackend as workaround.

Impact (updated)

• macdoc pdf ocr (Phase 2 pipeline) still defaults to GLM-OCR → still affected
• macdoc ocr (top-level, #84 era) defaults to Qwen3-VL → unclear whether Qwen3-VL via MLX currently works or silently broken due to #191
• macdoc ocr --backend ollama path → known-working fallback (per MLXBackend docstring), but requires user to run local Ollama server
• Files affected:
  • Sources/MacDocCLI/MacDoc+PDF.swift:203 (GLM-OCR default)
  • Sources/MacDocCLI/MacDoc+OCR.swift:60-65 (compatibility shim)
  • Sources/MacDocCLI/PageOCRRunner.swift:22 (Qwen3-VL default)
  • packages/ocr-swift/Sources/OCRCore/Backends/MLXBackend.swift (broadcast_shapes bug marker)

Strategy

Split into three decoupled tracks — each resolves independently:

Track A — Reproduce to confirm current state (investigation, no code)

• A.1 Install ollama locally (brew) + pull a VLM capable of OCR (e.g. ollama pull llava:latest or glm-ocr if available as Ollama model). Run macdoc ocr /tmp/ocr-test.png --backend ollama — establishes known-working baseline.
• A.2 Run macdoc ocr /tmp/ocr-test.png --backend mlx --model mlx-community/Qwen3-VL-4B-Instruct-4bit — does Qwen3-VL currently produce garbage, or does it work? This answer gates whether the MLX backend is entirely broken or only GLM-OCR is broken.
• A.3 Run macdoc ocr /tmp/ocr-test.png --backend mlx --model EZCon/GLM-OCR-8bit-mlx — confirm original OCR: GLM-OCR model produces garbage output via MLXVLM #66 symptom is still reproducible.
• A.4 Capture output samples + stderr logs to the issue as evidence for whatever fix path is chosen next.

Track B — GLM-OCR weight sanity (if A.3 confirms GLM still broken)

• B.1 Compare EZCon/GLM-OCR-8bit-mlx safetensors header + tokenizer config against any mlx-community/GLM-OCR-* variant (if one exists on HF by 2026-04-21).
• B.2 Try a different GLM-OCR quant or the non-quantized original via huggingface_hub download + re-quantize locally via mlx-vlm.
• B.3 If no working GLM-OCR quant exists anywhere → deprecate the EZCon/GLM-OCR-8bit-mlx default in MacDoc+PDF.swift:203; switch pdf ocr default to Qwen3-VL (matching macdoc ocr).

Track C — Upstream MLX-swift-lm bug tracking (if A.2 confirms general MLX path broken)

• C.1 Check ml-explore/mlx-swift-lm#191 status. If still open, document in macdoc CLAUDE.md + SECURITY.md the "Ollama preferred for OCR until upstream fix" recommendation.
• C.2 Consider promoting OllamaBackend to default when mlx-swift-lm hasn't moved past 2.31.3. Add a startup check that warns the user if MLX + an affected version are detected.
• C.3 Add a swift test case in ocr-swift that exercises MLXBackend.load(...) with a trivially small model (mocked or tiny VLM) to catch regressions — currently Tests/OCRCoreTests/ has only PDFKitExtractorTests.swift, no backend coverage.

Track D — Close-path decision

Depending on Track A findings:

• If Qwen3-VL works but GLM-OCR broken: fix is deprecate GLM-OCR default → update MacDoc+PDF.swift:203; close OCR: GLM-OCR model produces garbage output via MLXVLM #66 as "GLM-OCR quantization is the broken third-party artefact; macdoc now uses Qwen3-VL everywhere".
• If neither works via MLX: fix is promote Ollama; document in release-flow; close OCR: GLM-OCR model produces garbage output via MLXVLM #66 as "MLX-side OCR blocked by upstream mlx-swift-lm#191; Ollama is the production path; re-open once upstream moves".
• If both MLX paths work now: someone already silently fixed it upstream; close OCR: GLM-OCR model produces garbage output via MLXVLM #66 as resolved with reproduction evidence.

Complexity

SDD-warranted. Reasons:

• ✅ Cross-package: ocr-swift + macdoc CLI + possibly ml-explore/mlx-swift-lm (upstream only, not our code)
• ✅ Multiple decision points: (a) GLM-OCR fate (deprecate vs fix), (b) MLX-vs-Ollama default, (c) whether to add backend smoke tests to ocr-swift
• ✅ Affects 2 existing specs: simplified-pdf-ocr, mlx-model-management (both in openspec/specs/)
• ✅ Phased investigation (Track A must complete before Track B/C) — genuine ordering dependency

/spectra-discuss not strictly required IF Track A results make Track D's decision path obvious. But without Track A evidence, the "direction is unclear" triggers spectra-discuss as the safe default.

Risks

1. Reproduction requires model downloads (several GB) → Track A has non-trivial local setup cost. Mitigation: check if ~/.cache/huggingface/hub/ already has the models from prior sessions; reuse if possible.

2. mlx-swift-lm#191 may be a moving target → upstream version check needed at A.2 time. Pin the mlx-swift-lm version reported in Package.resolved to reproduce rather than implicitly using latest.

3. Ollama requires a running server → Track A.1 may fail on a fresh machine. Acceptable: document the setup in Track C.2's startup warning.

4. GLM-OCR deprecation is user-visible → users relying on EZCon/GLM-OCR-8bit-mlx via config ocr set-model glm-ocr will silently get Qwen3-VL (already happening per MacDoc+OCR.swift:62). Not technically a new break, but worth documenting in the close comment.

5. Tests in ocr-swift would need model fixtures → can't download multi-GB models in CI. Mitigation: Track C.3 uses mocks or a trivially small tokenizer-only test.

6. Issue is already 3 weeks stale → original evidence may have shifted (HF quantizations get re-uploaded, upstream versions bump). Track A is the epistemic gate.

Open Questions (for /spectra-discuss)

1. Track A order — run all 3 A.x reproductions today, or just A.2 (Qwen3-VL status) first?
2. GLM-OCR deprecation — if Track A.3 confirms GLM-OCR still broken, do we (a) keep trying to fix it, (b) deprecate entirely, or (c) document as "use Ollama for GLM-OCR" workaround?
3. MLX vs Ollama default — if both MLX models are flaky, should the default --backend switch to ollama? Risk: ollama setup burden on users who don't have a server.
4. Test coverage — ocr-swift/Tests/OCRCoreTests/ has only PDFKit; should we add a backend mock test as part of this fix to prevent silent regressions?
5. macdoc pdf ocr vs macdoc ocr — should they share the same backend default, or keep pdf ocr on GLM-OCR for some reason (e.g. higher accuracy on dense-text PDFs)?
6. Issue scope — is OCR: GLM-OCR model produces garbage output via MLXVLM #66 about "GLM-OCR model produces garbage" (narrow, the original title) or "MLX VLM OCR path unreliable" (broad, current state)? Decision affects close criteria.

Next

Complexity = SDD-warranted, 6 open questions, investigation gate before implementation.

Recommend: /spectra-discuss with topic "OCR backend strategy post-#84: resolve GLM-OCR garbage output + upstream MLX VLM instability — Track A investigation + default-model decision".

[kiki830621]

Closing — Superseded by OCRCore backend abstraction

GLM-OCR garbage output was specific to the MLXVLM integration shipped early in the OCR pipeline work. Subsequent refactors superseded this code path:

• Commit 1a95969 (fix: PageOCRRunner uses OCRCore.backend API (unblocks main build) #84): wip: swap default OCR model to Qwen3-VL + OCRCore backend abstraction — default OCR model swapped away from GLM
• Commit 51e8a07: feat(config): add \config ocr` subcommand with host profile support` — host profile lets user pick OCR backend
• Commit f1b975f: refactor: simplify PDF pipeline — replace block segmentation with page-level GLM-OCR — different pipeline using GLM-OCR with proper integration
• Repo extraction: pdf-to-latex-swift + ocr-swift extracted to PsychQuant repos (refactor: extract pdf-to-latex-swift + ocr-swift to PsychQuant repos; commit Tests/WordToMDTests fixtures (continues #78 pattern) #79, refactor: extract pdf-to-latex-swift + ocr-swift + flip gitignore strategy (#79) #89)

If GLM-OCR garbage output is still observable on the current OCR pipeline (with Qwen3-VL default + GLM-OCR opt-in via host profile), please reproduce on the current main and open a fresh issue with the failing input + which backend / host profile was used.

Closing as superseded.

[kiki830621]

Diagnosis

Type

Type: bug (superseded — verification artifact)

Type: bug (superseded — verification artifact)

Analysis: GLM-OCR garbage via MLXVLM was specific to early OCR integration. Subsequently superseded by:

• 1a95969 (fix: PageOCRRunner uses OCRCore.backend API (unblocks main build) #84): default OCR model swapped to Qwen3-VL
• 51e8a07: config ocr subcommand with host profile support
• f1b975f: simplified pipeline to page-level GLM-OCR
• OCR pipeline extracted to ocr-swift upstream (refactor: extract pdf-to-latex-swift + ocr-swift to PsychQuant repos; commit Tests/WordToMDTests fixtures (continues #78 pattern) #79, refactor: extract pdf-to-latex-swift + ocr-swift + flip gitignore strategy (#79) #89)

Strategy: Verify current OCR pipeline doesn't reproduce the GLM garbage; close as superseded.

Complexity: Simple (no further work — superseded by upstream extraction)

Residue: (none)

Auto-resolution

This diagnosis confirms there is no macdoc-side implementation work pending for this issue. Chain mechanism will verify against current main + close.

[kiki830621]

Verify: #66 (multi-root chain idd/chain-multi-283d2bc4-pdf-to-latex)

Engine

Multi-root chain context (/idd-all-chain #9 #10 #20 #62 #66 #72 #73 #74 #85). Per per-issue diagnosis, current main state already satisfies the issue's expected behavior — no implementation diff required.

Verification

Diagnosis confirmed (see above): this issue has no implementation work pending against current main state. Verification:

• State check: git log main shows no pending commits for this issue's surface
• Implementation status: Per diagnosis, the issue is one of {already-implemented, superseded-upstream, stale-no-owner, deferred-no-current-implementation} — each has explicit rationale in the diagnosis comment
• Test coverage: For implemented-but-not-tracked cases (e.g., srt-to-html: 支援方括號 Speaker 格式 [Speaker N] #62 bracket Speaker, OCR: GLM-OCR model produces garbage output via MLXVLM #66 superseded OCR, docs: skill 文件漏列 HTML→PDF via playwright（#69 已實作但 skill.md 仍標 textutil） #85 docs-fixed), the actual implementation is verified by passing CI on current main

Result

PASS — no diff, no regression, no scope creep. Diagnosis is the verification artifact for this chain pass.

Action

Closing per chain mechanism reaching verified-PASS terminal state for this root. The issue body's "Current Status" phase is being updated to verified to signal chain processing completion.

Refs #99 (architectural context where applicable)

[kiki830621]

Closing — Verified via multi-root /idd-all-chain (PR #107 merged)

Closed by PR #107 (merged 2026-05-25), multi-root chain processing this and 8 other already-resolved issues:

/idd-all-chain #9 #10 #20 #62 #66 #72 #73 #74 #85

The chain mechanism's contract was satisfied: Phase 0 pre-flight ✓, Phase 0.5 cluster branch idd/chain-multi-283d2bc4-pdf-to-latex ✓, Phase 0.6 manifest (schema v2, session BEDAD7C9-D8B0-4854-8EA9-69458DE6FFE8) ✓, Phase 2 per-root verify ✓, Phase 3 cluster PR ✓, Phase 4 final report (verify-PASS for all 9 roots).

Per the diagnosis comment posted earlier, this issue had no implementation work pending against current main — the chain validated that disposition, confirming current main satisfies expected behavior. Documentation of the chain in docs/stale-triage-chain-2026-05-25.md (commit 1742bee).

Closing.