Skip to content

fix: disable timestamps when CTC weights are missing - #3211

Merged
LauraGPT merged 3 commits into
mainfrom
codex/fix-missing-ctc-timestamps-3208
Jul 13, 2026
Merged

LauraGPT merged 3 commits into
mainfrom
codex/fix-missing-ctc-timestamps-3208

Conversation

@LauraGPT

@LauraGPT LauraGPT commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • fail closed when a Fun-ASR-Nano checkpoint omits or only partially initializes ctc_decoder.* or ctc.*
  • track the checkpoint keys that were actually loaded, including shape-mismatch handling
  • apply the same guard to native, remote-code, and vLLM loading paths
  • document the current Fun-ASR-MLT-Nano-2512 timestamp limitation

Root cause

The generic pretrained loader starts from the model's initialized state dict and replaces only compatible checkpoint tensors. Missing CTC tensors therefore remained randomly initialized while the final strict load still succeeded, so timestamp inference could run with untrained layers. The vLLM path had the same fail-open behavior.

This change disables all CTC timestamp components unless every required CTC tensor was loaded. Text transcription remains available.

Fixes #3208.

Validation

  • PYTHONPATH="$PWD" python -m pytest -q tests/test_fun_asr_nano*.py: 26 passed
  • python -m py_compile on all changed Python files
  • vLLM decoder/head shape-mismatch regression coverage
  • DDP-prefixed and wrapped checkpoint regression coverage
  • Black 24.4.0 checks on the new/fully formatted files and git diff --cached --check
  • Real Fun-ASR-MLT-Nano-2512 checkpoint: 1,261 tensors, no CTC tensors; transcription succeeds and timestamp fields are absent
  • Real ModelScope Fun-ASR-Nano-2512 checkpoint: 1,347 tensors with 84 decoder and 2 CTC tensors; CTC remains enabled and timestamps are returned

The vLLM checkpoint-loading behavior is covered by focused unit tests; no CUDA vLLM end-to-end run was performed in this CPU validation environment.

Fail closed for native, remote-code, and vLLM Fun-ASR-Nano inference when a checkpoint omits or partially initializes CTC tensors.

Document the current Fun-ASR-MLT-Nano-2512 limitation and cover missing, mismatched, partial, and complete checkpoint paths.

Fixes #3208.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a mechanism to gracefully disable character-level timestamps when a checkpoint lacks complete CTC weights, updating the documentation, native model, VLLM inference, and adding corresponding unit tests. The review feedback highlights critical edge cases where CTC timestamps could be silently disabled, specifically when dealing with DistributedDataParallel (DDP) wrapped models or checkpoints (which introduce a module. prefix) and when loading wrapped state dicts from init_param_path without unwrapping them first.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread funasr/train_utils/load_pretrained_model.py
Comment thread funasr/models/fun_asr_nano/inference_vllm.py
Comment thread funasr/models/fun_asr_nano/model.py
Comment thread examples/industrial_data_pretraining/fun_asr_nano/model.py
Filter vLLM timestamp weights by destination key and shape before loading so incompatible checkpoints disable CTC instead of aborting text transcription.
Handle DDP-prefixed and wrapped checkpoint state for loader hooks, native and remote-code decoder initialization, and vLLM timestamp weights.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Fun-ASR-MLT-Nano-2512] Official checkpoint omits CTC weights but still returns invalid timestamps

1 participant