Update MLX support for mlx-swift-lm 3.32.3

[mattt]

mlx-swift-lm 3.32.3 came out on September 30, and from: "3.31.4" resolves to it. It makes source-breaking changes, so the MLX trait doesn't compile against it: newCache(parameters:) now throws, prepare(_:cache:windowSize:) is replaced by prepare(_:cache:state:prefill:), and Generation has a new rejectedToolCall case.

This PR updates MLXLanguageModel for those changes and requires mlx-swift-lm 3.32.x. mlx-swift-lm's minor and patch versions follow mlx-swift's, so the requirement now uses .upToNextMinor to keep the next MLX update from breaking builds the same way. That version uses mlx-swift 0.32.3, which vendors MLX 0.32.2 and includes the fix for the fused attention causal mask (ml-explore/mlx#3271).

mlx-swift 0.32.3 declares swift-tools-version 6.3, so SwiftPM can't resolve it with earlier toolchains, even when the MLX trait is off. This PR raises AnyLanguageModel's requirement to Swift 6.3 (Xcode 26.4 or later), and updates the CI matrix and README to match.

A rejected tool call (for example, one that is malformed or incomplete) now throws RejectedToolCallError. Before, mlx-swift-lm dropped that output or passed it through as text. A call to a tool that the session doesn't declare still gets a "Tool not found" result, as with the other providers.

Models that place tokens with M-RoPE (Qwen3-VL, Qwen3.5, GLM-OCR) now throw ContinuationStateError when they continue a reused cache without the model state that goes with it. The session cache doesn't keep that state, so for those models generation falls back to an empty cache.

The structured generation path now synchronizes the current GPU stream instead of a new stream, which had no pending work.

[Copilot]

Copilot review overview

🟡 Changes recommended

Warm-cache paths do not preserve the model state required by mlx-swift-lm 3.32.3.

Review effort: Balanced
Findings: 1

Open (1)

• Warm-cache continuation drops LMOutput.State · New

What changed in this PR

Updates MLX integration for mlx-swift-lm 3.32.3 compatibility.

Changes:

• Adapts cache and prefill APIs.
• Handles rejected tool calls.
• Synchronizes the active GPU stream.

File	Description
Package.swift	Raises the mlx-swift-lm minimum version.
Sources/​AnyLanguageModel/​Models/​MLXLanguageModel.swift	Updates MLX generation, caching, tool handling, and synchronization.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Warm-cache reuse drops model state required by state-producing MLX models, causing subsequent generation to fail.

Review effort: Balanced
Findings: 1

Open (1)

• Warm-cache continuation drops LMOutput.State

Previously missed (2)

In code that hasn't changed since last review

Prewarm discards model state required with cached KV

Sources/​AnyLanguageModel/​Models/​MLXLanguageModel.swift:1471

This prewarm call discards the LMOutput.State returned by prepare and then stores the warmed KV cache alone. State-producing models in mlx-swift-lm 3.32.3 require that paired state when the cache is reused, so a prewarmed Qwen3-VL or GLM-OCR session can fail with ContinuationStateError.missingState. Store the returned state with newCache, or do not retain this prewarm cache when state cannot be preserved.

Incorrect upstream link for fused-attention causal-mask fix

Package.swift:44

The PR description attributes a fused-attention causal-mask fix to ml-explore/mlx#3271, but that linked pull request is titled “Nax Refactor” and does not describe such a fix. Please correct or remove the link so the dependency-upgrade rationale points to the intended upstream change.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Generation omits declared tool schemas, preventing undeclared calls from producing the intended rejection error.

Review effort: Balanced
Findings: 1

Open (1)

• Warm-cache continuation drops LMOutput.State

Previously missed (1)

In code that hasn't changed since last review

Pass declared tool schemas to generate to reject undeclared tools

Sources/​AnyLanguageModel/​Models/​MLXLanguageModel.swift:934

Pass the declared tool schemas to generate. In mlx-swift-lm 3.32.3, the default tools: nil accepts any parsed function name, so an undeclared tool is emitted as .toolCall rather than .rejectedToolCall; this path then returns a synthetic “Tool not found” result instead of throwing the promised RejectedToolCallError. Use an empty array when the session declares no tools so authorization still rejects tool-shaped output, and apply it to both the initial and retry calls.