Fix Ollama multi-turn, tool calling, and cross-backend tool argument encoding - #3
Merged
Merged
Conversation
`/api/chat` reads image data from each message's `images` field. Top-level `images` is an `/api/generate` field, so the server silently ignored it and every multimodal request was answered without the image ever being seen. Confirmed against a live Ollama: the old shape returns a normal completion, the new shape is actually parsed.
`extractPromptSegments` walked the transcript backwards and returned the first `.prompt` it found, which was then sent as a lone user message. Everything else was dropped: prior user turns, every assistant response, tool calls and their outputs, and the `.instructions` entry — so `LanguageModelSession(instructions:)` was silently a no-op for this backend and multi-turn conversations had no history. Add `Transcript.toOllamaMessages()`, mirroring the existing `toOpenAIMessages()` and `toAnthropicMessages()` converters, and use it in both `respond` and `streamResponse`.
`respond` executed tool calls and recorded them in the transcript, but only ever issued a single request. When the model answered with a tool call its `content` is empty, so the caller received `""` — the tool ran and its result was never used. Loop until the model answers without requesting tools, appending the assistant tool-call turn and each result as a `role: "tool"` message, matching the loop `OpenAILanguageModel` and `AnthropicLanguageModel` already use.
`streamResponse` sent the `tools` array but the chunk loop only ever read `chunk.message.content`, so `tool_calls` were silently discarded: no execution, no transcript entries, and an empty stream whenever the model chose a tool. Accumulate tool calls across chunks and run them through the same resolution path `respond` uses, looping until the model answers with text. Tool calls are appended to the transcript before their outputs so a Transcript-driven UI renders them in order, and `growStreamingTranscript` now runs per chunk to match `AnthropicLanguageModel`.
`toXMessages()` encoded `Transcript.ToolCall.arguments` with `GeneratedContent`'s `Codable` conformance, which writes its internal representation rather than the value it wraps. A call to `getWeather(city: "San Francisco")` was replayed to the provider as:
{"kind":{"type":"structure","properties":{"city":{"kind":{"type":"string","value":"San Francisco"}}}},"orderedKeys":["city"]}
Round-trip through `jsonString` instead, which yields `{"city":"San Francisco"}`.
Nothing rejects the malformed value — `arguments` is an opaque string to OpenAI and `input` a free-form object to Anthropic — so it degraded the model's view of its own history rather than erroring. OpenAI and Anthropic only hit this on a second `respond` after a tool turn, because their in-loop path reuses the raw provider message. Gemini rebuilds `toGeminiContent()` on every iteration of its tool loop, so it was affected from the first tool call onward.
The comments added in 0145ec6 describe `GeneratedContent`'s `Codable` conformance as encoding "its internal representation rather than the value it wraps", which reads as though the conformance is defective. It is not, and I asserted that before reading it. `GeneratedContent.encode(to:)` writes `{id?, kind}` and `Kind.encode(to:)` writes `{type, value}` or `{type, properties, orderedKeys}`, with a symmetric `init(from:)` that reads the same shape back. `Transcript`, `Transcript.ToolCall`, and `GeneratedContent` are all `Codable` precisely so a transcript can be persisted and restored losslessly. The format is deliberate and correct for that purpose. The bug in 0145ec6 was ours: we used that persistence format as the provider wire format for tool call arguments. Reword the comments to say so, since the original phrasing invites someone to "fix" `GeneratedContent.encode(to:)` and silently break transcript round-tripping. No behavior change — comments only.
ActuallyTaylor
commented
Aug 3, 2026
ActuallyTaylor
left a comment
Collaborator
Author
There was a problem hiding this comment.
Looks good enough. There are some problems I have with the general library right now but those don't relate to this PR. It matches the Anthropic Implementation quite well.
`llama_model_params.use_mmap` and `.use_mlock` were replaced upstream by a single `load_mode` enum, so `createModelParams` no longer compiled and the Llama trait could not be built at all. This is a pre-existing break rather than a regression: `Package.resolved` had no entry for llama.swift, so the trait had never been resolved or compiled in this repo, and `.upToNextMajor(from: "2.7484.0")` now resolves to 2.10241.0 (llama.cpp b10241). `use_mmap = true` / `use_mlock = false` maps to `LLAMA_LOAD_MODE_MMAP`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OllamaLanguageModelsent only the most recent.promptentry as a lone user message, so instructions, prior turns, and tool results never reached the server — multi-turn conversations had no history andLanguageModelSession(instructions:)was a silent no-op. Tool calls were executed but their results were never fed back (callers got""), streaming discardedtool_callsentirely, and images were sent at the top level where/api/chatignores them. Each of those is fixed in its own commit, with a newTranscript.toOllamaMessages()mirroring the existing OpenAI and Anthropic converters.The final commit fixes a separate bug in all five HTTP backends:
toXMessages()serializedTranscript.ToolCall.argumentswithGeneratedContent'sCodableconformance. That conformance is a lossless persistence format — it writeskind/orderedKeysso aTranscriptcan be saved and restored exactly — and using it as a provider wire format replayedgetWeather(city: "San Francisco")as{"kind":{"type":"structure",...}}. The conformance is correct; the misuse was ours, so the fix goes throughjsonStringat the wire boundary rather than changingGeneratedContent. Nothing rejects the malformed value, so it silently degraded the model's view of its own history — OpenAI and Anthropic only on a secondrespondafter a tool turn, Gemini from the first tool call onward.Verified against a live Ollama (
lfm2.5) and against fake Ollama/OpenAI/Anthropic servers that log request bodies, confirming each wire shape before and after;CI=1 swift testpasses 319 tests in 37 suites.🤖 Generated with Claude Code