Route subagents to a local model on a tailnet Strix Halo (DeepSeek V4.1 Flash via ds4-server)

[ParallelEntrepreneur]

Goal

Route Colonizer subagents (and optionally background work) to DeepSeek V4.1 Flash served on a local AMD Strix Halo box, reached over the operator's tailnet, for example subagent_model = strix/deepseek-v4-flash.

Model routing landed in #1, but it was verified only against a stub on the host. This server behaves very differently from that stub: it is on another machine, it is slow, it serves one session at a time and its context is capped. This issue is about making that setup work reliably, not just connect.

The endpoint

Server	antirez's DwarfStar ds4-server, ROCm build from antirez/ds4#1036, in Docker
Model	DeepSeek V4.1 Flash Q2 GGUF (763B total, 16B active per decoded token), experts streamed from SSD
Hardware	Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151), 128 GB unified memory
URL	http://<tailnet-ip>:8000 (tailnet only, plain HTTP, no auth)
APIs	Anthropic POST /v1/messages (tools, SSE, thinking), OpenAI chat/responses/completions, GET /v1/models
Model id	deepseek-v4-flash (a compatibility alias; the loaded GGUF decides the model)
Concurrency	one session (--batched-session 1); other requests queue
Context	131072 tokens allocated on the server
Speed	reported on the same chip in the PR: ~160-250 tok/s prefill, ~5-9 tok/s decode (ours not measured yet)
Thinking	on by default; disabled with a thinking: {"type": "disabled"} object

Provider entry this should work with:

{"id": "strix", "name": "Strix Halo (DeepSeek V4.1 Flash)", "preset": "local",
 "base_url": "http://<tailnet-ip>:8000", "auth": "none", "models": ["deepseek-v4-flash"]}

Gaps found in the code (at 59e4506)

1. A colony may not reach a tailnet or LAN provider. colony_routes only special-cases loopback hosts, rewriting them to host.microsandbox.internal and adding the host profile (providers.rs#L127-L135). Any other host goes out on the public profile (sessions.rs#L497-L522). It is unverified whether that profile allows 100.64.0.0/10 or RFC 1918 addresses. The bundled Headscale mesh also allocates from 100.64.0.0/10 (mesh.rs#L209), so a tailnet provider address can collide with the colony mesh. One option: treat private and CGNAT provider hosts like loopback and forward them from the mothership (which is on the tailnet) through host.microsandbox.internal. Another: add an explicit net rule for exactly the provider's host and port.
2. Long silent prefill vs. timeouts. The router uses the global fetch with no dispatcher (router.mjs#L137-L150). That means undici's default 300 s headers and body timeouts. A 50K-token subagent prompt at ~160 tok/s is about 5 minutes before the first token. Claude Code's own stream idle timeout needs raising too (ds4's client guide uses CLAUDE_STREAM_IDLE_TIMEOUT_MS=600000). Proposal: an optional per-provider timeout_secs, applied in the router and mapped to the Claude Code idle timeout when a slow provider is routed.
3. No per-provider concurrency. Parallelism is capped globally and per org (sessions.rs#L313-L328), not per provider. Several colonies sending subagents to a one-session server serialize behind each other and then hit the timeouts above. Proposal: an optional per-provider max_concurrent, coordinated by the mothership since each colony runs its own router, with waiting or fallback when it is full.
4. No per-model context window. The server allocates 131072 tokens, but Claude Code sizes compaction for Claude models, so a long subagent can exceed the allocation and fail. Proposal: an optional context_tokens per provider model, passed to the runner so the agent compacts early, or so the orchestrator refuses to route to a model that is too small.
5. The watchdog can read a slow model as a stall. Stall detection is wall-clock time without progress (watchdog.rs#L72-L78). A subagent at 5-9 tok/s, or one still in prefill, may be nudged or flagged. An in-flight provider request should count as activity.
6. No health check or fallback. When the box is rebooting or the model is still loading, the router answers 502 provider "…" is unreachable and the subagent fails. Proposal: probe GET /v1/models in Settings (reachable, model loaded) and at colony start, and optionally fall back to a configured Claude model.
7. Not yet exercised against a real non-Claude Anthropic-compatible server. The README says so. Claude Code sends tools, thinking, anthropic-beta headers and count_tokens (the router already estimates when the upstream returns 404, 405 or 501). All of these need an end-to-end check against ds4-server.

Acceptance

• The provider above can be added in Settings, and Settings shows whether it is reachable and which model is loaded.
• A colony with subagent_model = strix/deepseek-v4-flash completes a subagent task end to end, with the mesh module both on and off. The request arrives as model: deepseek-v4-flash and no credential headers are sent.
• A subagent with a ~60K-token prompt finishes without a router or Claude Code timeout.
• Two colonies routing to the provider at once: the second one waits or falls back instead of failing.
• The watchdog does not nudge a colony whose only activity is a live request to this provider.
• With the provider down, the session log says so clearly, and a configured fallback model takes over.
• README and docs/protocol.md §6.1 document the private-network provider path and the new provider fields.

Notes

• Given ~5-9 tok/s decode, this fits subagents and background work, not the orchestrator.
• Only one model fits in memory on a 128 GB Strix Halo at a time. The server stays single-purpose, so routing is the only place to manage contention.

[ParallelEntrepreneur]

Findings, verified on the mothership

Gap	Result
1. Reaching a tailnet/LAN provider	A colony on --net public or public,host cannot reach a tailnet (100.x) or LAN (192.168.x) listener. An explicit --net-rule allow@<ip>:tcp:<port> works, but not once the colony has joined the mesh: its kernel-mode tailscaled captures 100.x traffic. With tailscaled --tun=userspace-networking in the colony, the rule works again and the mothership still reaches the colony over the mesh.
2. Timeouts	The router's fetch has no dispatcher (undici 300 s). Claude Code additionally runs a 5-minute body idle timeout that is on for non-Anthropic providers (API_FORCE_IDLE_TIMEOUT), 300 s event and byte stream watchdogs (CLAUDE_STREAM_IDLE_TIMEOUT_MS, capped at 30 min for the byte watchdog), a first-byte deadline (CLAUDE_STREAM_FIRST_BYTE_TIMEOUT_MS), API_TIMEOUT_MS (10 min) and a 10-minute subagent stall timeout (CLAUDE_ASYNC_AGENT_STALL_TIMEOUT_MS).
3. Concurrency	Confirmed: only global and per-org limits; each colony has its own router, so coordination has to live on the mothership.
4. Context	Claude Code's window settings are per session, but CLAUDE_CODE_MAX_CONTEXT_TOKENS only applies to model IDs it does not recognise (strix/deepseek-v4-flash), so it can cap routed subagents without shrinking a Claude orchestrator.
5. Watchdog	Confirmed: wall-clock only.
6. Health/fallback	Confirmed: none.
7. Real server	Not yet: chi-NucBox-EVO-X2 (100.80.225.14) is offline.

Design: a provider gateway on the mothership

• Path. The colony router sends every routed (non-Anthropic) request to http://host.microsandbox.internal:<gateway_port>/providers/<id>/v1/... with a per-colony token. The gateway listens on mothership loopback, so it works with the mesh on or off, and the mothership (already on the tailnet) reaches tailnet and LAN providers itself. No per-provider net rules, no mesh collision.
• Credentials. The gateway adds the provider credential and strips client credentials and OAuth betas. Provider keys no longer enter colonies at all, not even as placeholders.
• New provider fields. timeout_secs (default 600), max_concurrent (default unlimited), queue_timeout_secs, context_tokens, fallback_model (e.g. sonnet).
• Concurrency. A per-provider semaphore across all colonies; requests wait up to queue_timeout_secs, then get 503 with x-colonizer-fallback: queue_timeout.
• Timeouts. The gateway applies timeout_secs to connect, headers and idle body. When a colony routes to a provider with timeout_secs above 300, the runner sets CLAUDE_STREAM_IDLE_TIMEOUT_MS, API_TIMEOUT_MS, CLAUDE_ASYNC_AGENT_STALL_TIMEOUT_MS to match (clamped to Claude Code's caps) and API_FORCE_IDLE_TIMEOUT=0.
• Context. CLAUDE_CODE_MAX_CONTEXT_TOKENS = the smallest context_tokens among providers the colony's model settings route to.
• Watchdog. The gateway tracks in-flight requests per colony; a colony with a live provider request counts as making progress.
• Health. GET /api/providers/{id}/health probes GET /v1/models (reachable, loaded models, latency); Settings shows it and colony start logs a warning when a routed provider is down.
• Fallback. On an unreachable provider or queue timeout, the router retries the request on fallback_model through the Anthropic path and logs why.

Follow-ups kept out of this change: SSE keep-alive pings during prefill, and switching colony tailscaled to userspace mode.

[ParallelEntrepreneur]

The gateway is in 52d7b46.

What shipped

• Gateway: it runs on the mothership, and colonies reach it at host.microsandbox.internal:41750 with a per-colony token. It reaches loopback, LAN and tailnet providers, and adds the provider key. Keys no longer enter colonies.
• Provider settings:
  • timeout_secs (default 600)
  • max_concurrent, with a queue on the mothership
  • queue_timeout_secs
  • context_tokens
  • fallback_model
• Health: GET /api/providers/{id}/health probes /v1/models. Settings shows the result next to live running/queued counts, and colony start logs a warning for an unreachable provider.
• Runner: for the routes a colony uses, it sets CLAUDE_STREAM_IDLE_TIMEOUT_MS, API_TIMEOUT_MS, API_FORCE_IDLE_TIMEOUT=0, CLAUDE_ASYNC_AGENT_STALL_TIMEOUT_MS and CLAUDE_CODE_MAX_CONTEXT_TOKENS. It retries on the Claude fallback when the gateway answers x-colonizer-fallback (unreachable, timeout, queue_timeout) or can't be reached.
• Watchdog: a colony with a gateway request in flight counts as busy, not stalled.
• Docs: protocol §6.5.

Verified with a real colony (haiku orchestrator, subagents on a stub provider bound to the LAN address only, max_concurrent 1, 6 s prefill):

• Health: reachable in 3 ms. Requests without a token, or with a forged one, get 401.
• Three parallel subagents: the provider never saw more than 1 concurrent request (load reached in_flight 1, queued 2). The request that waited past the 8 s queue timeout fell back with provider lan unavailable (queue_timeout); used claude-haiku-4-5.
• A 150 s silent request completed. With the watchdog's stall threshold set to 1 minute, the colony was never flagged.
• Inside the colony, Claude Code's env had the expected timeouts and CLAUDE_CODE_MAX_CONTEXT_TOKENS=131072. The provider key appeared nowhere in the colony env or session files.
• Tests: 24 Rust and 23 runner tests pass.

Still open

• A run against the real ds4-server. chi-NucBox-EVO-X2 has been offline for 5 days.
• SSE keep-alive pings during prefill.