bug(operator): kubectl rollout restart of ToolHive workloads is reverted; no supported bounce API

[RaviTharuma]

Status

Fix PR: #6378

• Maintainer direction (preserve kubectl.kubernetes.io/restartedAt): bug(operator): kubectl rollout restart of ToolHive workloads is reverted; no supported bounce API #6344 (comment)

• PR announcement: bug(operator): kubectl rollout restart of ToolHive workloads is reverted; no supported bounce API #6344 (comment)

• Reply to maintainer: bug(operator): kubectl rollout restart of ToolHive workloads is reverted; no supported bounce API #6344 (comment)

• PR cross-link comment: fix(operator): honor kubectl rollout restart on ToolHive workloads #6378 (comment)

---

Bug description

There is no supported way to bounce a ToolHive-managed MCP workload.

• kubectl rollout restart deploy/<name> / sts/<name> is reverted by the operator within about a second (it treats the restart annotation / template hash as drift).
• Patching StatefulSet replicas is also reverted ([BUG] Operator reverts manual StatefulSet replica scaling back to 1 #3329, closed as “operator is source of truth”).
• The documented-looking kubectl rollout restart path therefore does nothing useful.

The only thing that actually recycles a pod is kubectl delete pod -l toolhive-name=<name>. That is not documented, and for stdio servers you have to know which pod is the proxy vs the workload.

This matters for the known vMCP memory growth (#5860): the practical mitigation is a nightly pod delete, not a rollout.

Steps to reproduce

1. Deploy any MCPServer or VirtualMCPServer on the operator.
2. kubectl rollout restart deploy/<name> -n <ns>
3. Immediately kubectl get deploy <name> -n <ns> -o yaml | grep restartedAt (or watch ReplicaSets).

Expected behavior

Either:

• honor kubectl rollout restart (ignore kubectl.kubernetes.io/restartedAt in the drift check, or re-apply it), or
• document a first-class bounce: annotation on the CR (toolhive.stacklok.dev/restartedAt), kubectl annotate mcpserver …, or a condition the operator turns into a new ReplicaSet.

Actual behavior

Operator writes the Deployment/StatefulSet spec back to the desired template and the restart is gone. Users conclude “rollout restart is the runbook” and then wonder why memory/leak mitigations never fire.

Environment (if relevant)

• OS/version: Talos Linux, Kubernetes
• ToolHive version: operator v0.43.0 (behavior also seen on earlier 0.3x)

Additional context

Related: #3329 (replica scale-back). #5860 (vMCP leak — current workaround is kubectl delete pod -l toolhive-name=…, because rollout restart is a no-op).

Happy to take a docs-only fix if product intent is “never mutate child workloads”; then the CR annotation / thv command needs to exist so GitOps and CronJobs have something legal to do.

No credentials or private hostnames in this report.

[reyortiz3]

Thanks for the detailed report, @RaviTharuma, and for the concrete tie-in to #3329 and #5860 — the lack of a legal restart path is a real gap when the actual mitigation for a leak is a pod bounce.

We want to honor kubectl rollout restart rather than add a new bounce mechanism: preserve the kubectl.kubernetes.io/restartedAt annotation when the operator computes the desired pod template, instead of treating it as drift. That fixes the reported symptom directly and keeps the standard, already-scriptable kubectl/GitOps workflow working, without inventing new API surface.

This is a good candidate for a community contribution if you or anyone else would like to pick it up. Open a draft PR or comment here and we will help align on the exact reconciliation change.

Thanks again for taking the time to file it.

[RaviTharuma]

Opened #6378. The operator now preserves kubectl.kubernetes.io/restartedAt on desired pod templates (and on runner SSA) instead of treating it as drift, so kubectl rollout restart sticks. The existing CR annotation is unchanged.

[RaviTharuma]

@reyortiz3 following your direction: opened #6378. kubectl.kubernetes.io/restartedAt is preserved on desired templates and runner SSA so kubectl rollout restart sticks. No new bounce API.

Cross-links:

• Issue: bug(operator): kubectl rollout restart of ToolHive workloads is reverted; no supported bounce API #6344
• PR: fix(operator): honor kubectl rollout restart on ToolHive workloads #6378
• Your comment: bug(operator): kubectl rollout restart of ToolHive workloads is reverted; no supported bounce API #6344 (comment)
• First PR note: bug(operator): kubectl rollout restart of ToolHive workloads is reverted; no supported bounce API #6344 (comment)