Hot reconcile loop when podTemplateSpec is cleared on VirtualMCPServer

[jhrozek]

Summary

When a user sets spec.podTemplateSpec on a VirtualMCPServer and then clears it, the managed Deployment enters a hot reconcile loop: a no-op update fires every reconcile cycle, incrementing metadata.generation each time. On a kind cluster this saturates etcd write bandwidth and can starve leader-election lease renewal, causing the operator to lose leadership and restart.

Symptom

• kubectl get events -n <namespace> shows continuous DeploymentUpdated events at ~2/30s with no CR changes
• Deployment metadata.generation increments on every observation
• Operator logs show "Updating Deployment" in a tight loop
• On kind: etcd logs "server is overloaded", operator logs show lease renewal failures

Root cause

The interaction between the annotation-write gate in buildDeploymentMetadataForVmcp and the merge strategy in the update write path.

Write path (ensureDeployment, virtualmcpserver_controller.go):

deployment.Annotations = ctrlutil.MergeAnnotations(newDeployment.Annotations, deployment.Annotations)

MergeAnnotations starts from every key on the live object and overlays desired on top. Keys absent from desired pass through unchanged.

Write gate (buildDeploymentMetadataForVmcp, virtualmcpserver_deployment.go):

// only written when PodTemplateSpec is non-nil
if vmcp.Spec.PodTemplateSpec != nil && len(vmcp.Spec.PodTemplateSpec.Raw) > 0 {
    deploymentAnnotations[podTemplateSpecHashAnnotation] = hash
}

When the field is nil, the annotation is absent from newDeployment.Annotations.

Detection (podTemplateSpecNeedsUpdate, virtualmcpserver_controller.go):

if vmcp.Spec.PodTemplateSpec == nil || vmcp.Spec.PodTemplateSpec.Raw == nil {
    _, hadPrevious := deployment.Annotations[podTemplateSpecHashAnnotation]
    return hadPrevious  // always true — MergeAnnotations preserved it on the prior write
}

Trigger condition

1. Create a VirtualMCPServer with spec.podTemplateSpec set
2. Apply a change that clears the field to nil
3. The annotation strands on the Deployment; every subsequent reconcile sees drift and writes a no-op update

Also triggered on any upgrade that leaves the annotation on an existing Deployment while the current CR has the field unset.

Proposed fix

Explicitly delete the operator-owned annotation from the merged result when the desired state omits it:

merged := ctrlutil.MergeAnnotations(newDeployment.Annotations, deployment.Annotations)
if _, want := newDeployment.Annotations[podTemplateSpecHashAnnotation]; !want {
    delete(merged, podTemplateSpecHashAnnotation)
}
deployment.Annotations = merged

Alternatively, make podTemplateSpecNeedsUpdate symmetric (compare stored hash vs expected hash rather than checking presence) so a missing annotation with empty expected hash is treated as steady state, triggering exactly one cleanup write.

Related

Same MergeAnnotations-preserves-stale-key class of bug fixed for Services in #5731. That PR did not extend the fix to the Deployment write path.

See also: companion bugs for imagePullRefsHashAnnotation (#5817) and podTemplateMetadataNeedsUpdate (filed separately).