Skip to content

fix(operations): replace the component removal recipe that undoes itself #684

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Aleksei Sviridkin (lexfrei) merged 4 commits into from
Oct 6, 2026
Merged

fix(operations): replace the component removal recipe that undoes itself #684

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7d56bd4
fix(operations): replace the component removal recipe that undoes itself
lexfrei Sep 22, 2026
5111fe6
docs(operations): widen the uninstall warning and give it a done signal
lexfrei Sep 22, 2026
1d32ec8
docs(operations): correct the component removal path on v1.0
lexfrei Sep 22, 2026
4327720
docs(operations): warn what removing a v0 component destroys
lexfrei Sep 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 37 additions & 7 deletions content/en/docs/next/operations/configuration/components.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,7 @@ Bundles have optional components that need to be explicitly enabled (included) i
Regular bundle components can, on the other hand, be disabled (excluded) from the installation, when you don't need them.

Use `bundles.enabledPackages` and `bundles.disabledPackages` in the Platform Package values.
Every entry in those lists is a fully-qualified Package name — the same name you see with
`kubectl get package`. All platform packages live under the `cozystack.` prefix (for example,
`cozystack.metallb`, `cozystack.hetzner-robotlb`, `cozystack.nfs-driver`). Run
`kubectl get package` to see the exact names available on your cluster before editing
the Platform Package.
Every entry in those lists is a fully-qualified name under the `cozystack.` prefix (for example, `cozystack.metallb`, `cozystack.hetzner-robotlb`, `cozystack.nfs-driver`). Run `kubectl get packagesource` to see the exact names available on your cluster before editing the Platform Package. `kubectl get package` answers only for `disabledPackages`, because an optional component has no Package object until its name is already in `enabledPackages`.

For example, [installing Cozystack in Hetzner]({{% ref "/docs/next/install/providers/hetzner" %}})
requires swapping the default load balancer, MetalLB, with one made specifically for Hetzner, called RobotLB:
Expand All @@ -70,8 +66,42 @@ spec:

Disabling components must be done before installing Cozystack.
Applying updated configuration with `disabledPackages` will not remove components that are already installed.
To remove already installed components, delete the Helm release manually using this command:
Removing one that is already installed takes two steps. Add its name to `disabledPackages` in the Platform Package above, then wait for the operator to carry that edit across. The name appears in this output once it has:

@IvanHunters IvanHunters Oct 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[NIT] the readback confirms the edit propagated, not that the platform re-reconciled

jsonpath='{.spec.values.bundles.disabledPackages}' reads desired values on the cozystack-platform HelmRelease, which flip as soon as the operator carries the edit across, before helm-controller has re-rendered the release. In steady state this is harmless (the new values already exclude the name, so nothing re-creates the Package). The one window it misses is a concurrent platform upgrade still applying an older manifest, which can re-create a just-deleted Package; the consequence there is a silent re-install, not data loss. If you want the check to mean "the platform has actually stopped rendering it", gate on the release being reconciled (observedGeneration == generation, Ready=True), not on .spec.


```bash
kubectl delete hr -n <namespace> <component>
kubectl get helmrelease cozystack-platform --namespace cozy-system \
--output jsonpath='{.spec.values.bundles.disabledPackages}'
```

Then delete the Package object.

{{% alert title="Warning" color="warning" %}}
Deleting the Package uninstalls the component's Helm release, and that destroys more than the workloads. Anything the chart rendered as an ordinary template without `helm.sh/resource-policy: keep` goes with the release, CRDs and namespaces included, and Kubernetes deletes every custom resource of those CRD kinds along with them. The annotation is not the only thing that keeps an object alive. From v1.5.0 the platform installs a ValidatingAdmissionPolicy that denies DELETE on anything labelled `platform.cozystack.io/no-delete: "true"`, and several component charts render objects that carry it; `kubectl get <kind> --all-namespaces --selector platform.cozystack.io/no-delete=true` lists them for a given kind. One denial fails the whole uninstall: Helm deletes what it can and then errors out, and the controller keeps its finalizer and retries, so the HelmRelease sits in deletion and the wait below runs to its timeout. Taking the label off hands the object to the uninstall, which is the whole point of the guard, and it has to come off every labelled object in the release: the `cert-manager-issuers` release of `cozystack.cert-manager` labels three ClusterIssuers, and unlabelling one of them still leaves the other two to fail the uninstall. The command is `kubectl label <kind> <name> --namespace <ns> platform.cozystack.io/no-delete-`, without `--namespace` for cluster-scoped kinds. Weigh what that costs before doing it: `cozystack.cozystack-basics` labels two objects, the `tenant-root` Namespace and the `tenant-root` HelmRelease, and only the Namespace is one the uninstall would delete, since the HelmRelease also carries the keep annotation. Unlabelling that Namespace means the uninstall takes the root tenant and every application in it. Removing `cozystack.metallb` takes every CRD the MetalLB chart bundles, subcharts included, and with them every custom resource of those kinds cluster-wide; removing `cozystack.cozystack-basics` takes the `cozy-public` namespace and everything stored in it. Back up anything you still need first.
{{% /alert %}}

The namespace a component installs into is the exception: the operator applies that one itself, outside the component's release and with no ownerReference, so the uninstall never had it to remove.

List the releases the Package owns before deleting it. The operator labels every HelmRelease it renders with the name of the Package that produced it, and one Package can own several:

```bash
kubectl get helmrelease --all-namespaces --selector cozystack.io/package=<package-name> \
--output custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,SUSPENDED:.spec.suspend'
```

Clear `spec.suspend` on any release that shows `true` before going on. Flux skips the uninstall for a suspended HelmRelease and only drops its own finalizer, so that release disappears with everything it installed left behind and nothing left managing it.

@IvanHunters IvanHunters Oct 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] "Clear spec.suspend" has no command, and the obvious way to do it is a no-op

The step tells the reader to clear spec.suspend on any release the listing shows as true, and the warning beside it is correct that a suspended HelmRelease is dropped without an uninstall, orphaning everything it installed. But there is no command, and the path a reader is likeliest to try does not work: the operator copies the existing release's suspend state back on every reconcile (internal/operator/package_reconciler.go:455, hr.Spec.Suspend = existing.Spec.Suspend), so re-enabling the component or re-rendering the Package will not unsuspend it. Give the direct form, e.g. kubectl patch helmrelease <name> --namespace <ns> --type merge --patch '{"spec":{"suspend":false}}' (or flux resume hr), and note that unsuspending first triggers a reconcile of the release someone suspended on purpose. Same line exists in the v1.0-v1.6 pages.


```bash
kubectl delete package.cozystack.io <package-name>

@IvanHunters IvanHunters Sep 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] No signal for when the destructive uninstall has actually finished

internal/operator/package_reconciler.go registers no finalizer on Package (verified: the only finalizer usage in the tree belongs to an unrelated tapmaterializer reconciler), so kubectl delete package.cozystack.io returns as soon as the Package object itself is gone. The owned HelmRelease's deletion, and the helm uninstall inside it that destroys the CRDs/namespaces the Warning describes, proceeds asynchronously afterward via the ownerReference cascade. The recipe has no follow-up check for confirming the destructive step is actually done. Same text is duplicated identically at v1.1:84, v1.2:89, v1.3:89, v1.4:89, v1.5:89, v1.6:89.

@lexfrei Aleksei Sviridkin (lexfrei) Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a wait after the delete:

kubectl wait --for=delete helmrelease/<component> --namespace <namespace> --timeout=10m

with a line saying why the delete returns early.

```

Nothing holds a finalizer on the Package, so this command returns as soon as the object is gone and the uninstall it triggers runs afterwards. Wait on each release from the listing to know the destructive part has finished:

```bash
kubectl wait --for=delete helmrelease/<name> --namespace <namespace> --timeout=10m
```

`kubectl wait --for=delete` exits 0 for a name that was never there, silently and with nothing to tell it apart from a deletion it watched, so take both values from the listing rather than guessing them. A returned wait says the HelmRelease is gone, not that the uninstall ran.

Deleting the Package while the platform values still render it means the next platform upgrade brings it back, undoing the removal one level up. Nothing reports this at the time: the delete succeeds either way and the Package reappears whenever that upgrade happens to run.

`kubectl delete hr` is not a lighter-weight version of this. Flux uninstalls the release when an unsuspended HelmRelease goes away, so it destroys the same CRDs and custom resources, and then the Package recreates the HelmRelease and the chart reinstalls. The workloads come back, the custom resources do not. If you have run it before, those custom resources are already gone and have to be recreated from your own manifests or a backup.
6 changes: 5 additions & 1 deletion content/en/docs/v0/operations/configuration/components.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,11 @@ data:

Disabling components must be done before installing Cozystack.
Applying updated configuration with `bundle-disable` will not remove components that are already installed.
To remove already installed components, delete the Helm release manually using this command:
To remove already installed components, delete the Helm release manually.

{{% alert title="Warning" color="warning" %}}
Uninstalling the component's Helm release destroys more than the workloads. Anything the chart rendered as an ordinary template without `helm.sh/resource-policy: keep` goes with the release, CRDs and namespaces included, and Kubernetes deletes every custom resource of those CRD kinds along with them. Removing `metallb` takes every CRD the MetalLB chart bundles, subcharts included, and with them every custom resource of those kinds cluster-wide. Back up anything you still need first.
{{% /alert %}}

```bash
kubectl delete hr -n <namespace> <component>
Expand Down
4 changes: 2 additions & 2 deletions content/en/docs/v1.0/install/providers/hetzner.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -344,9 +344,9 @@ The final stage of deploying a Cozystack cluster on Hetzner is to install Cozyst
values:
bundles:
disabledPackages:
- metallb
- cozystack.metallb
enabledPackages:
- hetzner-robotlb
- cozystack.hetzner-robotlb
publishing:
host: "example.org"
apiServerEndpoint: "https://api.example.org:443"
Expand Down
34 changes: 28 additions & 6 deletions content/en/docs/v1.0/operations/configuration/components.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ Bundles have optional components that need to be explicitly enabled (included) i
Regular bundle components can, on the other hand, be disabled (excluded) from the installation, when you don't need them.

Use `bundles.enabledPackages` and `bundles.disabledPackages` in the Platform Package values.
Every entry in those lists is a fully-qualified name under the `cozystack.` prefix — run `kubectl get packagesource` to see the exact names on your cluster. `kubectl get package` answers only for `disabledPackages`, because an optional component has no Package object until its name is already in `enabledPackages`.
For example, [installing Cozystack in Hetzner]({{% ref "/docs/v1.0/install/providers/hetzner" %}})
requires swapping default load balancer, MetalLB, with one made specifically for Hetzner, called RobotLB:

Expand All @@ -56,16 +57,37 @@ spec:
values:
bundles:
disabledPackages:
- metallb
- cozystack.metallb
enabledPackages:
- hetzner-robotlb
- cozystack.hetzner-robotlb
# rest of the config
```

Disabling components must be done before installing Cozystack.
Applying updated configuration with `disabledPackages` will not remove components that are already installed.
To remove already installed components, delete the Helm release manually using this command:
Disabling a component before installing Cozystack keeps it out of the installation entirely.

On v1.0 the platform does not annotate the Packages it renders with `helm.sh/resource-policy: keep`, so adding a name to `disabledPackages` also removes the component when it is already installed. The next platform reconcile drops the Package, the operator's ownerReference takes the component's HelmRelease with it, and Flux uninstalls the release. There is no second command and no confirmation step, so back up anything you still need before making that edit.

{{% alert title="Warning" color="warning" %}}
Uninstalling the component's Helm release destroys more than the workloads. Anything the chart rendered as an ordinary template without `helm.sh/resource-policy: keep` goes with the release, CRDs and namespaces included, and Kubernetes deletes every custom resource of those CRD kinds along with them. Removing `cozystack.metallb` takes every CRD the MetalLB chart bundles, subcharts included, and with them every custom resource of those kinds cluster-wide; removing `cozystack.cozystack-basics` takes the `cozy-public` and `tenant-root` namespaces and everything stored in them, which is every application in the root tenant. Back up anything you still need first.

@IvanHunters IvanHunters Sep 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MAJOR] the v0 tree keeps the bare command and gets none of the warning

Out of this diff, and pre-existing, so a follow-up PR is a fine resolution. Raising it here because it is the same gap you folded v1.0 in to close last round, and v0 is the last tree still carrying the original line:

$ grep -n 'kubectl delete hr' content/en/docs/v0/operations/configuration/components.md
66:kubectl delete hr -n <namespace> <component>
$ grep -c '^content/en/docs/v0/' /tmp/...-files.txt
0

The page is published and carries no deprecation banner:

$ sed -n '236,239p' hugo.yaml
    - version: "v0"
      url: "/docs/v0/"
      id: "v0"
      order: 1

The destruction the Warning describes belongs to helm uninstall, not to Package objects, so it applies to v0 in full: at v0.41.11 the MetalLB chart renders its CRDs as ordinary templates with no resource-policy, and a v0 reader following line 66 loses every IPAddressPool, L2Advertisement and BGPPeer cluster-wide. "v0 predates Package objects" excuses the recipe, not the missing warning.

@lexfrei Aleksei Sviridkin (lexfrei) Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Folded in here rather than deferred, at f621e84. v0 keeps its recipe and gets the warning ahead of the command. I checked charts/metallb/charts/crds/templates/crds.yaml at v0.1.0, v0.10.0, v0.20.0, v0.30.0 and v0.41.11: no resource-policy at any of them, so the destruction is the same on every 0.x line. The v0 copy drops the cozystack.cozystack-basics half of the sentence, that package does not exist before v1.

{{% /alert %}}

The namespace a component installs into is the exception: the operator applies that one itself, outside the component's release and with no ownerReference, so the uninstall never had it to remove.

The removal starts when the operator picks the edit up, so take the component's releases first — the Package the selector needs goes away with the component. The operator labels every HelmRelease it renders with the name of the Package that produced it, and one Package can own several:

@IvanHunters IvanHunters Oct 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[NIT] "the Package the selector needs goes away" misstates why you list first

The selector cozystack.io/package=<package-name> matches HelmReleases by a label value you already know; it never reads the Package object. What disappears when the component is removed is the HelmReleases themselves, which is exactly why you must list them before making the edit. "the Package the selector needs goes away" points at the wrong object. The instruction to list first is right; the reason given is not.


```bash
kubectl delete hr -n <namespace> <component>
kubectl get helmrelease --all-namespaces --selector cozystack.io/package=<package-name> \
--output custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,SUSPENDED:.spec.suspend'
```

Clear `spec.suspend` on any release that shows `true` before making the edit. Flux skips the uninstall for a suspended HelmRelease and only drops its own finalizer, so that release disappears with everything it installed left behind and nothing left managing it.

Then wait on each release from the listing to know the destructive part has finished:

```bash
kubectl wait --for=delete helmrelease/<name> --namespace <namespace> --timeout=10m
```

`kubectl wait --for=delete` exits 0 for a name that was never there, silently and with nothing to tell it apart from a deletion it watched, so take both values from the listing rather than guessing them. A returned wait says the HelmRelease is gone, not that the uninstall ran.

`kubectl delete hr` is not a lighter-weight way to do the same thing. Flux uninstalls the release when an unsuspended HelmRelease goes away, so it destroys the same CRDs and custom resources, and then the Package recreates the HelmRelease and the chart reinstalls. The workloads come back, the custom resources do not. If you have run it before, those custom resources are already gone and have to be recreated from your own manifests or a backup.
4 changes: 2 additions & 2 deletions content/en/docs/v1.1/install/providers/hetzner.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -344,9 +344,9 @@ The final stage of deploying a Cozystack cluster on Hetzner is to install Cozyst
values:
bundles:
disabledPackages:
- metallb
- cozystack.metallb
enabledPackages:
- hetzner-robotlb
- cozystack.hetzner-robotlb
publishing:
host: "example.org"
apiServerEndpoint: "https://api.example.org:443"
Expand Down
47 changes: 42 additions & 5 deletions content/en/docs/v1.1/operations/configuration/components.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ Bundles have optional components that need to be explicitly enabled (included) i
Regular bundle components can, on the other hand, be disabled (excluded) from the installation, when you don't need them.

Use `bundles.enabledPackages` and `bundles.disabledPackages` in the Platform Package values.
Every entry in those lists is a fully-qualified name under the `cozystack.` prefix — run `kubectl get packagesource` to see the exact names on your cluster. `kubectl get package` answers only for `disabledPackages`, because an optional component has no Package object until its name is already in `enabledPackages`.
For example, [installing Cozystack in Hetzner]({{% ref "/docs/v1.1/install/providers/hetzner" %}})
requires swapping default load balancer, MetalLB, with one made specifically for Hetzner, called RobotLB:

Expand All @@ -56,16 +57,52 @@ spec:
values:
bundles:
disabledPackages:
- metallb
- cozystack.metallb
enabledPackages:
- hetzner-robotlb
- cozystack.hetzner-robotlb
# rest of the config
```

Disabling components must be done before installing Cozystack.
Applying updated configuration with `disabledPackages` will not remove components that are already installed.
To remove already installed components, delete the Helm release manually using this command:
From v1.1.5 onward, applying updated configuration with `disabledPackages` will not remove components that are already installed.
On v1.1.0 through v1.1.4 the platform does not annotate the Package with `helm.sh/resource-policy: keep`, so adding the name to `disabledPackages` does remove an installed component: the destruction described below happens at that point, and there is no second command to run. Back up anything you still need before making that edit. Take the listing below before making the edit and wait on the same releases afterwards: the Package its selector needs goes away with the component.

From v1.1.5 onward, removing an installed component takes two steps. Add its name to `disabledPackages` in the Platform Package above, then wait for the operator to carry that edit across. The name appears in this output once it has:

```bash
kubectl get helmrelease cozystack-platform --namespace cozy-system \
--output jsonpath='{.spec.values.bundles.disabledPackages}'
```

Then delete the Package object.

{{% alert title="Warning" color="warning" %}}
Deleting the Package uninstalls the component's Helm release, and that destroys more than the workloads. Anything the chart rendered as an ordinary template without `helm.sh/resource-policy: keep` goes with the release, CRDs and namespaces included, and Kubernetes deletes every custom resource of those CRD kinds along with them. Removing `cozystack.metallb` takes every CRD the MetalLB chart bundles, subcharts included, and with them every custom resource of those kinds cluster-wide; removing `cozystack.cozystack-basics` takes the `cozy-public` and `tenant-root` namespaces and everything stored in them, which is every application in the root tenant. Back up anything you still need first.
{{% /alert %}}

The namespace a component installs into is the exception: the operator applies that one itself, outside the component's release and with no ownerReference, so the uninstall never had it to remove.

List the releases the Package owns before deleting it. The operator labels every HelmRelease it renders with the name of the Package that produced it, and one Package can own several:

```bash
kubectl delete hr -n <namespace> <component>
kubectl get helmrelease --all-namespaces --selector cozystack.io/package=<package-name> \
--output custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,SUSPENDED:.spec.suspend'
```

Clear `spec.suspend` on any release that shows `true` before going on. Flux skips the uninstall for a suspended HelmRelease and only drops its own finalizer, so that release disappears with everything it installed left behind and nothing left managing it.

```bash
kubectl delete package.cozystack.io <package-name>
```

Nothing holds a finalizer on the Package, so this command returns as soon as the object is gone and the uninstall it triggers runs afterwards. Wait on each release from the listing to know the destructive part has finished:

```bash
kubectl wait --for=delete helmrelease/<name> --namespace <namespace> --timeout=10m
```

`kubectl wait --for=delete` exits 0 for a name that was never there, silently and with nothing to tell it apart from a deletion it watched, so take both values from the listing rather than guessing them. A returned wait says the HelmRelease is gone, not that the uninstall ran.

Deleting the Package while the platform values still render it means the next platform upgrade brings it back, undoing the removal one level up. Nothing reports this at the time: the delete succeeds either way and the Package reappears whenever that upgrade happens to run.

`kubectl delete hr` is not a lighter-weight version of this. Flux uninstalls the release when an unsuspended HelmRelease goes away, so it destroys the same CRDs and custom resources, and then the Package recreates the HelmRelease and the chart reinstalls. The workloads come back, the custom resources do not. If you have run it before, those custom resources are already gone and have to be recreated from your own manifests or a backup.
Loading
Loading