Skip to content

fix(cozystack): load drbd_transport_tcp explicitly - #252

Open
Aleksei Sviridkin (lexfrei) wants to merge 1 commit into
mainfrom
fix/drbd-transport-tcp
Open

Aleksei Sviridkin (lexfrei) wants to merge 1 commit into
mainfrom
fix/drbd-transport-tcp

Conversation

@lexfrei

@lexfrei Aleksei Sviridkin (lexfrei) commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

The cozystack preset now loads drbd_transport_tcp next to drbd. Without it DRBD can't connect to any peer on Talos v1.14.

Since v1.14.0 Talos builds the kernel with an empty modprobe path (siderolabs/pkgs#1565), so the kernel no longer loads modules on request_module(). DRBD loads its transport that way when a resource first connects to a peer. After upgrading a node from v1.13.6 to v1.14.2, drbdsetup new-peer failed with "Failed to create transport (drbd_transport_xxx module missing?)" and every resource on the node stayed in Connecting. Adding the module to machine.kernel.modules fixed it without a reboot.

The module ships in the drbd extension on v1.13 too, so nodes can take this config before the upgrade. Nodes that already list it through extraKernelModules are fine, since Talos tolerates duplicate module names.

Cozystack is moving its Talos image to v1.14.2 in cozystack/cozystack#4598, so this should land before that release.

Talos v1.14 ships a kernel with an empty modprobe path
(siderolabs/pkgs#1565), so /proc/sys/kernel/modprobe is empty and the
kernel no longer loads modules on request_module(). DRBD requests its
transport module that way when a resource first connects to a peer,
so on v1.14 drbdsetup new-peer fails with "Failed to create transport
(drbd_transport_xxx module missing?)" and every DRBD resource on the
node stays in Connecting.

Listing drbd_transport_tcp in machine.kernel.modules makes Talos load
it up front; applying that to an affected node restored connectivity
without a reboot. The module ships in the drbd extension for both
v1.13 and v1.14, so loading it explicitly is safe on either.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
@coderabbitai

coderabbitai Bot commented Sep 30, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 46 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: a69837d0-b52a-49a2-8560-c0b4944e83da

📥 Commits

Reviewing files that changed from the base of the PR and between dac1d2c and c4f62f4.

📒 Files selected for processing (8)
  • charts/cozystack/templates/_helpers.tpl
  • charts/cozystack/values.yaml
  • docs/manual-test-plan.md
  • pkg/engine/contract_machine_test.go
  • pkg/engine/testdata/golden/cozystack-controlplane-legacy.golden.yaml
  • pkg/engine/testdata/golden/cozystack-controlplane-multidoc.golden.yaml
  • pkg/engine/testdata/golden/cozystack-worker-legacy.golden.yaml
  • pkg/engine/testdata/golden/cozystack-worker-multidoc.golden.yaml

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@IvanHunters IvanHunters left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict

LGTM

Correct, minimal, well-targeted fix: the cozystack preset now pins drbd_transport_tcp beside drbd, verified safe on fresh v1.14 install and on the N-1 upgrade, with the changed contract tests proven non-vacuous.

Caveats

  • Upgrade path verified, not assumed. A node that already lists drbd_transport_tcp via extraKernelModules now renders it twice (built-in + append). Talos v1.14.0 tolerates this: KernelModuleConfigController builds each entry as NewKernelModuleSpec(ns, module.Name()) via WriterModify keyed by name, so duplicate names collapse to one resource, and v1alpha1_validation.go performs no module-uniqueness check; the list is a YAML sequence, so decode does not reject duplicate items either. Confirmed against pinned upstream source, not test-pinned (runtime behaviour, correctly out of unit scope).
  • Module name confirmed from DRBD9 source, not folklore: drbd/drbd_transport_tcp.c compiles to drbd_transport_tcp.ko, and drbd_transport.c:87 calls request_module("drbd_transport_%s", name) with name tcp, so the exact missing-on-v1.14 module is drbd_transport_tcp as written.
  • Changed-test non-vacuity: removing the template line turns TestContract_Machine_KernelModules_Cozystack (all 4 cells), _ExtraKernelModules_Cozystack_AppendValues, and _EmptyOmitsAppend RED, the last with expected 7 modules ... got 6, so the kernel:/certSANs: block-boundary count is real and not fooled by the added line.
  • Scope confirmed correct: rendered on CP and worker, legacy and multidoc, ordered after drbd core and before zfs; the generic chart (operator-supplied modules, no built-in pins) is untouched, so it correctly does not receive the entry, and drbd core is already pinned on the same node set so emitting the transport everywhere mirrors existing placement rather than over-broadening.
  • Hermetic: no cluster touched. talm renders Talos machine config applied client-side, so there is no helm-controller SSA / admission surface; the render plus the upstream-source checks above are complete for this repo. One unrelated suite failure (TestCommittedTextFilesIgnoresUntrackedArtefacts) is a host-git 2.23 artifact, not a PR defect; review-helper mutate --from-diff had nothing to grade (the diff adds a list entry, no guard or numeric bound); talm ships no AGENTS/CONTRIBUTING, so the cozystack release-note and make generate conventions do not apply, and the values.yaml edit is a plain comment, not a cozyvalues @param.

Aleksei Sviridkin (lexfrei) added a commit to cozystack/cozystack that referenced this pull request Oct 1, 2026
## What this PR does

Bumps the Talos image Cozystack builds from v1.13.6 to v1.14.2 to get a
newer DRBD. v1.13.6 ships DRBD 9.3.2, v1.13.7 and later and
v1.14.0/v1.14.1 ship 9.3.3, and v1.14.2 is the first release with 9.3.4.
ZFS moves from 2.4.3 to 2.4.4.

On 9.3.2 I've seen a peer reboot leave the DRBD sender thread spinning
on a failed send (EPIPE). What followed was RCU stalls, processes stuck
in D-state, a hanging `drbdsetup disconnect` and two-phase-commit
timeouts. The [9.3.4
ChangeLog](https://github.com/LINBIT/drbd/blob/drbd-9.3.4/ChangeLog) has
fixes for a sender thread pinning a CPU while its connection is down,
for two-phase-commit issues and a hanging `drbdadm disconnect`, and for
two nodes ending UpToDate with different data after a reconnect. None of
them are in 9.3.3. I matched the trace to those entries by their
descriptions only, not against the commits.

The base installer image had to change. Talos v1.14 no longer publishes
`ghcr.io/siderolabs/installer`, so `gen-profiles.sh` now uses
`ghcr.io/siderolabs/installer-base`, same as upstream's own v1.14 imager
profiles. I built the amd64 installer from the new profile with
`imager:v1.14.2` locally and it builds without errors.

PR CI will not boot the new image. It builds the installer and matchbox
images, but the e2e runs on the upstream `talos:v1.13.5` container from
`hack/e2e-compose.yaml`. The first run that boots the new kernel, DRBD
and ZFS is the nightly after merge, which builds the nocloud disk from
these profiles and runs it in QEMU.

On Talos v1.14.2 the `drbd_transport_tcp` module is no longer loaded on
demand. After upgrading a node from v1.13.6, `drbd` was loaded and the
transport module was not, `drbdsetup new-peer` failed with "Failed to
create transport (drbd_transport_xxx module missing?)", and every DRBD
resource on the node stayed in Connecting. Listing the module in
`machine.kernel.modules` fixed it without a reboot. The cause is
[siderolabs/pkgs#1565](siderolabs/pkgs#1565), in
Talos since v1.14.0: the kernel is built with an empty modprobe path, so
it no longer loads modules on `request_module()`, which is how DRBD asks
for its transport. Any module that used to load that way now has to be
listed explicitly (reported upstream as siderolabs/talos#14501). Outside
DRBD this includes `dm-thin-pool` and `dm-multipath`, which the v1.14.2
kernel builds as modules, so LVM-thin or multipath users on Talos need
them listed too. The e2e node config now loads `drbd_transport_tcp`
explicitly, and the talm preset (cozystack/talm#252) and the install
docs (cozystack/website#720) get the same line. The module ships in the
drbd extension on v1.13 too, so the extra line is safe before the
upgrade.

The order matters for host nodes. Either update the cozystack preset in
the talm project to a version with cozystack/talm#252 and re-render and
apply the node config, or add `drbd_transport_tcp` to
`machine.kernel.modules` by hand. Only then upgrade the node to v1.14.2.
A node upgraded first loses DRBD replication until the module is added.

The host network policy now also denies port 2383 to `world`. Talos
v1.14 moved etcd's `/metrics`, `/health` and gRPC-gateway JSON API from
2379 to a separate listener on 2383, and the policy only blocked 2379
and 2380. Without this, the etcd JSON API on v1.14 control-plane nodes
would be reachable from outside, protected only by client mTLS.

Two smaller changes come with the bump. The multus
`talos-cni-plugins-checked-against` marker moves to v1.14.2, because the
pkgs release behind Talos v1.14.2 still pins CNI plugins v1.9.1, same as
multus. The system memory limits doc now describes the v1.14 OOM
trigger, since v1.14.0 dropped its global memory PSI clause
([siderolabs/talos#13895](siderolabs/talos#13895)).

Before upgrading hosts, operators should know a few things from the
[v1.14.0 release
notes](https://github.com/siderolabs/talos/releases/tag/v1.14.0) and the
Talos compatibility code:

- Talos v1.14 accepts a host upgrade from v1.12.0 or later. It supports
Kubernetes 1.32 through 1.37, while v1.13 still accepted 1.31, so a
cluster on Kubernetes 1.31 has to upgrade Kubernetes first.
- etcd's HTTP endpoints (`/metrics`, `/health`) moved from port 2379 to
2383. The Cozystack etcd scrape proxy reads a separate metrics listener
on `127.0.0.1:2381`, which this change does not move. Anything else that
scraped or health-checked etcd on 2379 has to switch to 2383, and a
firewall outside the cluster that blocked 2379 should block 2383 too.
- etcd and kube-apiserver now require TLS 1.3, so a client that can only
speak TLS 1.2 stops connecting.
- Workload isolation (`SecurityProfileConfig`) stays off on upgraded
clusters, but `talosctl gen config` turns it on for new ones. I haven't
tested Cozystack with it enabled.
- `talosctl apply-config --mode=reboot` is gone.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#720
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [x] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
cozystack/talm#252
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The website PR refreshes the `next` version pins for v1.14.2 and adds
the module to the Talos install pages. The talm PR adds it to the
cozystack preset.

### Release note

```release-note
feat(talos): Bump Talos to v1.14.2, which ships DRBD 9.3.4 and ZFS 2.4.4. DRBD 9.3.4 fixes a sender thread spinning on a dead connection, two-phase-commit hangs and a data divergence after a reconnect. Before upgrading a node, add `drbd_transport_tcp` to its Talos `machine.kernel.modules` next to `drbd`: Talos v1.14.2 no longer loads it on demand, and without it DRBD cannot connect to any peer. We recommend skipping the v1.13 line and upgrading nodes straight to v1.14.2. Talos accepts a direct upgrade from v1.12.0 or later. Talos v1.14 moves etcd's HTTP metrics and health endpoints from port 2379 to 2383, and the Cozystack host network policy now blocks 2383 from outside the cluster.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Updates**
* Upgraded host node support to Talos v1.14.2 across machine images,
with refreshed firmware and storage extensions.
* Updated memory-pressure guidance for Talos v1.14, including
version-specific behavior and configuration recommendations.
  * Added protection for Talos v1.14’s etcd HTTP endpoint on port 2383.
* Updated cluster preparation to load the TCP transport module required
for DRBD.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants