docs: add Talos node upgrade and DRBD recovery notes - #26
Open
Aleksei Sviridkin (lexfrei) wants to merge 3 commits into
Open
Aleksei Sviridkin (lexfrei) wants to merge 3 commits into
Aleksei Sviridkin (lexfrei) wants to merge 3 commits into
Conversation
Node upgrades on DRBD-backed clusters fail in ways the Cozystack upgrade flow never sees: the Talos version decides the DRBD version, Talos 1.14 stops autoloading drbd_transport_tcp, kexec reboots can hang bare-metal nodes, and a second node reboot before storage has settled can take quorum away from a resource. Add a reference that covers the version path, the machine-config changes to apply before the first node upgrade, per-node gotchas and the gate between node reboots. Add a known failure for the LINSTOR controller crash-loop caused by orphan layer rows after an interrupted resource deletion. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
Talos 1.14 builds the kernel with an empty modprobe path, so DRBD can no longer load drbd_transport_tcp on the first peer connection and every new-peer call fails with "Failed to create transport". List the module next to drbd in the review checklists and verification step. Also recommend kernel.kexec_load_disabled on bare metal: kexec reboots during later upgrades can hang the node, and with kexec disabled Talos reboots through firmware. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
A node that reboots in the middle of a resync can leave the pair looping in Connecting, with both sides expecting the other to become SyncTarget. invalidate and discard-my-data do not get out of it; forcing the stale side down and letting LINSTOR recreate the replica does. Also record the DRBD 9.3.2 sender soft-lockup fixed in 9.3.4 and the Talos 1.14 drbd_transport_tcp autoload regression. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Aleksei Sviridkin (lexfrei)
marked this pull request as ready for review
September 30, 2026 10:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The skills had nothing on upgrading Talos under a running Cozystack cluster, and most of what broke for me on a bare-metal cluster with LINSTOR/DRBD storage was in that gap. The cluster-upgrade skill stops at
helm upgrade. The nodes, and the DRBD version that comes with each Talos release, are left to the operator.This adds what I hit during that upgrade:
drbd_transport_tcpin the talos-bootstrap module lists, plus kexec disabled on bare metal. Talos 1.14 no longer loads extension modules on demand (v1.14: kernel no longer loads extension modules on demand (drbd_transport_tcp) siderolabs/talos#14501), and the preset fix is still open in fix(cozystack): load drbd_transport_tcp explicitly talm#252.Version claims are checked against upstream sources. DRBD versions come from
Pkgfileat each siderolabs/extensions tag. The rest comes from the DRBD 9.3.4 ChangeLog,MinimumHostUpgradeVersionin Talos 1.14 and the checksum format in the KubeVirt 1.8.4 CRDs.tools/check-refs.shpasses. markdownlint only flags the repeated Symptom, Root cause and Recovery headings, and every entry in known-failures.md already uses them.