Hetzner: cpx (new gen) defaults + capacity-aware placement - #166
Merged
Conversation
ARM cax* capacity at Hetzner has been chronically tight across all EU locations, causing repeated `resource_unavailable` placement failures on apply. Switch the supernode/fullnode defaults to the spec-equivalent AMD shared-vCPU sizes which have far better availability: - supernode: cax41 (16/32) -> cpx51 (16/32) - fullnode: cax31 (8/16) -> cpx41 (8/16) The arch label logic already keys off the `^cax` regex, so labels will automatically flip to `arch:amd64`.
Hetzner is rolling out a new CX line that replaces the CPX series in many locations. CPX SKUs (cpx41/cpx51) are no longer creatable in fsn1/nbg1/hel1, returning `unsupported location for server type (invalid_input)` from the API. Switch defaults to the new CX equivalents at identical specs and slightly lower price: - supernode: cpx51 (16/32) -> cx53 (16/32) — €31.99 -> €22.99 - fullnode: cpx41 (8/16) -> cx43 (8/16) — €24.49 -> €12.49
Hetzner ARM (cax*) capacity has been chronically tight in EU and
the legacy CPX line (cpx41/cpx51) is no longer creatable in some
regions. Newer cpx42/cpx62 are widely available and still ~3-4x
cheaper than DigitalOcean for equivalent specs.
Two behavior changes layered together:
1. Default sizes flipped to the new-gen CPX:
- supernode: cpx62 (16 vCPU / 32 GB / 640 GB)
- fullnode: cpx42 (8 vCPU / 16 GB / 320 GB)
2. Capacity-aware placement: query `hcloud_datacenters` and
`hcloud_server_type` data sources at plan time, build the set
of locations whose datacenters currently report both SKUs as
`available`, and round-robin only across those. Falls back to
the full region list if every region is sold out.
- `lifecycle { ignore_changes = [location] }` keeps existing
servers pinned where they were placed even if next plan's
filter would prefer a different region (location is
replacement-forced otherwise).
- `hcloud_server_network` reads the actual server location
post-state so the network reference stays correct after any
drift between planned and real location.
Arch labels still flip to `arch:amd64` automatically via the
existing `^cax` regex.
Capture why we landed on cpx42/cpx62 (capacity vs cax/legacy cpx, cost vs DigitalOcean) so future operators don't have to relitigate the choice.
qu0b
approved these changes
Jul 27, 2026
barnabasbusa
pushed a commit
that referenced
this pull request
Aug 3, 2026
#180) Backports the generic improvements that ran ahead in `glamsterdam-devnets` and `bal-devnets` but never made it back here. Most of the earlier drift has already landed on master (coredevs SSH rosters, buildoor, the IPv6 rework, `sync_mapping`, the template-name guard, inventory-path derivation) — this is what was left. The buildoor gap below is a bug found while auditing rather than a straight copy. ## buildoor only covered 3 of 5 consensus clients `buildoor` wins on group precedence via `ansible_group_priority=100`, so any client whose args it does **not** re-merge silently loses that client's own args — bootnodes included. Only prysm, lighthouse and lodestar were re-merged, which left teku and grandine buildoor nodes unusable. Split teku/grandine into `*_container_command_extra_simple_args` (the pattern the other three already use), added their merge entries, and gave every client its subscribe-all-subnets flag — a builder has to see every attestation subnet to value a payload correctly. Every flag was verified against the real binaries rather than trusted from the source repo: | Client | Flags | Verified against | |---|---|---| | teku | `--Xfork-choice-updated-always-send-payload-attributes=true`, `--p2p-subscribe-all-subnets-enabled` | `consensys/teku:latest` (the first is a hidden `X` option, probed by running it) | | grandine | `--features=AlwaysPrepareExecutionPayload`, `--subscribe-all-subnets` | `ethpandaops/grandine:develop` (also confirmed repeated `--features` both parse) | | lighthouse | `--always-prepare-payload`, `--subscribe-all-subnets`, `--import-all-attestations` | `beacon_node/src/cli.rs` | | prysm | `--prepare-all-payloads`, `--subscribe-all-subnets`, `--p2p-colocation-whitelist` | `cmd/flags.go`, `config/features/flags.go` | | lodestar | `--emitPayloadAttributes`, `--subscribeAllSubnets` | already live on glamsterdam-devnet-7 | ## The rest - **mitogen 0.3.43 → 0.3.49** (`ansible.cfg` + `install_dependencies.sh` in lockstep) - **CI: install python build headers** — asdf compiles python from source and the runner image lacks `libbz2`/`readline`/`lzma`, so the build silently drops those modules and pip fails later - **`noqa` syntax** — `#noqa rule` is not recognised by ansible-lint; it needs `# noqa: rule` - **prysm colocation whitelist** — devnet peers are few and often share a /24, which prysm's colocation limit treats as a sybil cluster and refuses to peer with - **Skip DigitalOcean VPCs on hetzner-only devnets** (from bal-devnets) — an empty VPC per region is still billable surface and blocks project teardown - **`run.zsh`: `check_deps` and `send_funds`** — `check_deps` reports every missing tool with an install hint instead of dying on the first one mid-run ## Deliberately not included - **`validator_keys_container_name`** for teku/nimbus/grandine. It only exists in the unmerged ethpandaops/ansible-collection-general#562, so adding it now would ship a variable no released role reads. Worth a follow-up once that merges. - **Hetzner capacity-aware placement / cpx defaults** — already proposed in #166 and live on glamsterdam; not duplicated here. - **`actions/checkout` bump** — dependabot owned that in #172, now merged and pulled in here via master. - **Prefixing the remote terraform state key** — dropped at qu0b's request; the key is set per-devnet during fork setup anyway. ## Verification - `ansible-lint playbook.yaml --profile production` (the exact CI invocation) — passes - `terraform fmt -check` and `terraform validate` — clean - `zsh -n scripts/run.zsh`, plus `run.zsh check_deps` and `run.zsh help` executed - Jinja render check on all five buildoor merge expressions, confirming each client's base args survive the merge: ``` prysm -> ['--a', '--bootstrap-node=enr:x', '--prepare-all-payloads', '--subscribe-all-subnets'] lighthouse -> ['--b', '--always-prepare-payload', '--subscribe-all-subnets', '--import-all-attestations'] lodestar -> ['--c', '--emitPayloadAttributes', '--subscribeAllSubnets'] teku -> ['--d', '--Xfork-choice-updated-always-send-payload-attributes=true', '--p2p-subscribe-all-subnets-enabled'] grandine -> ['--e', '--features=AlwaysPrepareExecutionPayload', '--subscribe-all-subnets'] ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two changes layered together to address recurring
resource_unavailableandunsupported location for server typeerrors onterraform apply:1. New default SKUs
cax41(16 ARM / 32 GB / 320 GB)cpx62(16 AMD / 32 GB / 640 GB)cax31(8 ARM / 16 GB / 160 GB)cpx42(8 AMD / 16 GB / 320 GB)Hetzner ARM
cax*capacity has been chronically tight, and the legacycpx41/cpx51SKUs are no longer creatable in some EU locations. The new-gencpx42/cpx62are widely available.Cost vs DigitalOcean
cpx*is more expensive thancax*/cx*, but still dramatically cheaper than DigitalOcean for equivalent specs:cpx42≈ €25/mos-8vcpu-16gb≈ €88/mo (~$96)cpx62≈ €50/mos-16vcpu-32gb≈ €177/mo (~$192)So we're roughly 3–4× cheaper than DO while staying on a SKU that's actually in stock. For a multi-month devnet that's hundreds of euros saved per supernode.
The arch label logic at
hetzner.tfalready keys off the^caxregex, so labels automatically flip toarch:amd64.2. Capacity-aware placement
At every plan/apply, query
hcloud_datacentersandhcloud_server_typedata sources, build the set of locations whose datacenters currently report both SKUs asavailable, and round-robin new servers only across those. Falls back to the fullvar.hetzner_regionslist if every region is sold out (so plan doesn't error out — at that point you're stuck either way).lifecycle { ignore_changes = [location] }onhcloud_serverkeeps existing placements pinned even if next plan's filter would prefer a different region (location is replacement-forced onhcloud_serverotherwise).hcloud_server_networknow readshcloud_server.main[each.key].location(the actual post-state location) so the network reference stays correct under any drift.Test plan
resource_unavailable/unsupported locationerrorsecho 'local.hetzner_available_locations' | terraform consolereturns the expected non-empty list when capacity is healthy