Skip to content

Swap is force-recreated every ~2 hours by DAppNode host scripts #733

Description

@daferna

Swap is force-recreated every ~2 hours by DAppNode host scripts, which breaks validator block proposals.

Setup: DAppNode host, 32 GB RAM, Besu + Nimbus + Web3Signer, 7 mainnet keys.

I missed a proposal because the host was swapping (~5 GB of 16 GB /swapfile in use) while Besu was importing a heavy block and Nimbus timed out signing a blinded builder block (Unable to sign blinded block proposal using remote signer reason="Operation timed out"). Besu docs explicitly say to disable swap on a validator.

I then:

  • Commented swap out of /etc/fstab
  • swapoff /swapfile
  • systemctl mask swapfile.swap (/etc/systemd/system/swapfile.swap -> /dev/null)

None of that sticks. /swapfile is recreated and swapon’d about every 2 hours:

  • 2026-09-01 21:12:20 UTC — kernel: Adding 16777212k swap on /swapfile
  • 2026-09-01 23:12:11 UTC — file born; 23:12:19swapon again
    stat: Size 17179869184, mode 0600, root:root

swapfile.swap then shows as masked + active, because systemd sees the file in /proc/swaps. The mask is doing its job. Something else calls swapon(2) on the host.

That something is your own scripts:

/usr/src/dappnode/scripts/dappnode_install.sh
  fallocate -l ${SWAP}k /swapfile
  mkswap /swapfile
  swapon /swapfile
  echo '/swapfile none swap defaults 0 0' >> /etc/fstab

/usr/src/dappnode/DNCORE/scripts/upgrade/add_swap.sh
  dd if=/dev/zero of=/swapfile bs=1M count=$mb
  mkswap /swapfile
  swapon /swapfile
  echo "/swapfile swap swap defaults 0 0" >> /etc/fstab

add_swap.sh is clearly “if swapon --show is empty, build a 16G file and enable it.” There is no timer named swap in systemctl list-timers, no crontab, and no DNCORE compose volumes: for /swapfile. A short-lived privileged --rm container (not listed in docker ps -a) is almost certainly how dappmanager/core invokes this on the host. Wifi is the only core compose with privileged: true; it does not mention swap.

This is a bad default for a staking box:

  1. You fight the operator. Mask + fstab is the documented way to disable swap. You overwrite it.
  2. You append another fstab line every successful run.
  3. On 32 GB + Besu/RocksDB, swap turns a 200ms import into multi-second stalls and misses proposals. I had a clean proposal the next day after heap/governor fixes and swap off — until you turned swap back on.
  4. There is no UI toggle, no env var, no “I am a validator, do not touch swap.”

Ask:

  • Stop auto-creating /swapfile when the operator has masked swapfile.swap or removed it from fstab.
  • Expose an explicit setting: DAPPNODE_ENABLE_SWAP=false (or host UI) that add_swap.sh honors.
  • Do not >> /etc/fstab on every run.
  • Document that staking nodes should run without swap.

I have stubbed add_swap.sh to exit 0 and replaced /swapfile with an immutable directory so the next 2-hour job fails closed. I should not have to vandalize your upgrade scripts to keep a validator safe.

Happy to provide full journal slices and stat output.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions