Skip to content

v1.1.1 follow mode never issues forkchoice updates (EL can't sync); persistence service wedges after one batch on snapshot-restored datadirs #1156

Description

@Dynobit

Running ghcr.io/base/node-reth:v1.1.1 (and the ~Jun-5 build for comparison) on a
Base mainnet datadir restored from the official pruned snapshot
(base-mainnet-pruned-reth-1782806250.tar.zst, storage settings absent →
storage_v2:false), fed via base-consensus follow (BASE_NODE_SOURCE_L2_RPC),
host: Linux 6.19 x86_64, 62G RAM, ext4.

Bug 1 — v1.1.1 follow mode sends engine_newPayload only, never any
forkchoiceUpdated.
Payloads insert (payload_safety=unsafe, ~3ms each) but
nothing canonicalizes; the in-memory tree grows until cgroup OOM (55G in
minutes), and all progress rewinds on restart. The ~Jun-5 build performs the
expected per-payload lockstep (insert → canonical commit → Forkchoice updated) — but is unusable post-Beryl. Net effect: follow mode cannot EL-sync
on v1.1.1, and the warning ("Running without --proofs; ... does not support EL
sync") suggests this may be intended — if so, what is the supported way to
catch an EL up from a snapshot when devp2p peers are unreachable (residential
NAT; Base EL mesh is effectively unlistable)?

Bug 2 — persistence service dispatches exactly one batch per process, then
never again
(observed on the Jun-5 build where follow+FCU works). Kernel-level
evidence while ~300 blocks/process were canonically committed with
--engine.persistence-threshold 16..1024:

  • persistence and mdbx-rs-txn-mgr threads parked in futex_wait across all
    samples, /proc/<pid>/task/<tid>/io lifetime read_bytes=0 write_bytes=0;
  • device write counters ~0 the entire time; mdbx.dat mtime bumps only on
    graceful shutdown flush;
  • durable floor advances by exactly one threshold-batch (e.g. +16) shortly
    after start, then never again; memory then accumulates (~150MB/executed
    block) until OOM.
    Same behavior with --db.sync-mode safe-no-sync, thresholds 16/64/32 and
    1024/2048/1024, and paced insertion (--follow.insert-delay-ms 250..2000).
    Looks like the engine awaits a persistence ack that never arrives (or the
    dispatcher gates on something absent in snapshot-restored v1 datadirs — note
    db settings get → "No storage settings found", and the v1-vs-requested-v2
    mismatch WARN at startup).

Happy to provide full logs, metric dumps, or run instrumented builds — this
box reproduces both within minutes, every time.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions