[hl-node][Mainnet] Panic: too many blocks to request in gossip forward_client_blocks after starting bootstrap; visor restarts child

[dasoylemez-dev]

Summary

Running a Mainnet non-validator with hl-visor / hl-node, I hit a hard panic in gossip forward_client_blocks immediately after starting bootstrap, with message "too many blocks to request". The hl-node process then exits; hl-visor logs "visor child in bad state, restarting" with child_stuck: true and child_running: false.

Separately (and often for long stretches), the node logs "execution behind?" (Querying stakes for high round / Querying jailed validators for high round) and "querying client blocks" lines may stop for 30+ minutes while peer connection attempts show "Timed out connecting to peer" in a tight loop. Local HyperEVM RPC on port 3001 is often unavailable while catch-up is incomplete.

I do not see a documented knob to raise limits or chunk bootstrap client-block requests; this looks like an internal assertion / bounds check in the node.

Environment

• OS: Ubuntu 24.04
• Chain: Mainnet
• Command (via systemd): hl-visor run-non-validator --serve-eth-rpc (child: hl-node --chain Mainnet run-non-validator --serve-evm-rpc --replica-cmds-style actions)
• Hardware: 128 GB RAM class; hl-node RSS often ~27-35 GB when observed
• Gossip: ~/override_gossip_config.json with root_node_ips from API gossipRootIps, try_new_peers: true; n_gossip_peers experimented (default vs 20) — panic reproduced regardless
• Public IP: ~/hl/override_public_ip_address set to server IPv4 (avoid IPv6 issues)
• Firewall: inbound allows 4001/4002 (and broadly TCP/UDP per operator policy)
• Related ops: I avoid systemd unit name / cmdline containing hl-visor where possible to sidestep journalctl-like processes makes hl-visor panic when run as systemd service #94-style proc matching.

Panic / Restart Sequence

• Right after a successful peer greeting and bootstrap start:

"successfully received greeting from peer, send_abci=false Ip()
starting bootstrap @@ [start_round: Round(1226461227)] @ [end_round: Round(1226492203)]
tokio_spawn_forever crashing process name=gossip forward_client_blocks:
... task ... panicked with message "too many blocks to request"
client_block_reader error, ending task: Bounded TrySend error: closed @@ [tcp_stream.peer_addr(): Ok(:4001)]"

• Then (tens of seconds later), visor:

"visor child in bad state, restarting @@ [child_stuck: true] @ [child_low_memory: false] @ [memory_usage: ...] @ [child_running: false] ..."

Other Symptoms

• High volume of "gossip_server_connect_to_peer connected to abci stream" followed by "Timed out connecting to peer" (~5s pattern).
• "execution behind?" ERROR spam with Round(...) advancing faster than estimated_*_valid_until catches up.
• No new "querying client blocks" journal lines for long periods despite active peer attempts.

Expected Behavior

Non-validator should not panic when transitioning into bootstrap / client-block sync; large catch-up should be chunked or bounded without terminating the process. Visor restart then loses progress and amplifies sync pain.

Questions:

Is "too many blocks to request" a known condition when local state is far behind — is there a supported mitigation (config, flags, recommended topology)?

Can bootstrap / client-block fetching be made resilient (chunk smaller batches, retry without panic)?

Any metrics or log lines recommended to distinguish healthy catch-up vs stuck loops?

[kevin-matthews-vc]

Reopening this. Still reproducing on latest Mainnet and it's silently corrupting served EVM history.

We run a Mainnet non-validator (hl-visor run-non-validator --serve-eth-rpc, child hl-node --serve-evm-rpc --replica-cmds-style actions) and hit this panic roughly once a day. 3 times in the last week on one node, lifetime n_restarts: 40. The reason I'm posting is the impact is worse than just a restart. Every panic leaves a permanent hole in the served HyperEVM history.

The panic and restart look like this:

15:09:22 WARN  hl-node  forward_client_blocks no more blocks from reader, reconnecting to a new peer [current_node_ip: Ip(<peer>)]
15:09:49 WARN  hl-node  tokio_spawn_forever crashing process name=nv_stream_forward_client_blocks:
         err=... task panicked with message "too many blocks to request"
15:10:16 ERROR hl-visor visor child in bad state, restarting
         [child_stuck: true] [child_low_memory: false] [child_running: false]
         [visor_abci_state: height: 1022625514, consensus_time: 2026-06-04T13:54:19, wall_clock_time: 2026-06-04T15:10:06]

Notice the committed consensus_time is stuck at 13:54:19 while wall clock is 15:10:06. The node's state didn't advance for about 76 minutes, then it panicked and restarted.

After the restart the child fast-forwards to head instead of backfilling, so that 76 minute window is gone for good. eth_getBlockByNumber for any height inside it returns {"code":-32603,"message":"invalid block height: N"} and eth_getTransactionByHash returns null, while the blocks either side of it are fine. We've confirmed holes from three separate panics, each one contiguous, somewhere between 1k and 4.5k blocks (17 to 74 minutes). Anything older than the holes is fully present, so it's not pruning. It's the catch-up getting skipped.

Here's what the node is actually doing during the stall, from the hl-node logs. After a restart it's behind, so it enters bootstrap and pulls client blocks in batches of 100 rounds, but the peers keep dropping out:

ERROR forward_client_blocks ... reconnecting to a new peer: early eof
WARN  error connecting to candidate peer Ip(...): Timed out connecting to peer
ERROR forward_client_blocks ... Could not query client block range 1317232601-1317232700
ERROR Querying jailed validators for high round, is execution behind?

It keeps re-bootstrapping against one peer after another and barely makes progress while chain height keeps climbing, until the round gap gets big enough to trip too many blocks to request and the process dies. Looks like a bounds check on a large catch-up that kills the process instead of chunking the request or retrying.

To be clear it's not the box running out of resources during the stall. It's 8 vCPU / 64 GB, and across all three panic windows the stall shows about 99% CPU idle, load near 0, 30 to 60 GB RAM free, disk util under 1%, network at baseline. The node is just sitting on repeated 20s peer timeouts, not starved. We do separately see child_low_memory: true restarts at around 95% memory on this 64 GB host (looks like #136), but those are restart triggers. The data loss happens in the catch-up afterwards, not from memory pressure.

One thing that points at a fix: the size of the hole tracks how far behind the child got. When a restart catches up quickly there are no missing blocks at all. It's only the long stall into panic cycles that leave holes. So the panic itself is what's causing the data loss.

A few asks:

1. Make bootstrap / forward_client_blocks chunk or bound a large catch-up instead of panicking.
2. Give us a way to configure the "too many blocks" limit and the peer/stream timeout (related to Non-validator Node Timeout Deadline Too Fast #83).
3. An HTTP or S3 abci_state snapshot for bootstrap would help a lot (related to Non-validator node stuck in bootstrap loop — P2P state transfer always times out (829MB, 60s timeout) #143). Once this panic loses visor_abci_state a non-validator can end up stuck in a bootstrap loop it can't get out of.

Env is Ubuntu 24.04, Mainnet, hl-node binary from around 2026-05-30. Happy to share the full visor_child_stderr and the Datadog charts. Given this silently corrupts served history, please reopen.