erigon_proxy

A health-checking HTTP/WebSocket reverse proxy and load balancer for a fleet of Erigon Ethereum RPC nodes. It routes JSON-RPC traffic only to nodes that are currently at the chain tip, evicting nodes that fall behind and re-admitting them once they catch up — so clients see a single, always-fresh endpoint instead of a set of independently-lagging nodes.

Single file, pure Python + aiohttp, no database, no external state.

The problem it solves

Individual archive nodes lag the chain tip unpredictably — a background compaction stall, a slow import, a brief network blip — and a client that talks to one node directly will intermittently get stale answers. The usual fix, a plain round-robin load balancer, is worse: it happily routes to a node that's 50 blocks behind because it only checks that the TCP port is open.

erigon_proxy instead tracks every backend's block height on a fast internal sweep and routes only to nodes at (or acceptably near) the fleet tip. The interesting part is what "near the tip" means — see the health band below.

How health works

The health band, not exact-tip

Independent nodes import each new block with a small stagger: for a second or two around every ~12-second slot, the fleet is naturally spread across a few block numbers. Treating anything below the single fastest node as "unhealthy" therefore evicts nodes on normal jitter — and produces a constant eviction storm that gets worse on cheaper hardware (wider stagger), even though those nodes answer RPC perfectly well one or two blocks off the tip.

So health is a band around the fleet tip (the max block height observed across backends), with hysteresis:

• lag ≤ SOFT_LAG_MAX → healthy (within the propagation band).
• SOFT_LAG_MAX < lag < HARD_LAG_IMMEDIATE → soft lag: tolerated for up to LAG_GRACE_SECONDS, then evicted if it persists.
• lag ≥ HARD_LAG_IMMEDIATE → frozen/broken: evicted immediately.
• A recovering node is re-admitted once it stays within the band for READMIT_SECONDS (not at the exact tip — a moving tip made exact-tip re-admit nearly unreachable).
• The healthy pool is never reduced below MIN_HEALTHY_FLOOR by soft eviction — a slightly-stale node beats an empty pool returning 503s.

Routing always prefers the freshest healthy node, so widening the band never serves stale data when a fresher backend is available; it just keeps slightly-behind nodes in rotation as warm failover.

External anchor

Once every REFERENCE_CHECK_INTERVAL seconds the proxy makes a single call to an external reference provider, purely to detect the case where the whole fleet trails the real chain tip (an upstream/reorg situation the internal sweep can't see, since it measures the fleet against itself). This rate-limited call is kept off the hot path so the internal sweep can run as fast as desired.

WebSocket support

The proxy transparently relays WebSocket connections (eth_subscribe, long-lived duplex RPC) to a healthy backend:

• The freshest healthy node is chosen at connect time and pinned for the life of the connection (a subscription is stateful and can't be re-routed mid-stream).
• A periodic heartbeat (WS ping) keeps the connection alive across load-balancer idle timeouts and detects dead peers.
• If the pinned backend leaves rotation (lags, is evicted, or the proxy is draining), the client is closed so it reconnects onto a fresh node. Clients are therefore expected to implement reconnect-on-close.

WebSocket upgrades are handled automatically — there is no flag to enable them.

Installation

RPM (Fedora / EPEL)

The project ships as an RPM with a systemd unit and a sysconfig file. Build it from the spec, or install from a Copr repository if you publish one:

rpmbuild -ba erigon_proxy.spec
sudo dnf install ./erigon_proxy-*.noarch.rpm

The package installs:

• /usr/bin/erigon_proxy — the proxy
• /usr/lib/systemd/system/erigon_proxy.service — the service unit
• /etc/sysconfig/erigon_proxy — configuration (marked config(noreplace))

Only runtime dependency: python3-aiohttp.

Running directly

INSTANCES=10.0.0.1,10.0.0.2 REFERENCE_ADDR=example.com \
  SERVER_KEY=... BIND_ADDR=0.0.0.0 ./erigon_proxy

Configuration

All configuration is via environment variables, normally set in /etc/sysconfig/erigon_proxy and read by the systemd unit. SERVER_KEY is a secret, so the file is installed 0600.

Required (no defaults — the process will not start without them)

Variable	Meaning
INSTANCES	Comma-separated backend node IPs; each becomes http://<ip>:8545.
REFERENCE_ADDR	External reference provider host (used only for whole-fleet-lag detection).
SERVER_KEY	API key for the reference provider. Secret.
BIND_ADDR	Public interface to listen on (0.0.0.0, or a specific address).

Health band

Variable	Default	Meaning
SOFT_LAG_MAX	3	Blocks behind tip still considered healthy (propagation band).
HARD_LAG_IMMEDIATE	8	Blocks behind → evict immediately (frozen/broken).
LAG_GRACE_SECONDS	20	How long soft lag is tolerated before eviction (keep > one slot).
READMIT_SECONDS	5	How long a node must sit within the band before rejoining.
MAX_FLEET_LAG	5	Fleet tip this far behind the external reference → whole-fleet-lag warning.
MIN_HEALTHY_FLOOR	1	Never soft-evict below this many healthy nodes.

Operational

Variable	Default	Meaning
HEALTH_CHECK_INTERVAL	1	Internal sweep cadence (seconds).
HEALTH_CHECK_TIMEOUT	2	Per-node / per-reference call timeout (seconds).
REFERENCE_CHECK_INTERVAL	45	External anchor cadence (seconds).
ETH_GETLOGS_LIMIT	75	Max concurrent eth_getLogs in flight (caps backend disk-I/O amplification).
SHUTDOWN_DRAIN_SECONDS	40	On SIGTERM, keep serving this long while the LB stops routing (see draining).

WebSocket (optional; sane defaults apply if unset)

Variable	Default	Meaning
WS_HEARTBEAT_SECONDS	30	WS ping interval; keeps connections under the LB idle timeout.
WS_HEALTH_POLL_SECONDS	5	How often a WS connection re-checks its pinned backend's health.

Note: the WebSocket heartbeat is validated at startup against a hard-coded AWS_NLB_IDLE_TIMEOUT_SECONDS (350s). This ceiling is deliberately not configurable via environment, so the safety check can't be disabled by "raising the limit". If you front the proxy with something other than an AWS NLB (a different idle timeout), that constant in the source is the one line to change.

Management API

A separate, localhost-only management server runs on 127.0.0.1:8081:

Endpoint	Method	Purpose
/health	GET	Full per-backend health, fleet tip, external tip, and counts (JSON).
/loglevel	POST	Change log level at runtime, e.g. {"level": "DEBUG"}.
/servers/add	POST	Add a backend at runtime, e.g. {"url": "http://10.0.0.3:8545"}.
/servers/remove	POST	Remove a backend by URL.

The public proxy port additionally serves GET /_health as a load-balancer probe (returns 503 while draining or when no backend is healthy, 200 otherwise).

curl -s http://127.0.0.1:8081/health | jq

Logging

The proxy logs on state change — evictions, re-admissions, whole-fleet-lag, and health-count changes — so a quiet log means a stable fleet. Set the level to DEBUG (via /loglevel) to see a full per-sweep state dump.

Draining and restarts

On SIGTERM (a systemd stop or package restart) the proxy flips its /_health probe to 503 and keeps serving for SHUTDOWN_DRAIN_SECONDS so the load balancer stops routing new traffic before the process exits. SIGINT (Ctrl-C) skips the drain and exits promptly.

While draining, the health monitor is frozen: it holds its last-known-good view of the backends and stops probing. This is deliberate — a shutting-down process should keep serving the nodes it last knew were healthy and then leave, not keep re-evaluating a fleet whose network path may be tearing down around it. Without this, a transient blip during the drain window (or, on a host reboot, networking being torn down) would evict every backend at once and make the proxy return 503 for the very requests the drain window exists to serve.

SHUTDOWN_DRAIN_SECONDS should be set longer than the front load balancer's deregistration delay, so the balancer has fully stopped routing before the proxy stops accepting connections.

Deployment notes

• The proxy is designed to sit behind a TCP load balancer (e.g. an AWS NLB) with two or more instances for high availability. Because it's a plain Python aiohttp service it is architecture-agnostic (runs on x86-64 and ARM64 alike).
• It expects each backend to serve both HTTP-RPC and WebSocket on the same port (Erigon's --ws shares the HTTP-RPC port).
• The client's Host header is stripped before forwarding, so backends that enforce an http.vhosts allowlist accept the proxied request.

License

Licensed under the MIT License. This project is REUSE compliant.