Skip to content

smp-server: fix notification store key leak - #1907

Open
shumvgolove wants to merge 2 commits into
masterfrom
sh/fix-ntf-store
Open

shumvgolove wants to merge 2 commits into
masterfrom
sh/fix-ntf-store

Conversation

@shumvgolove

@shumvgolove shumvgolove commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

The notification store kept a key for every notifier that ever received a notification: expiry trimmed the list but not the key, and delivery emptied the list but not the key. deliverNtfsThread sends every key to getQueueNtfServices (one notifier_id IN ? query, read in full) every 1.5 s, so the cost of each tick grew with every notifier notified since restart.

  • Expiry removes a key when its list becomes empty, and storeNtf is atomic, so a notification cannot be stored to a removed key's TVar.
  • After each delivery tick, keys whose lists were emptied are removed (deleteEmptyNtfs), each in its own small transaction, only if the list is still empty.

Delivery loop with no traffic (ntfloop):

Keys Allocation CPU Peak memory in use
0 0 MiB/s 0 280 MiB
100k 168 MiB/s 0.42 cores 530 MiB
300k 263 MiB/s 0.76 cores 913 MiB

20k notifications delivered to a subscribed ntf client, then idle (ntfdeliver):

Before After
keys kept 20,000 0
idle allocation 36 MiB/s 0.2 MiB/s
idle CPU 0.18 cores 0.01 cores

New tests in MsgStoreTests (notification store): expired and empty keys are removed; only flushed keys that are still empty are removed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant