[ZMQ] append a message sequence number to every ZMQ notification

[jonasschnelli]

Currently, ZMQ listeners cannot detect if they have missed a message.
This PR adds a sequence number to each message (each message type has its own counter).

The sequence number is the last part of the multipart zmq message to not break the API, though, we could consider breaking the API in favor of moving the sequence number to the very beginning.

Todo:

• update release notes

[laanwj]

Concept ACK, I like adding the sequence nr as an extra part. Also ok with using 32 bit sequence numbers.

[promag]

I thought a subscriber can't miss a message.

Edit:

But it can: http://stackoverflow.com/a/15821036. So what can a subscriber do if it detects a missing message?

[jonasschnelli]

@promag:
I'm not sure if I would design an listening application that fully rely on getting all notifications. With this PR, you could at least detect if you got all (also check if you got the first) notifications and if not, you could poll/sync your data over RPC.

[laanwj]

I thought a subscriber can't miss a message.

The way we are using ZMQ, the sender will never block. This is sensible as we don't want to hang the application due to a slow client. But this means at some point one of the send queues will fill up and new messages will be discarded.

But it can: http://stackoverflow.com/a/15821036. So what can a subscriber do if it detects a missing message?

That's up to the application. Either stop with a fatal error, or re-start/re-sync. An event application usually does some synchronization at start-up, then starts processing incremental changes. After getting out of sync it can repeat the process. The important thing is being able to detect it and thus act on it.

[laanwj]

This needs mention in doc/release-notes.md as well as doc/zmq.md (there, don't forget to specify the size and endianness of the sequence number), I think it is ready for merge otherwise.

[maflcko]

What is the reason for using str() here?

[jonasschnelli]

Fixed @MarcoFalke nit.
Mentioned the change in doc/release-notes.md as well as in doc/zmq.md

[maflcko]

Nit: Add the b prefix, which is a noop in py2?

b"hashtx"

[maflcko]

Thanks for fixing!

[jonasschnelli]

@MarcoFalke: fixed nits.

[laanwj]

Why the rename to _keepalive?

[jonasschnelli]

I renamed it to _keepalive because it does not "close" the multipart message. All messaged are sent with ZMQ_SNDMORE and I wanted to increase the awareness of that by renaming the function.

[laanwj]

Here's a slight simplification of the code - gets rid of some duplication, as well as the _keepalive change:
laanwj@fb843df

[jonasschnelli]

@laanwj: Thanks. Much better. Stolen fb843df, squashed and force pushed.

[dcousens]

The way we are using ZMQ, the sender will never block. This is sensible as we don't want to hang the application due to a slow client.

Could we make that parameterizable?
In certain cases I'd rather have message reliability than having the node to be guaranteed to be running in real time.

[sipa]

I don't believe ZMQ ever guarantees reliability.

[laanwj]

In certain cases I'd rather have message reliability than having the node to be guaranteed to be running in real time.

Well notifications can be lost, through zmq or otherwise, for example if bitcoind needs to be restarted, or a myriad of other circumstances not under your control. At least you can detect it now:

• sequence number goes to 0 (and wasn't at 0xffffffff) -> bitcoind restarted
• sequence number skips a beat -> send buffer was full

Your application needs an (application dependent) way to resync anyhow. This is better than pretending to guarantee something.