Pending blocks might not be submitted to DA

[tzdybal]

Currently, blocks are submitted to DA via pendingBlocks queue. This queue is stored in memory, so in case of a restart of the aggregator node, the information might be lost. After the restart aggregator will continue to produce blocks, and submit those new blocks to DA, but all that were pending before the restart were lost so they will not be submitted to DA.
This causes issues since if any blocks are not in DA, full nodes are not able to sync.

Additionally:

• blocks are safely stored in database before submission to DA
• blocks are always pushed to DA in order (by height)
• DA submission of multiple blocks is atomic - it's impossible to submit only part of a batch
• rollkit can handle duplicate blocks as well as blocks out of order during sync process

Example:

1. Last produced block height: 1010.
2. pendingBlocks contains blocks between 1000 and 1010.
3. Node is restarted.
4. Block 1011 is produced and pushed to DA.
5. Blocks between 1000 and 1010 will never be submitted to DA.

[tzdybal]

Simple and effective solution is to modify the code to ensure that height of the latest block successfully submitted to DA is persisted in store. After restart we need to continue DA submission.

On the implementation level, there are several possibilities:

1. push blocks from store to pendingBlocks when node is (re)starting
2. get rid of pendingBlocks and always read blocks directly from store (this seems to be more generic approach).

[arhamj]

@tzdybal Can I pick this? Option 2 sounds better. We can have a cron-like go-routine which runs every X number of seconds and queries Y number of blocks from DB using a prefix filter. X and Y can vary based on the lag. What do you think?

[tzdybal]

Hi @arhamj, I'm currently tackling this issue. Thanks for understanding!