3

We are observing the following panic for a parachain node that is warp synced fully on both sides when we try to restart the node:

 Thread 'main' panicked at 'route from finalized to leaf should be available; qed: UnknownBlock("Header was not found in the database: 0x1e70664aec1b2dcbec1ac26210ddb6d7388700e33604edbfa3a55141b22c3837")', /root/.cargo/git/checkouts/cumulus-59522f43471fa161/f603a61/client/consensus/common/src/level_monitor.rs:123

This is pointing to this line in the cumulus project. My own reading from the code is that a recent header is missing simply because it wasn't yet flushed to the permanent db. But that to me is an expected situation rather than a panic. For two reasons:

  1. It happened at least two times consistently after we restarted a node which is sort of not accidental.
  2. I don't know of a safe way to restart a node but we do need nodes to be resilient to restarts and so on.

My questions are:

  1. Is it expected?
  2. Is it related to warp sync flag?
  3. How can we avoid it?
Improve this question

1 Answer 1

Reset to default
1

No this wasn't expected. This was/is a bug.

The problem here is that when a node is warp syncing, it imports a header around the tip of the chain. However, in the case of the parachain this header is not imported as finalized block. The level monitor will then see this header as a leaf and will try to go back down to the last finalized block (genesis block). However, the node hasn't yet synced all the blocks between genesis and the leaf and thus, it will fail to fetch this block from the db.

Here is a fix.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.