Skip to main content
edited body
Source Link
Alex Sed
  • 600
  • 2
  • 10

We are observing the following panic for a parachain node that is warp synced fully on both sides when we try to restart the node:

 Thread 'main' panicked at 'route from finalized to leaf should be available; qed: UnknownBlock("Header was not found in the database: 0x1e70664aec1b2dcbec1ac26210ddb6d7388700e33604edbfa3a55141b22c3837")', /root/.cargo/git/checkouts/cumulus-59522f43471fa161/f603a61/client/consensus/common/src/level_monitor.rs:123 

This is pointing to this line in the cumulus project. My own reading from the code is that a recent header is missing simply because it wasn't yet flushed to the permanent db. But that to me is an expected situation rather than a panic. For two reasons:

  1. It happened at least two times consistently after we restarted a node which is sort of not accidental.
  2. I don't know of a safe way to restart a node but we do need nodes to be resilient to restarts and so on.

My questions are:

  1. Is it expected?
  2. Is it related to warp sync flag?
  3. How we can we avoid it?

We are observing the following panic for a parachain node that is warp synced fully on both sides when we try to restart the node:

 Thread 'main' panicked at 'route from finalized to leaf should be available; qed: UnknownBlock("Header was not found in the database: 0x1e70664aec1b2dcbec1ac26210ddb6d7388700e33604edbfa3a55141b22c3837")', /root/.cargo/git/checkouts/cumulus-59522f43471fa161/f603a61/client/consensus/common/src/level_monitor.rs:123 

This is pointing to this line in the cumulus project. My own reading from the code is that a recent header is missing simply because it wasn't yet flushed to the permanent db. But that to me is an expected situation rather than a panic. For two reasons:

  1. It happened at least two times consistently after we restarted a node which is sort of not accidental.
  2. I don't know of a safe way to restart a node but we do need nodes to be resilient to restarts and so on.

My questions are:

  1. Is it expected?
  2. Is it related to warp sync flag?
  3. How we can avoid it?

We are observing the following panic for a parachain node that is warp synced fully on both sides when we try to restart the node:

 Thread 'main' panicked at 'route from finalized to leaf should be available; qed: UnknownBlock("Header was not found in the database: 0x1e70664aec1b2dcbec1ac26210ddb6d7388700e33604edbfa3a55141b22c3837")', /root/.cargo/git/checkouts/cumulus-59522f43471fa161/f603a61/client/consensus/common/src/level_monitor.rs:123 

This is pointing to this line in the cumulus project. My own reading from the code is that a recent header is missing simply because it wasn't yet flushed to the permanent db. But that to me is an expected situation rather than a panic. For two reasons:

  1. It happened at least two times consistently after we restarted a node which is sort of not accidental.
  2. I don't know of a safe way to restart a node but we do need nodes to be resilient to restarts and so on.

My questions are:

  1. Is it expected?
  2. Is it related to warp sync flag?
  3. How can we avoid it?
Source Link
Alex Sed
  • 600
  • 2
  • 10

Node panics on restart after warp synced with both relay and parachain

We are observing the following panic for a parachain node that is warp synced fully on both sides when we try to restart the node:

 Thread 'main' panicked at 'route from finalized to leaf should be available; qed: UnknownBlock("Header was not found in the database: 0x1e70664aec1b2dcbec1ac26210ddb6d7388700e33604edbfa3a55141b22c3837")', /root/.cargo/git/checkouts/cumulus-59522f43471fa161/f603a61/client/consensus/common/src/level_monitor.rs:123 

This is pointing to this line in the cumulus project. My own reading from the code is that a recent header is missing simply because it wasn't yet flushed to the permanent db. But that to me is an expected situation rather than a panic. For two reasons:

  1. It happened at least two times consistently after we restarted a node which is sort of not accidental.
  2. I don't know of a safe way to restart a node but we do need nodes to be resilient to restarts and so on.

My questions are:

  1. Is it expected?
  2. Is it related to warp sync flag?
  3. How we can avoid it?