How long does it take to restore the latest dump on an external node?

[breezytm]

Hi,

I am trying to deploy an external node. The restore seems to be taking quite some time. It has been 9 days since I started but only 5305150/ 30496435 has been processed. If my math is correct, it is importing 24,500 blocks/hr. At this rate it should take well over a month and half. If I am wrong, what am I missing?

[EmilLuta]

cc: @slowli, @tomg10

[slowli]

Hey, @breezytm! You're not wrong at all; that's within an expected rate of block processing on consumer-grade hardware. For this reason, we're currently developing snapshot recovery feature that will allow to recover the node state much faster (preliminary results are ~4–5 hours for the mainnet on consumer-grade hardware).

[breezytm]

all
Rather than taking the approach of backing up and restoring the dump, why not back up the data folder in its entirety as a .tar file and make it available for download. This method would be much quicker to download, extract, and start the node. This is literally the approach every other blockchain projects are doing in terms of snapshot.

[slowli]

First, there's no single data folder; the data stored partially in Postgres, partially in RocksDB. Second, for either of these databases, data consistency and portability are concerns that prevent "just" taking data from the disk at any point in time. Third, the amount of data in Postgres and RocksDB (several TB) makes this approach hardly desirable for non-archival nodes even if other concerns were sorted out.