Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use more cores with rayon while syncing with the CDN #3209

Open
wants to merge 1 commit into
base: mainnet-staging
Choose a base branch
from

Conversation

ljedrz
Copy link
Collaborator

@ljedrz ljedrz commented Apr 9, 2024

Since we perform the CDN sync before starting the node, we can use a rayon threadpool with all the cores dedicated to syncing without sacrificing any other functionality.

I tested it back in testnet3 and saw higher CPU utilization.

Signed-off-by: ljedrz <ljedrz@gmail.com>
@howardwu
Copy link
Contributor

howardwu commented Apr 9, 2024

How much faster is the rate of syncing? Can you share numbers for pre and post syncing for this PR?

@ljedrz
Copy link
Collaborator Author

ljedrz commented Apr 10, 2024

Back when I was testing on testnet3 I noted that it doubled my CPU utilization; the exact increase in speed would depend on the number and power of cores, but this results in both faster deserialization and ledger insertion.

@howardwu
Copy link
Contributor

Do you have empirical data that you can share to confirm this finding?

@ljedrz
Copy link
Collaborator Author

ljedrz commented Apr 22, 2024

Unfortunately no, not unless @zosorock has performed some measurements back when this was requested.

In any case, it's a pretty trivial change, so it can wait until the next time we're using a CDN if we'd like to collect some solid figures before merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants