Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(Prover CLI): delete cmd #1802

Merged
merged 45 commits into from
May 14, 2024
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
a7d0591
Support delete cmd
ilitteri Apr 25, 2024
e1ca28a
Add delete cmd implementation
ilitteri Apr 25, 2024
d026ca2
Implement queries for deleting batch data
ilitteri Apr 25, 2024
914e13d
Add queries for deleting the data
ilitteri Apr 29, 2024
68fa9d1
Implement `delete_prover_db`
ilitteri Apr 29, 2024
8d48303
Add prompt for user to confirm the action
ilitteri Apr 29, 2024
d849eb9
Fix queries
ilitteri Apr 29, 2024
2be245c
Fix command
ilitteri Apr 29, 2024
aae15a1
Refactor queries
ilitteri Apr 30, 2024
e891a0f
Implement functionallity for --failed flag
ilitteri Apr 30, 2024
487fd95
Merge branch 'main' of github.com:matter-labs/zksync-era into prover_…
ilitteri Apr 30, 2024
e557528
Merge branch 'main' into prover_cli_delete_cmd
ilitteri May 2, 2024
9f28e4c
zk lint rust
ilitteri May 2, 2024
46de113
Merge branch 'prover_cli_delete_cmd' of github.com:matter-labs/zksync…
ilitteri May 2, 2024
078508b
Merge branch 'main' into prover_cli_delete_cmd
ilitteri May 2, 2024
8dafaff
Merge branch 'main' into prover_cli_delete_cmd
ilitteri May 2, 2024
984e511
Merge branch 'main' of github.com:matter-labs/zksync-era into prover_…
ilitteri May 3, 2024
5b593be
Remove --failed functionality
ilitteri May 3, 2024
b463fcd
zk lint rust
ilitteri May 3, 2024
8c9afd7
Merge branch 'main' of github.com:matter-labs/zksync-era into prover_…
ilitteri May 3, 2024
0dc428d
Delete recursion tip data for batch
ilitteri May 3, 2024
0fdddae
Update deps
ilitteri May 3, 2024
2e39b45
Fix cmd
ilitteri May 6, 2024
77c8f2a
Move dep to workspace
ilitteri May 6, 2024
45df260
Merge branch 'main' into prover_cli_delete_cmd
ilitteri May 6, 2024
6c11115
Implement restart cmd
ilitteri May 7, 2024
39a113b
Handle restart cmd
ilitteri May 7, 2024
6fce480
Implement restart for prover jobs
ilitteri May 7, 2024
0497586
Merge branch 'main' into prover_cli_delete_cmd
ColoCarletti May 7, 2024
2a34b1a
Merge pull request #1868 from matter-labs/prover_cli_restart_batch
ilitteri May 7, 2024
dcb95c5
Merge branch 'main' of github.com:matter-labs/zksync-era into prover_…
ilitteri May 7, 2024
8eb838f
Merge branch 'main' into prover_cli_delete_cmd
ilitteri May 8, 2024
d42de00
Use singletoon connection pool
ilitteri May 8, 2024
7ebdfbe
Merge branch 'main' into prover_cli_delete_cmd
ilitteri May 8, 2024
de4aec6
Fix merge
ilitteri May 8, 2024
c19e780
Merge branch 'main' of github.com:matter-labs/zksync-era into prover_…
ilitteri May 8, 2024
c0ae857
Merge branch 'main' into prover_cli_delete_cmd
ColoCarletti May 9, 2024
f96f520
Merge branch 'main' into prover_cli_delete_cmd
ilitteri May 10, 2024
c15bc42
Merge branch 'main' into prover_cli_delete_cmd
ColoCarletti May 10, 2024
4ce5a2f
Update Cargo.lock
ilitteri May 13, 2024
80ddf0c
Merge branch 'prover_cli_delete_cmd' of github.com:matter-labs/zksync…
ilitteri May 13, 2024
7c1c0d4
Merge branch 'main' into prover_cli_delete_cmd
ilitteri May 13, 2024
018cfbf
Merge branch 'main' into prover_cli_delete_cmd
ilitteri May 13, 2024
60a07b0
Merge branch 'main' into prover_cli_delete_cmd
ilitteri May 13, 2024
d6fba63
Merge branch 'main' into prover_cli_delete_cmd
Artemka374 May 14, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
39 changes: 39 additions & 0 deletions prover/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion prover/prover_cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ categories.workspace = true
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
env_logger = "0.10"
log = "0.4"
dialoguer = "0.11"
ilitteri marked this conversation as resolved.
Show resolved Hide resolved

clap = { workspace = true, features = ["derive"] }
tracing.workspace = true
Expand All @@ -29,4 +30,3 @@ zksync_types.workspace = true
zksync_prover_fri_types.workspace = true
zksync_prover_interface.workspace = true
prover_dal.workspace = true

4 changes: 3 additions & 1 deletion prover/prover_cli/src/cli.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
use clap::{command, Parser, Subcommand};

use crate::commands::get_file_info;
use crate::commands::{delete, get_file_info};

pub const VERSION_STRING: &str = env!("CARGO_PKG_VERSION");

Expand All @@ -14,12 +14,14 @@ struct ProverCLI {
#[derive(Subcommand)]
enum ProverCommand {
FileInfo(get_file_info::Args),
Delete(delete::Args),
}

pub async fn start() -> anyhow::Result<()> {
let ProverCLI { command } = ProverCLI::parse();
match command {
ProverCommand::FileInfo(args) => get_file_info::run(args).await?,
ProverCommand::Delete(args) => delete::run(args).await?,
};

Ok(())
Expand Down
106 changes: 106 additions & 0 deletions prover/prover_cli/src/commands/delete.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
use anyhow::{bail, Context};
use clap::Args as ClapArgs;
use dialoguer::{theme::ColorfulTheme, Input};
use prover_dal::{
fri_proof_compressor_dal::ProofCompressionJobStatus,
fri_witness_generator_dal::FriWitnessJobStatus, Connection, ConnectionPool, Prover, ProverDal,
};
use zksync_config::PostgresConfig;
use zksync_env_config::FromEnv;
use zksync_types::{prover_dal::ProverJobStatus, L1BatchNumber};

#[derive(ClapArgs)]
pub(crate) struct Args {
/// Batch number to delete
#[clap(short, long, conflicts_with = "all")]
batch: Option<L1BatchNumber>,
/// Delete data from all batches
#[clap(short, long, conflicts_with = "batch", default_value_t = false)]
all: bool,
/// Delete only failed data, used only with --batch
#[clap(short, long, default_value_t = false, conflicts_with = "all")]
EmilLuta marked this conversation as resolved.
Show resolved Hide resolved
failed: bool,
}

pub(crate) async fn run(args: Args) -> anyhow::Result<()> {
if !args.all && args.batch.is_none() {
bail!("Either --all or --batch should be provided");
}

let confirmation = Input::<String>::with_theme(&ColorfulTheme::default())
.with_prompt("Are you sure you want to delete the data?")
.default("yes".to_owned())
ilitteri marked this conversation as resolved.
Show resolved Hide resolved
.interact_text()
.unwrap();

if confirmation != "yes" {
ilitteri marked this conversation as resolved.
Show resolved Hide resolved
println!("Aborted");
return Ok(());
}

let config = PostgresConfig::from_env()?;
let prover_connection_pool =
ConnectionPool::<Prover>::builder(config.prover_url()?, config.max_connections()?)
.build()
.await
.context("failed to build a prover_connection_pool")?;
let conn = prover_connection_pool.connection().await.unwrap();

if args.all {
delete_prover_db(conn).await?;
} else {
delete_batch_data(conn, args.batch.unwrap(), args.failed).await?;
}

Ok(())
}

async fn delete_prover_db(mut conn: Connection<'_, Prover>) -> anyhow::Result<()> {
conn.fri_gpu_prover_queue_dal()
.delete()
.await
.context("failed to delete gpu prover queue")?;
conn.fri_prover_jobs_dal()
.delete()
.await
.context("failed to delete prover jobs")?;
conn.fri_protocol_versions_dal()
.delete()
.await
.context("failed to delete protocol versions")?;
conn.fri_proof_compressor_dal()
.delete()
.await
.context("failed to delete proof compressor")?;
conn.fri_witness_generator_dal()
.delete()
.await
.context("failed to delete witness generator")?;
Ok(())
}

async fn delete_batch_data(
mut conn: Connection<'_, Prover>,
block_number: L1BatchNumber,
failed: bool,
) -> anyhow::Result<()> {
conn.fri_proof_compressor_dal()
.delete_batch_data_with_status(
block_number,
failed.then_some(ProofCompressionJobStatus::Failed),
)
.await
.context("failed to delete proof compressor data")?;
conn.fri_prover_jobs_dal()
.delete_batch_data_with_status(
block_number,
failed.then_some(ProverJobStatus::Failed(Default::default())),
)
.await
.context("failed to delete prover jobs data")?;
conn.fri_witness_generator_dal()
.delete_batch_data_with_status(block_number, failed.then_some(FriWitnessJobStatus::Failed))
.await
.context("failed to delete witness generator data")?;
Ok(())
}
1 change: 1 addition & 0 deletions prover/prover_cli/src/commands/mod.rs
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
pub(crate) mod delete;
pub(crate) mod get_file_info;

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.