proof_generation_data endpoint unavailable

[nikurt]

Hello 👋

I'd like to benchmark the proof generation for realistic payloads, and I've been following the instructions in https://github.com/matter-labs/zksync-era/tree/main/prover/prover_fri

I don't have a GPU available, and thus I'm venturing on the CPU adventure :)

What I did:

• Installed all dependencies
• zk init
• Switched to using the real prover as described in https://github.com/matter-labs/zksync-era/blob/main/docs/guides/development.md#using-dummy-prover
• zk init
• ./setup.sh
• zk server --components=api,eth,tree,state_keeper,housekeeper,commitment_generator,proof_data_handler
• zk f cargo run --release --bin zksync_prover_fri_gateway

Here I see the gateway reporting that the proof_generation_data endpoint is not available:

% zk f cargo run --release --bin zksync_prover_fri_gateway
    Finished release [optimized] target(s) in 0.26s
     Running `target/release/zksync_prover_fri_gateway`
2024-04-18T14:59:35.249435Z  INFO zksync_prover_fri_gateway: Starting Fri Prover Gateway
2024-04-18T14:59:35.249559Z  INFO zksync_prover_fri_gateway::api_data_fetcher: Starting periodic job: ProofGenDataFetcher with frequency: 120s
2024-04-18T14:59:35.249585Z  INFO zksync_prover_fri_gateway::api_data_fetcher: Sending request to http://127.0.0.1:3320/proof_generation_data
2024-04-18T14:59:35.249587Z  INFO zksync_prover_fri_gateway::api_data_fetcher: Starting periodic job: ProofSubmitter with frequency: 120s
2024-04-18T14:59:35.249600Z  INFO vise_exporter::exporter: Created metrics exporter with 16 metrics in 6 groups from crates zksync_db_connection 0.1.0, zksync_health_check 0.1.0, vise_exporter 0.1.0, zksync_prover_fri_gateway 0.1.0, zksync_object_store 0.1.0, ...
2024-04-18T14:59:35.250323Z  INFO vise_exporter::exporter: Starting Prometheus exporter web server on 0.0.0.0:3314
2024-04-18T14:59:35.251448Z ERROR zksync_prover_fri_gateway::api_data_fetcher: HTTP request failed due to error: HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:3320/proof_generation_data)

The server has corresponding errors in the logs:

% zk server --components=api,eth,tree,state_keeper,housekeeper,commitment_generator,proof_data_handler2024-04-18T14:59:19.181831Z  WARN zksync_queued_job_processor: Stop signal received, shut
ting down basic_witness_input_producer component while waiting for a new job
2024-04-18T14:59:19.182489Z  INFO zksync_core::api_server::healthcheck: Stop signal received, healthcheck server is shutting down
2024-04-18T14:59:19.182633Z  INFO zksync_core::api_server::healthcheck: Healthcheck server shut down                                                                                                                                        2024-04-18T14:59:19.182703Z  INFO zksync_server: Stopped                                                                                                                                                                                    
                                                                                                                      
(node:285371) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)                                                                                                                                                                  
    Finished release [optimized] target(s) in 0.29s                                                                                                                                                                                         
     Running `target/release/zksync_server --components=api,eth,tree,state_keeper,housekeeper,commitment_generator,proof_data_handler`                                                                                                      2024-04-18T14:59:20.335315Z  INFO zksync_server: No sentry URL was provided                                                                                                                                                                 2024-04-18T14:59:20.341076Z  INFO zksync_core: Starting the components: [HttpApi, WsApi, ContractVerificationApi, EthWatcher, EthTxAggregator, EthTxManager, Tree, StateKeeper, Housekeeper, CommitmentGenerator, ProofDataHandler]         2024-04-18T14:59:20.347050Z DEBUG zksync_health_check: Created app health with time limits: slow=500ms, hard=3s                                                                                                                             
...
2024-04-18T14:59:21.815864Z DEBUG zksync_health_check: Changed health of `commitment_generator` from {"status":"not_ready"} to {"status":"ready"}                                                                                           
2024-04-18T14:59:21.815868Z  INFO zksync_server: Running 35 core task handlers                                                                                                                                                              
2024-04-18T14:59:21.815873Z DEBUG zksync_core::api_server::healthcheck: Starting healthcheck server with checks AppHealthCheck { components: Mutex { data: [CheckHealth { name: "prometheus_exporter" }, CheckHealth { name: "http_api" }, CheckHealth { name: "ws_api" }, CheckHealth { name: "tree" }, CheckHealth { name: "commitment_generator" }, CheckHealth { name: "connection_pool" }], poisoned: false, .. }, slow_time_limit: 500ms, hard_time_limit: 3s } on 0.0.0.0:3071   
2024-04-18T14:59:35.249990Z  INFO zksync_core::proof_data_handler::request_processor: Received request for proof generation data: Json(ProofGenerationDataRequest)                                                                          
2024-04-18T14:59:35.251359Z ERROR zksync_core::proof_data_handler::request_processor: GCS error: KeyNotFound(Os { code: 2, kind: NotFound, message: "No such file or directory" })

Running witness processes fails:

2024-04-18T13:47:19.809313Z  INFO zksync_witness_generator: No sentry URL was provided                                                                                                                                                      
thread 'main' panicked at witness_generator/src/main.rs:135:9:                 
Could not find a protocol version for my commitments. Is gateway running?  Maybe you started this job before gateway updated the database? Commitments: L1VerifierConfig { params: VerifierParams { recursion_node_level_vk_hash: 0x5a3ef282
b21e12fe1f4438e5bb158fc5060b160559c5158c6389d62d9fe3d080, recursion_leaf_level_vk_hash: 0x400a4b532c6f072c00d1806ef299300d4c104f4ac55bd8698ade78894fcadc0a, recursion_circuits_set_vks_hash: 0x000000000000000000000000000000000000000000000
0000000000000000000 }, recursion_scheduler_level_vk_hash: 0x063c6fb5c70404c2867f413a8e35563ad3d040b1ad8c11786231bfdba7b472c7 }

My interpretation of this error is that L1BatchNumber(21) or (a similarly low number) is no longer available on some GCS bucket.

Is there an alternative way to get some blocks into the local database?
Is the default configuration not working correctly for this usecase?

[EmilLuta]

Hey @nikurt. Most likely, you've ran the setup multiple time and removed items from artifacts (which are your local "gcs" implementation, if you want).

I'd suggest you run from scratch zk && zk clean -all && zk init and make sure you're on the latest branch.

Let me know if you run into other issues!

[joshuacheong]

Hey @nikurt. Most likely, you've ran the setup multiple time and removed items from artifacts (which are your local "gcs" implementation, if you want).

I'd suggest you run from scratch zk && zk clean -all && zk init and make sure you're on the latest branch.

Let me know if you run into other issues!

Correcting a typo here
zk && zk clean --all && zk init