Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

syz-manager, syz-fuzzer: switch communication to flatrpc #4818

Merged
merged 1 commit into from
May 21, 2024

Conversation

dvyukov
Copy link
Collaborator

@dvyukov dvyukov commented May 17, 2024

syz-manager, syz-fuzzer: switch communication to flatrpc

Switch to flatrpc connection between manager and fuzzer.
With flatrpc we have a goroutine per connection instead of async RPC,
which makes things a bit simpler. Now don't reordered messages
(in particular start executing and finish executing for programs),
race on the program during printing is no longer possible
since we finish handlign start executing request before we even
receive finish executing.
We also don't need to lookup Runner for every RPC since it's
now local to the handling goroutine.
We also don't need to protect requests map since only single
goroutine accesses it.
We also send new programs to the fuzzer as soon as we receive
start executing message, which provides better buffering.
We also don't batch new requests and finish executing requests
in a single RPC, which makes things a bit simpler.

Update #1541

@dvyukov dvyukov changed the title syz-manager syz-manager, syz-fuzzer: switch communication to flatrpc May 17, 2024
@dvyukov dvyukov requested a review from a-nogikh May 17, 2024 14:10
@dvyukov dvyukov force-pushed the dvyukov-flatrpc-comm branch 4 times, most recently from 21ec294 to c3ee288 Compare May 17, 2024 14:35
@dvyukov
Copy link
Collaborator Author

dvyukov commented May 17, 2024

Note: this breaks syz-runtest. But since we don't use it, it looks pointless to fix. We better restore it as part of syz-manager. Now with dynamic request source, it should fit into syz-manager very nicely -- after checking we just switch to runtest source instead of fuzzer source.

@a-nogikh
Copy link
Collaborator

After checking we just switch to runtest source instead of fuzzer source.

We'd also need to deal with the web UI, corpus file operations and multiple ways in which the vm loop is interleaved with the fuzzer :( That's all doable, but unfortunately it's not "just switch" if we want to do it cleanly.

@dvyukov
Copy link
Collaborator Author

dvyukov commented May 17, 2024

ranch o

After checking we just switch to runtest source instead of fuzzer source.

We'd also need to deal with the web UI, corpus file operations and multiple ways in which the vm loop is interleaved with the fuzzer :( That's all doable, but unfortunately it's not "just switch" if we want to do it cleanly.

I wouldn't bother about web UI. What do we need to do for the VM loop?

@a-nogikh
Copy link
Collaborator

Decouple it from syz-hub and bug reproduction (which we don't need for runtest).

@dvyukov dvyukov mentioned this pull request May 17, 2024
a-nogikh
a-nogikh previously approved these changes May 21, 2024
syz-manager/rpc.go Outdated Show resolved Hide resolved
syz-manager/rpc.go Show resolved Hide resolved
syz-manager/rpc.go Show resolved Hide resolved
Switch to flatrpc connection between manager and fuzzer.
With flatrpc we have a goroutine per connection instead of async RPC,
which makes things a bit simpler. Now don't reordered messages
(in particular start executing and finish executing for programs),
race on the program during printing is no longer possible
since we finish handlign start executing request before we even
receive finish executing.
We also don't need to lookup Runner for every RPC since it's
now local to the handling goroutine.
We also don't need to protect requests map since only single
goroutine accesses it.
We also send new programs to the fuzzer as soon as we receive
start executing message, which provides better buffering.
We also don't batch new requests and finish executing requests
in a single RPC, which makes things a bit simpler.
In my local run this reduces syz-manager heap size from 1.3GB to 1.1GB.

Update google#1541
Copy link

codecov bot commented May 21, 2024

Codecov Report

Attention: Patch coverage is 0.20619% with 484 lines in your changes are missing coverage. Please review.

Project coverage is 61.1%. Comparing base (c0f1611) to head (d98a661).
Report is 1 commits behind head on master.

Additional details and impacted files
Files Coverage Δ
syz-manager/stats.go 0.0% <ø> (ø)
syz-manager/manager.go 0.0% <0.0%> (ø)
syz-fuzzer/fuzzer.go 4.7% <1.4%> (+0.7%) ⬆️
syz-fuzzer/proc.go 0.0% <0.0%> (ø)
syz-manager/rpc.go 0.0% <0.0%> (ø)

... and 1 file with indirect coverage changes

@dvyukov dvyukov enabled auto-merge May 21, 2024 08:45
@dvyukov dvyukov added this pull request to the merge queue May 21, 2024
Merged via the queue into google:master with commit 98aa8d4 May 21, 2024
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants