Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dnn: better net exporter that works with netron #25582

Merged
merged 5 commits into from
May 17, 2024

Conversation

fengyuentau
Copy link
Member

Current net exporter dump and dumpToFile exports the network structure (and its params) to a .dot file which works with graphviz. This is hard to use and not friendly to new user. What's worse, the produced picture is not looking pretty.

This PR introduces new exporter dumpToPbtxt and uses this new exporter by default with environment variable OPENCV_DNN_NETWORK_DUMP. It mimics the string output of a onnx model but modified with dnn-specific changes, see below for an example.

image

Usage

Call cv::dnn::Net::dumpToPbtxt:

TEST(DumpNet, dumpToPbtxt) {
    std::string path = "/path/to/model.onnx";
    auto net = readNet(path);

    Mat input(std::vector<int>{1, 3, 640, 480}, CV_32F);
    net.setInput(input);

    net.dumpToPbtxt("yunet.pbtxt");
}

Set export OPENCV_DNN_NETWORK_DUMP=1

TEST(DumpNet, env) {
    std::string path = "/path/to/model.onnx";
    auto net = readNet(path);

    Mat input(std::vector<int>{1, 3, 640, 480}, CV_32F);
    net.setInput(input);

    net.forward();
}

Note:

  • pbtxt is registered as one of the ONNX model suffix in netron. So you can see module: ai.onnx and such in the model.
  • We can get the string output of an ONNX model with the following script
import onnx
net = onnx.load("/path/to/model.onnx")
net_str = str(net)
file = open("/path/to/model.pbtxt", "w")
file.write(net_str)
file.close()

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
  • The PR is proposed to the proper branch
  • There is a reference to the original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake

@fengyuentau
Copy link
Member Author

Here is one example I get from this patch models.zip. They all can be opened with Netron.

Note that I did not test other models.

modules/dnn/src/net_impl.cpp Outdated Show resolved Hide resolved
modules/dnn/src/net_impl.cpp Show resolved Hide resolved
@asmorkalov
Copy link
Contributor

I propose to add dump call to apps/model-diagnostics to get graph every time, when the network was loaded or do it by option.

@asmorkalov asmorkalov added this to the 4.10.0 milestone May 14, 2024
modules/dnn/src/net_impl.cpp Outdated Show resolved Hide resolved
@fengyuentau
Copy link
Member Author

fengyuentau commented May 15, 2024

I propose to add dump call to apps/model-diagnostics to get graph every time, when the network was loaded or do it by option.

Unfortunately it cannot be added to apps/model-diagnostics. I took a look at the app, and found it only reads the net but does not call net.setInput(), which needs to be called first in order to get the graph.

However, we can add option in the app asking for the input dimension and then we can call the new exporter to get the graph. In this case, I need CommandLineParser supporting reading dictionary-like paramter, e.g. input_name:dim0,dim1,dim2,dim3, and reading multiple of them for multiple inputs of a net, which does not look like being supported yet. We need to figure out a solution to solve this before adding graph-exporting call to the app.

@asmorkalov
Copy link
Contributor

@fengyuentau Let's extend the tool or add the call to some generic sample. It'll be great to have the option available without codding. It helps to triage incoming issues.

@fengyuentau
Copy link
Member Author

@fengyuentau Let's extend the tool or add the call to some generic sample. It'll be great to have the option available without codding. It helps to triage incoming issues.

I mean we can extend the tool only after we figure out how to set the input for the net. Current CommandLineParser does not satisfy our needs to have inputs unless we code everytime with the input of the model.

add the call to some generic sample

Do you mean existing ones in samples/dnn/ ?

@asmorkalov
Copy link
Contributor

I propose to add several cmdline options line --input_shape1, --input_shape2, ... --input_shape5 and parse formula like 128x128x3 with sscanf. It should be enough to define inputs without CommadLineParser modification.

@fengyuentau
Copy link
Member Author

I propose to add several cmdline options line --input_shape1, --input_shape2, ... --input_shape5 and parse formula like 128x128x3 with sscanf. It should be enough to define inputs without CommadLineParser modification.

Ok. Although this is not a perfact solution, it can satisfy most of our needs.

@fengyuentau
Copy link
Member Author

I propose to add several cmdline options line --input_shape1, --input_shape2, ... --input_shape5 and parse formula like 128x128x3 with sscanf. It should be enough to define inputs without CommadLineParser modification.

@asmorkalov It is implemented. It works as follows

./build/bin/opencv_model_diagnostics -model=/workspace/opencv_zoo/models/face_detection_yunet/face_detection_yunet_2023mar.onnx -input0_name=input -input0_shape=1,3,640,640

Copy link
Contributor

@asmorkalov asmorkalov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@asmorkalov asmorkalov merged commit bc0618b into opencv:4.x May 17, 2024
28 checks passed
@asmorkalov asmorkalov self-assigned this May 17, 2024
klatism pushed a commit to klatism/opencv that referenced this pull request May 17, 2024
Current net exporter `dump` and `dumpToFile` exports the network structure (and its params) to a .dot file which works with `graphviz`. This is hard to use and not friendly to new user. What's worse, the produced picture is not looking pretty.
dnn: better net exporter that works with netron opencv#25582

This PR introduces new exporter `dumpToPbtxt` and uses this new exporter by default with environment variable `OPENCV_DNN_NETWORK_DUMP`. It mimics the string output of a onnx model but modified with dnn-specific changes, see below for an example.

![image](https://github.com/opencv/opencv/assets/17219438/0644bed1-da71-4019-8466-88390698e4df)

## Usage

Call `cv::dnn::Net::dumpToPbtxt`:

```cpp
TEST(DumpNet, dumpToPbtxt) {
    std::string path = "/path/to/model.onnx";
    auto net = readNet(path);

    Mat input(std::vector<int>{1, 3, 640, 480}, CV_32F);
    net.setInput(input);

    net.dumpToPbtxt("yunet.pbtxt");
}
```

Set `export OPENCV_DNN_NETWORK_DUMP=1`

```cpp
TEST(DumpNet, env) {
    std::string path = "/path/to/model.onnx";
    auto net = readNet(path);

    Mat input(std::vector<int>{1, 3, 640, 480}, CV_32F);
    net.setInput(input);

    net.forward();
}
```

---

Note:
- `pbtxt` is registered as one of the ONNX model suffix in netron. So you can see `module: ai.onnx` and such in the model.
- We can get the string output of an ONNX model with the following script

```python
import onnx
net = onnx.load("/path/to/model.onnx")
net_str = str(net)
file = open("/path/to/model.pbtxt", "w")
file.write(net_str)
file.close()
```

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
@fengyuentau fengyuentau deleted the dnn/dump_pbtxt branch May 22, 2024 06:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants