[GSoC 2024] New Idea: Porting and Optimizing OpenVINO for RISC-V devices with Vector extension

[BHbean]

Hello everyone,

I am a student who is pursuing my postgraduate in CS at Institute of Software Chinese Academy of Science. I have great passion in taking part in the OpenVINO community, which perfectly matches my wishes - making AI deployment easier and AI inference faster. By going through the idealist, I found that Project 8 (Improve Generative AI workload performance on ARM devices with OpenVINO) greatly interested and inspired me. I am wondering if I could support a new backend for OpenVINO, so I proposed my new idea for GSoC 2024.

Idea Description

RISC-V is a new rising platform as many edge devices (like Lichee Pi 4A, Kendryte K230 and so on) are using RISC-V chips now, as well as some laptops (for example ROMA book from Alibaba and Ruyi BOOK from ISCAS) and servers (for instance SG2042 from SOPHGO). With the powerful Vector extension (known as RVV) for HPC, I believe that RVV chips are of vital importance for future computing devices. There are also open-source projects that can run basic models on RVV backend, for example ncnn and OpenCV so it might be a great chance to make preparations now. This project can be devided into mainly two parts:

• Porting: Porting OpenVINO to RISC-V devices (maybe in plugins like ARM), on which some basic models (like Bert or something) can run correctly through OpenVINO framework
• Optimizing: Improving the inference performance of specific models through vectorization or compilation techniques

As a draft proposal at an initial stage, some specific models are mainly focused. However, more supported models or performance improvement can be achieved as the progress of the development of both RISC-V and OpenVINO.

Good Fit

• Strong interest in accelerating model inference on various devices
• RISC-V dev experience: Maintaining RVV support for libjpeg-turbo
• Test Envs: Having RVV development boards
• Experience on compiler construction based on MLIR
• Contributing to OpenVINO ( [PY API] Fix the preoblem that Node.get_attributes() cannot return all attributes #23530 )

Discussion

• If this idea cannot be accepted in the end, I would also be happy to take participate in Project 8 to get involved in OpenVINO community. (I have a Macbook Air with M1 chip)
• I am currently working on Pyhon API issue of OpenVINO, which seems less relavant to this idea and Project 8. Could you please offer other GFIs that is relavant to the two ideas to help me start?

I hope to take OpenVINO as my first gate to AI/ML system sincerely, and maybe extend further research for faster AI inference on various devices. I am looking forward to your suggestions and feedback!

Best regards,
Zhiyuan Tan

[zhuo-yoyowz]

Hi Zhiyuan, nice to receive your message showing interest in our Project 8. Alexandr Voron, Dmitry Gorokhov are our mentors for this project. You can reach out to them for more detailed discussions. @alvoron , @dmitry-gorokhov

[dmitry-gorokhov]

Hi Zhiyuan, I think this is amazing idea. We have big interest in making OpenVINO truly cross-platform inference solution and RISC-V is one of the top priorities. Let us prepare new GSOC project targeting RISC-V so you will be able to apply on it.

[BHbean]

Thanks a lot for your reply! I am in great excitement to receive this news, and am looking forward to making a detailed proposal for the new GSoC project!

To make better preparations for the application, there may be some questions to discuss with you:

• Is there any related pre-tasks that can be assigned for me? Although I have been working on one, it might not be so helpful to get me familiar with the particular parts of OpenVINO that is essential for beginning the new RISC-V related project.
• What is the possible ways to implement this feature? Maybe similar technical raodmap can be used as ARM's, where device-specific opsets and transformations might be needed for porting. As for kernel optimization, RVV extension could give a helping hand by vectorizing dense calculations.

[dmitry-gorokhov]

Hi @BHbean, sorry for late response. I prepared some project idea targeting RISC-V platform support in OpenVINO:

RISC-V is a rapidly evolving instruction set architecture based on open-source standards. RISC-V devices are adopting throughout the industry from edge domain to servers. Running of DL/AI workloads on RISC-V platforms has great protentional due to their power efficiency. OpenVINO is aiming to become truly cross-platform inference solution and RISC-V is one of the top priorities.
The goal of this project is to improve OpenVINO runtime performance on RISC-V devices for wide set of models/inference workloads. In bounds of these project, we should determine actual set of DL models/workloads which will be used as representative benchmark to showcase final value of the contribution. Performance enhancements should be done based on the following optimizations strategies:

1. Adopt new (or improve adoption of existing) third-party libraries with RISC-V optimized primitives (e.g. https://github.com/T-head-Semi/csi-nn2)
2. Port existing x86/ARM optimized kernels implemented inside OpenVINO to RISC-V isa (using RVV vector intrinsics: https://github.com/riscv-non-isa/rvv-intrinsic-doc and JIT code generation: https://github.com/herumi/xbyak_riscv)
3. Implement device specific transformation passes to change execution graph in a way it better fits RISC-V backend optimizations.

Optimizations might include improvements in terms of latency/throughput metrics, faster model compilation time and lower memory consumption.

Expected outcomes:

• More operations leverages optimized RISC-V kernels from third-party libraries
• Some of the most important (based on determined benchmark) X86/ARM optimizations are ported to RISC-V
• Existing transformation pipeline is adopted for RICS-V backend requirements
• Demonstrated better OpenVINO runtime performance on RISC-V CPU devices for determined benchmarking scope

Does this project idea corresponds to what you have expected/proposed? If yes please submit your proposal based on that idea as soon as possible given the fact Apr 2 is the deadline already. We would be really happy to see you as a contributor in bounds OpenVINO GCOS program!

Regarding pre-tasks: we definetelly can prepare some of them to help you ramp-up. Let us prepare some list in bounds of our Good First Issue project https://github.com/orgs/openvinotoolkit/projects/3.

[BHbean]

@dmitry-gorokhov Thanks for your reply! it is just in time to tell me all of these things, as I still have some time to modify my proposal.

Actually I have almost finished my poposal and I will send it to you via email for your review as long as I finished. This draft proposal is ddesigned on my own without any knowledge on this newly proposed idea, so it might not perfectly match this idea now. Therefore, modifications must be taken as soon as possible for me, and your advice is of vital importance!

As for the warm-up task, I have finished the cross-compilation for RISC-V (see this issue) and ready to open PR for oneDNN repo. So there may not be enough time for me to finish this GFI now, but I will take a look after the deadline.

[zhuo-yoyowz]

@BHbean Yes, just a kind reminder. Tomorrow is the application deadline. Please submit your proposal before the deadline. Thanks!

[BHbean]

Thanks for your kind reminder! I am working really hard on it right now. I think I can make it before the deadline! :)

[RitikaxShakya]

@zhuo-yoyowz Hello! sorry for interrupting here in this discussion, I am interested in project 20 and I've prepared a draft proposal outlining my approach and have sent it to your mail. Your expertise and feedback on refining proposal would be immensely valuable.