Reaching out to mentors

[RamyaKappagantu]

Hi,
I'm Ramya, currently pursuing my PG in AI from Georgian college. I'm interested to work on the project "A general OpenVINO performance benchmarking suite for Vision-LLM models" and would like to reach out to the mentors. How can I reach out to them?

[adrianboguszewski]

Hi Ramya, you can use discussions for that. The mentors for this project are @junwenwu and @helena-intel

[RamyaKappagantu]

Hi @junwenwu and @helena-intel ,
I'm Ramya, currently pursuing my Postgraduation in Artificial Intelligence - Design and Implementation from Georgian College. I'm very much looking forward to work on the project "A general openvino performance benchmarking suite for vision LLM models" and have previously worked with Openvino in benchmarking the performance of person detection and person re-identification models. I believe that this project will align very well with my previous experience and current learning curve.
In this process, I would like to know if there are any mandatory pre-requisite tasks related to the project that you can direct me to so that I can start working on it?

[junwenwu]

Hi @RamyaKappagantu I’m very happy to see that you are interested in this project! As described in the project proposal, there is no mandatory pre-requisite for this project, but it will be nice if you can start to get yourself familiar with Intel OpenVINO toolkit and the LLM benchmark suite here: https://github.com/openvinotoolkit/openvino.genai
Also I don’t know how familiar you are with Vision-transformer models, if this is not something you have worked on, please get yourself familiar with some of the model topologies. Thanks again for your interest!

[RamyaKappagantu]

Thank you @junwenwu for your response. I've started setting up my environment for zero-shot image classification and tried my hands on a few hugging face models. I'm also getting myself familiar with Vision LLMs and working on a few code samples. Should I directly work on my project proposal since you mentioned that there's no pre-requisite? I want to get it reviewed before the contributor phase begins.

[junwenwu]

Sounds good. We can definitely review your proposal before the contributor phase begins.

[RamyaKappagantu]

Hi @junwenwu and @helena-intel,
Can we use the existing library that has been developed in the repo: https://github.com/openvinotoolkit/openvino.genai/tree/master/llm_bench/python/who_what_benchmark and add extensions to work for joint-modal tasks or create a library similar to whowhatbench to work for unimodal and joint-modal tasks?

[junwenwu]

Sure, that will be perfect. Now this benchmark repo supports some of the vision transformer models, such as SDXL/SDXL-turbo diffusion models for text-to-image generation tasks. If you can expand it to support more models and additional use cases, that will be great.

[RamyaKappagantu]

Sounds good. I'm almost done with the initial draft of my proposal and planning to get it reviewed before submitting it.

[RamyaKappagantu]

Hi @junwenwu and @helena-intel,
could you please look into the proposal I submitted and provide feedback on what can be improved? I would like to submit the final proposal for this project.

[junwenwu]

@RamyaKappagantu Thanks for your interest in submitting the proposal. Your background looks quite relevant and the proposed timeline makes sense. Here are some comments:

1. In the proposal, please list your goals as basic goal and stretch goal, so that you can make sure you will be able to have a deliverable while provide yourself more challenges.
2. For the accuracy evaluation, I’m not sure if whowhat benchmark works for the visual based modal. You might need to develop your own accuracy measurement.

Best,
Junwen

[RamyaKappagantu]

Thanks @junwenwu for your feedback. I'll make the required additions and send the updated proposal. Does rest of the document make sense and looks good?

[RamyaKappagantu]

Hi @junwenwu and @helena-intel ,
I've submitted the proposal through the website. Looking forward to working on this project and gain a deeper understanding on VisionLLMs.

[junwenwu]

@RamyaKappagantu Thanks for letting us know.

[RamyaKappagantu]

Hi @junwenwu and @helena-intel ,
Hope you're doing good. My project didn't make it to GSoC and I can see in the latest thread that this project wasn't assigned to anyone. The reason I chose this project at first place was because of my experience in benchmarking ML/DL pre-trained models and knowledge that I have on LLMs (specifically Vision LLMs) through my curriculum. If possible, can I still work with you on this project? This will help me in gaining a broader knowledge in Vision LLMs and also might help me for next year:)
Please let me know if this works.

[adrianboguszewski]

Hi @RamyaKappagantu. We wanted to have you on board, but unfortunately, Google assigned us 11 of 17 projects and that change impacted you :(
But feel free to work with our mentors anyway :)

[junwenwu]

Thanks @RamyaKappagantu. Sure I will be glad to continue working with you on this project.

[RamyaKappagantu]

Thank you @junwenwu , could you guide me on how we can get started and how can we communicate for the project?