[Bug]: First Inference on GPU device is slow when using ov::set_batch for openvino 2024

[hihapaddy]

OpenVINO Version

2024.1.0

Operating System

Ubuntu 20.04 (LTS)

Device used for inference

GPU

Framework

None

Model used

face-reidentification-retail-0095

Issue description

We test inference on openvino 2023.0.1 and it works good for face recognition.
However, we switch to use openvino 2024.1.0 and it's first inference delayed about several minutes.
We found compile_model not compile full resource for inference, so first called infer() will take several minutes to prepare opencl related resource.

Step-by-step reproduction

No response

Relevant log output

No response

Issue submission checklist

• I'm reporting an issue. It's not a question.
• I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
• There is reproducer code and related data files such as images, videos, models, etc.

[hihapaddy]

We found this issue may be caused by ov::set_batch.

[hihapaddy]

@ilya-lavrenov
We provide a jupyter notebook to test set_batch feature.
By the way, we found another issue that set_batch will also influence outputs of some model.
For example, the result of face-reidentification-retail-0095 on GPU and CPU only the same in first inference. Thus, the face recognition on GPU will not work when switching to openvino 2024.1.

Here is the sample code.
https://drive.google.com/file/d/16wRtWgo6V0GV32ScTUoID--txYnoHg2_/view?usp=drive_link