[Detector Support]: Creation of Openvino models with integrated intel GPU fails on latest Linux kernel 6.8.2

[marco-calautti]

Describe the problem you are having

After updating my system (Arch), the kernel got updated to 6.8.2, and frigate is not able anymore to compile the openvino detector model for my integrated intel gpu, failing with the message logs shown below.
After reverting the kernel to 6.7.8, everything works normally again. Note that ffmpeg hardware acceleration is not affected, but only the openvino detectors are.

My specs: CPU intel i3 6100T with integrated GPU HD Graphics 530.

This issue in intel compute-runtime might be related: intel/compute-runtime#710

Version

0.13.2-6476f8a

Frigate config file

mqtt:
  enabled: True
  host: mosquitto

detectors:
  ov:
    type: openvino
    device: GPU
    model:
      path: /openvino-model/ssdlite_mobilenet_v2.xml

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt
  
go2rtc:
  streams:
    cam_salotto: # this needs to be named exactly as the camera, if you want this stream to be the live stream in frigate integration
      - rtsp://frigate:xxx@192.168.1.22:554/stream1
      - "ffmpeg:cam_salotto#audio=aac"
    cam_salotto_sub: #used only to detect movement and audio
      - rtsp://frigate:xxx@192.168.1.22:554/stream2
      - "ffmpeg:cam_salotto_sub#audio=aac"

    cam_terrazzo: # this needs to be named exactly as the camera, if you want this stream to be the live stream in frigate integration
      - rtsp://frigate:xxx@192.168.1.25:554/stream1
      - "ffmpeg:cam_terrazzo#audio=aac"
    cam_terrazzo_sub: #used only to detect movement and audio
      - rtsp://frigate:xxxB@192.168.1.25:554/stream2
      - "ffmpeg:cam_terrazzo_sub#audio=aac"

ffmpeg:
  hwaccel_args: preset-vaapi

record:
  enabled: True
  retain:
    days: 0
    mode: all
  events:
    retain:
      default: 30
      mode: motion

# Include all cameras by default in Birdseye view
birdseye:
  enabled: True
  mode: continuous

cameras:
  cam_salotto:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/cam_salotto_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
            - audio
        - path: rtsp://127.0.0.1:8554/cam_salotto
          input_args: preset-rtsp-restream
          roles:
            - record
    onvif: # this enables camera features control like object movement tracking
      host: 192.168.1.22
      port: 2020
      user: frigate
      password: xxx

    audio:
      enabled: True
      min_volume: 5
    record:
      enabled: True
    detect:
      enabled: True
    objects:
      track:
        - person
      filters:
        person:
          threshold: 0.75    
    live:
        stream_name: cam_salotto
  
  cam_terrazzo:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/cam_terrazzo_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
            - audio
        - path: rtsp://127.0.0.1:8554/cam_terrazzo
          input_args: preset-rtsp-restream
          roles:
            - record
    onvif: # this enables camera features control like object movement tracking
      host: 192.168.1.25
      port: 2020
      user: frigate
      password: xxx

    audio:
      enabled: True
      min_volume: 5
    record:
      enabled: True
    detect:
      enabled: True
    objects:
      track:
        - person
      filters:
        person:
          threshold: 0.75
          mask:
            - 0,69,348,0,0,0
            - 597,0,588,29,532,0
    live:
        stream_name: cam_terrazzo

docker-compose file or Docker CLI command

frigate:
    container_name: frigate
    cap_add:
      - PERFMON # to see gpu stats
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: ${FRIGATE_SHM_SIZE} # update for your cameras based on calculation above
    devices:
      - /dev/bus/usb:/dev/bus/usb
      - ${FRIGATE_DRI_CARD} # this is /dev/dri/card0
      - ${FRIGATE_DRI_RENDER} # this is /dev/dri/renderD128
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ${BASE_STORAGE_DIR}/${HASS_DATA_DIR}/${FRIGATE_DATA_DIR}/${FRIGATE_CONFIG_DIR}:/config
      - ${FRIGATE_STORAGE_DEVICE}:/media/frigate
      - type: tmpfs # Optional: reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: ${FRIGATE_TMPFS_SIZE}
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    networks:
      - homeassistant-net

Relevant log output

frigate  | 2024-04-02 08:57:44.141725869  Process detector:ov:
frigate  | 2024-04-02 08:57:44.143750259  Traceback (most recent call last):
frigate  | 2024-04-02 08:57:44.143842791    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
frigate  | 2024-04-02 08:57:44.143846325      self.run()
frigate  | 2024-04-02 08:57:44.143848743    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
frigate  | 2024-04-02 08:57:44.143850429      self._target(*self._args, **self._kwargs)
frigate  | 2024-04-02 08:57:44.143856204    File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
frigate  | 2024-04-02 08:57:44.143886819      object_detector = LocalObjectDetector(detector_config=detector_config)
frigate  | 2024-04-02 08:57:44.143889641    File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
frigate  | 2024-04-02 08:57:44.143908891      self.detect_api = create_detector(detector_config)
frigate  | 2024-04-02 08:57:44.143912045    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
frigate  | 2024-04-02 08:57:44.143914599      return api(detector_config)
frigate  | 2024-04-02 08:57:44.143968297    File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 32, in __init__
frigate  | 2024-04-02 08:57:44.143971237      self.interpreter = self.ov_core.compile_model(
frigate  | 2024-04-02 08:57:44.143974192    File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 399, in compile_model
frigate  | 2024-04-02 08:57:44.143976497      super().compile_model(model, device_name, {} if config is None else config),
frigate  | 2024-04-02 08:57:44.144153790  RuntimeError: Failed to create plugin /usr/local/lib/python3.9/dist-packages/openvino/libs/libopenvino_intel_gpu_plugin.so for device GPU
frigate  | 2024-04-02 08:57:44.144156352  Please, check your environment
frigate  | 2024-04-02 08:57:44.144158442  Check 'error_code == 0' failed at src/plugins/intel_gpu/src/runtime/ocl/ocl_device_detector.cpp:194:
frigate  | 2024-04-02 08:57:44.144160354  [GPU] No supported OCL devices found or unexpected error happened during devices query.
frigate  | 2024-04-02 08:57:44.144162338  [GPU] Please check OpenVINO documentation for GPU drivers setup guide.
frigate  | 2024-04-02 08:57:44.144163985  [GPU] clGetPlatformIDs error code: -1001

Operating system

Other Linux

Install method

Docker Compose

Coral version

CPU (no coral)

Any other information that may be helpful

No response

[marco-calautti]

Any reason why this has been ignored? Especially considering that kernel 6.7 has been declared EOL, I think it is important for frigate to work as expected on 6.8 and greater.

[NickM-27]

None of the maintainers use openvino and building your own model has even more variables to it. If the built in model doesn't work then that's very concerning, building your own model is a lot more specific and any number of things could be the reason that it doesn't work

You're asking a very specific and specialized question that few people can give an informed answer to

Edit: looks like the title wasn't correct and it is the default model being used.

[NickM-27]

It looks like this issue is outside of Frigate intel/compute-runtime#710 (comment)

[esand]

@NickM-27 As mentioned below by @alexeynl, it seems that updating Intel compute package might be required as well. It seems Jellyfin also updated (jellyfin/jellyfin-packaging#15) to address the issue. Not saying a kernel fix isn't also necessary, but if Intel compute is part of Frigate, I'd just like to +1 requesting it get updated.

[alexeynl]

I use image Docker images from documentation ghcr.io/blakeblackshear/frigate:stable. I have migrated Docker host to Ubuntu 24.04 with 6.8 kernel and faced with the same issue.
To get rid from this error I have installed latest compute-runtime deb packages inside Docker container according installation steps from https://github.com/intel/compute-runtime/releases.

1. Download packages to temp directory of Docker host:

mkdir neo
cd neo
wget https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.16510.2/intel-igc-core_1.0.16510.2_amd64.deb
wget https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.16510.2/intel-igc-opencl_1.0.16510.2_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/24.13.29138.7/intel-level-zero-gpu-dbgsym_1.3.29138.7_amd64.ddeb
wget https://github.com/intel/compute-runtime/releases/download/24.13.29138.7/intel-level-zero-gpu_1.3.29138.7_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/24.13.29138.7/intel-opencl-icd-dbgsym_24.13.29138.7_amd64.ddeb
wget https://github.com/intel/compute-runtime/releases/download/24.13.29138.7/intel-opencl-icd_24.13.29138.7_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/24.13.29138.7/libigdgmm12_22.3.18_amd64.deb

2. Next you need repack downloaded libigdgmm12_22.3.18_amd64.deb package as states here otherwise you will get the error during installation inside docker container:

dpkg-deb: error: archive 'libigdgmm12_22.3.18_amd64.deb' uses unknown compression for member 'control.tar.zst', giving up

sudo apt install binutils
mkdir deb_repack
cp ./libigdgmm12_22.3.18_amd64.deb ./deb_repack/libigdgmm12_22.3.18_amd64.deb
cd deb_repack
ar x libigdgmm12_22.3.18_amd64.deb
zstd -d < control.tar.zst | xz > control.tar.xz
zstd -d < data.tar.zst | xz > data.tar.xz
ar -m -c -a sdsd libigdgmm12_22.3.18_amd64_repacked.deb debian-binary control.tar.xz data.tar.xz
cp libigdgmm12_22.3.18_amd64_repacked.deb ../libigdgmm12_22.3.18_amd64.deb
cd ..
rm -rf ./deb_repack/

3. Run Frigate docker container as following to prevent from constant restart due the error and crash:

sudo docker run -d ghcr.io/blakeblackshear/frigate:stable tail -f /dev/null

4. Copy deb files to Frigate docker container:

cd ..
sudo docker cp neo frigate:/

5. Install deb packages inside Frigate docker container:

sudo docker exec -it frigate sh -c 'dpkg -i /neo/*.deb'

6. Restart the container and check its status:

sudo docker restart frigate
sudo docker ps | grep frigate

This is just workaround. For permanent fix Docker image mainteiner must build and push image with latest intel compute-runtime.

[marco-calautti]

A simpler, temporary fix, as suggested in the intel-compute-runtime repo is to set the following env variables (in the case of frigate, I guess this should be enough in the docker configuration):
NEOReadDebugKeys=1
OverrideGpuAddressSpace=48

Anyway, since the frigate image contains the intel-compute-runtime, I also believe it is frigate's responsibility to update it, to properly fix the bug.

[alexeynl]

this should be enough in the docker configuration):
NEOReadDebugKeys=1
OverrideGpuAddressSpace=48

I can confirm this workaround works. I have added to my docker compose file these variables:

frigate:
...
    environment:
      NEOReadDebugKeys: "1"
      OverrideGpuAddressSpace: "48"

Checked variables exist inside container after down and up docker compose:
sudo docker exec -it frigate sh -c 'env'

HOSTNAME=6dc8233bedd6
HOME=/root
OverrideGpuAddressSpace=48
FRIGATE_RTSP_PASSWORD=password
NVIDIA_DRIVER_CAPABILITIES=compute,video,utility
TERM=xterm
PATH=/usr/lib/btbn-ffmpeg/bin:/usr/local/go2rtc/bin:/usr/local/nginx/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
NEOReadDebugKeys=1
S6_LOGGING_SCRIPT=T 1 n0 s10000000 T
PWD=/opt/frigate
NVIDIA_VISIBLE_DEVICES=all

Frigate is running now and no errors in logs.

[SamTheGreatCoder]

Vouching for this answer as this was able to get both QSV hardware accelerated decoding and an OpenVINO detector working again on my setup. I'm running Frigate through a Portainer stack running through an LXC container on Proxmox 8.2.2, with kernel version 6.8.4-3-pve, on an Intel i7-7700K.

[phammichael]

This answer does not work for me. I am running an unprivileged LXC with a privileged Docker container. I've confirmed in the Docker shell the env variables are passed through correctly. When I set OpenVINO to GPU in Frigate's config file, I get an error about no available OpenVINO device and to check the driver. In the LXC terminal, my iGPU is detected by clinfo and vainfo.

My system:
J5005 Gemini Lake
Kernel 6.8.4-3-pve
Unprivileged LXC
Privileged Docker container for Frigate
Latest Intel runtime compute 24.13.29138.7 from https://github.com/intel/compute-runtime/releases
intel-media-va-driver-non-free