Grounding DINO missing custom kernels

[sam-ulrich1]

System Info

Latest including master as of 5/11/24
Ubuntu, 4090 GPU

Who can help?

@amyeroberts

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

When running any GroundingDINO model you get the following. I have confirmed the kernels do not exist in the transformers repo:

Could not load the custom kernel for multi-scale deformable attention: [Errno 2] No such file or directory: '/.../transformers-main/src/transformers/kernels/grounding_dino/vision.cpp'".
Could not load the custom kernel for multi-scale deformable attention: [Errno 2] No such file or directory: '.../transformers-main/src/transformers/kernels/grounding_dino/vision.cpp'".

Expected behavior

Either the kernels are not necessary and should be removed or they are necessary and causing performance issues. GroundingDINO does seem to under perform in transformers but this is purely anecdotal

[NielsRogge]

cc @EduardoPach

Regarding

GroundingDINO does seem to under perform in transformers but this is purely anecdotal

Could you clarify? Text queries need to be lowercased + end with a dot

[EduardoPach]

Hey @sam-ulrich1 thanks for opening this Issue! GroundingDino was missing the ms_deform_attn_kernels this was something that we overlooked when we merged the model to the library. I have opened this PR #30768 to fix it.