[BUG] This module is not using GPU at all.

[KVignesh122]

rembg_session = rembg.new_session()
rembg.remove(data=Image.open(input_path), session=rembg_session, only_mask=True)

rembg_session.inner_session.get_providers() # prints out ['CPUExecutionProvider']

I also saw that there were no spikes at all on my GPU RAM graph.

I am running this code on GColab with T4 runtime, here is the link to a project [Video Background Removal] I have created for my non-tech colleagues: https://colab.research.google.com/drive/16AslpibFerebpJXULY0C8oCPH8Dqrvim?usp=sharing

I have GColab pro and tried it on other GPUs too, but rembg did not use any GPU at all.

[ProgrammingLife]

The same problem here. I've installed onnxruntime-gpu and rembg[gpu] successfully, but it doesn't use gpu.
Maybe we should run it in some other way than just rembg i ...?

[jalsop24]

I think this is an issue with how onnxruntime-gpu installs.

Try installing your usual dependencies and then afterwards run pip install --force-reinstall onnxruntime-gpu

[KVignesh122]

I have tried all possible ways actually.

1. Pip Installing rembg automatically disables support for GPU
Installing rembg[gpu] automatically installs both onnxruntime-gpu AND onnxruntime. I read somewhere that maybe this may be causing a conflict of packages issue and in turn disabling GPU support.

For instance, if I pip install rembg[gpu] as suggested and run this code:

import torch
import onnxruntime as ort
print(ort.get_available_providers()) # Available providers are ['AzureExecutionProvider', 'CPUExecutionProvider']

So I pip installed the base rembg[gpu] without any of the other dependencies, and then manually installed all the dependencies without onnxruntime:

pip install onnxruntime-gpu
pip install rembg[gpu] --no-deps
pip install jsonschema numpy opencv-python-headless pillow pooch pymatting scikit-image scipy tqdm

Doing this shows that

import torch
import onnxruntime as ort
print(ort.get_available_providers()) # Available providers are ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'AzureExecutionProvider', 'CPUExecutionProvider']

So clearly, normal pip installing rembg[gpu] naturally removes 'CUDAExecutionProvider' to begin with and falls back to CPU support only... Same result if I pip install --force-reinstall onnxruntime-gpu @jalsop24 :(

2. Tried using torch 2.1 and CUDA 11
Understanding this, I also read that maybe there is a problem with CUDA12 support in PyTorch version 2.2.x so I forced the program to run on PyTorch 2.1.2 with CUDA version 11.8 instead:

pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu118
pip install nvidia-cudnn-cu11
pip install tensorrt-cu11

import torch
print(torch.__version__) # Prints out 2.1.2+cu118

But, nope still no GPU usage.

3. Hardcoded 'CUDAExecutionProvider' into rembg_session.inner_session._providers
Tried this too:

rembg_session = rembg.new_session()
rembg_session.inner_session._providers = ['CUDAExecutionProvider']
print(rembg_session.inner_session.get_providers())

Ouput:

*************** EP Error ***************
EP Error /onnxruntime_src/onnxruntime/python/onnxruntime_pybind_state.cc:456 void onnxruntime::python::RegisterTensorRTPluginsAsCustomOps(onnxruntime::python::PySessionOptions&, const ProviderOptions&) Please install TensorRT libraries as mentioned in the GPU requirements page, make sure they're in the PATH or LD_LIBRARY_PATH, and that your GPU is supported.
 when using ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'AzureExecutionProvider', 'CPUExecutionProvider']
Falling back to ['CUDAExecutionProvider', 'CPUExecutionProvider'] and retrying.
****************************************
['CUDAExecutionProvider']

The EP Error is not making any sense also, because I already installed the "TensorRT libraries as mentioned in the GPU requirements page" but anyways it falls back to ['CUDAExecutionProvider', 'CPUExecutionProvider']. But somehow, during runtime, the provider falls back to 'CPUExecutionProvider' only once again and no GPU usage. I tried to inspect all the codefiles on how the provider is selected but I still could not figure out this issue...

[Bouts2019]

same issue

[Bonitodelcapo]

Same issue here.
Checked the installation matrix and installed the provided wheels for Jetson from the Jetson Zoo.
Then i pip installed rembg[gpu], which as dependency has onnxruntime, so it installed also the cpu package.
I tried to pip uninstall the onnxruntime package and to force reinstall the onnxruntime-gpu but then rembg gets lost in a loop by the import.

So only CPU is working for Jetson

[Mr47hsy]

Here is my solution:

Based On @KVignesh122 's works. thank you buddy!

# based on cuda 11.8, other versions may have compatibility issues
# for cudnn version check https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements
# and https://docs.nvidia.com/deeplearning/cudnn/archives/cudnn-895/install-guide/index.html
sudo apt-get install libcudnn8=8.9.2.26-1+cuda11.8 -y
# directly installing rembg[gpu] will cause dependency issues
# so install it separately
pip3 install onnxruntime-gpu==1.18.0
pip3 install rembg[gpu]==2.0.50 --no-deps
pip3 install numpy opencv-python-headless pillow pooch pymatting scikit-image scipy tqdm

# test & enjoy
python3 -c "from rembg import remove, new_session; from PIL import Image; output = remove(Image.open('i.png'), session=new_session('u2net', ['CUDAExecutionProvider'])); output.save('o.png')"

[KVig122]

print(new_session('u2net', ['CUDAExecutionProvider'])).inner_session.get_providers())
# You still get ['CPUExecutionProvider']

Solution still doesn't work buddy @Mr47hsy 😭

[Mr47hsy]

@KVig122
hi , it was worked in my side:

Try to identify the specific cause by:

import  onnxruntime as ort

print(f"onnxruntime device: {ort.get_device()}") # output: GPU
print(f'ort avail providers: {ort.get_available_providers()}') # output: ['CUDAExecutionProvider', 'CPUExecutionProvider']

ort_session = ort.InferenceSession('/root/.u2net/u2net.onnx', providers=["CUDAExecutionProvider"])
print(ort_session.get_providers())

[WindowsNT]

Same error here. Excellent results, but CPU only.