Multiple CNN models concurrently on the GPU

[B-JackMao]

I want to know if TFLITE currently supports running two or more CNN tasks in parallel on a mobile phone's GPU. Can TFLITE run multiple CNN models concurrently on the GPU to accelerate inference?

[sawantkumar]

Hi @B-JackMao ,

I haven't seen any tflite official documentation mentioning the concurrent model execution on GPU. If i find something i will let you know.

[sawantkumar]

Hi @pkgoogle ,

I saw examples of running two models concurrently on dedicated desktop gpu's using tensorflow but i am not sure if its possible in tflite on android.

[pkgoogle]

Hi @B-JackMao, the GPU Delegate will automatically execute tasks in parallel. Please review the speed section of the description here: https://www.tensorflow.org/lite/performance/gpu . Of course depending on the nature of the tasks and the various dependencies it may or may not be easily parallelizable but the algorithm will try its best to. Let us know if that answers your question, thanks!