Custom model orchestration

[tingc9]

Hi,
I'm trying to dynamically load and unload models to better make use of GPU memory.
While I can POST the v2 load endpoint from the host like mentioned here (https://github.com/triton-inference-server/server/blob/main/docs/protocol/extension_model_repository.md), I receive connection refused errors when I attempt to do so from within a python backend model running on tritonserver.
Essentially, can tritonserver api endpoints be called from within a running python model?

[dyastremsky]

You are using a model to load/unload other models? That sounds like poor separation of duties and can run into a lot of trouble. The intent of the model loading/unloading API is to be called by the user. Could you describe your use case further?

Python backend models are created in their own processes, so it's possible they are not communicating on the same network as Triton server.

CC: @Tabrizian

[outtanames]

Ray was meant to deal with situations like this. https://github.com/autonomi-ai/nos supports multiple models on the same HW (with loading/unloading) if you want to try it out.

[Tabrizian]

@tingc9 We've added model load/unload support in Python backend starting from 23.07 which might be what you're looking for: https://github.com/triton-inference-server/python_backend?tab=readme-ov-file#model-loading-api