Any specific optimization did in kserve to support LLM inference?

[Jeffwan]

Hi community,

I am wondering any specific optimization did in kserve to support LLM applications? Is there a feature list?

[Jeffwan]

/kind question

[yuzisun]

Hi @Jeffwan ! KServe's position is still on LLM inference orchestration and serverless autoscaling, so we are focusing on supporting OpenAI protocol by integrating optimized LLM serving runtime like vLLM/ Tensorrt-LLM/TGI, improving LLM container cold startup time and enabling autoscaling based on custom metrics like the number of input/output tokens. We will update our roadmap with a list features we have planned.