Custom ClusterServingRuntime not being selected based on modelFormat Name/Version without runtime specification

[supertetelman]

/kind bug

I created a ClusterServingRuntime that looks like this:

apiVersion: serving.kserve.io/v1alpha1
kind: ClusterServingRuntime
metadata:
  name: nvidia-nim-llm-24.01
spec:
  supportedModelFormats:
    - name: nvidia-nim-llm
      version: "24.01"
      autoSelect: true
      priority: 1

And an InferenceService like this:

kind: InferenceService
metadata:
  name: my-model
spec:
  predictor:
    model:
      modelFormat:
        name: nvidia-nim-llm
        version: "24.01"

And I get this error:

v1beta1Controllers  no runtime found to support predictor with model type: {nvidia-nim-llm 0xc002c46540}

Based on my understanding of the docs, if I specify the same model format/version as supported in my runtime as I list in my predictor, KServe should select it for use. I was not expecting to get this error. If I manually specify the name of the runtime in my yaml it works fine, but I would like to just specify the modelFormat name/version.

Running this through Kubeflow v1.8.0 and on the latest KServe v1.12.1.

[yuzisun]

Did you create isvc first or the cluster serving runtime ?

[yuzisun]

Also the version here means for the model framework version like pytorch version or tensorrt version, I think what you specified is the nim serving time version which we have another field named “runtimeVersion”. It is not the reason why this fails though but just want to clarify.