Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support download huggingface models in storage initializer #3545

Open
lizzzcai opened this issue Mar 24, 2024 · 2 comments · May be fixed by #3584
Open

Support download huggingface models in storage initializer #3545

lizzzcai opened this issue Mar 24, 2024 · 2 comments · May be fixed by #3584

Comments

@lizzzcai
Copy link
Member

lizzzcai commented Mar 24, 2024

/kind feature

Describe the solution you'd like

I would like to use storage initializer to pull the huggingface models which fits the KServe experience. StorageUri will be in this format: hf://<repo>/<model>:<hash(optional)>

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: bloom
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      #runtime: kserve-huggingface
      storageUri: hf://bigscience/bloom-560m:<hash>
      resources:
        limits:
          cpu: "2"
          memory: 4Gi
        requests:
          cpu: "1"
          memory: 2Gi

Anything else you would like to add:

Another option is to support any Git LFS.

Links to the design documents:
[Optional, start with the short-form RFC template to outline your ideas and get early feedback.]
[Required, use the longer-form design doc template to specify and discuss your design in more detail]

@andyi2it
Copy link
Contributor

/assign @andyi2it

@jiangxiaobin96
Copy link

Hello, I have some questions about kServe model storage.
If I have a model installed from huggingface and a python file to load model, how to use InferenceService CRD?
Like the following YAML, where is the python file to load model?

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: bloom
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      #runtime: kserve-huggingface
      storageUri: hf://bigscience/bloom-560m:<hash>
      resources:
        limits:
          cpu: "2"
          memory: 4Gi
        requests:
          cpu: "1"
          memory: 2Gi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants