LlavaNext Model cannot be started

[paulcx]

System Info

image: ghcr.io/huggingface/text-generation-inference:sha-40213c9
model: https://huggingface.co/llava-hf/llava-v1.6-34b-hf

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

...
17T07:25:20.920584343Z [rank0]:   File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/weights.py", line 112, in get_sharded

17T07:25:20.920592356Z [rank0]:     filename, tensor_name = self.get_filename(tensor_name)

17T07:25:20.920599543Z 

17T07:25:20.920606030Z [rank0]:   File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/weights.py", line 63, in get_filename

17T07:25:20.920613791Z [rank0]:     raise RuntimeError(f"weight {tensor_name} does not exist")

17T07:25:20.920621329Z 

17T07:25:20.920627955Z [rank0]: RuntimeError: weight language_model.suffix.weight does not exist

Expected behavior

working on sha-eade737 but broken on sha-40213c9. Is cuda-graph not yet supported for llavanext? Both images will report errors if cuda-graph is enabled or the input image is not a square picture.

[ktrapeznikov]

same issue here