ValueError: QuantLinear() does not have a parameter or a buffer named weight.

[ciliamadani]

I'm facing an error while trying to load a HF model from disk:

quantized_model_dir = "/home/ubuntu/chatbot/backend/Llama-2-13B-chat-GPTQ-4bit"
tokenizer_dir = "tokenizer"

tokenizer  = AutoTokenizer.from_pretrained(tokenizer_dir)
model = AutoGPTQForCausalLM.from_quantized(quantized_model_dir, device="cuda:0")

The model and tokenizer were saved to disk using the following code:

tokenizer_dir = "tokenizer"
quantized_model_dir = "Llama-2-13B-chat-GPTQ-4bit"

model_id = "TheBloke/Llama-2-13B-chat-GPTQ"
model_basename = "model"

tokenizer  = AutoTokenizer.from_pretrained(model_id)
tokenizer.save_pretrained(tokenizer_dir)


model = AutoGPTQForCausalLM.from_quantized(model_id,
        model_basename=model_basename,
        use_safetensors=True,
        trust_remote_code=True,
        #device="cuda:0",
        quantize_config=None)

# save quantized model
model.save_quantized(quantized_model_dir)

# save quantized model using safetensors
model.save_quantized(quantized_model_dir, use_safetensors=True)

hardware info
Tesla T4
Driver Version: 530.30.02
CUDA Version: 12.1

software info
Python 3.10.12
ubuntu 22.04
auto_gptq : .7.0.dev0
PyTorch version: 2.1.1+cu121
CUDA version: 12.1

Error trace
model = AutoGPTQForCausalLM.from_quantized(quantized_model_dir, device="cpu")
File "/home/ubuntu/.local/lib/python3.10/site-packages/auto_gptq/modeling/auto.py", line 129, in from_quantized
return quant_func(
File "/home/ubuntu/.local/lib/python3.10/site-packages/auto_gptq/modeling/_base.py", line 946, in from_quantized
accelerate.utils.modeling.load_checkpoint_in_model(
File "/home/ubuntu/.local/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 1491, in load_checkpoint_in_model
set_module_tensor_to_device(model, param_name, "meta", dtype=new_dtype)
File "/home/ubuntu/.local/lib/python3.10/site-packages/auto_gptq/utils/patch_utils.py", line 44, in set_module_tensor_to_device_patched
raise ValueError(f"{module} does not have a parameter or a buffer named {tensor_name}.")
ValueError: QuantLinear() does not have a parameter or a buffer named weight.

[ciliamadani]

This was solved by installing auto_gptq from source using
pip install git+https://github.com/PanQiWei/AutoGPTQ.git@v0.4.2