AttributeError: 'LlamaAttention' object has no attribute 'qkv_proj'

[ciliamadani]

Hello, I'm trying to load a quantized model from the disk of an Ubuntu ec2 instance, however, I keep getting the attached error message, does anybody know the solution?

For context, I installed, the auto_gptq using the following command: pip install "auto-gptq==0.4.2"
transformers[sentencepiece]==4.32.1
optimum==1.12.0
accelerate==0.22.0
safetensors>=0.3.1
torch ==2.1.2

Related Code

 local_model_path ="/home/ubuntu/.cache/huggingface/hub/models--TheBloke--Llama-2-13B-chat-GPTQ/snapshots/ea078917a7e91c896787c73dba935f032ae658e9"
quantized_model_dir = "/home/ubuntu/chatbot/backend/Llama-2-13B-chat-GPTQ-4bit"

    tokenizer  = AutoTokenizer.from_pretrained(local_model_path)
    model = AutoGPTQForCausalLM.from_quantized(quantized_model_dir, device="cuda:0")  

Error message
File "/home/ubuntu/.local/lib/python3.10/site-packages/auto_gptq/modeling/auto.py", line 108, in from_quantized
return quant_func(
File "/home/ubuntu/.local/lib/python3.10/site-packages/auto_gptq/modeling/_base.py", line 875, in from_quantized
accelerate.utils.modeling.load_checkpoint_in_model(
File "/home/ubuntu/.local/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 1392, in load_checkpoint_in_model
set_module_tensor_to_device(
File "/home/ubuntu/.local/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 265, in set_module_tensor_to_device
new_module = getattr(module, split)
File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1269, in getattr
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LlamaAttention' object has no attribute 'qkv_proj'

[ciliamadani]

UPDATE
This was solved by installing auto_gptq from source using
pip install git+https://github.com/PanQiWei/AutoGPTQ.git@v0.4.2