Fix for fields being lists #563
Conversation
|
Isn't that a mistake on @TheBloke s part? I don't think it makes sense for these values to be lists in the quantize_config.json |
|
Yes sorry that was an error in my code for a little while. I thought I had fixed the repos that had that, but obviously not. Please don't PR AutoGPTQ for it I've been away but will be back fairly soon and will fix any repos with that issue In the meantime, the issue won't affect loading the model through Transformers (which uses config.json not quantize_config.json) - and that still uses AutoGPTQ as the backend |
|
Yes, had a similar thought, but ultimately wasn't sure. Glad we have sorted this out. In the meantime, people can use this branch in case they want to evaluate affected models. Thanks, everyone! |
| if isinstance(self.__dict__[check_field], list) and \ | ||
| len(self.__dict__[check_field]) == 1: | ||
| self.__dict__[check_field] = self.__dict__[check_field][0] |
There was a problem hiding this comment.
why not use getattr(self, check_field)?
There was a problem hiding this comment.
This is not really something that needs to be fixed in AutoGPTQ anyway. But I guess it's good to point out better patterns for their future code.
Description
This fixes an error happening with TheBloke/stablelm-zephyr-3b-GPTQ model where certain fields are single element lists.
How to test
I tested it through lm-eval on arc-challenge:
lm-eval --model hf --model_args pretrained=TheBloke/stablelm-zephyr-3b-GPTQ,trust_remote_code=True,autogptq=model.safetensors,quantization_config={'bits':4} --tasks arc_challenge --device cuda:0 --batch_size auto --output /tmp/results/stablelm-zephyr-3b-GPTQ