-
Notifications
You must be signed in to change notification settings - Fork 856
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] ValueError: [quantize] The last dimension of the matrix needs to be divisible by the quantization group size 64. #1033
Comments
It's not a bug.. at the risk of being redundant, the last dimension of the matrix has to be divisible by the quantization group size. For the size 4304 there is no supported group size which divides it (e.g. none of 32, 64, 128). It's not on our roadmap to support irregular sizes... but we can leave this issue open to help prioritize if it's something we should consider in the future. |
It can be divided by 16, would an implementation for that be complicated to implement? |
Yes, it's not a bug. It's more of a feature request / clarification. Because all SigLip based VLM are not quantisable because of this, which include Idefics 2, NanoLlava and Deepseek VL. |
Is there a way to skip particular target layer or Block X in the model in MLX? Not all layers of the same type like class_predicate does. |
You can use
|
Thank you very much, I will give it a try ASAP! |
It works wonders! 💯 Also found a better way, skipping the entire block:
|
Describe the bug
When I try to quantize a VLM model that use SigLIP it throws a value error because it has intermediate size of 4304 which is not divisible by 64 or 128.
To Reproduce
Include code snippet
Expected behavior
Sucessfully quantize model.
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.
Traceback
The text was updated successfully, but these errors were encountered: