Quantized Phi3: Features to add

[EricLBuehler]

• Support for LongRope (this is supported with ISQ in non-GGUF models, though)
  • The challenge is that the scalings information is not present in the GGUF file.
• X-LoRA: Add X-LoRA support for GGUF #293
  • Again, we already have ISQ support, so this is not critical.

[polarathene]

• The challenge is that the scalings information is not present in the GGUF file.

I'm not too familiar with what you're talking about, but I have seen this as settings in https://github.com/oobabooga/text-generation-webui

No clue if that's related. Presumably user config/overrides might be an alternative option for supplying that information (eg: JSON) or if it's usually a field supported in GGUF, those that want that could presumably patch the metadata directly?

[EricLBuehler]

I am not familiar with this interface, but from my experience implementing LongRope, I think they are using a bit of a hack to get this to work because GGUF does not contain the necessary data. We actually load the full long and short factors for non-GGUF phi3 so it is flexible - and guaranteed to be correct unlike theirs - for any sequence length. I have looked at the GGUF metadata, and it does not contain these long/short factors.

We could of course implement something like this. It would not be ideal, but it would provide a way for users to put the context length about where they need it.

[polarathene]

I think they are using a bit of a hack to get this to work because GGUF does not contain the necessary data.

I don't know if it's actually used with GGUF models. The interface is simple enough to use, but some parts aren't very clear.

---

We could of course implement something like this.

I would suggest deferring until actual feature request is raised 😅

You can just document the caveat with Phi3 quantized? Potentially add a debug! tracing/log call as additional context.