Chat format ignored

[woheller69]

Trying example: chatbot_using_local_model.py with WizardLM2 (WizardLM-2-7B.Q8_0.gguf)
gives:

Using fallback chat format: None
User: 

but the example defines CHATML as format:

predefined_messages_formatter_type=MessagesFormatterType.CHATML

is chat format ignored?

[woheller69]

Switching on debug output it seems to be applied correctly. Does the None come from llama-cpp-python or from llama-cpp-agent?

[woheller69]

Regarding order:

1. tries to get template from gguf
2. tries to guess format
3. uses predefined_messages_formatter

Right?

[Maximilian-Winter]

@woheller69 I currently use my own messages formatter system and the completion endpoints, not the chat completion.

I think the Using fallback chat format: None is from llama-cpp-python.

You can ignore it, since I don't use the chat completion endpoints.

[woheller69]

OK, this means it has to be always specified. It would be nice if in future there was an option to apply the format from the gguf because it is often difficult to find out which template is required. And the newer ggufs usually contain the information...