MeetKai Functionary GGUF models - chat template format unknown

[Anto79-ops]

LocalAI version:

quay.io/go-skynet/local-ai:master-cublas-cuda12-ffmpeg

Using the Jan 21,2024 build

Environment, CPU architecture, OS, and Version:

Docker Ubuntu, AMD, 4060 dual GPU

I cannot run the gguf model using standard chat templates

When use the standard chat and chat-block tmpls as shown here, this error results for a querry

anto@ubuntu-cyberpc:~/MyAI/Auto79_Localai/docker/models$ curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "functionary7bq8",
     "messages": [{"role": "user", "content": "How are you?"}],
     "temperature": 0.9
   }'
{"created":1705952676,"object":"chat.completion","id":"3b96e0cd-646d-4a40-b689-0b6f891fa638","model":"functionary7bq8","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":""}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

The devs over at MeetKai say here that this model uses a jinja template in tokenizer_config.json:

https://huggingface.co/meetkai/functionary-medium-v2.2/blob/main/tokenizer_config.json#L69

and it would have to be converted to the format of LocalAI

any suggestions how to do this conversion? thanks

[Anto79-ops]

anybody have an idea about this? thanks

[Zen3515]

Sources

• https://huggingface.co/meetkai/functionary-small-v2.5/blob/main/tokenizer/tokenizer_config.json
• https://localai.io/advanced/#prompt-templates
• https://github.com/MeetKai/functionary/blob/main/tests/prompt_test_v2.llama3.txt

{% set loop_messages = messages %}
{% for message in loop_messages %}
{%   set content = '' + message['role'] + '\n\n'+ message['content'] | trim + '' %}
{%   if loop.index0 == 0 %}
{%     set content = bos_token + content %}
{%   endif %}
{{ content }}
{% endfor %}
{% if add_generation_prompt %}
{{ 'assistant\n\n' }}
{% endif %}

What I got so far

roles:
  user: "user"
  assistant: "assistant"
  system: "system"
  tool: "tool"
template:
  chat_message: |
    <|begin_of_text|>{{.RoleName}}


    {{if .Content}}{{.Content}}{{end}}<|end_of_text|>
  chat: |
    <|begin_of_text|>{{.RoleName}}


    {{if .Content}}{{.Content}}{{end}}assistant
    
    <|end_of_text|>
  completion: |
    {{.Input}}

IMHO, it may be better to just run them and use the code (apply_chat_template) to convert as shown in https://huggingface.co/meetkai/functionary-small-v2.5#prompt-template

[Zen3515]

On second thought, it uses llama, so this should work

https://github.com/mudler/LocalAI/blob/5b75bf16c72c1c796e261abab0763dd477c46781/gallery/llama3-instruct.yaml

---
# name: "llama3-instruct"

# config_file: |
  mmap: true
  template:
    chat_message: |
      <|start_header_id|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "tool"}}tool{{else if eq .RoleName "user"}}user{{end}}<|end_header_id|>

      {{ if .FunctionCall -}}
      Function call:
      {{ else if eq .RoleName "tool" -}}
      Function response:
      {{ end -}}
      {{ if .Content -}}
      {{.Content -}}
      {{ else if .FunctionCall -}}
      {{ toJson .FunctionCall -}}
      {{ end -}}
      <|eot_id|>
    function: |
      <|start_header_id|>system<|end_header_id|>

      You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
      <tools>
      {{range .Functions}}
      {'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
      {{end}}
      </tools>
      Use the following pydantic model json schema for each tool call you will make:
      {'title': 'FunctionCall', 'type': 'object', 'properties': {'arguments': {'title': 'Arguments', 'type': 'object'}, 'name': {'title': 'Name', 'type': 'string'}}, 'required': ['arguments', 'name']}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
      Function call:
    chat: |
      <|begin_of_text|>{{.Input }}
      <|start_header_id|>assistant<|end_header_id|>
    completion: |
      {{.Input}}
  context_size: 8192
  f16: true
  stopwords:
  - <|im_end|>
  - <dummy32000>
  - "<|eot_id|>"
  - <|end_of_text|>

[Zen3515]

jekalmin/extended_openai_conversation#174 (comment)

Also in a very similar situation

[Anto79-ops]

thanks @Zen3515

I used the older MeetKai functionary with that integration, and it did do the functions well.

At the moment im using the Llama3 8b (q8) with the HomeLLM integration and seems to working well, mostly.

Have you tried this model mediumV2.2 with HA?

[Zen3515]

Oh, I just noticed that it's also you in that comment.

No, I just got a 24GB card and set it up for the first time, and thinking about how I should approach this.

Just out of curiosity, one of the hard requirements of mine is that the model needs to support multiple languages.
What exactly is the Llama3 8b (q8) you use, is it multilingual?

[Anto79-ops]

check out this repo here, and its a good place to start:

https://github.com/acon96/home-llm

I'm currently using this with the Llama3 8b Q8.

I'm sorry, I don't know how well it performs other than Englsih, BUT if you read the discusison page at the repo and maybe the readme there, there are recommeded models for non-english listed.

[Zen3515]

Hello, I got it up with one docker image. So, I decided to stop trying to make it work with Local AI. Since it's easy enough.

In case anyone is interested, here is the image I made for this. It's simply a vllm image with functionary git on top.
https://hub.docker.com/r/zen3515/functionary-vllm

So far, I'm impressed with functionary a lot, the flow is so natural, it asks you what you missed, this reminds me of the good old days when I was using dialogflow with overly complicated rules.
All this in Thai language!

The only downside is it doesn't know very well about home assistant function name.
But that can be fixed using some prompt.