You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After upgrading the wasi-nn-ggml plugin to b 2715, the API response is not correct.
Example 1: <|eot_id|> from Llama-3-8b
curl -X POST http://localhost:8080/v1/chat/completions \
-H 'accept:application/json' \
-H 'Content-Type: application/json' \
-d '{"messages":[{"role":"system", "content": "You are a helpful, respectful and honest assistant"}, {"role":"user", "content": "Hello"}], "model":"Meta-Llama-3-8B-Instruct-Q5_K_M"}'
{"id":"chatcmpl-910f1f25-66c8-4e7b-89e3-f00c545a6b73","object":"chat.completion","created":1713947997,"model":"Meta-Llama-3-8B-Instruct-Q5_K_M","choices":[{"index":0,"message":{"role":"assistant","content":"I'm here to help with any questions you have. What would you like to know?<|eot_id|>"},"finish_reason":"stop"}],"usage":{"prompt_tokens":622,"completion_tokens":20,"total_tokens":642}}%
Example 2: </s> from Llama-2-7b and Llama-2-13b
curl -X POST http://localhost:8080/v1/chat/completions \
-H 'accept:application/json' \
-H 'Content-Type: application/json' \
-d '{"messages":[{"role":"system", "content": "You are a helpful, respectful and honest assistant"}, {"role":"user", "content": "Hello"}], "model":"llama-2"}'
{"id":"chatcmpl-129c334b-20b5-4555-91b9-74af02bd447c","object":"chat.completion","created":1713949744,"model":"llama-2","choices":[{"index":0,"message":{"role":"assistant","content":"Hello there! *adjusts glasses* It's a pleasure to make your acquaintance. Is there anything I can help you with or would you like to chat? I'm here to assist you in any way I can, so feel free to ask me anything.</s>"},"finish_reason":"stop"}],"usage":{"prompt_tokens":32,"completion_tokens":59,"total_tokens":91}}
Current State
No response
Expected State
No response
Reproduction steps
See above
Screenshots
Any logs you want to share for showing the specific issue
No response
Components
Others
WasmEdge Version or Commit you used
0.13.5
Operating system information
Ubuntu 22.04
Hardware Architecture
Arm
Compiler flags and options
No response
The text was updated successfully, but these errors were encountered:
alabulei1
changed the title
bug: with Wasi-nn-ggml plugin: b2715, when I using curl to send a API request, it responses the top sign of the model
bug: with Wasi-nn-ggml plugin: b2715, when I using curl to send a API request, it responses the stop sign of the model
Apr 24, 2024
Summary
After upgrading the wasi-nn-ggml plugin to b 2715, the API response is not correct.
Example 1:
<|eot_id|>
from Llama-3-8bExample 2:
</s>
from Llama-2-7b and Llama-2-13bCurrent State
No response
Expected State
No response
Reproduction steps
See above
Screenshots
Any logs you want to share for showing the specific issue
No response
Components
Others
WasmEdge Version or Commit you used
0.13.5
Operating system information
Ubuntu 22.04
Hardware Architecture
Arm
Compiler flags and options
No response
The text was updated successfully, but these errors were encountered: