You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
Describe the bug
After set logprobs = true and top_logprobs = 5. The number of top logrobs response is not correct, only 4 for each token. I suppose it should be 5 for each token.
Reproduction
start a api_server such as internlm2-chat-20b, such as lmdeploy serve api_server /nvme/qa_test_models/internlm/internlm2-chat-20b --tp 2
open swagger and send request to /v1/chat/completions like this:
check the response, only 4 top probs return. I want 5 top probs
And I cannot get logprobs return by using /v1/chat/completions api, the script is:
from lmdeploy.serve.openai.api_client import APIClient
api_client = APIClient('http://localhost:23333')
for output in api_client.chat_completions_v1(model='internlm2',
messages='Shanghai is',
logprobs=True,
top_logprobs=5,
max_tokens=20):
continue
print(output)
logprobs in response is none, it's unexpected.
{'id': '1', 'object': 'chat.completion', 'created': 1714991049, 'model': 'internlm2', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': ' a city of contrasts. The city is both ancient and modern, traditional and innovative, and a world'}, 'logprobs': None, 'finish_reason': 'length'}], 'usage': {'prompt_tokens': 4, 'total_tokens': 25, 'completion_tokens': 21}}
Checklist
Describe the bug
After set logprobs = true and top_logprobs = 5. The number of top logrobs response is not correct, only 4 for each token. I suppose it should be 5 for each token.
Reproduction
check the response, only 4 top probs return. I want 5 top probs
logprobs in response is none, it's unexpected.
Environment
Error traceback
No response
The text was updated successfully, but these errors were encountered: