New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] 关于流式并发相关 #1557

Open

2 tasks

Dingxiangxiang opened this issue May 8, 2024 · 1 comment

Assignees

Dingxiangxiang commented May 8, 2024

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.

Describe the bug

lmdeploy 4.0版本，pipline的方式流式输出，好像不支持并发，请问大佬们是怎么解决的

Reproduction

lmdeploy pipline推理的

Environment

lmdeploy 4.0

Error traceback

No response

lvhan028 assigned AllentDan

Collaborator

AllentDan commented May 8, 2024

麻烦贴一下使用的脚本。stream_infer 本身就是支持多个 prompt 输入的

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment