Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

使用NPU swift infer 后端支持VLLM吗 #886

Open
zhengxingmao opened this issue May 9, 2024 · 5 comments
Open

使用NPU swift infer 后端支持VLLM吗 #886

zhengxingmao opened this issue May 9, 2024 · 5 comments
Assignees
@Jintao-Huang
Labels
question Further information is requested

Comments

@zhengxingmao
Copy link

使用NPU swift infer 后端支持VLLM吗
@Jintao-Huang
Copy link
Collaborator

不支持的

@Jintao-Huang Jintao-Huang self-assigned this May 9, 2024
@Jintao-Huang Jintao-Huang added the question Further information is requested label May 9, 2024
@zhengxingmao
Copy link
Author

有计划支持吗？
另外swift infer在使用GPU卡中后端使用vllm 支持OpenAI API服务方式吗？

@Jintao-Huang
Copy link
Collaborator

gpu是支持的

是vllm本身不支持npu啦, 不是ms-swift不支持

@Jintao-Huang
Copy link
Collaborator

使用swift deploy

@zhengxingmao
Copy link
Author

zhengxingmao commented May 11, 2024 •
edited

好的，那么NPU推理时可以指定多卡吗

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Jintao-Huang Jintao-Huang

Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zhengxingmao @Jintao-Huang