Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inference gRPC/Rest client to support FP16 #3643

Closed
yuzisun opened this issue Apr 28, 2024 · 0 comments · Fixed by #3695 · May be fixed by #3685
Closed

Inference gRPC/Rest client to support FP16 #3643

yuzisun opened this issue Apr 28, 2024 · 0 comments · Fixed by #3695 · May be fixed by #3685

Comments

@yuzisun
Copy link
Member

yuzisun commented Apr 28, 2024

/kind feature

Describe the solution you'd like
Triton client supports FP16, kserve rest/grpc should also support transporting FP16.
https://github.com/kserve/kserve/blob/master/python/kserve/kserve/protocol/infer_type.py#L278-L296

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Links to the design documents:
[Optional, start with the short-form RFC template to outline your ideas and get early feedback.]
[Required, use the longer-form design doc template to specify and discuss your design in more detail]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant