Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Track costs for streaming with OpenAI #214

Open
brenkao opened this issue May 7, 2024 · 2 comments
Open

Track costs for streaming with OpenAI #214

brenkao opened this issue May 7, 2024 · 2 comments
Labels
Feature Request New feature or request good first issue Good for newcomers

Comments

@brenkao
Copy link
Collaborator

brenkao commented May 7, 2024

Is your feature request related to a problem? Please describe.
Prior versions of openai did not have usage stats when streaming.

Describe the solution you'd like
Add stream_options: {"include_usage": true}. Add total_cost as a property of OpenAICallResponseChunk.

Additional context
OpenAI Cookbook Reference

@brenkao brenkao added Feature Request New feature or request good first issue Good for newcomers labels May 7, 2024
@willbakst
Copy link
Contributor

We may also want to consider updating the generator to just return total cost separate from the response chunk so the generator would have type Generator[BaseCallResponseChunkT, None, Optional[float]] and then return total cost at the end of the generator if available, otherwise return None.

@willbakst willbakst changed the title Cost tracking for streaming OpenAI Feature Request: Track costs for streaming with OpenAI May 8, 2024
@willbakst willbakst changed the title Feature Request: Track costs for streaming with OpenAI [FEATURE REQUEST] Track costs for streaming with OpenAI May 8, 2024
@willbakst willbakst changed the title [FEATURE REQUEST] Track costs for streaming with OpenAI Track costs for streaming with OpenAI May 10, 2024
@brenkao
Copy link
Collaborator Author

brenkao commented May 14, 2024

The stream_options: {"include_usage": true} has been implemented with #239

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature Request New feature or request good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

2 participants