Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add websocket support #8

Open
pors opened this issue Apr 25, 2023 · 2 comments
Open

Add websocket support #8

pors opened this issue Apr 25, 2023 · 2 comments

Comments

@pors
Copy link
Contributor

pors commented Apr 25, 2023

To support chat-like langchain applications we need state.

Fast API sessions don't seem very straightforward, so websockets may be the way to go: https://fastapi.tiangolo.com/advanced/websockets/

See also https://github.com/hwchase17/chat-langchain for a reference on how it could be done.

@andrictham
Copy link

How about SSE support as well?

In a LLM chat scenario, tokens only need to be streamed from server to client, not the other way around, so SSE might work better than WebSockets, especially in a serverless environment where long-running sessions are difficult to keep alive.

@msoedov
Copy link
Owner

msoedov commented May 10, 2023

Hi all. WS support requires more invasive changes than I expected. Nevertheless, passing the memory with the request to maintain the state is possible. I made these changes available at #10

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants