Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Search image data from /api/conversation/custom when using langchain #751

Closed
3 tasks
adamdougal opened this issue Apr 24, 2024 · 1 comment
Closed
3 tasks
Labels
subtask A subtask

Comments

@adamdougal
Copy link
Collaborator

adamdougal commented Apr 24, 2024

Required by #323

Description

When calling the /api/conversation/custom endpoint and using the langchain orchestrator, generate embeddings of the question using computer vision and query the image embeddings using the question embeddings. We should continue to create question embeddings using text-embeddings-ada-002 in addition to be used to search the image description.

Tasks

  • Generate question embeddings using computer vision
  • Query the search index using the generated embeddings
  • Pass any returned images to gpt-4-vision along with the chat
@cecheta
Copy link
Collaborator

cecheta commented May 22, 2024

This will be covered by #750, as the changes are in the question answer tool

@cecheta cecheta closed this as completed May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
subtask A subtask
Projects
None yet
Development

No branches or pull requests

2 participants