Getting no documents retrieved with retrieval chain

[Asma-droid]

hello,

I'am newest in langchain and langserver. I think that there is a problem when returning meta data. The API return only the answer and the question

hereis my server code

from fastapi import FastAPI
from fastapi.responses import RedirectResponse
from langserve import add_routes
from langchain.vectorstores import Chroma
from langchain.llms import HuggingFaceEndpoint
import os
from langchain.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough, RunnableParallel
from langchain.schema import StrOutputParser
from langchain.embeddings import HuggingFaceInferenceAPIEmbeddings

from langchain.chains import RetrievalQA
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.llms import LlamaCpp
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores import Chroma
from langchain.document_loaders import PyPDFDirectoryLoader
from langchain.llms import CTransformers
import pickle


app = FastAPI()

#Step 06:Downlaod the Embeddings
embeddings = HuggingFaceEmbeddings(
    model_name="thenlper/gte-large",
    model_kwargs={"device": "cuda"},
    encode_kwargs={"normalize_embeddings": True},
    )


#Import Model
llm = LlamaCpp(
    streaming = True,
    model_path="/srv/nas_data/atrabelsi/Whisper-Analysis-Final/Meeting_summarization/summary/mistral-7b-instruct-v0.1.Q5_K_M.gguf",
    temperature=0.01,
    top_p=1,
    verbose=True,
    n_ctx=4096
)


chroma_index =  Chroma(persist_directory='/srv/nas_data/atrabelsi/Whisper-Analysis-Final/RAG/LangServeDemo/langserve_index', 
                   embedding_function=embeddings)

retriever = chroma_index.as_retriever(search_kwargs={"k": 1})

prompt_template = """\
Use the provided context to answer the user's question. If you don't know the answer, say you don't know.

Context:
{context}

Question:
{question}"""

rag_prompt = ChatPromptTemplate.from_template(prompt_template)


with open('./langserve_index_2/data.pkl', 'rb') as file: 
    # Call load method to deserialze 
    data = pickle.load(file) 
    
    
def format_docs(data):
    return "\n\n".join(doc.page_content for doc in data)


rag_chain_from_docs = (
    RunnablePassthrough.assign(context=(lambda x: format_docs(x["context"])))
    | rag_prompt
    | llm
    | StrOutputParser()
)


rag_chain_with_source = RunnableParallel(
    {"context": retriever, "question": RunnablePassthrough()}
).assign(answer=rag_chain_from_docs)


@app.get("/")
async def redirect_root_to_docs():
    return RedirectResponse("/docs")


# Edit this to add the chain you want to add
add_routes(app, rag_chain_with_source, path="/rag")

if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="0.0.0.0", port=8010)

i have got empty context and empty documents

[eyurtsev]

Hi @Asma-droid,

At first glance, this sounds like an error user code rather than an issue with LangServe.

To determine whether that's the case, first try to run the chain from your notebook and don't use a server.

from langchain.vectorstores import Chroma
from langchain.llms import HuggingFaceEndpoint
import os
from langchain.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough, RunnableParallel
from langchain.schema import StrOutputParser
from langchain.embeddings import HuggingFaceInferenceAPIEmbeddings

from langchain.chains import RetrievalQA
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.llms import LlamaCpp
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores import Chroma
from langchain.document_loaders import PyPDFDirectoryLoader
from langchain.llms import CTransformers
import pickle



#Step 06:Downlaod the Embeddings
embeddings = HuggingFaceEmbeddings(
    model_name="thenlper/gte-large",
    model_kwargs={"device": "cuda"},
    encode_kwargs={"normalize_embeddings": True},
    )


#Import Model
llm = LlamaCpp(
    streaming = True,
    model_path="/srv/nas_data/atrabelsi/Whisper-Analysis-Final/Meeting_summarization/summary/mistral-7b-instruct-v0.1.Q5_K_M.gguf",
    temperature=0.01,
    top_p=1,
    verbose=True,
    n_ctx=4096
)


chroma_index =  Chroma(persist_directory='/srv/nas_data/atrabelsi/Whisper-Analysis-Final/RAG/LangServeDemo/langserve_index', 
                   embedding_function=embeddings)

retriever = chroma_index.as_retriever(search_kwargs={"k": 1})

prompt_template = """\
Use the provided context to answer the user's question. If you don't know the answer, say you don't know.

Context:
{context}

Question:
{question}"""

rag_prompt = ChatPromptTemplate.from_template(prompt_template)


with open('./langserve_index_2/data.pkl', 'rb') as file: 
    # Call load method to deserialze 
    data = pickle.load(file) 
    
    
def format_docs(data):
    return "\n\n".join(doc.page_content for doc in data)


rag_chain_from_docs = (
    RunnablePassthrough.assign(context=(lambda x: format_docs(x["context"])))
    | rag_prompt
    | llm
    | StrOutputParser()
)


rag_chain_with_source = RunnableParallel(
    {"context": retriever, "question": RunnablePassthrough()}
).assign(answer=rag_chain_from_docs)

What do you get when you inovke this chain?

rag_chain_with_source.invoke(...)``

[Asma-droid]

hello @eyurtsev Thanks for your quick response. When i run just rag_chain_with_source.invoke(...)`` i got the array with context, metadta, question and answer

[Asma-droid]

hello again @eyurtsev. The problem is solved by using an other embedding
embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2") and chromadb by FAISS

I can not understand the logic. Have you an explication please ?

[eyurtsev]

If it's solved by using a different embedding then the issue is unlikely to be LangServe related.

The code that you posted looks correct from first glance. I'd suggest verifying the individual steps to make sure that the embeddings are generated correctly etc.

Instead of using LCEL, you could use .invoke on each of the steps separately to see if you can identify / isolate the issue.

[AhmadHakami]

rag_chain_from_docs = (
    RunnablePassthrough.assign(context=(lambda x: format_docs(x["context"])))
    | rag_prompt
    | llm
    | StrOutputParser()
)

rag_chain_with_source = RunnableParallel(
    {"context": retriever, "question": RunnablePassthrough()}
).assign(answer=rag_chain_from_docs)

this approach leads us to another problem which is the answer generated with rag_chain_with_source.invoke(question) provides any answer with any sources and page numbers even if it is incorrect or not retrieved from the data

it should return based on the file information sorry i can not answer your question without mentioning sources or page numbers

how can this be done?

[Aillian]

is this issue solved? i am facing the same problem

[AhmadHakami]

is this issue solved? i am facing the same problem

hi ali, actually this problem happened to me because the vector store method, when it returns the chunks to my prompt/question sometimes it contains many chunks that dont represent or answer my question

after improving the extraction and re ranking steps i get accurate answers and get nothing if the document doesnt have it

take a look here for the vector store methods and with experiment/playing with parameters you can find the accurate one for your task

also i recommend to you to train your own tokenizer model for feature extraction so it can be representative to the target domain

good luck :)