ValueError: doc_id <id> not found while using KnowledgeGraphIndex

[anirbanbasu]

Hi,

I have a StorageContext with a graph_store set to a Neo4jGraphStore, a document_store set to a RedisDocumentStore, an index_store set to a RedisIndexStore.

I am using local Neo4j and Redis Stack installations.

With that said, this is what I am doing.

1. I use a SentenceSplitter to split my documents into text chunk nodes, by calling get_nodes_from_documents() on the SentenceSplitter object built using SentenceSplitter.from_defaults(). For brevity, I am skipping the arguments passed to the function calls.
2. I build a KnowledgeGraphIndex from those text chunk nodes with the storage_context of the knowledge graph index set to the StorageContext object previously instantiated.
3. I explicitly add the text chunk nodes by calling storage_context.document_store.add_documents() on the built knowledge graph index. This is where I think I am already doing what the resolution for this closed issue suggests.
4. I am calling persist() explicitly on the storage_context.document_store and storage_context.index_store, although this documentation tells that this is not necessary.

Database connection issues are not there as I can clearly see databases being populated and updated.

All goes well when I try to use the knowledge graph index built in step 2 above. However, my problem appears on the step 3 below.

1. I try to reload the knowledge graph index using the index_id I obtained from the previously built index object, by calling load_index_from_storage() on the storage context object described above. Index loads perfectly fine.
2. The loaded index can be initialised without a problem, e.g., calling the as_chat_engine() on it.
3. However, when I try to actually query or chat something, it fails to look up the graph triplets (which is not a problem if I continue to use the index built as described above, without trying to reload it) and throws an error ValueError: doc_id <id> not found.

I can confirm every time from Redis that the specific <id> reported in the error message actually exists amongst the document key(s). Redis also has the index stored along with metadata and ref_doc_info, which get updated if I run the steps above. The graph database is also up and running. I guess it is not the problem at all?

❓😕 Why is the document not found when it clearly exists? I must be missing something trivial.

Thanks!

[dosubot[bot]]

Hey @anirbanbasu, great to see you diving deep into the world of LlamaIndex again! How's the coding journey treating you this time?

Based on the information you've provided, it seems like you've followed the correct steps to set up and use the KnowledgeGraphIndex. However, the ValueError: doc_id <id> not found error you're encountering when querying or chatting with a reloaded KnowledgeGraphIndex could be due to several reasons. Here are some potential causes and solutions:

1. Index Prefix Mismatch: Ensure that the index_prefix and prefix_ending used to initialize the RedisVectorStore match exactly what was used when the documents were initially added. A mismatch in these prefixes would lead to the system looking in the wrong place for the documents.

2. Document ID Handling: The document ID (doc_id) is stored and queried as a TAG field in Redis. Ensure that the doc_id used in queries exactly matches the doc_id stored in Redis, including case sensitivity and any potential leading or trailing spaces.

3. Redis Index Existence: The method _index_exists checks if the index exists by listing all indices and checking if the specified index name is among them. If the index somehow was deleted or not properly created, attempts to query it would result in errors.

4. Redis Search Module: The implementation relies on the Redis Search module for indexing and querying documents. Ensure that this module is correctly installed and enabled in your Redis instance.

5. Document Eviction: If Redis is configured with an eviction policy that might remove keys to free up memory, it's possible that documents could be evicted from the index.

6. Index Creation and Overwriting: If the index is supposed to be overwritten (_overwrite is True), the existing index is deleted and a new one is created. Ensure that this process completes successfully without errors.

7. Query Execution: When executing a query, the system constructs a Redis query string and parameters based on the input query embedding and filters. Ensure that the query parameters are correctly formed and that the Redis query execution is successful without errors.

To diagnose and resolve the issue, you may need to verify the index prefix and document IDs, check the Redis logs and the output of the Redis Search module for any errors or warnings, ensure that the Redis instance has the Search module installed and that it's accessible from the application, and consider adding more logging around the index creation, document addition, and querying processes to capture any potential issues.

You can refer to the RedisVectorStore implementation for more details.

As for the load_index_from_storage() function, it is designed to load an index from storage based on the provided storage context and service context. The steps it takes to load an index from storage are as follows:

1. Construct a Storage Context.
2. Construct an Index.
3. Persist Storage to Disk.
4. Load Storage Context from Disk.
5. Load Index from Storage.
6. Validation and Usage.

You can refer to the test cases for more details on how this function works.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

Sources

• llama-index-legacy/llama_index/legacy/vector_stores/redis.py
• llama-index-core/tests/indices/test_loading.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[anirbanbasu]

@dosu-bot, thank you for your response. I shall elaborate on the potential causes and solutions that you have raised in your reply.

1. Index Prefix Mismatch: ❓ You have mentioned about RedisVectorStore. Do I need to use it to be able to look up the documents correctly?
2. Document ID Handling: ❗This is something I cannot ensure because the knowledge graph index automatically makes those calls. What I can tell is that the document ID that the index fails to look up was previously saved in Redis.
3. Redis Index Existence: 🆗 The knowledge graph index is loaded without a problem. Querying Redis from Redis Insight also shows that it exists, although I have not run _index_exists check.
4. Redis Search Module: 🆗 Yes, it is installed and enabled.
5. Document Eviction: 🆗 According to the default configuration, it is set to maxmemory-policy noeviction. I have not modified it.
6. Index Creation and Overwriting: 🆗 Yes, it is being created and/or overwritten correctly. I also tried manually removing all keys using Redis Insight to allow only the newly created indices to exist.
7. Query Execution: ❓ How should I ensure that since the knowledge graph index's chat or query engines handle that automatically? Of course, the LLM and the embedding functions used in the creation of the index are the same as those used during the queries.

[dosubot]

Thank you for providing a detailed description of your issue. Let's address your questions one by one:

1. Index Prefix Mismatch: The RedisVectorStore is not necessary for looking up documents correctly. The RedisDocumentStore should be sufficient for storing and retrieving documents. The RedisVectorStore is used for storing vectors, not documents. The RedisIndexStore is used for storing indices, which are used to quickly look up documents based on their vector representations. If you're using the RedisDocumentStore and RedisIndexStore correctly, there should be no need for the RedisVectorStore in your case.

2. Document ID Handling: The document ID used in queries should match the doc_id stored in Redis. The KnowledgeGraphIndex does not automatically generate or modify these IDs. It uses the IDs provided when adding documents to the index. If you're seeing a mismatch, it could be due to the way you're generating or handling these IDs in your application. Make sure you're using the same ID when adding a document to the index and when querying for that document.

3. Query Execution: The KnowledgeGraphIndex handles the formation of query parameters and the execution of Redis queries. If there's an error in this process, it's likely due to a bug in the KnowledgeGraphIndex code. However, based on your description, it seems like the index is being created and/or overwritten correctly, so this might not be the issue.

Given the information you've provided, it's difficult to pinpoint the exact cause of the issue. However, here are a few things you can try:

• Check Redis Configuration: Double-check your Redis configuration to make sure it's set up correctly. Make sure the Redis Search module is installed and enabled, and that the document eviction policy is set to noeviction.

• Check Document ID Consistency: Make sure the document ID used in queries matches the doc_id stored in Redis. This might involve checking the process that generates or assigns these IDs to ensure they are consistent and correctly referenced during queries.

• Debugging and Logging: Add more logging around the areas where the error occurs to get more insight into the state of the index and the operations leading up to the error. This can help identify if the issue is with the document IDs, the Redis data, or the index's internal state.

If after checking these areas the issue persists, consider isolating the problem by creating a minimal example that reproduces the error and reviewing the differences between the creation and reloading processes of the index.

I hope this helps! If you have any more questions or need further clarification, please don't hesitate to ask.

Sources

• llama-index-core/llama_index/core/indices/knowledge_graph/base.py
• llama-index-integrations/storage/kvstore/llama-index-storage-kvstore-redis/llama_index/storage/kvstore/redis/base.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[anirbanbasu]

@dosu-bot, thanks for your response. I have enabled debug logging and tracing using langfuse. It is clear that the document retrieval stage fails, but I cannot see any error leading up to it.

Running HGETALL "<ns>/doc" on Redis where <ns> is the namespace I used when constructing both the RedisDocumentStore and the RedisIndexStore reveals something like this.

1) "484b0420-e7ee-420d-8819-8d6ecfc11a45"
2) "{\"__data__\": {\"id_\": \"484b0420-e7ee-420d-8819-8d6ecfc11a45\", \"embedding\": null, \"metadata\": {}, \"excluded_embed_metadata_keys\": [], \"excluded_llm_metadata_keys\": [], \"relationships\": {\"1\": {\"node_id\": \"46495290\", \"node_type\": \"4\", \"metadata\": {}, \"hash\": \"32dd2474b9aecd84d3e4dd16b3a64f837073417e845103e243cccb1997995016\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"169b44d3-e7a3-44f7-a531-bea80181ca64\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5f061c839c8502b04986873df0c8227098f058ff64221abe039bd81619a818a5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"<the-actual-text-in-the-chunk>\", \"start_char_idx\": 1802, \"end_char_idx\": 3678, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}, \"__type__\": \"1\"}"
3) "169b44d3-e7a3-44f7-a531-bea80181ca64"
4) "{\"__data__\": {\"id_\": \"169b44d3-e7a3-44f7-a531-bea80181ca64\", \"embedding\": null, \"metadata\": {}, \"excluded_embed_metadata_keys\": [], \"excluded_llm_metadata_keys\": [], \"relationships\": {\"1\": {\"node_id\": \"46495290\", \"node_type\": \"4\", \"metadata\": {}, \"hash\": \"32dd2474b9aecd84d3e4dd16b3a64f837073417e845103e243cccb1997995016\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"484b0420-e7ee-420d-8819-8d6ecfc11a45\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fb8c9f2f06e38b9d9ce6c03d1e8965f969f5553f2e24ff286f319dea0bfc2828\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"<the-actual-text-in-the-chunk>\", \"start_char_idx\": 0, \"end_char_idx\": 1799, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}, \"__type__\": \"1\"}"

Yet, the error ValueError: doc_id 484b0420-e7ee-420d-8819-8d6ecfc11a45 not found continues to exist. You can clearly see that the doc_id with value 484b0420-e7ee-420d-8819-8d6ecfc11a45 exists.

Running FT._LIST returns one output by the name <ns>.

To dig a bit further, FT.SEARCH <ns> "*" outputs four matches <ns>/doc, <ns>/metadata, <ns>/ref_doc_info and <ns>/index. The <ns>/doc obviously has the doc_id that seems to be not found. I suppose Redis search module is working fine since FT.SEARCH is resulting in valid outputs?

I tried running document_exists() on the RedisDocumentStore using that document ID. It can be found. Yet, it cannot be found with the query engine or chat engine from the knowledge graph index. 😕

Do these give you any idea why the document ID cannot be found? Am I missing something?

[anirbanbasu]

I solved the problem. There is no storage_context.document_store but storage_context.docstore! 😮 I wonder how the documents were getting stored in Redis.