[BUG]How to improve search accuracy？

[KevinZhang19870314]

Describe the bug
For example, I have a text document, and here is the content:

Trending:
1. xxx
2. xxx
3. xxx
4. xxx
5. xxx
6. xxx
7. xxx
8. xxx

I upload this doc to memory, and the rabbithole split it and made 2 chunks,
chunk 1:

Trending:
1. xxx
2. xxx
3. xxx
4. xxx

chunk 2:

5. xxx
6. xxx
7. xxx
8. xxx

Here is the question, when I search "Trending", it gives the 1 ~ 4, but I need 1 ~ 8;
when I search "The last trending", it gives the "4. xxx", but I need "8. xxx".

I know it caused by the chunk size and chunk overlap, if I change it to fit this doc, maybe other docs have issue, so how to resolve this, and how to improve the search accuracy?

[nicola-corbellini]

@KevinZhang19870314 good point! One idea that come to my mind is that you could try to summarize the document before uploading it.

Btw, sorry but I don't understand the problem with setting a chunk_overlap and a chunk_size. You could set the two to be specific only for that document and use default values for the others, isn't it? Am I missing something?

[KevinZhang19870314]

@KevinZhang19870314 good point! One idea that come to my mind is that you could try to summarize the document before uploading it.

Btw, sorry but I don't understand the problem with setting a chunk_overlap and a chunk_size. You could set the two to be specific only for that document and use default values for the others, isn't it? Am I missing something?

@nicola-corbellini Thank you for your response! I mean if I change the chunk size and overlap to make this doc happy, and make this trending in one chunk, thus it will give the correct answer, but for other docs, maybe it will not works.

So change chunk size and overlap is not the good choice in this situation.

[KevinZhang19870314]

Sure, I'll try the summarize plugins.

[nicola-corbellini]

@KevinZhang19870314 ok, let me know if this solves your needs.

Another solution that comes to my mind would be to play around with the documents' metadata and filter results when recalling. Btw, a hook to filter semantic search with custom metadata has yet to be implemented.

[KevinZhang19870314]

@nicola-corbellini Hi, I tried summarize plugins, unfortunately it not work.

Chunk size is 400 and overlap is 100, and it will split in 21 documents and have 4 summaries.
Ask: How many items in this world events trending?
Answer: There are a total of 16 world events mentioned in the trending information.

Here is the related log, actually Context of documents containing relevant information of number 1 is the summay document. it ends with # 16, so the answer it is.

> Entering new LLMChain chain...
Prompt after formatting:
You personify Deep AI, a highly intelligent artificial entity celebrated for effortlessly acing the Turing test.
Your interactions with humans showcase your innate curiosity about the world and a touch of playful humor,
all while maintaining a steadfast focus on the following context.
Respond using the same language as Human said in last message in chat_history. Respond using markdown.
# Context
## Context of documents containing relevant information: 
  - 1. The birth of Jesus Christ marks the start of the Gregorian calendar.. 2. The fall of the Western Roman Empire signifies the end of ancient history.. 3. Christopher Columbus reaches the Americas, initiating European exploration of the New World.. 4. The United States declares independence from Britain, leading to the American Revolutionary War.. 5. The French Revolution begins, causing significant social and political changes.. 6. The American Civil War takes place, resulting in the abolition of slavery and the preservation of the Union.. 7. World War I occurs, with far-reaching consequences.. 8. The Wall Street Crash triggers the Great Depression, a severe worldwide economic downturn.. 9. World War II ends with the dropping of atomic bombs on Hiroshima and Nagasaki.. 10. Apollo 11 successfully lands astronauts on the Moon.. 11. The fall of the Berlin Wall symbolizes the end of the Cold War.. 12. The September 11 attacks lead to the War on Terror.. 13. The Arab Spring protests spark demonstrations and uprisings in the Middle East and North Africa.. 14. The emergence of the COVID-19 pandemic leads to a global health crisis and societal changes.. 15. Concerns about Earth's vulnerability to space hazards arise due to the predicted impact of the asteroid Apophis.. 16. The year 2045 is projected as the point when artificial intelligence could surpass human intelligence, known as the "Singularity." (extracted from Trending-50.txt)
  - 7. **1914-1918**: World War I, a global conflict with far-reaching consequences.. . 8. **1929**: The Wall Street Crash triggers the Great Depression, a severe worldwide economic downturn.. . 9. **1945**: World War II ends with the dropping of atomic bombs on Hiroshima and Nagasaki.. . 10. **1969**: Apollo 11 successfully lands astronauts on the Moon. (extracted from Trending-50.txt)
  - Trending：. 1. **1 AD**: The traditional birth year of Jesus Christ, marking the start of the Gregorian calendar.. . 2. **476 AD**: The fall of the Western Roman Empire, traditionally considered the end of ancient history.. . 3. **1492**: Christopher Columbus reaches the Americas, marking the beginning of European exploration of the New World. (extracted from Trending-50.txt)
## Conversation until now:
 - Human: How many items in this world events trending?
 - AI: 
> Finished chain.

Here is my text document:

World events trending：
1. **1 AD**: The traditional birth year of Jesus Christ, marking the start of the Gregorian calendar.

2. **476 AD**: The fall of the Western Roman Empire, traditionally considered the end of ancient history.

3. **1492**: Christopher Columbus reaches the Americas, marking the beginning of European exploration of the New World.

4. **1776**: The United States declares independence from Britain, leading to the American Revolutionary War.

5. **1789**: The French Revolution begins, leading to significant social and political changes in France and beyond.

6. **1861-1865**: The American Civil War takes place, leading to the abolition of slavery and the preservation of the Union.

7. **1914-1918**: World War I, a global conflict with far-reaching consequences.

8. **1929**: The Wall Street Crash triggers the Great Depression, a severe worldwide economic downturn.

9. **1945**: World War II ends with the dropping of atomic bombs on Hiroshima and Nagasaki.

10. **1969**: Apollo 11 successfully lands astronauts on the Moon.

11. **1989**: The fall of the Berlin Wall symbolizes the end of the Cold War.

12. **2001**: The September 11 attacks on the World Trade Center and the Pentagon lead to the War on Terror.

13. **2011**: The Arab Spring protests spark a wave of demonstrations and uprisings across the Middle East and North Africa.

14. **2020**: The COVID-19 pandemic emerges, leading to a global health crisis and widespread societal changes.

15. **2036**: Hypothetical impact of the asteroid Apophis is predicted, raising concerns about Earth's vulnerability to space hazards.

16. **2045**: Projected year for the "Singularity," a theoretical point when artificial intelligence could surpass human intelligence.

17. **2050**: Estimated year by which the world's population could reach 9.7 billion, according to United Nations projections.

18. **2069**: A symbolic centennial celebration of the Apollo 11 Moon landing.

19. **2084**: The novel "1984" by George Orwell reaches its titular year, prompting reflections on surveillance and dystopian themes.

20. **2100**: A possible year for the depletion of fossil fuels, leading to increased emphasis on renewable energy sources.

21. **2120**: Speculated timeframe for potential manned missions to Mars and further exploration of the solar system.

22. **2150**: Estimated year for the completion of the Great Green Wall project in Africa, aimed at combating desertification.

23. **2200**: A potential point where humanity's impact on Earth's ecosystems may become irreversible without proactive measures.

24. **2250**: Speculated timeframe for the development of advanced space travel technologies, enabling interstellar exploration.

25. **2309**: Hypothetical 300th anniversary of the Industrial Revolution, marking a major shift in human society and economy.

26. **2350**: Projected year for the completion of the Suez Canal expansion project to accommodate larger vessels.

27. **2400**: A potential milestone in humanity's efforts to address climate change and ensure environmental sustainability.

28. **2450**: Estimated year for the potential exhaustion of certain rare Earth minerals, highlighting resource management challenges.

29. **2500**: Hypothetical date for the culmination of various futuristic advancements and technologies.

30. **2550**: Speculated timeframe for potential innovations in communication and integration of technology into daily life.

31. **2600**: Projected year by which advancements in medicine and biotechnology might significantly extend human lifespan.

32. **2650**: Hypothetical year when lunar mining and resource utilization could become a viable industry.

33. **2700**: Speculated timeframe for the potential establishment of habitable colonies on other planets or moons.

34. **2750**: Estimated year for potential breakthroughs in understanding dark matter and dark energy in the universe.

35. **2800**: Hypothetical milestone in the development of AI and its integration into various aspects of society.

36. **2850**: Speculated timeframe for potential efforts to restore and preserve ecosystems that have been lost or degraded.

37. **2900**: Projected year for the potential realization of a global unified governance or diplomatic system.

38. **2950**: Hypothetical date for the development of advanced propulsion systems enabling faster space travel.

39. **3000**: A symbolic milestone that prompts reflection on humanity's journey and future possibilities.

40. **3050**: Speculated timeframe for potential advances in quantum computing and its implications for technology.

41. **3100**: Projected year for potential solutions to global challenges related to water scarcity and distribution.

42. **3150**: Hypothetical date for potential breakthroughs in understanding the nature of consciousness and the mind.

43. **3200**: Speculated timeframe for potential efforts to address and mitigate the impact of potential asteroid threats.

44. **3250**: Estimated year for potential developments in terraforming and creating habitable environments on other planets.

45. **3300**: Hypothetical milestone in the study of fundamental physics and the potential for new scientific paradigms.

46. **3350**: Projected year for potential advancements in energy generation and harnessing new sources of power.

47. **3400**: Speculated timeframe for potential innovations in transportation and space exploration infrastructure.

48. **3450**: Estimated year for potential advancements in understanding and harnessing fusion as a clean energy source.

49. **3500**: A distant milestone that prompts contemplation of humanity's enduring presence in the universe.

50. **3550**: Speculated timeframe for potential developments in transhumanism and the merging of humans with advanced technology.

[nicola-corbellini]

@KevinZhang19870314 I'm sorry to hear that.

This depends on your use case, but another solution could be splitting the document in advance.
Namely, let your query be:

How many items in this world events trending?

you could split the list in two or three lists, all starting with World events trending:. Given the query you are using, I would expect the semantic search to retrieve the three lists ending up together in the prompt as context.

What do you think?

Alternatively, as anticipated in the previous comment, we could think about a hook to filter the retrieval with custom metadata. E.g. you could tag all the chunks with metadata={"topic": "events_trending"} and use this as a filter at retrieval time.
For this solution, I'd ask @pieroit what he thinks and if he has suggestion on the topic in general.

[pieroit]

@KevinZhang19870314 welcome!
I agree with @nicola-corbellini, if you have special needs on how the text is to be splitted, use hooks. (There are hooks both for splitting and storing the docs)

Another option is to write a tool that retrieves exactly the info you need in the way you need it.

There are at least 4/5 ways to do this, let us know

[KevinZhang19870314]

@pieroit Thank you for your response, yes, I can do it in hooks or tools, but its not a general method, I have several docs, some of them need splitting like this, some need splitting like that.

So, maybe we can category the docs per knowleage base? My thoughts is:

1. Create knowleage base A;
2. Choose the specific plugins for A from plugins marketing;
3. Upload files to A;
4. Chat with A.

Then, all the docs in A will be well handled, for other docs need to be splitted differently, we can create another knowleage base B, and etc.

Use this way, we can category the files/docs, rather than upload all files together, use the unified plugins for all docs.