Occasional final response outputting entities

[kevinl424]

• final response occasionally outputs the knowledge entities instead of a correct final response to the question
• tweaking and modifying the system persona does not seem to help
• https://github.com/seyeong-han/KnowledgeGraphRAG did not appear to have this issue, final response in this older agent outputted a valid answer each time
• issue seemed to arise after the top K entity fix where only the top 20 entities were fed into the LLM

[kingjulio8238]

Could this bug potentially arise from the order in which we pass the top K entities into the model? Not including it first or last could potentially help with this

[kevinl424]

Update
Using gpt-3.5-preview and any more advanced models resolves the issue and the final response works quite well. Running Llama 3 8B outputs initial response fine, but when fed all the persona and knowledge entity store information, it tends to get confused and output the wrong thing.

Seems to be largely due to model ability. A potential solution can be some sort of method to detect what ollama models the user has pulled, and be able to select from there. This allows users to have much larger models (llama 3 70B, etc.) to output valid responses more consistently while still running locally. Should also probably include a note that llama 3 8B does not always work with the final response.

[kevinl424]

Further testing using requests library to get response (allows isolation of number tokens used in prompt) suggest that the context is not overflowing. Would like to try 70B but as noted in Ollama repo, you need "32 GB to run the 33B models" and more for 70B.