Benchmark existing techniques using evaluation harness

[mrm1001]

Context on benchmark work

• goal number 1 is to give user practical guidance on what techniques to try out on their dataset/use case

• goal number 2 is to show that there is not a “silver bullet” type of solution, that it depends on the dataset and use case, but that Haystack can support them all

• goal number 3 is to showcase advanced evaluation/experimentation API (most advanced compared to competitors)

• it’s not a research paper, so should not be too “academic” (i.e. not too restricted in terms of metrics or datasets to use, not meant to be peer-reviewed or submitted to an academic conference)

• Datasets

Tasks

Beta Give feedback

1. Create a new repository that will have all the code for the benchmark study
2. Select 4 or 5 datasets #7628
   P1 +1
   
3. Run evaluations on selected datasets to optimise basic RAG pipeline #7629
   P1 +1