quickstart basic - missing qa/llm_tests:?

[richardehughes]

Ran:
llmtune generate config
llmtune run ./config.yml

Things worked well (once I fixed my mistake with Mistral/huggingface repo permissions). The job ran very fast and put results into the "experiment" directory. But the experiment/XXX/results/ directory only has a "results.csv" file in it. I expected there to be results from the qa/llm_tests section in the config.yml file, which looks like this:
qa:
llm_tests:
- jaccard_similarity
- dot_product
- rouge_score
- word_overlap
- verb_percent
- adjective_percent
- noun_percent
- summary_length

Do I have to do something extra to get the qa to run?

[benjaminye]

Hi @richardehughes, we decided to take little pause in adding new features to the CLI until we finish our test suite (for the program, that is). This should be done by today -- and we'll release the LLM tests integration under a new minor version next week.

But you can still run all the test by importing the module directly like so:

from src.qa.qa import LLMTestSuite
from src.qa.qa_tests import LengthTest, JaccardSimilarityTest

# Load prompts, ground_truths, and model_preds from the results.csv file
# Your code here

# Running LLM Tests
tests = [LengthTest(), JaccardSimilarityTest()]
test_suite = LLMTestSuite(tests, prompts, ground_truths, model_preds)
test_suite.run_tests()
test_suite.print_test_results()
test_suite.save_test_results("path/to/save/test_results.csv")

[benjaminye]

This is addressed in release v0.2.2

https://github.com/georgian-io/LLM-Finetuning-Toolkit/releases/tag/v0.2.2