Make it easy to run functional tests with multiple combinations of configurations.

[superhindupur]

Motivation

Currently, the functional tests spin up a new instance of the mock frontend API service for each combination of app configuration.

For e.g., we have one group of tests running for SHOULD_USE_DATA= true and another for false.

With the addition of functional tests for Semantic Kernel in #800, it became apparent that there are multiple combinations of app configs that we want to test, and the current method of spinning up a new mock server instance will not scale. Making it easy to write functional tests for multiple combinations of configs will encourage developers to write more of them, ultimately increasing the stability of CWYD.

One way of solving this problem would be to:

• Allow declaring app config per test file rather than per test folder.
• Allow reloading app config while the server is running, removing the need for restarting a server when there is a new config combination.

How would you feel if this feature request was implemented?

Requirements

A list of requirements to consider this feature delivered

• The ability to run functional tests with multiple app config combinations without needing to run each combination on a new mock server instance.

Tasks

To be filled in by the engineer picking up the issue

• Task 1
• Task 2
• ...

[ross-p-smith]

Closing this as this is a nice to have, but not been prioritised