Create QA Kedro test projects for stress testing and performance and evaluation

[AhdraMeraliQB]

Description

There are several features across the Kedro organisation that could benefit from manual testing on large projects to evaluate performance. The proposal is to create several kedro projects of varying size that can be used to test and experiment with.

This could be particularly useful for testing Viz features, CC @rashidakanchwala, @NeroOkwa

These would ideally live in a separate repo within the organisation, and make use of many available features (namespaces, hooks, parameters, etc). I propose the following scales of project:

• 10 nodes
• 100 nodes
• 1000 nodes
• 10000 nodes

Numbers inspired by this comment

Note: After completion of this, it should become part of the release process to migrate these projects to the newest version. This provides the added opportunity to test project migration during major releases.

[noklam]

I found that I have an old project that dynamically creates some big number of nodes. The question is how realistic this pipeline need to be? Is it mainly for QA purpose or it should reflect a realistic data science project that make use of the features in a reasonable way?
https://github.com/noklam/kedro-example/tree/master/stress-test-pipeline

[astrojuanlu]

To do a comprehensive performance analysis, we would need a family of projects, comprising

• An empty project (so that we can measure just "cold startup" time, essentially what I did in Kedro initialization is slow #3033 (comment))
• 1 pipeline with increasingly large number of nodes (essentially what @AhdraMeraliQB proposed in Create QA Kedro test projects for stress testing and performance and evaluation #3790 (comment))
• N pipelines of 1 node
• 1 pipeline and 1 node with increasingly large number of datasets
• ...possibly more?