-
-
Notifications
You must be signed in to change notification settings - Fork 28.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add weighting to slow tests to better split buckets #117234
base: dev
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What problem is this tackling?
Some of the tests take a lot longer so we end up with one bucket that runs for 15m and one bucket that runs for 6m Ideally each bucket takes ~8m |
Aah, so we count slower tests as more tests so they are more evenly. Sounds cool! |
# | ||
TEST_WEIGHTS = { | ||
"tests/test_circular_imports.py": 45, | ||
"tests/test_bootstrap.py": 2, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A lot of these can be optimized and than removed from this list.
The idea longer term is to set the durations to 100 for each group and than make a script that uses the GitHub api to pull the log file for the last run and update it every so often. We probably only need the top 100 for each one as anything past that will not be significant enough to matter. |
Proposed change
Add weighting to slow tests to better split buckets
Some of the tests take a lot longer so we end up with one bucket that runs for 15m and one bucket that runs for 6m
Ideally each bucket takes ~8m
Test runs are ranging from ~6m-~11m instead of ~5m-~15m after adding weighting to the slowest tests. While we could add a few more to the list, its a bit of diminishing returns to expand the list, and its much better to start improving the tests.
Type of change
Additional information
Checklist
ruff format homeassistant tests
)If user exposed functionality or configuration variables are added/changed:
If the code communicates with devices, web services, or third-party tools:
Updated and included derived files by running:
python3 -m script.hassfest
.requirements_all.txt
.Updated by running
python3 -m script.gen_requirements_all
..coveragerc
.To help with the load of incoming pull requests: