Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add weighting to slow tests to better split buckets #117234

Draft
wants to merge 33 commits into
base: dev
Choose a base branch
from
Draft

Conversation

bdraco
Copy link
Member

@bdraco bdraco commented May 11, 2024

Proposed change

Add weighting to slow tests to better split buckets

Some of the tests take a lot longer so we end up with one bucket that runs for 15m and one bucket that runs for 6m

Ideally each bucket takes ~8m

Test runs are ranging from ~6m-~11m instead of ~5m-~15m after adding weighting to the slowest tests. While we could add a few more to the list, its a bit of diminishing returns to expand the list, and its much better to start improving the tests.

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

  • This PR fixes or closes issue: fixes #
  • This PR is related to issue:
  • Link to documentation pull request:

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • I have followed the perfect PR recommendations
  • The code has been formatted using Ruff (ruff format homeassistant tests)
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
  • Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

@home-assistant home-assistant bot added cla-signed small-pr PRs with less than 30 lines. labels May 11, 2024
Copy link
Member

@joostlek joostlek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What problem is this tackling?

@bdraco
Copy link
Member Author

bdraco commented May 11, 2024

Some of the tests take a lot longer so we end up with one bucket that runs for 15m and one bucket that runs for 6m

Ideally each bucket takes ~8m

@joostlek
Copy link
Member

Aah, so we count slower tests as more tests so they are more evenly. Sounds cool!

#
TEST_WEIGHTS = {
"tests/test_circular_imports.py": 45,
"tests/test_bootstrap.py": 2,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A lot of these can be optimized and than removed from this list.

@bdraco
Copy link
Member Author

bdraco commented May 12, 2024

The idea longer term is to set the durations to 100 for each group and than make a script that uses the GitHub api to pull the log file for the last run and update it every so often. We probably only need the top 100 for each one as anything past that will not be significant enough to matter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla-signed small-pr PRs with less than 30 lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants