New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Codecov test coverage #1397
Conversation
I ran this on my fork: With unit-test coverage only, we have 29.67% coverage for backend. With integration test coverage, we have 58.16% coverage for backend. NOTE: for the CodeCov to actually work on CI, someone with write access to the repo, a.k.a. maintainer, needs to authorize CodeCov to access this repo (by logging into https://app.codecov.io/ using their GitHub account, and link opendevin/opendevin), and then add the token to repository secrets: https://docs.codecov.com/docs/adding-the-codecov-token |
As someone who got bit by the massive codecov breach this makes me a bit nervous 😅 Are there any open source python coverage tools we could use? Or is this the best option? |
That's very interesting. I've been using codecov for years but I wasn't aware of this. If I understand correctly, this is more of a concern when hosting codecov in your own network? On a public GitHub repo, there's no concern regarding code leakage. What about secrets? According to https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions, secrets are not available to GitHub Actions (CI) unless they are explicitly provided. Currently we only expose secrets (LLM api key) to dogfood tasks (issue solver and pr reviewer). The test actions that CodeCov runs in, have no access to secrets.
Codecov-action, the GitHub Action that we use to upload the xml reports, is indeed open-source: https://github.com/codecov/codecov-action Codecov server itself is also "source available": https://github.com/codecov/self-hosted |
There are alternatives to codecov but according to https://github.com/marketplace?category=code-quality&type=actions&verification=&query=sort%3Apopularity-desc+coverage, codecov is the most popular one. Apparently, it's far more popular than other ones. |
The unit test failure seems to be a transient failure (might be fixed by #1381). I don't have permission to rerun the CI. |
Thanks @li-boxuan ! |
Now that we have a fair amount of test coverage, we could start tracking coverage using Codecov. I know we have frontend tests as well but they don't seem to be part of CI yet. I'll leave that to our frontend experts to solve :)