Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] Fix Kotest based tests #5022

Merged
merged 9 commits into from
May 31, 2024
Merged

[core] Fix Kotest based tests #5022

merged 9 commits into from
May 31, 2024

Conversation

adangel
Copy link
Member

@adangel adangel commented May 17, 2024

Describe the PR

The changes are in pmd-java only:

  • Renamed BranchingExprsTestCases to BranchingExprsTests so that it is picked up by surefire
  • Fixed BranchingExprsTests (the test code sample wouldn't actually be compileable, so I guess, expected a wildcard for the type is ok)
  • Added a check in pmd-java's ParserTestSpec, so that test sets ("container" in kotest speak) that don't have any test cases, are failing. This makes sure, that the tests are reported correctly via surefire (only for single test cases the succeed counter is incremented). Now such tests don't end up with ("Tests run: 0...") anymore.
  • Fixed tests in pmd-java, that didn't define test cases yet. I simply wrapped the test code inside a "doTest", which registers a test case. Some tests did this already, but many didn't. This created many whitespace changes due to indentation/reformatting.
  • Update build-tools to pick up Improve surefire reporter build-tools#31 - note: this is only cosmetical, so that junit suites are displayed as indented cases.

Related issues

Ready?

  • Added unit tests for fixed bug/feature
  • Passing all unit tests
  • Complete build ./mvnw clean verify passes (checked automatically by github actions)
  • Added (in-code) documentation (if needed)

@adangel adangel added the in:pmd-internals Affects PMD's internals label May 17, 2024
@adangel adangel added this to the 7.2.0 milestone May 17, 2024
@pmd-test
Copy link

1 Message
📖 No regression tested rules have been changed.

Generated by 🚫 Danger

- introduce parserTestContainer
- use either parserTest or parserTestContainer with should/doTest
- parserTestGroup is now private
  - this is to limit DSL options - with too many options maintainability suffers
  - parserTestContainer should be used instead
  - there were only 2 test classes, that used parserTestGroup
@oowekyala oowekyala added the in:testing About tests of pmd, eg the module pmd-lang-test or pmd-test label May 21, 2024
@oowekyala
Copy link
Member

This looks good, thanks :)

@adangel
Copy link
Member Author

adangel commented May 21, 2024

FYI - I was now investigating the root cause for the following problem: We have a disabled test in TypeDisambiguationTest (disabled via bang) - this is listed correctly when the test is running, but it is missing in the summary. So, disabled kotest tests can be overseen very easily:

$ ./mvnw surefire:test -pl pmd-java -Dtest=TypeDisambiguationTest,CtorInvocMirrorTest,ReportTest
...
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running net.sourceforge.pmd.lang.java.ReportTest
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.136 s -- in net.sourceforge.pmd.lang.java.ReportTest
[INFO] Running net.sourceforge.pmd.lang.java.ast.TypeDisambiguationTest
[INFO]     └─ ↷  TODO Import on demand of class defined in same compilation unit that has an extends clause
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 2.856 s -- in net.sourceforge.pmd.lang.java.ast.TypeDisambiguationTest
[INFO] Running net.sourceforge.pmd.lang.java.types.internal.infer.ast.CtorInvocMirrorTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.206 s -- in net.sourceforge.pmd.lang.java.types.internal.infer.ast.CtorInvocMirrorTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0

It turns out, that the surefire/junit-platform integration is tailored against methods based test frameworks, where each test is based on a method. For kotest, when it reports the tests up to junit-platform, it only provides a class source (and not a method source): https://github.com/kotest/kotest/blob/6f371fbd0af65577f9a5092d3abab017613afa98/kotest-runner/kotest-runner-junit5/src/jvmMain/kotlin/io/kotest/runner/junit/platform/descriptors.kt#L34
When surefire receives such a TestDescriptor, it tries to convert it here https://github.com/apache/maven-surefire/blob/790e332790b1d601cb58e566c29eef3e47353f7d/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java#L366 into classname and methodname, but the method name will always be null for kotest tests. This is used to create a ReportEntry here: https://github.com/apache/maven-surefire/blob/790e332790b1d601cb58e566c29eef3e47353f7d/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java#L218 - so the report entry for the test has a classname, but no method name. These ReportEntries are collected and converted into TestMethodStats instances (https://github.com/apache/maven-surefire/blob/790e332790b1d601cb58e566c29eef3e47353f7d/maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire/report/TestSetRunListener.java#L288). The "classMethodName" will be something like "TestClassName.null" - for all test cases in the same kotest class. These TestMethodStats are later on accumulated: https://github.com/apache/maven-surefire/blob/790e332790b1d601cb58e566c29eef3e47353f7d/maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire/report/DefaultReporterFactory.java#L243 - the grouping happens by "ClassMethodName" (Map<String, List<TestMethodStats>> mergedTestHistoryResult). We still have the one skipped TestMethodStat and a couple of other, but they are dealt here as multiple executions of the same test (probably for the feature to automatically repeat failing tests x times until they succeed - flaky tests). But the tests should actually be dealt as separate test cases...
For now, I don't see how this could be fixed without a change to either surefire or kotest. I think, this problem is also the cause for kotest/kotest#4024 ...

@adangel adangel merged commit 910c098 into pmd:master May 31, 2024
3 checks passed
@adangel adangel deleted the kotest-fixes branch May 31, 2024 09:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in:pmd-internals Affects PMD's internals in:testing About tests of pmd, eg the module pmd-lang-test or pmd-test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[core] Kotest tests aren't picked up by surefire
3 participants