[Possible issue] TEST-junit-jupiter.xml replaced when using the rerunFailingTestsCount from maven-surefire-plugin

[eliasnogueira]

Hi all! 👋🏼

I'm facing a problem (won't say it's a bug because I don't know if it's by design) and need your help figuring out whether it's by design or a bug.

The problem

When the <rerunFailingTestsCount> configuration from the maven-surefire-plugin. is in use, the following happens when I look at the contents of TEST-junit-jupiter.xml:

• First run: generates all the data, meaning all the failures, errors, and successful tests
• N runs (number of retries): shows only the retried ones, ignoring the successful ones from the first run

Why this is a problem

Even though I can see the individual TEST-*.xml files in the target/surefire-reports and it shows the tests regardless its run status the TEST-junit-jupiter.xml shows only the failed ones.

In terms of reporting, it's bad as I can't see the successful ones.

My specific case

My case is that I enrich each <testcase> entry in the TEST-junit-jupiter.xml generated file by adding new <property> entries. So I'm missing all the successful executions.

Experiment

It only happens when I use the <rerunFailingTestsCount> configuration from the maven-surefire-plugin.
When I develop a custom retry extension by implementing the handleTestExecutionException from TestExecutionExceptionHandler, it works as expected, meaning I see all tests.

Confirmation I want

It seems to be a "problem" with the maven-surefire-plugin, but I would like any input/experience/etc. for anyone to understand whether it would be acceptable behaviour.

I'm open to help with anything.
Thank you!

[marcphilipp]

TEST-junit-jupiter.xml sounds like it's written by JUnit's LegacyXmlReportGeneratingListener, not Surefire's XML reporting support. Is that correct?

If so, how are you using that? Could you please share a reproducer project?