Provide an injection point for custom test reporting functionality #15775
Conversation
The current test logging implementation makes it difficult to know what tests failed during test execution. This especially affects GitHub Actions, which lack any test reporting functionality, and require developers to download logs and search for failures. The injection point provides the ability to mitigate the issue by allowing to inject a custom test reporter project before a test run is terminated (in an event of a failure).
|
/cc: @radical |
| <!-- | ||
| Allow running an arbitrary target after the tests have been executed. This is useful for running custom test reporting tools. | ||
| --> | ||
| <MSBuild Projects="$(OnTestsExecutedProject)" |
There was a problem hiding this comment.
I assume you need this push mechanism here because the Error statement below ends all execution and so you can't sequence a target after it?
If that's the case, I would rather put the Error task into a separate target that runs after this one so that we can sequence additional targets between them, i.e.:
<Target Name="OnTestsExecutedAnalysis" BeforeTargets="RunTestsReportError" DepensOnTargets="RunTests">
...
</Target>There was a problem hiding this comment.
That allows for more extensibility in general and avoids the use of the MSBuild task here.
There was a problem hiding this comment.
Thanks for the idea, though it would be a more invasive change. Are we comfortable with it?
There was a problem hiding this comment.
Yes I'm fine with changing targets here. If we keep the RunTests target which depends on the both other targets then there shouldn't be any potential fallout.
Example: RunTests -> ExecuteTests, ReportTestResults
| <!-- | ||
| Allow running an arbitrary target after the tests have been executed. This is useful for running custom test reporting tools. | ||
| --> | ||
| <MSBuild Projects="$(OnTestsExecutedProject)" |
There was a problem hiding this comment.
At least for xunit.v3 where we expect most repos to use MTP, I have same comment as in the Aspire PR. Whatever logic that will be done, why can't it be served via MTP extensibility?
If the whole issue is that Arcade redirects stdout to a log file, it's already possible to disable that with <TestCaptureOutput>false</TestCaptureOutput>
There was a problem hiding this comment.
Whatever logic that will be done, why can't it be served via MTP extensibility?
Are there any samples to show how this can be achieved with the MTPE?
If the whole issue is that Arcade redirects stdout to a log file, it's already possible to disable that with
<TestCaptureOutput>false</TestCaptureOutput>
No, this is not the purpose of this change. I want to be able to execute an arbitrary target (in the case of Aspire - report successful and failed tests) between the finish of the tests execution and the reported error.
There was a problem hiding this comment.
And you are sure want that even though it will only work with Arcade's Test target but not with dotnet test or inside VS with Test Explorer?
The current test logging implementation makes it difficult to know what tests failed during test execution.
This especially affects GitHub Actions, which lack any test reporting functionality, and require developers to download logs and search for failures.
The injection point provides the ability to mitigate the issue by allowing to inject a custom test reporter project before a test run is terminated (in an event of a failure).
Before:
After
Examples
A real-life example can be seen here: https://github.com/RussKie/aspire/actions/runs/14617272672/job/41008327227#step:6:303. This is achieved with https://github.com/dotnet/aspire/pull/8811/files#diff-2f47f8038028665e98b009c5afd2a2fba53b4c62d24a96729a50567dbe4e80de.