Skip to content

Provide an injection point for custom test reporting functionality #15775

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Provide an injection point for custom test reporting functionality #15775

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions Documentation/ArcadeSdk.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1008,6 +1008,57 @@ Properties that define TargetFramework for use by projects so their targeting ea

Set to `true` to indicate that the project is a test utility project. Such are projects that offer utilities to run tests. Example: `Microsoft.DotNet.XUnitExtensions.csproj` in Arcade. This makes are mark them as non-shipping and excluded from source build.

### `OnTestsExecutedProject` (string)

The path to an MSBuild projects in the consumer repository with `OnTestsExecuted` target, that will be executed whenever project's test have been executed. This can be useful for custom test reporting functionality.

For example:

`tests/Directory.Build.props` in the consumer project:
```xml
<Project>
<Import Project="$([MSBuild]::GetPathOfFileAbove('Directory.Build.props', '$(MSBuildThisFileDirectory)../'))" />

<PropertyGroup>
<OnTestsExecutedProperties>$(OnTestsExecutedProperties);TestReporter=true</OnTestsExecutedProperties>
<OnTestsExecutedProject>$(RepositoryEngineeringDir)TestReporter.proj</OnTestsExecutedProject>
</PropertyGroup>
</Project>
```

`eng/TestReporter.proj` in the consumer project:
```xml
<Project>

<!--
Available properties:
- Project: The project name of the test project.
- LogFile: The full path to the test log file (stdout).
- TrxFile: The full path to the test trx file.
- ExitCode: The exit code of the test run.
- TestEnvironment: The test environment (e.g., 'net9.0|x64').
- TestAssembly: The full path to the test assembly that was executed.
- TargetFramework: The target framework of the test assembly (e.g., 'net9.0').
- any custom properties defined in `OnTestsExecutedProperties`.
-->
<Target Name="OnTestsExecuted" DependsOnTargets="TestsExecuted;TestsFailed">
</Target>

<Target Name="TestsExecuted" Condition=" '$(ExitCode)' == 0 ">
<Message Importance="high" Text="Running TestReporter for successful tests in $(TestEnvironment)..." />
</Target>

<Target Name="TestsFailed" Condition=" '$(ExitCode)' != 0 ">
<Message Importance="high" Text="Running TestReporter for failed tests in $(TestEnvironment)..." />
</Target>

</Project>
```

### `OnTestsExecutedProperties` (list of strings)

List of custom properties to be passed to `OnTestsExecutedProject` whenever project's test have been executed.

### `SkipTests` (bool)

Set to `true` in a test project to skip running tests.
Expand Down
10 changes: 10 additions & 0 deletions src/Microsoft.DotNet.Arcade.Sdk/tools/VSTest.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,16 @@
<_ResultsFileToDisplay Condition="!Exists('$(_ResultsFileToDisplay)')">%(TestToRun.ResultsStdOutPath)</_ResultsFileToDisplay>
</PropertyGroup>

<!--
Allow running an arbitrary target after the tests have been executed. This is useful for running custom test reporting tools.
-->
<MSBuild Projects="$(OnTestsExecutedProject)"
Properties="$(OnTestsExecutedProperties);Project=$(MSBuildProjectName);LogFile=%(TestToRun.ResultsStdOutPath);TrxFile=%(TestToRun.ResultsTrxPath);ExitCode=$(_TestErrorCode);TestEnvironment=$(_TestEnvironment);TestAssembly=$(_TestAssembly);TargetFramework=%(TestToRun.TargetFramework)"
Targets="OnTestsExecuted"
SkipNonexistentProjects="true"
Condition=" Exists('$(OnTestsExecutedProject)') "
/>

<!--
Ideally we would set ContinueOnError="ErrorAndContinue" so that when a test fails in multi-targeted test project
we'll still run tests for all target frameworks. ErrorAndContinue doesn't work well on Linux though: https://github.com/Microsoft/msbuild/issues/3961.
Expand Down
10 changes: 10 additions & 0 deletions src/Microsoft.DotNet.Arcade.Sdk/tools/XUnit/XUnit.Runner.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,16 @@
<_ResultsFileToDisplay Condition="!Exists('$(_ResultsFileToDisplay)')">%(TestToRun.ResultsStdOutPath)</_ResultsFileToDisplay>
</PropertyGroup>

<!--
Allow running an arbitrary target after the tests have been executed. This is useful for running custom test reporting tools.
-->
<MSBuild Projects="$(OnTestsExecutedProject)"
Properties="$(OnTestsExecutedProperties);Project=$(MSBuildProjectName);LogFile=%(TestToRun.ResultsStdOutPath);TrxFile=%(TestToRun.ResultsTrxPath);ExitCode=$(_TestErrorCode);TestEnvironment=$(_TestEnvironment);TestAssembly=$(_TestAssembly);TargetFramework=%(TestToRun.TargetFramework)"
Targets="OnTestsExecuted"
SkipNonexistentProjects="true"
Condition=" Exists('$(OnTestsExecutedProject)') "
/>

<!--
Ideally we would set ContinueOnError="ErrorAndContinue" so that when a test fails in multi-targeted test project
we'll still run tests for all target frameworks. ErrorAndContinue doesn't work well on Linux though: https://github.com/Microsoft/msbuild/issues/3961.
Expand Down
10 changes: 10 additions & 0 deletions src/Microsoft.DotNet.Arcade.Sdk/tools/XUnitV3/XUnitV3.Runner.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,16 @@
<_ResultsFileToDisplay Condition="!Exists('$(_ResultsFileToDisplay)')">%(TestToRun.ResultsStdOutPath)</_ResultsFileToDisplay>
</PropertyGroup>

<!--
Allow running an arbitrary target after the tests have been executed. This is useful for running custom test reporting tools.
-->
<MSBuild Projects="$(OnTestsExecutedProject)"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume you need this push mechanism here because the Error statement below ends all execution and so you can't sequence a target after it?

If that's the case, I would rather put the Error task into a separate target that runs after this one so that we can sequence additional targets between them, i.e.:

<Target Name="OnTestsExecutedAnalysis" BeforeTargets="RunTestsReportError" DepensOnTargets="RunTests">

...

</Target>

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That allows for more extensibility in general and avoids the use of the MSBuild task here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the idea, though it would be a more invasive change. Are we comfortable with it?

Copy link
Member

@ViktorHofer ViktorHofer Apr 24, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I'm fine with changing targets here. If we keep the RunTests target which depends on the both other targets then there shouldn't be any potential fallout.

Example: RunTests -> ExecuteTests, ReportTestResults

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least for xunit.v3 where we expect most repos to use MTP, I have same comment as in the Aspire PR. Whatever logic that will be done, why can't it be served via MTP extensibility?

If the whole issue is that Arcade redirects stdout to a log file, it's already possible to disable that with <TestCaptureOutput>false</TestCaptureOutput>

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whatever logic that will be done, why can't it be served via MTP extensibility?

Are there any samples to show how this can be achieved with the MTPE?

If the whole issue is that Arcade redirects stdout to a log file, it's already possible to disable that with <TestCaptureOutput>false</TestCaptureOutput>

No, this is not the purpose of this change. I want to be able to execute an arbitrary target (in the case of Aspire - report successful and failed tests) between the finish of the tests execution and the reported error.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And you are sure want that even though it will only work with Arcade's Test target but not with dotnet test or inside VS with Test Explorer?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

Properties="$(OnTestsExecutedProperties);Project=$(MSBuildProjectName);LogFile=%(TestToRun.ResultsStdOutPath);TrxFile=%(TestToRun.ResultsTrxPath);ExitCode=$(_TestErrorCode);TestEnvironment=$(_TestEnvironment);TestAssembly=$(_TestAssembly);TargetFramework=%(TestToRun.TargetFramework)"
Targets="OnTestsExecuted"
SkipNonexistentProjects="true"
Condition=" Exists('$(OnTestsExecutedProject)') "
/>

<!--
Ideally we would set ContinueOnError="ErrorAndContinue" so that when a test fails in multi-targeted test project
we'll still run tests for all target frameworks. ErrorAndContinue doesn't work well on Linux though: https://github.com/Microsoft/msbuild/issues/3961.
Expand Down
Loading