-
-
Notifications
You must be signed in to change notification settings - Fork 1k
Add Roslyn analyzers to detect incorrect usage of BenchmarkDotNet #2837
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
@dotnet-policy-service agree |
08ffcd6
to
e0bd1d6
Compare
...markDotNet.Analyzers.Tests/AnalyzerTests/Attributes/ParamsAllValuesAttributeAnalyzerTests.cs
Outdated
Show resolved
Hide resolved
...markDotNet.Analyzers.Tests/AnalyzerTests/Attributes/ParamsAllValuesAttributeAnalyzerTests.cs
Outdated
Show resolved
Hide resolved
...hmarkDotNet.Analyzers/BenchmarkDotNet.Analyzers.Tests/BenchmarkDotNet.Analyzers.Tests.csproj
Outdated
Show resolved
Hide resolved
src/BenchmarkDotNet.Disassembler.x64/BenchmarkDotNet.Disassembler.x64.csproj
Outdated
Show resolved
Hide resolved
I'm not sure whether the analyzers should be automatically enabled with the base BenchmarkDotNet package or be opt-in via its own NuGet package, what do you think? |
They should be enabled by default. |
So maybe the VSIX package project can be removed then as the analyzer can be referenced through an analyzer project reference. |
I actually think the analyzer should be included directly into the annotations package. Otherwise, it was found that a separate analyzer package pulls in too many unnecessary dependencies. It's a bit complicated to set up the build to do it, though, so I can do it separately after this is merged if you want. [Edit] Or I can push to your branch after your changes are complete. |
I'm getting |
4cec602
to
de94c5b
Compare
You need to import common.props in the analyzer project, too. |
de94c5b
to
c18a417
Compare
Solved it. I also needed to add the public key of the assembly to the InternalsVisibleTo attribute. |
Do I just reference the analyzer project from the annotations project? Or do I need to do something special for the analyzers to activate for the user? Mind that we of course don't want the analyzers to activate for the annotations project, but transitively for the user. |
Yes that's sufficient for now. |
Also you should move the analyzers test project to under the tests/ directory (and move the analyzers project up 1 level). |
src/BenchmarkDotNet.Analyzers/General/BenchmarkClassAnalyzer.cs
Outdated
Show resolved
Hide resolved
} | ||
|
||
var benchmarkMethods = benchmarkMethodsBuilder.ToImmutable(); | ||
if (benchmarkMethods.Length == 0) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Base/derived classes without benchmarks may still apply other attributes like [GlobalSetup]
, [Params]
, etc. These types should be analyzed as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would split each attribute analysis to a separate method and call them all.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it valid that the top class (the one referenced in the BenchmarkRunner
) does not have any benchmark methods, but its ancestor classes do?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes
src/BenchmarkDotNet.Annotations/BenchmarkDotNet.Annotations.csproj
Outdated
Show resolved
Hide resolved
… from BenchmarkDotNet.Annotations
Is |
It's per category. |
… must be non-abstract and generic * Benchmark classes are allowed to be generic if they are either abstract or annotated with at least one [GenericTypeArguments] attribute * Assume that a class can be annotated with more than one [GenericTypeArguments] attribute
* Add a rule that the benchmark class referenced in the type argument of the BenchmarkRunner.Run method cannot be abstract
…hrough its ancestors
…arameter created by using a typeof expression
…ed with the Benchmark attribute when analyzing GenericTypeArguments attribute rules
if (actualValueTypeSymbol != null && actualValueTypeSymbol.TypeKind != TypeKind.Error) | ||
{ | ||
var conversionSummary = context.Compilation.ClassifyConversion(actualValueTypeSymbol, expectedValueTypeSymbol); | ||
if (!conversionSummary.IsImplicit) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this work for this?
[Params(1, 2)]
public byte num;
I think the compiler treats the numbers as int
, and thus they are not implicitly convertible to byte
, but it works today because we generate source with integer literals, which are implicitly convertible to byte
(if they are in range).
Please add tests for it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, I'll test it out tomorrow. I've only added a test for the other way around, byte to int.
I remember when I first started building this analyzer, I actually tested for the exact type, but then I inspected the source code that BDN generated and realized the requirement is implicit conversion so I adjusted the logic and added applicable tests.
This PR introduces an extensive set of analyzers that warns the user of incorrect usage of BenchmarkDotNet. This is something that has been asked since 2017 but has yet to be included as of this date. BDN has a set of validators that use reflection to detect errors but they are only triggered after the benchmark code has been compiled and is about to run.
I had the idea to implement this in 2022 but the testing framework back then wasn't trivial to use so I gave up in the end. Today, the Roslyn analyzer testing is completely testing framework-agnostic, making things considerably easier. It's also trivial to add multiple source files, references and framework assemblies in order to test your analyzer precisely the way you want.
All unit tests are implemented using xUnit v2.
With these analyzers, developers can detect errors early and solve them immediately. The descriptions are very clear and succinct, guiding the user and explaining the reasoning behind the specific rule.
Here's a list of currently implemented analyzers. There are still some remaining but I believe this is a good start and covers most common usage errors. The rest is up for grabs and can be added along the way.
BenchmarkRunner.Run<BenchmarkClass>()
and the benchmark classBenchmarkClass
(or any of its inherited classes) has no public methods marked with the[Benchmark]
attributeTODO
[ArgumentsSource]
points to a valid method[ParamsSource]
points to a valid method[GenericTypeArguments]
attribute per annotated classSee #2666 for discussion as well as #389.