Skip to content

Conversation

silkfire
Copy link

@silkfire silkfire commented Oct 10, 2025

This PR introduces an extensive set of analyzers that warns the user of incorrect usage of BenchmarkDotNet. This is something that has been asked since 2017 but has yet to be included as of this date. BDN has a set of validators that use reflection to detect errors but they are only triggered after the benchmark code has been compiled and is about to run.

I had the idea to implement this in 2022 but the testing framework back then wasn't trivial to use so I gave up in the end. Today, the Roslyn analyzer testing is completely testing framework-agnostic, making things considerably easier. It's also trivial to add multiple source files, references and framework assemblies in order to test your analyzer precisely the way you want.
All unit tests are implemented using xUnit v2.

With these analyzers, developers can detect errors early and solve them immediately. The descriptions are very clear and succinct, guiding the user and explaining the reasoning behind the specific rule.

Here's a list of currently implemented analyzers. There are still some remaining but I believe this is a good start and covers most common usage errors. The rest is up for grabs and can be added along the way.

Rule ID Category Severity Title Description Comment
BDN1000 Usage Error Benchmark class (or any of its ancestors) has no annotated method(s) The referenced benchmark class (or any of its inherited classes) must have at least one method annotated with the [Benchmark] attribute Triggers when invoking BenchmarkRunner.Run<BenchmarkClass>() and the benchmark class BenchmarkClass (or any of its inherited classes) has no public methods marked with the [Benchmark] attribute
BDN1003 Usage Error Benchmark classes must be public A benchmark class referenced in the BenchmarkRunner.Run method must be public
BDN1003 Usage Error Benchmark classes must be non-abstract A benchmark class referenced in the BenchmarkRunner.Run method must be non-abstract
BDN1004 Usage Error Benchmark classes must be unsealed A benchmark class referenced in the BenchmarkRunner.Run method must be unsealed
BDN1100 Usage Error Benchmark classes annotated with a [GenericTypeArguments] attribute must be non-abstract A benchmark class annotated with a [GenericTypeArguments] attribute must be non-abstract
BDN1101 Usage Error Benchmark classes can only be generic if they're either abstract or annotated with a [GenericTypeArguments] attribute Benchmark classes can only be generic if they're either abstract or annotated with a [GenericTypeArguments] attribute
BDN1102 Usage Error Benchmark classes annotated with a [GenericTypeArguments] attribute must be generic A benchmark class annotated with a [GenericTypeArguments] attribute must be generic, having between one to three type parameters
BDN1103 Usage Error Number of type arguments passed to a [GenericTypeArguments] attribute must match the number of type parameters on the targeted benchmark class The number of type arguments passed to a [GenericTypeArguments] attribute must match the number of type parameters on the targeted benchmark class
BDN1104 Usage Error Benchmark methods must be public A method annotated with the [Benchmark] attribute must be public
BDN1105 Usage Error Benchmark methods must be non-generic A method annotated with the [Benchmark] attribute must be non-generic
BDN1106 Usage Error Benchmark classes must be non-static A benchmark class must be an instance class
BDN1107 Usage Error Only one benchmark method can be baseline Only one benchmark method can be marked as baseline
BDN1200 Usage Error Only one parameter attribute can be applied to a field Parameter attributes are mutually exclusive; only one of the attributes [Params], [ParamsSource] or [ParamsAllValues] can be applied to a field at any one time
BDN1201 Usage Error Only one parameter attribute can be applied to a property Parameter attributes are mutually exclusive; only one of the attributes [Params], [ParamsSource] or [ParamsAllValues] can be applied to a property at any one time
BDN1202 Usage Error Fields annotated with a parameter attribute must be public A field annotated with a parameter attribute must be public
BDN1203 Usage Error Properties annotated with a parameter attribute must be public A property annotated with a parameter attribute must be public
BDN1204 Usage Error Fields annotated with a parameter attribute cannot be read-only Parameter attributes are not valid on fields with a readonly modifier
BDN1205 Usage Error Parameter attributes are not valid on constant field declarations Parameter attributes are not valid on constant field declarations
BDN1206 Usage Error Properties annotated with a parameter attribute cannot have an init-only setter A property annotated with a parameter attribute must have a public, assignable setter i.e. { set; }
BDN1207 Usage Error Properties annotated with a parameter attribute must have a public setter A property annotated with a parameter attribute must have a public setter; make sure that the access modifier of the setter is empty and that the property is not an auto-property or an expression-bodied property.
BDN1300 Usage Error The [Params] attribute must include at least one value The [Params] attribute requires at least one value. No values were provided, or an empty array was specified.
BDN1301 Usage Error Type of all value(s) passed to the [Params] attribute must match the type of (or be implicitly convertible to) the annotated field or property The type of each value provided to the [Params] attribute must match the type of (or be implicitly convertible to) the field or property it is applied to
BDN1302 Usage Warning Unnecessary single value passed to [Params] attribute Providing a single value to the [Params] attribute is unnecessary. This attribute is only useful when provided two or more values.
BDN1303 Usage Error The [ParamsAllValues] attribute cannot be applied to fields or properties of enum types marked with [Flags] The [ParamsAllValues] attribute cannot be applied to a field or property of an enum type marked with the [Flags] attribute. Use this attribute only with non-flags enum types, as [Flags] enums support bitwise combinations that cannot be exhaustively enumerated.
BDN1304 Usage Error The [ParamsAllValues] attribute is only valid on fields or properties of enum or bool type and nullable type for another allowed type The [ParamsAllValues] attribute can only be applied to a field or property of enum or bool type (or nullable of these types)
BDN1400 Usage Error [Arguments] attribute can only be used on methods annotated with the [Benchmark] attribute The [Arguments] attribute can only be used on methods annotated with the [Benchmark] attribute
BDN1401 Usage Error Benchmark methods without [Arguments] attribute(s) cannot declare parameters This method declares one or more parameters but is not annotated with any [Arguments] attributes. To ensure correct argument binding, methods with parameters must explicitly be annotated with one or more [Arguments] attributes. Either add the [Arguments] attribute(s) or remove the parameters.
BDN1402 Usage Error Number of values passed to an [Arguments] attribute must match the number of parameters declared in the targeted benchmark method The number of values passed to an [Arguments] attribute must match the number of parameters declared in the targeted benchmark method
BDN1403 Usage Error Values passed to an [Arguments] attribute must match exactly the parameters declared in the targeted benchmark method in both type (or be implicitly convertible to) and order The values passed to an [Arguments] attribute must match the parameters declared in the targeted benchmark method in both type (or be implicitly convertible to) and order

TODO

  • Check that the argument to [ArgumentsSource] points to a valid method
  • Check that the argument to [ParamsSource] points to a valid method
  • Check that each corresponding type argument is the same across all usages of a [GenericTypeArguments] attribute per annotated class

See #2666 for discussion as well as #389.

@silkfire
Copy link
Author

@dotnet-policy-service agree

@timcassell timcassell linked an issue Oct 10, 2025 that may be closed by this pull request
@silkfire
Copy link
Author

I'm not sure whether the analyzers should be automatically enabled with the base BenchmarkDotNet package or be opt-in via its own NuGet package, what do you think?

@timcassell
Copy link
Collaborator

They should be enabled by default.

@silkfire
Copy link
Author

So maybe the VSIX package project can be removed then as the analyzer can be referenced through an analyzer project reference.

@timcassell
Copy link
Collaborator

timcassell commented Oct 10, 2025

So maybe the VSIX package project can be removed then as the analyzer can be referenced through an analyzer project reference.

I actually think the analyzer should be included directly into the annotations package. Otherwise, it was found that a separate analyzer package pulls in too many unnecessary dependencies. It's a bit complicated to set up the build to do it, though, so I can do it separately after this is merged if you want. [Edit] Or I can push to your branch after your changes are complete.

@silkfire
Copy link
Author

I'm getting Referenced assembly 'BenchmarkDotNet.Analyzers, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' does not have a strong name. when trying to compile the test project.

@timcassell
Copy link
Collaborator

You need to import common.props in the analyzer project, too.

@silkfire
Copy link
Author

Solved it. I also needed to add the public key of the assembly to the InternalsVisibleTo attribute.

@silkfire
Copy link
Author

silkfire commented Oct 10, 2025

So maybe the VSIX package project can be removed then as the analyzer can be referenced through an analyzer project reference.

I actually think the analyzer should be included directly into the annotations package. Otherwise, it was found that a separate analyzer package pulls in too many unnecessary dependencies. It's a bit complicated to set up the build to do it, though, so I can do it separately after this is merged if you want. [Edit] Or I can push to your branch after your changes are complete.

Do I just reference the analyzer project from the annotations project? Or do I need to do something special for the analyzers to activate for the user? Mind that we of course don't want the analyzers to activate for the annotations project, but transitively for the user.

@timcassell
Copy link
Collaborator

Do I just reference the analyzer project from the annotations project?

Yes that's sufficient for now.

@timcassell
Copy link
Collaborator

Also you should move the analyzers test project to under the tests/ directory (and move the analyzers project up 1 level).

}

var benchmarkMethods = benchmarkMethodsBuilder.ToImmutable();
if (benchmarkMethods.Length == 0)
Copy link
Collaborator

@timcassell timcassell Oct 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Base/derived classes without benchmarks may still apply other attributes like [GlobalSetup], [Params], etc. These types should be analyzed as well.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would split each attribute analysis to a separate method and call them all.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it valid that the top class (the one referenced in the BenchmarkRunner) does not have any benchmark methods, but its ancestor classes do?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

@silkfire
Copy link
Author

Is Baseline = true unique per benchmark category or per benchmark class?

@timcassell
Copy link
Collaborator

Is Baseline = true unique per benchmark category or per benchmark class?

It's per category.

… must be non-abstract and generic

* Benchmark classes are allowed to be generic if they are either abstract or annotated with at least one [GenericTypeArguments] attribute
* Assume that a class can be annotated with more than one [GenericTypeArguments] attribute
* Add a rule that the benchmark class referenced in the type argument of the BenchmarkRunner.Run method cannot be abstract
…arameter created by using a typeof expression
…ed with the Benchmark attribute when analyzing GenericTypeArguments attribute rules
if (actualValueTypeSymbol != null && actualValueTypeSymbol.TypeKind != TypeKind.Error)
{
var conversionSummary = context.Compilation.ClassifyConversion(actualValueTypeSymbol, expectedValueTypeSymbol);
if (!conversionSummary.IsImplicit)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this work for this?

[Params(1, 2)]
public byte num;

I think the compiler treats the numbers as int, and thus they are not implicitly convertible to byte, but it works today because we generate source with integer literals, which are implicitly convertible to byte (if they are in range).

Please add tests for it.

Copy link
Author

@silkfire silkfire Oct 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, I'll test it out tomorrow. I've only added a test for the other way around, byte to int.

I remember when I first started building this analyzer, I actually tested for the exact type, but then I inspected the source code that BDN generated and realized the requirement is implicit conversion so I adjusted the logic and added applicable tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Analyzers for Framework

2 participants