39

I am using .NET Core 3.1 in my project (web api, VS2019) and XUnit 2.4.1. Recently I was thinking about adding some performance tests and I came accross this library - BenchmarkDotNet. Since I've already been using XUnit for other tests I wanted to run BenchmarkDotNet from within the XUnit [Fact]s.

I found this post where it is explained that shadow copies of assemblies must be turned off for xunit. So I tried following:

  1. To keep it simple I have created fresh new .net core 3.1 console application project with sample method I wanted to benchmark:
[SimpleJob(RuntimeMoniker.NetCoreApp31)] [MinColumn, MaxColumn, MedianColumn, KurtosisColumn] [HtmlExporter] public class TestScenarios { [Params("test")] public string TextToHash { get; set; } [Benchmark] public string CalculateSha256() { var engine = SHA256.Create(); var hash = engine.ComputeHash(Encoding.ASCII.GetBytes(TextToHash)); return Encoding.ASCII.GetString(hash); } }

Then in Program.cs I have:

class Program { static void Main(string[] args) { BenchmarkRunner.Run<TestScenarios>(); } }

When compiling and runnning the application in Release I verified that BenchmarkDotNet.Artifacts folder with log and benchmark output files were correctly created.

  1. I've added XUnit project to the solution and created simple [Fact] method:
public class DotNetBenchmarkTest { [Fact] public void TestMethod() { BenchmarkRunner.Run<TestScenarios>(); } }

Also building in Release configuration and running this method from test explorer will create BenchmarkDotNet.Artifacts folder in bin/Release/dotnetcoreapp3.1/ but log files are empty.

  1. I've also tried to add xunit.runner.json file with following contents to the root of my XUnit project:
{ "$schema": "https://xunit.net/schema/current/xunit.runner.schema.json", "shadowCopy": false, "methodDisplay": "classAndMethod", "diagnosticMessages": true }

but seems to have no effect on the behavior - I verified that file xunit.runner.json is copied to the output bin directory (Properties->Copy to Output Directory->Copy always).

  1. I also installed xUnit.net Console Runner v2.4.1 and tried to run the tests from the console:
xunit .\path\assembly-name.dll -noshadow

Again no luck. The option -noshadow I found at this github link. Seems like the option was available in the older version but is not listed my help output for xunit console runner v2.4.1 but I gave it a try.

So my question is if it is possible to get DotNetBenchmarkTest working altogether with XUnit and if so, can you point me what am I doing wrong?

Improve this question
3
  • 2
    Did you get a resolution to this? Running via tests, my Benchmark methods just say "N/A" Commented Oct 25, 2020 at 18:11
  • Also make sure you are running in Release mode. At least that is what I had to do. dotnet test -c Release. Commented Jul 7, 2021 at 13:18
  • Try to check the log file in your BenchmarkDotNet.Artifacts folder and deal with any issue there. Mine is "Assembly xx which defines benchmarks references xx; If you own this dependency, please, build it in RELEASE. If you don't, you can disable this policy by using 'config.WithOptions(ConfigOptions.DisableOptimizationsValidator)'. I ended up using the config according to their documentation. Commented Dec 8, 2021 at 22:36

3 Answers 3

Reset to default
2

I have created a wrapper test fixture that runs the benchmarks, collects the output and prints the summary at the end:

public class Benchmarks { private readonly ITestOutputHelper output; public Benchmarks(ITestOutputHelper output) { this.output = output; } [Fact] public void Run_Benchmarks() { var logger = new AccumulationLogger(); var config = ManualConfig.Create(DefaultConfig.Instance) .AddLogger(logger) .WithOptions(ConfigOptions.DisableOptimizationsValidator); BenchmarkRunner.Run<FooBenchmarks>(config); // write benchmark summary output.WriteLine(logger.GetLog()); } } [MemoryDiagnoser] public class FooBenchmarks { [GlobalSetup] public void Setup() { ModuleInitializer.Initialize(); ... } [Benchmark] public async Task ProcessRequest_Benchmark() { ... } }
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

0

I am running BenchmarkDotNet benchmarks within XUnit tests.

Make sure that your project is referencing these NuGet packages

  • xunit
  • xunit.runner.visualstudio
  • Microsoft.NET.Test.Sdk

Then you can run the xunit tests from Visual Studio or with dotnet test via command-line since this is .NET Core project.

xunit.console.runner is capable of running .NET Framework projects.

Check here and here for more explanation.

Improve this answer

Comments

0

After build, run it without restore and build. I am using a filter here to select my test and referencing the .runsettings file.

dotnet test -c Release --no-restore --no-build --filter Category=TimeEstimationBenchmark --settings "C:\Users\ZZZ\OneDrive - XXX\Desktop\.runsettings"

The run settings should be like this:

<?xml version="1.0" encoding="utf-8"?> <RunSettings> <RunConfiguration> <!-- Disables AppDomain to avoid shadow copying issues with BenchmarkDotNet --> <DisableAppDomain>true</DisableAppDomain> <!-- Disables shadow copying for test assemblies --> <DisableShadowCopy>true</DisableShadowCopy> </RunConfiguration> </RunSettings>
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.