feat: Add filter support for TestAdapter

[filzrev]

This PR add --filter parameter support for dotnet test.
It intended to resolve #2767 issue (and some part of issue #2662)

What's changed in this PR.

1. Add benchmark filter logics
Add following files to support benchmark filtering.

• TestCaseFilter.cs
• LoggerHelper.cs

Note: These files are based on xUnit v2 TestAdapter Code (License: Apache 2.0)

2. Modify VSTestAdapter.cs
Add benchmark filter logics to DiscoverTests and RunTests

3. BenchmarkEnumerator.cs
Remove code for Debug configuration.
(By #2774 changes. It can load DLLs that are build with Debug configurations)

4. Disable TestTfmsInParallel MSBuild property
It's enabled by default on .NET 9 or later.
But it's not expected multiple benchmarks are executed in parallel.

I've added code to disable this setting to following files.

• BenchmarkDotNet.Samples.csproj
• BenchmarkDotNet.Samples.FSharp.csproj
• BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.props

5. Temporary comment out IntroVisualStudioProfiler benchmark
Because it cause errors when benchmarks are enumerated/evaluated twice. (#2758)

Tasks that is not scope of this PR

• Benchmark progress is not shown when running benchmark with dotnet test by default
  -> It need to specify --logger:"console;verbosity=normal" explicitly
• DisplayName filter is same as FullyQualifiedName
  -> Currently there is no attribute that can be used for DisplayName

What's Tested

Currently it's hard to write unit tests that use TestAdapter.
So it need to run tests manually.

TestExplorer

1. All benchmarks(net462/net80) are displayed when using Release configuration.

1. It can run single selected test and multiple selected tests.
2. When running multiple TFMs benchmarks. It's sequentially executed. (Not executed in parallel)

dotnet test

1. Verify default behavior is not changed when run command without --filter parameter

   dotnet test -c Release -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed" --list-tests
   dotnet test -c Release -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed"

2. Filter by partial query works as expected.

   dotnet test -c Release --filter IntroBasic -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed" --list-tests
   dotnet test -c Release --filter IntroBasic -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed"

3. Filter by property value works as expected.

   dotnet test -c Release --filter FullyQualifiedName=BenchmarkDotNet.Samples.IntroBasic.Sleep -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed" --list-tests
   dotnet test -c Release --filter DisplayName=BenchmarkDotNet.Samples.IntroBasic.Sleep -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed" --list-tests

   dotnet test -c Release --filter FullyQualifiedName=BenchmarkDotNet.Samples.IntroBasic.Sleep -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed"
   dotnet test -c Release --filter DisplayName=BenchmarkDotNet.Samples.IntroBasic.Sleep -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed"

4. Filter by Category works as expected

   dotnet test -c Release --filter Category=Fast -tl:off --no-build --framework net8.0 --logger:"console;verbosity=normal" --list-tests
   dotnet test -c Release --filter Category=Fast -tl:off --no-build --framework net8.0 --logger:"console;verbosity=normal"

5. Verify it can run .NET462 benchmarks also

   dotnet test -c Release --filter IntroBasic -tl:off --no-build --framework net462 --logger:"console;verbosity=detailed" --list-tests
   dotnet test -c Release --filter IntroBasic -tl:off --no-build --framework net462 --logger:"console;verbosity=detailed"

[timcassell]

1. All benchmarks(net462/net8.0) are displayed on Debug/Release configuration.

In the original PR, not including Debug without in-process was on purpose. See comment chain #2438 (comment). We decided to leave it as a future improvement to include debugger attachment. I think that should be done in a separate PR.

[filzrev]

In the original PR, not including Debug without in-process was on purpose.

Thanks for pointed out it's expected behavior that display only InProcess jobs on Debug configuration.
I've add commit to revert related changes.

[filzrev]

I've added commit(8551dbf) to resolve issue console logs are outputted twice.

It use reflection to modify ImmutableConfig's logger field with following reasons

• ImmutableConfig member can't modify fields after construction
• To use existing ImmutableConfigBuilder. It requires BenchmarkDotNet side code modification. or need to a lot of duplicated code copy.

When it failed to get logger fields of ImmutableConfig. It skip steps to remove ConsoleLogger.

[timcassell]

Is that related to filter? Can it be done in a separate PR?

Also, I don't like using reflection to do that. We can instead construct a new config object, copying all the properties except ones we want to exclude. We can also add APIs to ManualConfig to do this kind of thing.

[filzrev]

Is that related to filter? Can it be done in a separate PR?
It's existing TestAdapter's problem. (Duplicated logs are outputted to Output Window and Console)

So it might better to create separated PR.

[filzrev]

I've added commit (2391e11)

It set <TestTfmsInParallel>false</TestTfmsInParallel> setting automatically if value is not explicitly set. when package is referenced by using <PackageReference>.

Note:
It's not works when reference TestAdapter project with ProjectReference.
So BenchmarkDotNet.Samples.csproj settings are remained as is)

[timcassell]

Thanks @filzrev!