Test class name missing in TRX report for parametrized fixture when running through MTP

[radim-bernatik-veeam]

In the following example is situation, when class name is not fully included in resulting TRX report. Only the namespace of the class is included, but the class name is missing. There are newest packages for MTP (2.0.1) and NUnit3TestAdapter (6.0.0-beta.1). But the problem was firstly recognized by me on MTP/1.8.1 and NUnit3TestAdapter/5.0.1. I also checked NET 8 and NET 10 rc1 frameworks, issue is reproducible on both.

In the TRX report for both instances of the Test1 test is only className="One", when running using MTP.
The TRX report fragment:

    <UnitTest name="Test1" storage="d:\git\mtp\one\bin\debug\net10.0\one.dll" id="f250b33b-8a35-b41a-e7bf-8e9547f93a26">
      <Execution id="983facad-df20-4239-a725-afd3d6150424" />
      <TestMethod codeBase="D:\Git\MTP\One\bin\Debug\net10.0\One.dll" adapterTypeName="executor://NUnitExtension/1.0.0+c7d97261880dc4b9bcf3f48c2cabc5293016adba" className="One" name="Test1" />
    </UnitTest>

Execution command: dotnet exec One\bin\Debug\net10.0\One.dll --results-directory TestResults --report-trx

On the other side, when same example is executed by vstest.console.exe command, the TRX report contains proper className="One.Tests(One)" and className="One.Tests(Two)".
The TRX report fragment:

  <TestDefinitions>
    <UnitTest name="Test1" storage="d:\git\mtp\one\bin\debug\net10.0\one.dll" id="31e09d51-24f9-f1bf-3546-ea0d27e9d1e2">
      <Execution id="205cb3bb-29b2-47ff-8f7a-16721f04c50a" />
      <TestMethod codeBase="D:\Git\MTP\One\bin\Debug\net10.0\One.dll" adapterTypeName="executor://nunit3testexecutor/" className="One.Tests(One)" name="Test1" />
    </UnitTest>
    <UnitTest name="Test1" storage="d:\git\mtp\one\bin\debug\net10.0\one.dll" id="b28b10d5-d1e1-b8e3-f896-8c4751d57b96">
      <Execution id="8cd2ab2d-6490-4b08-8438-500a777931a3" />
      <TestMethod codeBase="D:\Git\MTP\One\bin\Debug\net10.0\One.dll" adapterTypeName="executor://nunit3testexecutor/" className="One.Tests(Two)" name="Test1" />
    </UnitTest>
  </TestDefinitions>

Execution command: vstest.console.exe One\bin\Debug\net10.0\One.dll /logger:trx

In the example code bellow there are two TestFixtureData provider methods, one with used SetArgDisplayNames method, and one without. Reports above are from usage the one with custom test data names GetTestCasesWithNames. But this has no impact on the results. Even if test data contains custom name or not, or even if TestFixture attributes is used directly with parameters, in all cases the .Tests(One) or .Tests(1) is missing from the TRX report's TestMethod node className attribute value (when MTP is used, old vstest.console.exe runner works without issues in all cases).

I don't know if this issue is in MTP or NUnit3TestAdapter code, but I guess the NUnit3TestAdapter one, because the strange behavior is triggered by usage of parametrized fixture attributes.

I think this issue is quite critical, because it prevents usage of parametrized fixtures in MTP. When class name is not fully recognized in TRX report, we cannot map the test result to proper source file and address possible issues. It also leads to test method name misidentification because the class name is missing from the full tests name. It is nowhere in TRX report.

Example unit test class - UnitTest1.cs

using NUnit.Framework;
using System.Collections;

namespace One
{
    //[TestFixture(1)]
    //[TestFixture(2)]
    //[TestFixtureSource(nameof(GetTestCases))]
    [TestFixtureSource(nameof(GetTestCasesWithNames))]
    public class Tests
    {
        private int _counter;

        public Tests(int counter)
        {
            _counter = counter;
        }

        [Test]
        public void Test1()
        {
            Console.WriteLine($"Test1:{_counter}");
            Assert.Pass();
        }

        private static IEnumerable<TestFixtureData> GetTestCases()
        {
            yield return new TestFixtureData(1);
            yield return new TestFixtureData(2);
        }

        private static IEnumerable<TestFixtureData> GetTestCasesWithNames()
        {
            yield return new TestFixtureData(1).SetArgDisplayNames("One");
            yield return new TestFixtureData(2).SetArgDisplayNames("Two");
        }
    }
}

Example project file - One.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <IsPackable>false</IsPackable>
    <IsTestProject>true</IsTestProject>
    <EnableNUnitRunner>true</EnableNUnitRunner>
    <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="coverlet.collector" Version="6.0.0" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.13.0" />
    <PackageReference Include="NUnit" Version="3.12.0" />
    <PackageReference Include="NUnit.Analyzers" Version="3.8.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="6.0.0-beta.1" />
    <PackageReference Include="Microsoft.Testing.Extensions.Telemetry" Version="2.0.1" />
    <PackageReference Include="Microsoft.Testing.Extensions.TrxReport" Version="2.0.1" />
    <PackageReference Include="Microsoft.Testing.Extensions.VSTestBridge" Version="2.0.1" />
    <PackageReference Include="Microsoft.Testing.Platform" Version="2.0.1" />
    <PackageReference Include="Microsoft.Testing.Platform.MSBuild" Version="2.0.1" />
  </ItemGroup>
</Project>

[OsirisTerje]

Can you enable the Dump (runsettings) and look at the results there? If the same results are there, it is in the framework, if it is correct there, it is the adapter.

https://docs.nunit.org/articles/vs-test-adapter/Tips-And-Tricks.html#dumpxmltestdiscovery-and-dumpxmltestresults

Also, if you skip using MTP, does it then work as expected?

PS:

On the other side, when same example is executed by vstest.console.exe command, the TRX report contains proper className="One.Tests(One)" and className="One.Tests(Two)".

This indicates it is related to using MTP at least.

[radim-bernatik-veeam]

Here is the output from dump file:

      <test-suite type='TestSuite' id='0-1006' name='One' fullname='One' runstate='Runnable' testcasecount='2'>
         <test-suite type='ParameterizedFixture' id='0-1004' name='Tests' fullname='One.Tests' runstate='Runnable' testcasecount='2'>
            <test-suite type='TestFixture' id='0-1000' name='Tests(One)' fullname='One.Tests(One)' classname='One.Tests' runstate='Runnable' testcasecount='1'>
               <test-case id='0-1001' name='Test1' fullname='One.Tests(One).Test1' methodname='Test1' classname='One.Tests' runstate='Runnable' seed='1839287280' />
            </test-suite>
            <test-suite type='TestFixture' id='0-1002' name='Tests(Two)' fullname='One.Tests(Two)' classname='One.Tests' runstate='Runnable' testcasecount='1'>
               <test-case id='0-1003' name='Test1' fullname='One.Tests(Two).Test1' methodname='Test1' classname='One.Tests' runstate='Runnable' seed='514243834' />
            </test-suite>
         </test-suite>
      </test-suite>

Looks like the information is in the dump in correct form fullname='One.Tests(One).Test1', classname='One.Tests'. But I am not export on the dump file format, I used it for the first time now.

Also, if you skip using MTP, does it then work as expected?

What do you exactly mean by this? Isn't running using vstest.console.exe enough? When running through vstest.console, the TRX report contain correct values for parametrized fixture.

[OsirisTerje]

Thanks! The dump information is fine.

That narrows down the issue to be either in the adapter or in the MTP.

Isn't running using vstest.console.exe enough?

It is a strong indication that MTP has something to do with this. But, there are differences between running dotnet test and vstestconsole. You can turn off MTP in the csproj file too, and still use the test explorer and dotnet test.

[OsirisTerje]

Repro added here: https://github.com/nunit/nunit3-vs-adapter.issues/tree/master/Issue1351 (Thanks for the repro code @radim-bernatik-veeam !)

@Youssef1313 The dump file have the correct names, and the adapter itself doesn't change the names, it just sends them further.

I haven't debugged through this yet, but nice if you can comment on whether this can be in the bridge or the mtp itself.

[radim-bernatik-veeam]

I have made additional test. Switched to NET 8 (vstest mode is not supported in NET 10 at all), set the <TestingPlatformDotnetTestSupport>false</TestingPlatformDotnetTestSupport> in csproj file and run dotnet test --logger trx command.

The tests were executed in vstest mode through dotnet test command. Result in TRX is correct. So it is really connected to MTP.

[Youssef1313]

The FQN given to MTP by NUnit is:

Issue1351.Tests(One).Test1

Is that really expected?

[radim-bernatik-veeam]

Yes, I think Issue1351.Tests(One).Test1 is expected. The namespace is Issue1351, class name is Tests, the parameter is given by TestFixtureSource attribute with data from GetTestCasesWithNames function where data are named by SetArgDisplayNames("One"), thus the test class name is expanded by (One). The tests itself is Test1, of course.

The FQN given to MTP by NUnit is:

If I understand correctly, the problem is then in MTP, not in NUnit adapter. Am I right?

[Youssef1313]

A FQN isn't supposed to have parameters in the first place. So it's an unexpected NUnit behavior.

We might consider implementing a workaround for that in VSTestBridge. Such workaround might end up breaking F# scenarios though, where parentheses can be part of identifier names.

[OsirisTerje]

In NUnit a test may have any name (not quite true, but nearly so), it is not strictly following C# FQN. However, it most cases it does follow the FQN, and most of the variations are like shown above.

The typical names are like:

#	Case	Example C#	NUnit Internal FQN
1	Simple	MyTest()	MyNamespace.MyClass.MyTest
2	Nested class	Outer.Inner.MyTest()	MyNamespace.Outer+Inner.MyTest
3	Generic class	GenericTests<int>.MyTest()	MyNamespace.GenericTests<System.Int32>.MyTest
4	Generic method	MyTest<int>()	MyNamespace.MyClass.MyTest<System.Int32>
5	Parameterized test	[TestCase(1, "A")]	MyNamespace.MyClass.MyTest(1,"A")
6	Parameterized fixture	[TestFixture(42)]	MyNamespace.MyFixture(42).MyTest
7	Generic fixture + parameterized test	[TestFixture(typeof(int))][TestCase(1)]	MyNamespace.GenericFixture<System.Int32>.MyTest(1)

So the case above is (6).

In addition, you can set the name to practically anything you like, but in the most weird cases that will not even pass through the adapter. Only the NUnit.Console and NUnitLite can make sense of those.

[Youssef1313]

This would be resolved if you provide ManagedType/ManagedMethod on your end (#1259). Otherwise, we need some way that let you to provide the bridge with the "correct" class name and method name.

[OsirisTerje]

What would the ManageType and ManagedMethod be of this: MyNamespace.MyFixture(42).MyTest

[Youssef1313]

It's whatever ManagedNameHelper.GetManagedName returns. You don't need to calculate it yourself.