CollectionPerMethod support | parallel test execution

[abakumovoleg]

I was trying to use solution from https://www.meziantou.net/parallelize-test-cases-execution-in-xunit.htm which allow to run tests inside class in parallel.

It works fine with default XUnitTestFramework, but doesn't work with xunit.DependencyInjection with errors like

'The following constructor parameters did not have matching fixture data: ITest test
Exception doesn't have a stacktrace'

internal sealed class ParallelTestFramework : XunitTestFramework
{
    public ParallelTestFramework(IMessageSink diagnosticMessageSink)
        : base(diagnosticMessageSink)
    {
    }

    protected override ITestFrameworkExecutor CreateExecutor(AssemblyName assemblyName)
    {
        return new CustomTestExecutor(assemblyName, SourceInformationProvider, DiagnosticMessageSink);
    }

    private sealed class CustomTestExecutor : DependencyInjectionTestFrameworkExecutor
    {
        public CustomTestExecutor(AssemblyName assemblyName, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink)
            : base(assemblyName, sourceInformationProvider, diagnosticMessageSink)
        {
        }

        protected override void RunTestCases(IEnumerable<IXunitTestCase> testCases, IMessageSink executionMessageSink, ITestFrameworkExecutionOptions executionOptions)
        {
            try
            {
                var newTestCases = SetUpTestCaseParallelization(testCases);

                base.RunTestCases(newTestCases, executionMessageSink, executionOptions);
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
                throw;
            }
        }

        /// <summary>
        /// By default, all test cases in a test class share the same collection instance which ensures they run synchronously.
        /// By providing a unique test collection instance to every test case in a test class you can make them all run in parallel.
        /// </summary>
        private IEnumerable<IXunitTestCase> SetUpTestCaseParallelization(IEnumerable<IXunitTestCase> testCases)
        {
            var result = new List<IXunitTestCase>();
            foreach (var testCase in testCases)
            {
                var oldTestMethod = testCase.TestMethod;
                var oldTestClass = oldTestMethod.TestClass;
                var oldTestCollection = oldTestMethod.TestClass.TestCollection;

                // If the collection is explicitly set, don't try to parallelize test execution
                if (oldTestCollection.CollectionDefinition != null || oldTestClass.Class.GetCustomAttributes(typeof(CollectionAttribute)).Any())
                {
                    result.Add(testCase);
                    continue;
                }

                // Create a new collection with a unique id for the test case.
                var newTestCollection =
                        new TestCollection(
                            oldTestCollection.TestAssembly,
                            oldTestCollection.CollectionDefinition,
                            displayName: $"{oldTestCollection.DisplayName} {oldTestCollection.UniqueID}");
                newTestCollection.UniqueID = Guid.NewGuid();

                // Duplicate the test and assign it to the new collection
                var newTestClass = new TestClass(newTestCollection, oldTestClass.Class);
                
                var newTestMethod = new TestMethod(newTestClass, oldTestMethod.Method);
                switch (testCase)
                {
                    // Used by Theory having DisableDiscoveryEnumeration or non-serializable data
                    case XunitTheoryTestCase xunitTheoryTestCase:
                        result.Add(new XunitTheoryTestCase(
                            DiagnosticMessageSink,
                            GetTestMethodDisplay(xunitTheoryTestCase),
                            GetTestMethodDisplayOptions(xunitTheoryTestCase),
                            newTestMethod));
                        break;

                    // Used by all other tests
                    case XunitTestCase xunitTestCase:
                        result.Add(new XunitTestCase(
                            DiagnosticMessageSink,
                            GetTestMethodDisplay(xunitTestCase),
                            GetTestMethodDisplayOptions(xunitTestCase),
                            newTestMethod,
                            xunitTestCase.TestMethodArguments));
                        break;

                    // TODO If you use custom attribute, you may need to add cases here

                    default:
                        throw new ArgumentOutOfRangeException("Test case " + testCase.GetType() + " not supported");
                }
            }

            return result;

            static TestMethodDisplay GetTestMethodDisplay(TestMethodTestCase testCase)
            {
                return (TestMethodDisplay)typeof(TestMethodTestCase)
                    .GetProperty("DefaultMethodDisplay", BindingFlags.Instance | BindingFlags.NonPublic)!
                    .GetValue(testCase)!;
            }

            static TestMethodDisplayOptions GetTestMethodDisplayOptions(TestMethodTestCase testCase)
            {
                return (TestMethodDisplayOptions)typeof(TestMethodTestCase)
                    .GetProperty("DefaultMethodDisplayOptions", BindingFlags.Instance | BindingFlags.NonPublic)!
                    .GetValue(testCase)!;
            }
        }
    }
}

The above solution is to create unique collection for each test method, but it doesn't compatible with method RunTestCases in DependencyInjectionTestFrameworkExecutor

        var hostMap = testCases
            .GroupBy(tc => tc.TestMethod.TestClass, TestClassComparer.Instance)
            .ToDictionary(group => group.Key, group => GetHost(exceptions, () => _hostManager.GetHost(group.Key.Class.ToRuntimeType())));

Here we group tests by classes, not by collections.

If I remove this grouping just by ommiting TestClassComparer.Instance everything will works fine.

        var hostMap = testCases
            .GroupBy(tc => tc.TestMethod.TestClass)
            .ToDictionary(group => group.Key, group => GetHost(exceptions, () => _hostManager.GetHost(group.Key.Class.ToRuntimeType())));            

Why we need to group tests by class?

[pengweiqhca]

Feel free to raise a PR!

[abakumovoleg]

#93

[pengweiqhca]

I has release 8.9.0, please verify that your problem has been resolved.

[Rick-van-Dam]

When I enable ParallelizationMode I do see the tests are being run in parallel however they only finish when all tests have finished executing.

This is a problem for me because iam running a api test with a database behind it. I pool the databases so the tests have to spend less time on running migrations. After a test is finished the database is cleaned and returned to the pool. However since the tests never finish before all tests have finished running the database is never returned to the pool.

To give a example of the current faulty behavior lets say I have 100 tests and I run 20 tests in parallel this is what happens:
20 tests running 0 finished
40 tests running 0 finished
60 tests running 0 finished
80 tests running 0 finished
100 tests running 0 finished
suddenly finished tests quickly jumps to 100

In the above example 100 databases are created.

I would expect more something like this:
20 tests running 0 finished
20 tests running 20 finished
20 tests running 40 finished
20 tests running 60 finished
20 tests running 80 finished
0 tests running 100 finished

In the above example only 20 databases are created because of pooling.

[pengweiqhca]

Upgrade Xunit.DependencyInjection to 9.3.0, xunit to 2.8.0