Skip to content

Add regression tests for 8 previously untested bug fixes#15627

Draft
nohwnd wants to merge 1 commit intomicrosoft:mainfrom
nohwnd:ics-regression-tests-v3
Draft

Add regression tests for 8 previously untested bug fixes#15627
nohwnd wants to merge 1 commit intomicrosoft:mainfrom
nohwnd:ics-regression-tests-v3

Conversation

@nohwnd
Copy link
Copy Markdown
Member

@nohwnd nohwnd commented Apr 1, 2026

Summary

Add 25 regression tests (22 unit + 3 integration) across 13 files for 11 previously untested bug fixes.
Each test verifies the specific behavior that was fixed and would fail if the fix were reverted.

Unit Tests

Issue Component Tests What the test verifies
#4461 CommunicationUtilities 2 Socket IOException from child exit not propagated
#2319 TrxLogger 3 ErrorStackTrace settable before ErrorMessage
#5132 TrxLogger 2 WarnOnFileOverwrite parameter parsed correctly
#4243 TrxLogger 3 TestOutcome.Error is value 0, not Min alias
#5184 TestHostProvider 2 Stderr forwarded as Informational, not Error
#2479 TestHostProvider 1 ARM64 on Windows uses DLL hosting
#2483 HtmlLogger 2 Initialize creates results directory
#3136 HtmlLogger 1 Transform handles invalid XML char references
#3454 ObjectModel 1 PortablePdbReader null arg validation
#4424 AdapterUtilities 3 Serialization ctor marked Obsolete on NET8+
#813 vstest.console 2 Code coverage excludes auto-generated code

Integration Tests

Issue Test Asset What the test verifies
#4461 crash Testhost crash output has no socket stack traces
#5184 StderrOutputProject (new) Stderr from test doesn't fail the test run
#3136 SpecialCharOutputProject (new) HTML logger handles special chars without XmlException

Copilot AI review requested due to automatic review settings April 1, 2026 13:33
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds regression coverage for a set of previously untested bug fixes across multiple components (TestHostProvider, CommunicationUtilities, TrxLogger, HtmlLogger, ObjectModel), ensuring the fixed behaviors remain protected from regressions.

Changes:

  • Introduces new “RegressionBugFixTests” test classes in 5 unit-test projects.
  • Adds targeted assertions for specific historical bugs (stderr forwarding level, ARM64 hosting selection, socket IO exception propagation, TRX logger behaviors, HtmlLogger directory creation, PortablePdbReader null validation).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
test/Microsoft.TestPlatform.TestHostProvider.UnitTests/Hosting/RegressionBugFixTests.cs Adds regression tests for stderr forwarding level and ARM64 Windows hosting selection.
test/Microsoft.TestPlatform.ObjectModel.UnitTests/RegressionBugFixTests.cs Adds regression coverage for PortablePdbReader constructor null-argument validation.
test/Microsoft.TestPlatform.Extensions.TrxLogger.UnitTests/RegressionBugFixTests.cs Adds regression tests for TRX error info ordering, WarnOnFileOverwrite parsing, and TestOutcome enum semantics.
test/Microsoft.TestPlatform.Extensions.HtmlLogger.UnitTests/RegressionBugFixTests.cs Adds regression tests ensuring HtmlLogger.Initialize creates the results directory.
test/Microsoft.TestPlatform.CommunicationUtilities.UnitTests/RegressionBugFixTests.cs Adds regression tests around socket/IO exception propagation behavior in MessageLoopAsync.

Comment on lines +80 to +88
// GH-2479: On ARM64 Windows, testhost.exe must not be used.
// The fix added an IsWinOnArm() guard that checks PROCESSOR_ARCHITECTURE.
//
// When PROCESSOR_ARCHITECTURE at Machine level does NOT contain "arm"
// (i.e., running on x64), the guard returns false and testhost.exe CAN be used.
// So this test verifies the inverse: on non-ARM, with testhost.exe present,
// it IS used (confirming the guard activates only on ARM).
//
// On ARM64 hardware, IsWinOnArm() returns true and testhost.exe is skipped.
Copy link

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GH-2479 test setup doesn’t appear to force the precondition where testhost.exe is discoverable, so it may pass even if the regression returns (e.g., if the implementation can’t find testhost.exe and falls back to dotnet + testhost.dll regardless). Also, the comments describe verifying the inverse on non-ARM, but the mocks set Architecture to ARM64. To make this a true regression test: (1) explicitly mock the PROCESSOR_ARCHITECTURE machine-level lookup via IEnvironmentVariableHelper to an ARM value to deterministically exercise the guard, and (2) configure IFileHelper.Exists(...) to return true for the computed testhost.exe path (or via a predicate matching ...EndsWith(\"testhost.exe\")) so the code path would choose it if the guard were removed. Update the comments to match the actual scenario being tested.

Copilot uses AI. Check for mistakes.
Comment on lines +102 to +103
mockEnvironment.SetupGet(e => e.Architecture).Returns(PlatformArchitecture.ARM64);
mockEnvironment.Setup(ev => ev.OperatingSystem).Returns(PlatformOperatingSystem.Windows);
Copy link

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GH-2479 test setup doesn’t appear to force the precondition where testhost.exe is discoverable, so it may pass even if the regression returns (e.g., if the implementation can’t find testhost.exe and falls back to dotnet + testhost.dll regardless). Also, the comments describe verifying the inverse on non-ARM, but the mocks set Architecture to ARM64. To make this a true regression test: (1) explicitly mock the PROCESSOR_ARCHITECTURE machine-level lookup via IEnvironmentVariableHelper to an ARM value to deterministically exercise the guard, and (2) configure IFileHelper.Exists(...) to return true for the computed testhost.exe path (or via a predicate matching ...EndsWith(\"testhost.exe\")) so the code path would choose it if the guard were removed. Update the comments to match the actual scenario being tested.

Copilot uses AI. Check for mistakes.
Comment on lines +108 to +109
mockFileHelper.Setup(fh => fh.Exists(testhostDllPath)).Returns(true);
mockFileHelper.Setup(fh => fh.Exists(dotnetPath)).Returns(true);
Copy link

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GH-2479 test setup doesn’t appear to force the precondition where testhost.exe is discoverable, so it may pass even if the regression returns (e.g., if the implementation can’t find testhost.exe and falls back to dotnet + testhost.dll regardless). Also, the comments describe verifying the inverse on non-ARM, but the mocks set Architecture to ARM64. To make this a true regression test: (1) explicitly mock the PROCESSOR_ARCHITECTURE machine-level lookup via IEnvironmentVariableHelper to an ARM value to deterministically exercise the guard, and (2) configure IFileHelper.Exists(...) to return true for the computed testhost.exe path (or via a predicate matching ...EndsWith(\"testhost.exe\")) so the code path would choose it if the guard were removed. Update the comments to match the actual scenario being tested.

Copilot uses AI. Check for mistakes.
Comment on lines +141 to +145
// In both cases, the filename must NOT end with testhost.exe.
Assert.IsNotNull(startInfo);
Assert.IsFalse(
startInfo.FileName!.EndsWith("testhost.exe", StringComparison.OrdinalIgnoreCase),
"GH-2479: ARM64 on Windows must NOT use testhost.exe.");
Copy link

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GH-2479 test setup doesn’t appear to force the precondition where testhost.exe is discoverable, so it may pass even if the regression returns (e.g., if the implementation can’t find testhost.exe and falls back to dotnet + testhost.dll regardless). Also, the comments describe verifying the inverse on non-ARM, but the mocks set Architecture to ARM64. To make this a true regression test: (1) explicitly mock the PROCESSOR_ARCHITECTURE machine-level lookup via IEnvironmentVariableHelper to an ARM value to deterministically exercise the guard, and (2) configure IFileHelper.Exists(...) to return true for the computed testhost.exe path (or via a predicate matching ...EndsWith(\"testhost.exe\")) so the code path would choose it if the guard were removed. Update the comments to match the actual scenario being tested.

Copilot uses AI. Check for mistakes.
Comment on lines +100 to +109
var dotnetPath = @"c:\tmp\dotnet.exe";

mockEnvironment.SetupGet(e => e.Architecture).Returns(PlatformArchitecture.ARM64);
mockEnvironment.Setup(ev => ev.OperatingSystem).Returns(PlatformOperatingSystem.Windows);
mockRunsettingHelper.SetupGet(r => r.IsDefaultTargetArchitecture).Returns(false);
mockProcessHelper.Setup(ph => ph.GetCurrentProcessFileName()).Returns(dotnetPath);
mockProcessHelper.Setup(ph => ph.GetTestEngineDirectory()).Returns(dotnetPath);
mockProcessHelper.Setup(ph => ph.GetCurrentProcessArchitecture()).Returns(PlatformArchitecture.ARM64);
mockFileHelper.Setup(fh => fh.Exists(testhostDllPath)).Returns(true);
mockFileHelper.Setup(fh => fh.Exists(dotnetPath)).Returns(true);
Copy link

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GetTestEngineDirectory() is being set to a full executable path (c:\\tmp\\dotnet.exe) rather than a directory path. If the production code combines this value with filenames, it can generate invalid paths (and reduce the test’s fidelity). Return an actual directory (e.g., c:\\tmp or a temp directory) from GetTestEngineDirectory().

Suggested change
var dotnetPath = @"c:\tmp\dotnet.exe";
mockEnvironment.SetupGet(e => e.Architecture).Returns(PlatformArchitecture.ARM64);
mockEnvironment.Setup(ev => ev.OperatingSystem).Returns(PlatformOperatingSystem.Windows);
mockRunsettingHelper.SetupGet(r => r.IsDefaultTargetArchitecture).Returns(false);
mockProcessHelper.Setup(ph => ph.GetCurrentProcessFileName()).Returns(dotnetPath);
mockProcessHelper.Setup(ph => ph.GetTestEngineDirectory()).Returns(dotnetPath);
mockProcessHelper.Setup(ph => ph.GetCurrentProcessArchitecture()).Returns(PlatformArchitecture.ARM64);
mockFileHelper.Setup(fh => fh.Exists(testhostDllPath)).Returns(true);
mockFileHelper.Setup(fh => fh.Exists(dotnetPath)).Returns(true);
var dotnetExePath = Path.Combine(temp, "dotnet.exe");
var dotnetDirectory = temp;
mockEnvironment.SetupGet(e => e.Architecture).Returns(PlatformArchitecture.ARM64);
mockEnvironment.Setup(ev => ev.OperatingSystem).Returns(PlatformOperatingSystem.Windows);
mockRunsettingHelper.SetupGet(r => r.IsDefaultTargetArchitecture).Returns(false);
mockProcessHelper.Setup(ph => ph.GetCurrentProcessFileName()).Returns(dotnetExePath);
mockProcessHelper.Setup(ph => ph.GetTestEngineDirectory()).Returns(dotnetDirectory);
mockProcessHelper.Setup(ph => ph.GetCurrentProcessArchitecture()).Returns(PlatformArchitecture.ARM64);
mockFileHelper.Setup(fh => fh.Exists(testhostDllPath)).Returns(true);
mockFileHelper.Setup(fh => fh.Exists(dotnetExePath)).Returns(true);

Copilot uses AI. Check for mistakes.
mockEnvironment.Setup(ev => ev.OperatingSystem).Returns(PlatformOperatingSystem.Windows);
mockRunsettingHelper.SetupGet(r => r.IsDefaultTargetArchitecture).Returns(false);
mockProcessHelper.Setup(ph => ph.GetCurrentProcessFileName()).Returns(dotnetPath);
mockProcessHelper.Setup(ph => ph.GetTestEngineDirectory()).Returns(dotnetPath);
Copy link

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GetTestEngineDirectory() is being set to a full executable path (c:\\tmp\\dotnet.exe) rather than a directory path. If the production code combines this value with filenames, it can generate invalid paths (and reduce the test’s fidelity). Return an actual directory (e.g., c:\\tmp or a temp directory) from GetTestEngineDirectory().

Suggested change
mockProcessHelper.Setup(ph => ph.GetTestEngineDirectory()).Returns(dotnetPath);
mockProcessHelper.Setup(ph => ph.GetTestEngineDirectory()).Returns(temp);

Copilot uses AI. Check for mistakes.
{
if (Directory.Exists(nonExistentDir))
{
Directory.Delete(nonExistentDir);
Copy link

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test cleanup deletes the directory non-recursively. If Initialize (or future changes) creates any files/subdirectories under nonExistentDir, Directory.Delete(nonExistentDir) will throw and fail the test. Use the recursive overload (and/or ensure the directory is empty) to make the test robust.

Suggested change
Directory.Delete(nonExistentDir);
Directory.Delete(nonExistentDir, recursive: true);

Copilot uses AI. Check for mistakes.

Exception? capturedError = null;

serverSide.GetStream().Write(new byte[] { 1 }, 0, 1);
Copy link

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second socket-based test uses a synchronous Write with no flush and no cancellation token. Depending on timing/buffering, this can make the test more prone to intermittent hangs until CancelAfter triggers. Prefer mirroring the first test’s approach (WriteAsync + FlushAsync using TestContext.CancellationToken) to more reliably trigger the Poll/NotifyDataAvailable path and reduce flakiness.

Suggested change
serverSide.GetStream().Write(new byte[] { 1 }, 0, 1);
await serverSide.GetStream().WriteAsync(new byte[] { 1 }, 0, 1, TestContext.CancellationToken);
await serverSide.GetStream().FlushAsync(TestContext.CancellationToken);

Copilot uses AI. Check for mistakes.
using var cts = CancellationTokenSource.CreateLinkedTokenSource(TestContext.CancellationToken);
cts.CancelAfter(TimeSpan.FromSeconds(5));

await client.MessageLoopAsync(channel.Object, error => capturedError = error, cts.Token);
Copy link

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second socket-based test uses a synchronous Write with no flush and no cancellation token. Depending on timing/buffering, this can make the test more prone to intermittent hangs until CancelAfter triggers. Prefer mirroring the first test’s approach (WriteAsync + FlushAsync using TestContext.CancellationToken) to more reliably trigger the Poll/NotifyDataAvailable path and reduce flakiness.

Copilot uses AI. Check for mistakes.
@nohwnd nohwnd force-pushed the ics-regression-tests-v3 branch from 5e77a3d to 322bd8d Compare April 1, 2026 14:04
Tests verify the specific behavior that was fixed — each test would
fail if the corresponding fix were reverted.

- microsoftGH-4461: Socket IOException not propagated when child exits (2 tests)
- microsoftGH-2319: ErrorStackTrace settable before ErrorMessage (3 tests)
- microsoftGH-5132: WarnOnFileOverwrite parameter parsed and applied (2 tests)
- microsoftGH-4243: TestOutcome.Error is value 0, no Min/Max aliases (3 tests)
- microsoftGH-5184: Stderr forwarded as Informational, not Error (2 tests)
- microsoftGH-2479: ARM64 on Windows uses DLL hosting, not testhost.exe (1 test)
- microsoftGH-2483: HtmlLogger.Initialize creates results directory (2 tests)
- microsoftGH-3454: PortablePdbReader(MetadataReaderProvider) validates null (1 test)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 1, 2026 15:40
@nohwnd nohwnd force-pushed the ics-regression-tests-v3 branch from 322bd8d to 4c37368 Compare April 1, 2026 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants