assemblies-valid fails with .NET 10

[tmds]

: Executing /tmp/tmp.7RHfirdF6s/.dotnet/dotnet with arguments build -p:UseRazorBuildServer=false -p:UseSharedCompilation=false -m:1 in working directory /tmp/tmp.7RHfirdF6s/dotnet-regular-tests/assemblies-valid
Welcome to .NET 10.0!
---------------------
SDK Version: 10.0.100-alpha.1.25077.1
----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate run 'dotnet dev-certs https --trust'
Learn about HTTPS: https://aka.ms/dotnet-https
----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
Determining projects to restore...
Restored /tmp/tmp.7RHfirdF6s/dotnet-regular-tests/assemblies-valid/assemblies-valid.csproj (in 3.95 sec).
/tmp/tmp.7RHfirdF6s/.dotnet/sdk/10.0.100-alpha.1.25077.1/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(3265): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/tmp/tmp.7RHfirdF6s/dotnet-regular-tests/assemblies-valid/assemblies-valid.csproj]
assemblies-valid -> /tmp/tmp.7RHfirdF6s/dotnet-regular-tests/assemblies-valid/bin/Debug/net10.0/assemblies-valid.dll
Build succeeded.
0 Warning(s)
0 Error(s)
Time Elapsed 00:00:15.97
Process Exit Code: 0
Executing /tmp/tmp.7RHfirdF6s/.dotnet/dotnet with arguments test --no-restore --no-build in working directory /tmp/tmp.7RHfirdF6s/dotnet-regular-tests/assemblies-valid
Test run for /tmp/tmp.7RHfirdF6s/dotnet-regular-tests/assemblies-valid/bin/Debug/net10.0/assemblies-valid.dll (.NETCoreAppVersion=v10.0)
VSTest version 17.14.0-preview-25073-07 (x64)
Starting test execution please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.55]     AssembliesValid.AssembliesValid.ValidateAssemblies [FAIL]
Failed AssembliesValid.AssembliesValid.ValidateAssemblies [203 ms]
Error Message:
Assert.True() Failure
Expected: True
Actual:   False
Stack Trace:
at AssembliesValid.AssembliesValid.ValidateAssemblies() in /tmp/tmp.7RHfirdF6s/dotnet-regular-tests/assemblies-valid/AssembliesValid.cs:line 120
at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj IntPtr* args) in /_/src/runtime/src/coreclr/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.CoreCLR.cs:line 36
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj BindingFlags invokeAttr) in /_/src/runtime/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs:line 57
Failed!  - Failed:     1 Passed:     0 Skipped:     0 Total:     1 Duration: 203 ms - assemblies-valid.dll (net10.0)
Process Exit Code: 1

The failing assert is:

dotnet-regular-tests/assemblies-valid/AssembliesValid.cs

Line 120 in 5043d97

Assert.True(allOkay);

cc @omajid @nicrowe00

[omajid]

Console.WriteLine doesn't get logged by xunit, I suppose?

diff --git a/assemblies-valid/AssembliesValid.cs b/assemblies-valid/AssembliesValid.cs
index 344de1a..2400be1 100644
--- a/assemblies-valid/AssembliesValid.cs
+++ b/assemblies-valid/AssembliesValid.cs
@@ -106,11 +106,11 @@ namespace AssembliesValid
 
                         if (valid)
                         {
-                            Console.WriteLine($"{assembly}: OK");
+                            Console.Error.WriteLine($"{assembly}: OK");
                         }
                         else
                         {
-                            Console.WriteLine($"error: {assembly} hasMethods: {hasMethods}, hasAot: {hasAot}, inReleaseMode: {inReleaseMode}");
+                            Console.Error.WriteLine($"error: {assembly} hasMethods: {hasMethods}, hasAot: {hasAot}, inReleaseMode: {inReleaseMode}");
                             allOkay = false;
                         }
                     }

shows:

error: /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/10.0.0-alpha.2.25073.4/Microsoft.AspNetCore.Metadata.dll hasMethods: True, hasAot: False, inReleaseMode: True
error: /usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/10.0.0-alpha.2.25073.4/Microsoft.AspNetCore.ResponseCaching.Abstractions.dll hasMethods: True, hasAot: False, inReleaseMode: True

Essentially, test is flagging that the 2 dlls don't contain AOT (ReadyToRun) data.

[tmds]

Console.WriteLine doesn't get logged by xunit, I suppose?

It depends on how the test runner gets invoked. ITestOutputHelper is for associating output with a specific test: https://xunit.net/docs/capturing-output.

Essentially, test is flagging that the 2 dlls don't contain AOT (ReadyToRun) data.

We should take a look at what is in these assemblies to see if there is something AOT-able.

[omajid]

ITestOutputHelper is for associating output with a specific test: https://xunit.net/docs/capturing-output.

Thanks. I did some testing and I am seeing some strange behaviour when testing with dotnet test -p:TestTargetFramework=netX.0 | cat to save the output of test for logging. Sometimes output from Console is shown in teh test log, and sometimes it isn't. Same with ITestOutputHelper. So I made a table:

.NET Version	Console shows output	ITestOutputHelper shows output
.NET 8.0.112	Yes	No
.NET 9.0.102	No	No
A .NET 10 build from a few days ago	No	Yes

It doesn't look like a simple Console -> ITestOutputHelper will fully solve our problems.

[tmds]

I've seen some weirdness here too. I assume due to the terminal logger (that became the default in .NET 9). Disabling the terminal logger may help.

Sometimes output from Console is shown in teh test log, and sometimes it isn't. Same with ITestOutputHelper. So I made a table:

Is this for a test that fails?

[omajid]

Is this for a test that fails?

I was testing the current version of assemblies-valid (so it fails on some versions and passes on others). That's a good point though, so I will shortly test what happens if the test is passing vs failing too.

[omajid]

I re-ran some unit tests. Looks like relying on ITestOutputHelper for failing test output is a good strategy, though we won't get more logs for tests that pass:

✅ = Output from unit test is displayed
❌ = Output from unit test is not displayed

.NET Version	Console + Passing test	Console + Failing test	ITestOutputHelper + Passing test	ITestOutputHelper + failing test
.NET 8	✅	✅	❌	✅
.NET 9	❌	❌	❌	✅
.NET 10	❌	❌	❌	✅

[omajid]

Upstream tracker for the assemblies missing AOT: dotnet/aspnetcore#60174