Running coverlet global tool over dotnet build intermittently fails to open file when calculating coverage

[jkoritzinsky]

In my project https://github.com/SharpGenTools/SharpGenTools I've recently switched to using Coverlet for my code-coverage instrumentation. I've been encountering an intermittent issue where the Coverlet global tool is unable to open the hits file since it is being used by another process.

This only happens when I'm using the Coverlet global tool to instrument my MSBuild tasks while running dotnet build.

Here's the repro steps that I have:

1. Clone SharpGenTools
2. In Powershell run ./build ; ./test. That will build SharpGenTools and then run the tests. It will likely fail to build the outerloop tests because the process will be unable to access the hits file.

Alternatively, run the following commands from the cloned build:

./build
./build/deploy-test-packages Debug
./build/build-outerloop-native
dotnet restore ./SdkTests/SdkTests.sln
coverlet ".\SdkTests\RestoredPackages\sharpgentools.sdk\2.0.0-local\tools\netstandard1.3\SharpGenTools.Sdk.dll" -t "dotnet" -a "build ./SdkTests/SdkTests.sln /nodeReuse:false --no-restore" -f opencover -o ./artifacts/coverage/outerloop.xml --include-test-assembly --include-directory .\SdkTests\RestoredPackages\sharpgentools.sdk\2.0.0-local\tools\netstandard1.3

The last command will fail with the issue.

I can also privately send you a minidump reproing the issue if you want.

[jkoritzinsky]

Sometimes you'll get a different crash (with the same exception) when the instrumentation tracker in MSBuild loses the race and can't access the hits file to save the data on the shutdown of the dotnet.exe process. I also have a minidump saved for this scenario.

[cbroxton]

Could you post some more info about the error you're getting? I think we're getting an error which sounds similar to what you described in our Azure Devops pipeline, see below...

2019-07-23T12:53:55.0463811Z Test Run Successful. 2019-07-23T12:53:55.0481876Z Test execution time: 30.7144 Seconds 2019-07-23T12:53:55.1195193Z 2019-07-23T12:53:55.1377050Z Calculating coverage result... 2019-07-23T12:53:55.5143800Z C:\Users\DevOpsAgent2\.nuget\packages\coverlet.msbuild\2.6.3\build\coverlet.msbuild.targets(41,5): error : Unable to read beyond the end of the stream. [C:\agent_2\_work\1\s\xxx\xxx\xxx\xxx\xxx.Test.csproj] 2019-07-23T12:53:55.5144045Z C:\Users\DevOpsAgent2\.nuget\packages\coverlet.msbuild\2.6.3\build\coverlet.msbuild.targets(41,5): error : at System.IO.BinaryReader.FillBuffer(Int32 numBytes) [C:\agent_2\_work\1\s\xxx\xxx\xxx\xxx\xxx.Test.csproj] 2019-07-23T12:53:55.5144131Z C:\Users\DevOpsAgent2\.nuget\packages\coverlet.msbuild\2.6.3\build\coverlet.msbuild.targets(41,5): error : at System.IO.BinaryReader.ReadInt32() [C:\agent_2\_work\1\s\xxx\xxx\xxx\xxx\xxx.Test.csproj] 2019-07-23T12:53:55.5154396Z C:\Users\DevOpsAgent2\.nuget\packages\coverlet.msbuild\2.6.3\build\coverlet.msbuild.targets(41,5): error : at Coverlet.Core.Coverage.CalculateCoverage() in C:\Users\toni\Workspace\coverlet\src\coverlet.core\Coverage.cs:line 253 [C:\agent_2\_work\1\s\xxx\xxx\xxx\xxx\xxx.Test.csproj] 2019-07-23T12:53:55.5154672Z C:\Users\DevOpsAgent2\.nuget\packages\coverlet.msbuild\2.6.3\build\coverlet.msbuild.targets(41,5): error : at Coverlet.Core.Coverage.GetCoverageResult() in C:\Users\toni\Workspace\coverlet\src\coverlet.core\Coverage.cs:line 129 [C:\agent_2\_work\1\s\xxx\xxx\xxx\xxx\xxx.Test.csproj] 2019-07-23T12:53:55.5154809Z C:\Users\DevOpsAgent2\.nuget\packages\coverlet.msbuild\2.6.3\build\coverlet.msbuild.targets(41,5): error : at Coverlet.MSbuild.Tasks.CoverageResultTask.Execute() in C:\Users\toni\Workspace\coverlet\src\coverlet.msbuild.tasks\CoverageResultTask.cs:line 84 [C:\agent_2\_work\1\s\xxx\xxx\xxx\xxx\xxx.Test.csproj] 2019-07-23T12:53:55.7264131Z ##[error]Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 1

[MarcoRossignoli]

I don't know if they are same problem, we have a know issue for `Unable to read... "#210 that should be fixed by collectors, I'm OOF at the moment I'll take a look next week, sorry for the delay guys and thanks for help on investigation

[jkoritzinsky]

The issue that I found seemed to be caused by the fact that dotnet build defaults to parallel builds. The following two things happened:

• The MSBuild nodes don't shut down immediately and coverlet isn't resilient to a subprocess writing the hits file when the main program ends (the first issue I mentioned)
• The coverlet instrumenting isn't resilient to the same assembly being loaded multiple times and being unloaded (and writing the hits file) in parallel (the second issue I mentioned)

I worked around it for me by running the build process with -m:1 to force the build to run sequentially on one node.

[cbroxton]

Thanks for that, I've added the -m:1 flag to the dotnet build step of our Azure Devops pipeline and based on initial tests it seems to have solved the issue. I couldn't find any docs on how exactly that flag works though.

[StevenTCramer]

@cbroxton I'm hitting the same issue. Can you explain what you mean but adding the -m:1 flag?

Adding it where? I am looking at the docs and it doesn't seem to exist.

[MarcoRossignoli]

-m:1 forces sequential builds using only one core

some infos https://docs.microsoft.com/en-us/visualstudio/msbuild/building-multiple-projects-in-parallel-with-msbuild?view=vs-2019

[StevenTCramer]

The process cannot access the file 'C:\agent_work\6\s\test\TestApp.EndToEnd.Tests\bin\Debug\netcoreapp3.0\BlazorState.dll' because it is being used by another process.

Still happens with the maxcpucount set to 1

If anyone cares to review.... https://timewarpenterprises.visualstudio.com/Blazor-State/_build/results?buildId=593
without coverlet they all pass.

[MarcoRossignoli]

@StevenTCramer any news?
Close this issue as solved, I mean with a workaround on coverlet issue with parallel instrumentation/test run
This will be likely resolved if we complete #357