slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Repository	Step Name	Console log	Pull Request
1554878	dotnet/aspnetcore	Tests: macOS	Log
1554620	dotnet/aspnetcore	Tests: macOS	Log
1554582	dotnet/maui	Controls Layout	Log
1554548	dotnet/maui	Controls Shell	Log
1554509	dotnet/maui	Controls Shell	Log
1554500	dotnet/maui	Controls CollectionView	Log	dotnet/maui#37404
1554496	dotnet/maui	Controls Shell	Log
1554435	dotnet/aspnetcore	Tests: macOS	Log
1554285	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132353
1554278	dotnet/runtime	osx-arm64 Release Libraries_CheckedCoreCLR	Log
1554246	dotnet/maui	Controls Border,BoxView,Brush,Button	Log
1554269	dotnet/aspnetcore	Tests: macOS	Log
1554258	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129220
1554239	dotnet/aspnetcore	Tests: macOS	Log
1554194	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131507
1554097	dotnet/runtime	osx-arm64 Release Libraries_CheckedCoreCLR	Log
1554076	dotnet/sdk	Mac arm64	Log	dotnet/sdk#55785
1553914	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132326
1553863	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132319
1553795	dotnet/maui	Controls Layout	Log	dotnet/maui#37438
1553745	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132098
1553731	dotnet/aspnetcore	Tests: macOS	Log
1553554	dotnet/maui	Controls CollectionView		dotnet/maui#37034
1553504	dotnet/aspnetcore	Tests: macOS	Log
1553489	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129853
1553382	dotnet/maui	Controls Layout	Log	dotnet/maui#37182
1553310	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130859
1553213	dotnet/maui	Controls CollectionView	Log	dotnet/maui#37307
1553185	dotnet/maui	Controls Shell	Log	dotnet/maui#37397
1553173	dotnet/maui	Controls Layout	Log	dotnet/maui#37176
1553131	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132245
1553134	dotnet/aspnetcore	Tests: macOS	Log
1553010	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132245
1552964	dotnet/maui	Controls Shell	Log
1552851	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#123920
1552837	dotnet/maui	Controls Shell	Log
1552812	dotnet/maui	Controls Shell	Log
1552769	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132197
1552604	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132286
1552539	dotnet/maui	Controls ScrollView	Log
1552533	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132287
1552384	dotnet/maui	Controls CollectionView	Log
1550827	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#128675
1552405	dotnet/aspnetcore	Tests: macOS	Log
1552304	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#132234
1552215	dotnet/maui	Controls Shell	Log	dotnet/maui#37405
1552284	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#131552
1551891	dotnet/maui	Controls Border,BoxView,Brush,Button	Log	dotnet/maui#37393
1552244	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131538
1552121	dotnet/maui	Controls Shell	Log	dotnet/maui#37034
1552114	dotnet/runtime	osx-x64 Release NativeAOT_Libs	Log
1552008	dotnet/maui	Controls Shell	Log
1549740	dotnet/maui	Controls CollectionView	Log	dotnet/maui#37307
1551997	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1551875	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#132270
1551775	dotnet/maui	Controls Shell	Log	dotnet/maui#37382
1551777	dotnet/maui	Controls Shell	Log	dotnet/maui#36907
1551774	dotnet/maui	Controls Shell	Log	dotnet/maui#37111
1551678	dotnet/maui	Controls Image,ImageButton,IndicatorView,InputTransparent,IsEnabled,IsVisible	Log
1551696	dotnet/maui	Controls Shell	Log	dotnet/maui#37365
1548230	dotnet/maui	Controls Shell	Log
1551663	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132214
1551570	dotnet/runtime	tvossimulator-x64 Release AllSubsets_Mono_RuntimeTests	Log
1551485	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#132007
1551461	dotnet/maui	Controls Layout	Log	dotnet/maui#37278
1551468	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#129853
1551428	dotnet/maui	Controls Border,BoxView,Brush,Button	Log	dotnet/maui#37378
1551356	dotnet/maui	Controls Shell	Log
1551361	dotnet/aspnetcore	Tests: macOS	Log
1551324	dotnet/aspnetcore	Tests: macOS	Log
1551113	dotnet/aspnetcore	Tests: macOS	Log
1551088	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132250
1551025	dotnet/maui	Controls Shell	Log
1550966	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132114
1550963	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#131783
1550932	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132221
1550805	dotnet/aspnetcore	Tests: macOS	Log
1550735	dotnet/maui	Controls Shell	Log
1550778	dotnet/aspnetcore	Tests: macOS	Log
1550743	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132210
1550711	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#132227
1550629	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#131508
1550583	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132219
1550470	dotnet/runtime	Mono crossaot Product Build osx x64 release	Log	dotnet/runtime#132160
1550405	dotnet/maui	Controls Shell	Log
1550408	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131783
1550397	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131197
1550332	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132216
1550330	dotnet/aspnetcore	Tests: macOS	Log
1550329	dotnet/aspnetcore	Tests: macOS	Log
1550309	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#131538
1550248	dotnet/maui	Controls Shell	Log
1550200	dotnet/maui	Controls Shell	Log	dotnet/maui#37034
1550100	dotnet/maui	Controls CollectionView	Log
1550160	dotnet/aspnetcore	Tests: macOS	Log
1550121	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129734
1550029	dotnet/maui	Controls Image,ImageButton,IndicatorView,InputTransparent,IsEnabled,IsVisible	Log	dotnet/maui#37310
1550103	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1550066	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR		dotnet/runtime#130910
1549985	dotnet/aspnetcore	Tests: macOS	Log
Displaying 100 of 957 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	182	957

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.