macos-15 seems to have slowed down

[amlcurran]

Description

Hi, apologies if this isn't the right place to report this.

Since Jan 28th, we've had ~33% of our iOS app builds be slower on the macos-15 image. Since this morning (30th Jan), that has now increased to 100% of our builds. Whilst previously our builds were taking 20 minutes, they have all started hitting our timeout, which is 30 mins.

Given that this was intermittently starting from the 28th, but since this morning increase to 100%, is there any changes to the image that has happened recently to cause this? I notice that all our failing builds are using an updated image version.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• macOS 15
• macOS 15 Arm64
• Windows Server 2019
• Windows Server 2022
• Windows Server 2025

Image version and build link

All builds are using Current runner version: '2.321.0'

Failing builds are using
Image: macos-15-arm64
Version: 20250127.616

Passing builds are using
Image: macos-15-arm64
Version: 20250120.596

Is it regression?

Yes

Expected behavior

No performance regressions in images

Actual behavior

A performance regression of at least 50%

Repro steps

Use the latest macos-15 image
Note that builds are slower

[erik-bershel]

Hey @amlcurran!

Seems to be a duplicate for this: #11493 (possibly, but not certain; would be if the issue is platform related or due to changes on 🍎 side)

I'll leave your request for now, as the issue is under investigation. It would be great if you could provide a couple of links to your project/code/actions or at least minimal steps for comparison (something that has become noticeably more time-consuming in your workflow).

[amlcurran]

Hi @erik-bershel appreciate the quick response!

Is there any way I can get runs links to you without exposing the link publicly?

I'm still doing some diving into precisely what has slowed down within the build and will have a look if I find anything.

[ddramowicz]

Hi - our team has also encountered this exact issue in the last 24 hours or so (our repo is also private). However we haven't changed anything on our side and we are now consistently seeing our iOS builds run for about double the time than previously.

[erik-bershel]

@amlcurran
Honestly, I don't even know what to suggest. 😅 I should say in advance that I don't have access to private projects in any case.

• If your project is public, but you don't want to draw attention to it, then you can just tag me somewhere in the comments in this project - I'll notice.
• If your project is private, then feel free to share links - no one in this repository has access to private launches and code.

We can ask for links to private projects to collect some technical information not associated with the project code via internal channels, but we do not get direct access to code and build logs in any case.

In this case, those links might be helpful (the more information, the better).

[shagedorn]

We also see a noticeable slow down across all our workflows (unit tests, UI tests, signed builds), in macOS 15 large runners (using Xcode 16.1, with no setup changes in weeks).

While it seems that pretty much everything is slower than usual, it seems that Swift Package Manager checkouts are particularly slow. But also our UI tests are so slow that they basically cannot be used at all.

[amlcurran]

@erik-bershel thanks - I'm afraid I'd have to get approval to share links which would take an age (as we don't want to expose the structure & names of our private organisation repos)

I'll keep looking into what specifically has slowed down

[amlcurran]

@erik-bershel some initial investigations -

• Compiling our JS bundle has gone from 2 min 15 sec up to 3 min 30 sec
• There seem to be some instances where Touching files - particularly privacy bundles - have shot up in duration, however this isn't 100% consistent.
• These changes wouldn't by themselves wouldn't take us over our timeout limit; so I think there is some general slow down too

Before:

[21:14:50]: ▸ [lottie-react-native-Lottie_React_Native_Privacy] Touching Lottie_React_Native_Privacy.bundle
[21:14:50]: <next command line>

After:

[09:52:50]: ▸ [lottie-react-native-Lottie_React_Native_Privacy] Touching Lottie_React_Native_Privacy.bundle
[09:55:14]: <next command line>

Before:

[21:14:52]: ▸ [PromisesObjC-FBLPromises_Privacy] Touching FBLPromises_Privacy.bundle
[21:14:52]: <next command line>

After:

[09:59:42]: ▸ [PromisesObjC-FBLPromises_Privacy] Touching FBLPromises_Privacy.bundle
[09:59:54]: <next command line>

[erik-bershel]

Hey @shagedorn! 👋

Do you mean -xlarge or -large by macOS 15 large runners?
Do you have any links on the examples maybe? 🙏

[shagedorn]

macos-15-xlarge, sorry for not being clear on that.

One example is this unit test build + run/execution. This kind of workflow runs several times a day, with typical overall runtimes between 10 and 12 minutes. Today, we're in the 18-20 minutes range, and have several test failures because of timeouts in our test code that usually suffice, but start to fail once things get too slow.

From the build reports that we archive (I cannot share those publicly), we also see that basically everything is slower by almost 2x: The actual test execution as well as the build time. A random good run from yesterday finished executing test in 4:38 min, while the same test suite with only minor additions today ran for 8:38 min.

[icecoffin]

Since Jan 28th, we've had ~33% of our iOS app builds be slower on the macos-15 image. Since this morning (30th Jan), that has now increased to 100% of our builds. Whilst previously our builds were taking 20 minutes, they have all started hitting our timeout, which is 30 mins.

We're also experiencing a similar issue, with even the timings being similar (~20 to ~30 minutes). We're using macos-15-xlarge.

Runner image version 20250120.596 - good
Runner image version 20250127.616 - slow

[Vyazovoy]

Private repo, macos-15-xlarge runners, same problem. Our fastlane lane timed out with this command:

INFO [2025-01-30 16:00:32.22]: $ xcodebuild -showBuildSettings -workspace ./<redacted>.xcworkspace -scheme <redacted> -derivedDataPath /Users/runner/work/_temp/<redacted>/DerivedData 2>&1
WARN [2025-01-30 16:00:35.23]: Command timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout...
WARN [2025-01-30 16:00:41.24]: Command timed out after 6 seconds on try 2 of 4, trying again with a 12 second timeout...
WARN [2025-01-30 16:00:53.27]: Command timed out after 12 seconds on try 3 of 4, trying again with a 24 second timeout...
WARN [2025-01-30 16:01:17.29]: Command timed out after 24 seconds on try 4 of 4

Image Version: 20250127.616

And yes, version: 20250120.596 two days ago was working normally

[Nathan-Molby]

Seeing the same issue on macos-15-arm64. Version 20250120.596 is good, 20250127.616 consistently is hitting 30 minute timeouts for tests that used to take 10 minutes.

[erik-bershel]

⏰ ⏰ ⏰

We are rolling back the specified images ( macos-15-arm64, macos-14-arm64).

The changes should take effect in about the next 12 hours. Single agents with the old version may occasionally be encountered over the next 24 hours. Additional details will be provided following further investigation.

[shagedorn]

Thank you. Things look normal again on our end.

[KangxuanYe]

We saw two OS versions on macOS-14-arm64 in deployment.

20250127.793 is still broken for us.
20250120.774 is good.

I am wondering when would all of images be reverted back to 20250120.774? And is this the version you are reverting to? @erik-bershel

[erik-bershel]

We saw two OS versions on macOS-14-arm64 in deployment.

20250127.793 is still broken for us.
20250120.774 is good.

I am wondering when would all of images be reverted back to 20250120.774? And is this the version you are reverting to? >@erik-bershel

I checked - there are currently no environments with version 20250127.793 of macos-14-arm64. Most likely, you came across unused remaining virtual machines - the update process is done by replacing used environments.

[KangxuanYe]

Starting: Initialize job
Agent name: 'Azure Pipelines 44'
Agent machine name: 'Mac-1738779492346'
Current agent version: '4.251.0'
Operating System
macOS
14.7.2
23H311
Runner Image
Image: macos-14-arm64
Version: 20250127.793
Included Software: https://github.com/actions/runner-images/blob/macos-14-arm64/20250127.793/images/macos/macos-14-arm64-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/macos-14-arm64%2F20250127.793
Runner Image Provisioner
2.0.422.1+55c30c14fe2a0a1547db1b656933ae07d97649a9
Current image version: '20250127.793'

Here are some runs we got from pipeline this morning and it appears that there are still many of them using 20250127.793. When I clicked readme, it does show nothing to me. I am wondering how soon those unused/remaining VM can all be returned back to 20250120.774?