Azure DevOps pipelines running out of memory

[lukelloydagi]

Is anyone else having issues with Azure DevOps pipelines running out of memory recently?

I'm trying to figure out if it's MegaLinter that's grown or underlying MS infrastructure (which we've seen a lot of lately).

Would reducing CPU usage with PARALLEL_PROCESS_NUMBER reduce memory usage too?

[lukelloydagi]

Well...that answers that, the PARALLEL_PROCESS_NUMBER setting made no difference 😞

[lukelloydagi]

Tried using the megalinter-dotnet image to see if a smaller image size would make a difference....it did not 😞

[nvuillam]

@lukelloydagi Are you using custom self-hosted Azure Runners ?

I have many project using Azure, and have not detected similar issues :/

[lukelloydagi]

@nvuillam, I've reverted to v8.1 and I don't seem to have the issue, I only have the issue with v8.2+ using either default or dotnet flavor. Is there anything in those releases that's likely to be the cause? If not I will have to raise an issue with MS 😭

[nvuillam]

@lukelloydagi just to see if it is dotnet related, please can you try to run it with DOCUMENTATION or CI_LIGHT flavor ?

[lukelloydagi]

@lukelloydagi just to see if it is dotnet related, please can you try to run it with DOCUMENTATION or CI_LIGHT flavor ?

@nvuillam, I can do but none of those flavors contain arm-ttk linters so would not lint my codebase.

It's odd as it seems to only have the issue with v8.2+ on one of my bigger ARM template repos.
It's only configured to lint changed files so size of repo shouldn't matter?
Also I would have thought size of repo (and image for that matter) would only effect disk space not memory?

[lukelloydagi]

@nvuillam, if I reduce the linters (by only specifying a list using the ENABLE configuration variable) then I also don't get the issue. It therefore clearly relates to the linters that are running against the repo and maybe it's the linter(s) that are scanning the entire repo.

Has anything changed between v8.2+ in how the linters are running/consuming memory?

[nvuillam]

@lukelloydagi If a linter cli_lint_mode is in project mode, it does not lint only new files, but all files

By playing with enabling/disabling linters, did you identify the linter(s) that makes your CI job fail ?

I don't think we changed anything memory related between 8.2 & 8.3 :/ What changed are the linter versions, that have been upgraded, so maybe one of them has new performance issues or a memory leak :/

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed in 14 days if no further activity occurs.
Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[lukelloydagi]

@lukelloydagi Are you using custom self-hosted Azure Runners ?

I have many project using Azure, and have not detected similar issues :/

Sorry for the delay in responding, I've just been kind of living with it for now as it only affects my one repo but it's starting to become annoying so assigning some time to fix.

I'm using the Azure hosted pipeline agents but I am going to spin up a self-hosted agent as I'm sure it will fix the issue by giving it a higher spec VM.

I'm still trying to find an alternative way around too as I don't really want the additional cost of running self-hosted agents :(

[lukelloydagi]

@nvuillam, interestingly I've just updated my pipeline to use v8.4.2 and the problems seems to disappear 😎 Maybe there was a memory leak somewhere in v8.2 & v8.3 🤷‍♂️

[nvuillam]

@nvuillam, interestingly I've just updated my pipeline to use v8.4.2 and the problems seems to disappear 😎 Maybe there was a memory leak somewhere in v8.2 & v8.3 🤷‍♂️

Wonderful, thanks for the feedback 😊