Convert consolidated pipeline to a YAML based pipeline and enable nightly builds #4292

aishwaryabh · 2025-03-10T19:59:56Z

Issue describing the changes in this PR

resolves #4270 #4271
This PR converts the consolidated pipeline which was initially based off of azure classic pipelines UX to a yaml based pipeline.

I have updated the ArtifactAssembler within core tools to be able to process each artifact type one at a time. For example, if I were to pass in Azure.Functions.Cli.win-x64 when running the artifact assembler, the artifact assembler would only assemble artifacts for that win-x64. This enables us to use parallelization when running the consolidated pipeline, which significantly improves the run time overall.

The consolidated pipeline also publishes the artifacts to the pipeline directly, rather than publishing to a storage account. Please look at this run for an example of what this would look like.

This pipeline also runs significantly faster than the azure classic based pipeline, as this pipeline takes on average 35-40 min, whereas previous runs would easily take over an hour. This runtime will increase even more once the E2E tests are updated to run in a more efficient and parallel manner outside of the consolidated pipeline.

Finally, nightly builds are enabled for the consolidated artifact pipeline to run every night, after the in-proc and main artifacts have been created from their nightly builds. This PR also adds a tag called nightly-build to those builds for the in-proc and main branch to distinguish scheduled builds from manually triggered ones. That way the consolidated pipeline knows which builds to pick for the nightly builds.

Pull request checklist

My changes do not require documentation changes
- Otherwise: Documentation issue linked to PR
My changes do not need to be backported to a previous version
- Otherwise: Backport tracked by issue/PR #issue_or_pr
I have added all required tests (Unit tests, E2E tests)

…tools into aibhandari/pipeline-updates

…ts yaml

liliankasem · 2025-03-10T21:24:54Z

eng/ci/consolidate-artifacts.yml

+          artifact: drop
+          displayName: "Download out-of-proc artifact"
+
+        - powershell: |


Can we convert all of of the inline pwsh scropts into .ps1 script files and use the powershell task to run the script?

Also a side note: use pwsh task instead of powershell for inline scripts. The former is the newer cross-plat pscore. The latter is the old windows-only powershell.

liliankasem · 2025-03-10T21:25:49Z

eng/ci/consolidate-artifacts.yml

+
+      - job: AssembleArtifacts
+        strategy:
+          matrix:


Could this be simplied with a template where you pass in an array of the platforms/arch you want to build?

liliankasem · 2025-03-10T21:26:21Z

eng/ci/consolidate-artifacts.yml

+            arguments: '-StagingDirectory "$(Pipeline.Workspace)\staging\coretools-cli"'
+            workingDirectory: '$(Build.SourcesDirectory)/src/Azure.Functions.ArtifactAssembler'
+          condition: and(succeeded(), or(eq(variables['artifactName'], 'Azure.Functions.Cli.win-x64'), eq(variables['artifactName'], 'Azure.Functions.Cli.win-x86')))
+        - task: DotNetCoreCLI@2


nit: blank lines between each task to improve readability

liliankasem · 2025-03-10T21:27:37Z

eng/ci/consolidate-artifacts.yml

+    stages:
+    - stage: ConsolidateArticacts
+      displayName: "Assemble Artifacts"
+      jobs:


can each job be its own file

Consider breaking things out into job files and step files similar to how we do in the host:

https://github.com/Azure/azure-functions-host/tree/dev/eng/ci/templates

jviau · 2025-03-11T16:28:26Z

eng/ci/consolidate-artifacts.yml

+        steps:
+        - checkout: none
+
+        - download: core-tools-default


Use templateContext.inputs instead

jviau · 2025-03-11T16:31:39Z

eng/ci/consolidate-artifacts.yml

+          artifact: drop
+          displayName: "Download out-of-proc artifact"
+
+        - powershell: |


Also a side note: use pwsh task instead of powershell for inline scripts. The former is the newer cross-plat pscore. The latter is the old windows-only powershell.

jviau · 2025-03-11T16:37:37Z

eng/ci/consolidate-artifacts.yml

+            arguments: '-ArtifactsPath "$(Pipeline.Workspace)\staging\coretools-cli"'
+            workingDirectory: '$(Build.SourcesDirectory)/src/Azure.Functions.ArtifactAssembler'
+
+        - task: EsrpCodeSigning@5


Please use the sign-files.yml@eng template:

https://github.com/Azure/azure-functions-host/blob/dev/eng/ci/templates/official/jobs/build-artifacts-windows.yml#L91

This gives us a single place to manage and update our ESRP connection.

jviau · 2025-03-11T16:39:57Z

eng/ci/consolidate-artifacts.yml

+            filePath: '$(Build.SourcesDirectory)/src/Azure.Functions.ArtifactAssembler/PipelineHelpers/generateMetadataFile.ps1'
+            arguments: '-StagingDirectory "$(Pipeline.Workspace)\staging"'
+            workingDirectory: '$(Pipeline.Workspace)'
+            condition: and(succeeded(), eq(variables['artifactName'], 'Azure.Functions.Cli.min.win-x64')) 


When possible, use templating syntax for conditions:

- ${{ if eq(variables.artifactName, 'Azure.Functions.Cli.min.win-x64') }}: - task: PowerShell@2

The benefits with this are:

It is calculated at templating time, so if the if-statement does not pass, this task doesn't show up at all in the pipeline. (the condition: syntax it shows up, but as a grey skipped step).

No need to worry about and(succeeded()) - that will be implied.

jviau · 2025-03-11T16:45:34Z

eng/ci/consolidate-artifacts.yml

+          inputs:
+            command: run
+            projects: '$(Build.SourcesDirectory)/src/Azure.Functions.ArtifactAssembler/Azure.Functions.ArtifactAssembler.csproj'
+            arguments: 'zip  -c release'


I thought you had to supply app arguments like so for dotnet run:

Suggested change

arguments: 'zip -c release'

arguments: '-c release -- zip'

The -- delimits arguments for run vs arguments for the application, as documented here https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-run

jviau · 2025-03-11T16:50:43Z

eng/ci/consolidate-artifacts.yml

+
+        - pwsh: |
+            # Guardian files in TEMP add up and can cause the AntiMalware task to timeout.
+            Get-ChildItem -Path $env:TEMP -Filter 'MpCmdRun.*' -Recurse -ErrorAction SilentlyContinue | Remove-Item -Recurse -Force


Are these artifacts you have generated in this pipeline? Or from SDL? We shouldn't be deleting SDL files.

jviau · 2025-03-11T16:53:08Z

eng/ci/consolidate-artifacts.yml

+          inputs:
+            command: run
+            projects: '$(Build.SourcesDirectory)/src/Azure.Functions.ArtifactAssembler/Azure.Functions.ArtifactAssembler.csproj'
+            arguments: 'zip  -c release'


Can gain slight improvements by building ArtifactAssembler once, and then using dotnet run --no-build -c release after that.

jviau · 2025-03-11T16:55:42Z

eng/ci/templates/official/jobs/build-test.yml

@@ -81,6 +81,17 @@ jobs:
  - pwsh: |
      .\check-vulnerabilities.ps1
    displayName: "Check for security vulnerabilities"
+  - pwsh: |


Can also condition this step:

- ${{ if eq(variables['Build.Reason'], 'Schedule') }}: - pwsh: | Write-Host "##vso[build.addbuildtag]nightly-build" displayName: Tag nightly

jviau · 2025-03-11T16:58:34Z

src/Azure.Functions.ArtifactAssembler/ArtifactAssembler.cs

+            }
+            else
+            {
+                await Task.Run(() => FileUtilities.CopyDirectory(artifactZipPath, destinationDirectory));


await Task.Run here doesn't offer any benefit. It just adds extra thread pool scheduling. The only time to use Task.Run is if you really don't want that work on the current thread (and want to delegate it to the thread pool). An example is to do a fire & forget as part of an HTTP request. Or in old AspNet (not AspNetCore), HTTP requests had a special sync context and it was crucial to offload work to the threadpool to free up that sync context.

But given this is just a cli tool, there is no real multi-threaded-ness going on here and no restrictions on the current thread.

Suggested change

await Task.Run(() => FileUtilities.CopyDirectory(artifactZipPath, destinationDirectory));

FileUtilities.CopyDirectory(artifactZipPath, destinationDirectory);

jviau · 2025-03-11T17:00:20Z

src/Azure.Functions.ArtifactAssembler/ArtifactAssembler.cs

@@ -225,7 +274,7 @@ private async Task CreateVisualStudioCoreToolsAsync()
            Directory.CreateDirectory(consolidatedArtifactDirPath);

            // Copy in-proc8 files and delete directory after
-            await Task.Run(() => FileUtilities.CopyDirectory(inProcArtifactDirPath, createDirectory ? Path.Combine(consolidatedArtifactDirPath, Constants.InProc8DirectoryName): consolidatedArtifactDirPath));
+            await Task.Run(() => FileUtilities.CopyDirectory(inProcArtifactDirPath, createDirectory ? Path.Combine(consolidatedArtifactDirPath, Constants.InProc8DirectoryName) : consolidatedArtifactDirPath));


Suggested change

await Task.Run(() => FileUtilities.CopyDirectory(inProcArtifactDirPath, createDirectory ? Path.Combine(consolidatedArtifactDirPath, Constants.InProc8DirectoryName) : consolidatedArtifactDirPath));

FileUtilities.CopyDirectory(inProcArtifactDirPath, createDirectory ? Path.Combine(consolidatedArtifactDirPath, Constants.InProc8DirectoryName) : consolidatedArtifactDirPath);

liliankasem · 2025-03-12T18:05:21Z

eng/ci/consolidate-artifacts.yml

+      - template: /eng/ci/templates/official/jobs/assemble-artifacts.yml@self
+        parameters:
+          arch: 'win-x64'
+          displayName: 'winx64'


Is display name needed? You can use the arch value as the display name for the job, no?

that's what I thought but display name has to strictly be alpha-numeric (so no - or . allowed which is in all of the arch names)

liliankasem · 2025-03-12T18:07:31Z

eng/ci/consolidate-artifacts.yml

+          arch: 'win-x64'
+          displayName: 'winx64'
+
+      - template: /eng/ci/templates/official/jobs/assemble-artifacts.yml@self


This is looking better! Now we can go a step further and create a variable:

variables: archs: - min.win-x64 - osx-x64 - linux-x64

Then loop for each arch

jobs: - ${{ each arch in variables.archs }}: - template: /eng/ci/templates/official/jobs/assemble-artifacts.yml@self parameters: arch: ${{ arch }}

aishwaryabh added 5 commits March 6, 2025 11:20

add tag to nightly build

466ba4d

consolidate-artifacts.yml file

1f01b96

Merge branch 'main' of https://github.com/Azure/azure-functions-core-…

7224ca4

…tools into aibhandari/pipeline-updates

making artifact assembler run parallel and update consolidate artifac…

53dc17f

…ts yaml

fixing typo

9d5b2a5

aishwaryabh requested a review from a team as a code owner March 10, 2025 19:59

update read me

0982b1a

liliankasem reviewed Mar 10, 2025

View reviewed changes

jviau requested changes Mar 11, 2025

View reviewed changes

microsoft-github-policy-service bot added the Needs: Author Feedback label Mar 11, 2025

addressing comments; lets see if this compiles lol

12ac035

microsoft-github-policy-service bot removed the Needs: Author Feedback label Mar 11, 2025

aishwaryabh added 12 commits March 11, 2025 11:51

fixing typos

759144f

making sure each job is alphanumeric

90b9519

display name

26ad90b

adding assemble artifacts path

c1b1c30

trying to assemble artifact input

8a758e2

add artifact name

d285852

fix input

8eb1a1c

trying to print out conetnt

3590901

fix typo

aad4681

fixing path

d598255

addressing more comments

b3245fa

remove no build param

ed62ce7

liliankasem reviewed Mar 12, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert consolidated pipeline to a YAML based pipeline and enable nightly builds #4292

Convert consolidated pipeline to a YAML based pipeline and enable nightly builds #4292

aishwaryabh commented Mar 10, 2025 •

edited

Loading

liliankasem Mar 10, 2025

jviau Mar 11, 2025

liliankasem Mar 10, 2025

liliankasem Mar 10, 2025

liliankasem Mar 10, 2025

liliankasem Mar 10, 2025

jviau Mar 11, 2025

jviau Mar 11, 2025

jviau Mar 11, 2025

jviau Mar 11, 2025

jviau Mar 11, 2025

jviau Mar 11, 2025

jviau Mar 11, 2025

jviau Mar 11, 2025

jviau Mar 11, 2025

jviau Mar 11, 2025

liliankasem Mar 12, 2025 •

edited

Loading

aishwaryabh Mar 12, 2025 •

edited

Loading

liliankasem Mar 12, 2025

	await Task.Run(() => FileUtilities.CopyDirectory(artifactZipPath, destinationDirectory));
	FileUtilities.CopyDirectory(artifactZipPath, destinationDirectory);

	await Task.Run(() => FileUtilities.CopyDirectory(inProcArtifactDirPath, createDirectory ? Path.Combine(consolidatedArtifactDirPath, Constants.InProc8DirectoryName) : consolidatedArtifactDirPath));
	FileUtilities.CopyDirectory(inProcArtifactDirPath, createDirectory ? Path.Combine(consolidatedArtifactDirPath, Constants.InProc8DirectoryName) : consolidatedArtifactDirPath);

Convert consolidated pipeline to a YAML based pipeline and enable nightly builds #4292

Are you sure you want to change the base?

Convert consolidated pipeline to a YAML based pipeline and enable nightly builds #4292

Conversation

aishwaryabh commented Mar 10, 2025 • edited Loading

Issue describing the changes in this PR

Pull request checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liliankasem Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

aishwaryabh Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aishwaryabh commented Mar 10, 2025 •

edited

Loading

liliankasem Mar 12, 2025 •

edited

Loading

aishwaryabh Mar 12, 2025 •

edited

Loading