Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipelines | Fix dotnet Installation on ARM64 #3175

Open
wants to merge 42 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 36 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
bef94f5
Use script to install .NET because of ARM bug
MichelZ Nov 26, 2024
a7fdb5e
YAML
MichelZ Nov 26, 2024
826652a
Add back steps:
MichelZ Nov 26, 2024
db380f5
Fix name
MichelZ Nov 26, 2024
211a8cb
Try without ./
MichelZ Nov 26, 2024
f59a9b0
Maybe they need to hang back a level
MichelZ Nov 26, 2024
04732fc
Replace CURL
MichelZ Nov 26, 2024
383a994
Switch to powershell
MichelZ Nov 26, 2024
6e279f6
Need a Windows & non-Windows version
MichelZ Nov 26, 2024
296ff01
YAML
MichelZ Nov 26, 2024
b3c8c91
Use .0 for WIndows and .X for other
MichelZ Nov 26, 2024
6e63c28
Update install command to include Quality and only install Runtime wh…
MichelZ Nov 27, 2024
2a90fc7
Improve way to specify SDK vs Runtime
MichelZ Nov 27, 2024
ee60407
Make it a string
MichelZ Nov 27, 2024
cc301b7
Use whole number only
MichelZ Nov 27, 2024
0a9e5ca
We don't need to use "
MichelZ Nov 27, 2024
2490f31
Add additional options, make output cleaner
MichelZ Nov 27, 2024
9f69854
Runtime variables like Agent.OS do not work
MichelZ Nov 27, 2024
56d7dfc
Path to ensure-dotnet-version
MichelZ Nov 27, 2024
77da2e5
Condition not possible on template
MichelZ Nov 27, 2024
94761c0
Improve version detection
MichelZ Nov 27, 2024
bf4be0e
String
MichelZ Nov 27, 2024
eaa6db4
booleans...
MichelZ Nov 27, 2024
c0e0541
Use SDK for X86, use installDir
MichelZ Nov 27, 2024
dc2df9c
InstallDir
MichelZ Nov 27, 2024
675118f
PS String comparison
MichelZ Nov 27, 2024
7428b7b
Cosmetics
MichelZ Nov 27, 2024
88e057e
Add new steps to sln, formatting
benrr101 Feb 21, 2025
3e2e54b
Cleaning up dotnet install task.
benrr101 Feb 21, 2025
2c09144
Fixing call to the new task
benrr101 Feb 25, 2025
77c6c7b
Same operation in a different file
benrr101 Feb 25, 2025
9fc3ea5
Default to LTS ?
benrr101 Feb 25, 2025
48e0e44
Version vs channel is kind of annoying
benrr101 Feb 25, 2025
cc99e09
Fix logic for determining which task to use, just always using 9.x
benrr101 Feb 25, 2025
a1ad4e3
???
benrr101 Feb 25, 2025
d5242fd
Reinstate installing dotnet as part of ci-project-build-steps
benrr101 Feb 26, 2025
e78556f
Just don't use preview builds.
benrr101 Feb 27, 2025
cf842e2
Debug agent OS
benrr101 Feb 28, 2025
e5f9a13
debug part 2
benrr101 Mar 7, 2025
603f62d
Debug part 3?
benrr101 Mar 7, 2025
cf27dfb
DEBUG PART 4.
benrr101 Mar 7, 2025
b31abca
This should be the right way to get the agent os. Powershell task sho…
benrr101 Mar 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,10 @@ jobs:
- task: NuGetAuthenticate@1
displayName: 'NuGet Authenticate'

- task: UseDotNet@2
displayName: 'Use .NET SDK 8.0.x'
inputs:
version: 8.0.x
- template: ../steps/ensure-dotnet-version.yml@self
parameters:
packageType: 'sdk'
version: '8.0.x'

- ${{ parameters.downloadArtifactsSteps }}

Expand Down
34 changes: 14 additions & 20 deletions eng/pipelines/common/templates/jobs/ci-run-tests-job.yml
Original file line number Diff line number Diff line change
Expand Up @@ -231,26 +231,20 @@ jobs:
referenceType: ${{ parameters.buildType }}
testSet: ${{ parameters.testSet }}
operatingSystem: ${{ parameters.operatingSystem }}

- ${{ if and(eq(parameters.enableX86Test, true), eq(parameters.operatingSystem, 'Windows')) }}: # run x86 tests
- powershell: |
dotnet --info

Invoke-WebRequest https://dot.net/v1/dotnet-install.ps1 -OutFile dotnet-install.ps1

# install .net x86
$version = "LTS"
if (!"${{parameters.targetFramework }}".StartsWith("net4"))
{
$version = "${{parameters.targetFramework }}".Substring(3, "${{parameters.targetFramework }}".Length-3)
}

.\dotnet-install.ps1 -Channel $version -Architecture x86 -InstallDir "$(dotnetx86RootPath)"

$(dotnetx86RootPath)dotnet.exe --info
displayName: 'Install .NET x86 '
condition: ne(variables['dotnetx86RootPath'], '')


- ${{ if and(eq(parameters.enableX86Test, true), eq(parameters.operatingSystem, 'Windows')) }}:
# Set up for x86 tests by manually installing dotnet for x86 to an alternative location. This
# is only used to execute the test runtime (framework to test is specified in build params), so
# it should be acceptable to just install a specific version in all cases.
- ${{ if ne(variables['dotnetx86RootPath'], '') }}:
- template: ../steps/ensure-dotnet-version.yml
parameters:
installDir: $(dotnetx86RootPath)
packageType: sdk
usePreview: false
version: 9.x
windowsArchitecture: x86

- template: ../steps/run-all-tests-step.yml@self
parameters:
debug: ${{ parameters.debug }}
Expand Down
16 changes: 8 additions & 8 deletions eng/pipelines/common/templates/steps/ci-prebuild-step.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,17 +20,17 @@ parameters:
- Package

steps:
- task: UseDotNet@2
displayName: 'Use .NET 9.x sdk'
inputs:
- template: ensure-dotnet-version.yml
parameters:
packageType: sdk
version: '9.x'
usePreview: false
version: 9.x

- task: UseDotNet@2
displayName: 'Install .NET 8.x runtime'
inputs:
- template: ensure-dotnet-version.yml
parameters:
packageType: runtime
version: '8.x'
usePreview: false
version: 8.x

- ${{if eq(parameters.debug, true)}}:
- powershell: |
Expand Down
18 changes: 8 additions & 10 deletions eng/pipelines/common/templates/steps/ci-project-build-step.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,17 +34,15 @@ parameters:
- all

steps:
- task: UseDotNet@2
displayName: 'Use .NET 9.x sdk'
inputs:
packageType: sdk
version: '9.x'
- template: ./ensure-dotnet-version.yml@self
parameters:
packageType: 'sdk'
version: '9.x'

- task: UseDotNet@2
displayName: 'Install .NET 8.x runtime'
inputs:
packageType: runtime
version: '8.x'
- template: ./ensure-dotnet-version.yml@self
parameters:
packageType: 'runtime'
version: '8.x'

- ${{ if or(eq(parameters.operatingSystem, 'Windows'), eq(parameters.operatingSystem, 'deferedToRuntime')) }}:
- ${{ if or(eq(parameters.build, 'MDS'), eq(parameters.build, 'all')) }}:
Expand Down
102 changes: 102 additions & 0 deletions eng/pipelines/common/templates/steps/ensure-dotnet-version.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
#################################################################################
# Licensed to the .NET Foundation under one or more agreements. #
# The .NET Foundation licenses this file to you under the MIT license. #
# See the LICENSE file in the project root for more information. #
#################################################################################

# Installs dotnet sdk/runtime using UseDotNet on non-windows agents, but uses a custom ps script
# for installation on Windows.
#
# Reason for not using UseDotNet task:
# [BUG]: UseDotNet task installs x86 build on Windows arm64
# https://github.com/microsoft/azure-pipelines-tasks/issues/20300

parameters:
- # Directory where dotnet binaries should be installed. If not specified, defaults to the
# agent's tools directory.
name: installDir
type: string
default: ''

- # Whether to install the full SDK or the "dotnet" runtime
name: packageType
type: string
values:
- runtime
- sdk

- # Whether to enable preview versions of dotnet
name: usePreview
type: boolean

- # Version to install - supports full major.minor.patch, but can be shortened to major.minor
# to get the latest within a given channel.
name: version
type: string

- # Specifies the architecture to install, only applies on Windows agents. If not provided,
# defaults to the same architecture as the agent.
name: windowsArchitecture
type: string
default: '<auto>'
values:
- '<auto>'
- 'x86'
- 'x64'
- 'arm64'

steps:
- ${{ if eq(variables['Agent.OS'], 'Windows_NT') }}:
- powershell: |-
# Propagate parameters from pipeline #######################
$agentToolsDir = '$(Agent.ToolsDirectory)'
echo "agentToolsDir= $agentToolsDir"

$architecture = '${{ parameters.windowsArchitecture }}'
echo "architecture= $architecture"

$installDir = '${{ parameters.installDir }}'
echo "installDir= $installDir"

$packageType = '${{ parameters.packageType }}'
echo "packageType= $packageType"

$usePreview = ${{ parameters.usePreview }}
echo "usePreview= $usePreview"

$version = ${{ parameters.version }}
echo "version= $version"

# Install dotnet ###########################################
Invoke-WebRequest -Uri "https://dot.net/v1/dotnet-install.ps1" -OutFile "dotnet-install.ps1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this will end up becoming a blocker for onebranch conversion... we can cross that bridge when we get to it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, uhm ... maybe. I'll have to dig through the network restrictions list again.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you leverage 1es artifact instead? this one: https://dev.azure.com/devdiv/XlabImageFactory/_git/XlabImageFactory?path=/artifacts/windows-dotnetcore-sdk/dotnet-install.ps1 supports fetching all SDK variants and handles arm64 too.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fetching anything external (assuming "dot.net" is considered external) from an official OB pipeline will not fly. But if it's just a test pipeline, might be okay.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can try that, though it's not a publicly accessible ADO instance.


$installDir = If ($installDir) { $installDir } Else { $agentToolsDir }
$installParams = @{
Architecture = $architecture
Channel = $version
InstallDir = $installDir
Quality = If ($usePreview) { "GA" } Else { "preview" }
}

if ($packageType -eq 'runtime') {
$installParams.add('Runtime', 'dotnet')
}

& ./dotnet-install.ps1 @installParams
Remove-Item dotnet-install.ps1 -ErrorAction:Ignore

# Propagate values back out to pipeline ####################
Write-Host "##vso[task.setvariable variable=DOTNET_ROOT]${$installDir}"
Write-Host "##vso[task.prependpath]${$installDir}"
displayName: 'Install dotnet ${{ parameters.version }} ${{ parameters.packageType }} for Windows ${{ parameters.windowsArchitecture }}'

- ${{ else }}:
- task: UseDotNet@2
displayName: 'Install dotnet ${{ parameters.version }} ${{ parameters.packageType }}'
inputs:
${{ if ne(parameters.installDir, '') }}:
installationPath: '${{ parameters.installDir }}'
packageType: '${{ parameters.packageType }}'
version: '${{ parameters.version }}'

includePreviewVersions: '${{ parameters.usePreview }}'
14 changes: 6 additions & 8 deletions eng/pipelines/common/templates/steps/pre-build-step.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,14 @@
# See the LICENSE file in the project root for more information. #
#################################################################################
steps:
- task: UseDotNet@2
displayName: 'Use .NET 9.x sdk'
inputs:
packageType: sdk
- template: ./ensure-dotnet-version.yml@self
parameters:
packageType: 'sdk'
version: '9.x'

- task: UseDotNet@2
displayName: 'Install .NET 8.x runtime'
inputs:
packageType: runtime
- template: ./ensure-dotnet-version.yml@self
parameters:
packageType: 'runtime'
version: '8.x'

- script: SET
Expand Down
2 changes: 2 additions & 0 deletions src/Microsoft.Data.SqlClient.sln
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,8 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "steps", "steps", "{EABE3A3E
..\eng\pipelines\common\templates\steps\run-all-tests-step.yml = ..\eng\pipelines\common\templates\steps\run-all-tests-step.yml
..\eng\pipelines\common\templates\steps\update-config-file-step.yml = ..\eng\pipelines\common\templates\steps\update-config-file-step.yml
..\eng\pipelines\common\templates\steps\update-nuget-config-local-feed-step.yml = ..\eng\pipelines\common\templates\steps\update-nuget-config-local-feed-step.yml
..\eng\pipelines\common\templates\steps\configure-sql-server-macos-step.yml = ..\eng\pipelines\common\templates\steps\configure-sql-server-macos-step.yml
..\eng\pipelines\common\templates\steps\ensure-dotnet-version.yml = ..\eng\pipelines\common\templates\steps\ensure-dotnet-version.yml
EndProjectSection
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "libraries", "libraries", "{75BAE755-3A1F-41F2-9176-9F8FF9FEE2DD}"
Expand Down