vcpkg_download_distfile and .netrc authorization

[mickes27]

Is your feature request related to a problem? Please describe.

Hello. Right now the only way of authorization for vcpkg_download_distfile is via HEADERS. It would be nice to allow authorization via .netrc file, which I think is not possible to do with HTTP headers. I am not using aria2 to download, in the logs I see curl.

Proposed solution

Add some option to vcpkg_download_distfile, which allows to enable .netrc authorization, with optional .netrc file location (--netrc or --netrc-file in case of curl).

Describe alternatives you've considered

Custom-made script which replaces vcpkg_download_distfile completly. I think that vcpkg_download_distfile uses vcpkg x-download, which is not possible to anyhow update.

Additional context

No response

[Neumann-A]

As said on discord you can currently achieve the same by using asset caching.

set X_VCPKG_ASSET_SOURCES=clear;x-script,<path_to_your_custom_download_script> {url} {sha512} {dst};x-block-origin

Where your custom download script should download from url, do a hash check against sha512 and then copy the file to dst.

Or you just skip the hash check and just directly do set/export X_VCPKG_ASSET_SOURCES=clear;x-script,curl -n -o {dst} {url};x-block-origin

[FrankXie05]

cc @vicroms @BillyONeal

[vicroms]

@mickes27

Elaborating on @Neumann-A's response. As a workaround you can leverage the asset caching feature with a custom script. I had Copilot create a PowerShell script (it looks correct to me, but I haven't tested it yet).

You would create this script with the name asset-cache-scripts.ps1 and set the following environment variables:

• VCPKG_ASSET_CACHE_DIR, where your downloads cache will be located
• NETRC_FILE, the path to your .netrc file, and
• VCPKG_KEEP_ENV_VARS with the value NETRC_FILE;VCPKG_ASSET_CACHE_DIR so that the script's required variables are forwarded to vcpkg
• X_VCPKG_ASSET_SOURCES with this configuration clear;x-script,pwsh C:/path/to/asset-cache-script.ps1 {url} {sha512} {dst};x-block-origin.

Example:

set $env:NETRC_FILE="C:\Users\<user>\.netrc"
set $env:VCPKG_ASSET_CACHE_DIR="C:\vcpkg-cache\assets"
set $env:VCPKG_KEEP_ENV_VARS="NETRC_FILE;VCPKG_ASSET_CACHE_DIR"
set $env:X_VCPKG_ASSET_SOURCES="clear;x-script,pwsh C:/path/to/asset-cache-script.ps1 {url} {sha512} {dst};x-block-origin"
vcpkg install ...

asset-cache-script.ps1

param (
    [string]$url,
    [string]$sha,
    [string]$dst
)

# Get the cache directory from the environment variable
$cacheDir = $env:VCPKG_ASSET_CACHE_DIR

if (-not $cacheDir) {
    Write-Error "Environment variable VCPKG_ASSET_CACHE_DIR is not set."
    exit 1
}

# Ensure the cache directory exists
if (-not (Test-Path -Path $cacheDir)) {
    New-Item -ItemType Directory -Path $cacheDir -Force
}

# Define the cache file path
$cacheFile = Join-Path -Path $cacheDir -ChildPath $sha

# Check if the file already exists in the cache
if (Test-Path -Path $cacheFile) {
    Write-Output "File found in cache. Copying to destination."
    Copy-Item -Path $cacheFile -Destination $dst -Force
    Write-Output "File copied from cache to $dst"
    exit 0
}

# Create a temporary file path
$tempFile = [System.IO.Path]::GetTempFileName()

# Get the .netrc file path from the environment variable
$netrcFile = $env:NETRC_FILE

if (-not $netrcFile) {
    Write-Error "Environment variable NETRC_FILE is not set."
    exit 1
}

if (-not (Test-Path -Path $netrcFile)) {
    Write-Error "The .netrc file specified in NETRC_FILE does not exist."
    exit 1
}

# Download the file using curl.exe with .netrc for authorization
curl.exe -L -s --netrc-file $netrcFile -o $tempFile $url

# Calculate the SHA512 hash of the downloaded file
$downloadedSha = Get-FileHash -Path $tempFile -Algorithm SHA512 | Select-Object -ExpandProperty Hash
$downloadedSha = $downloadedSha.ToLower()

# Compare the calculated SHA512 hash with the provided SHA
if ($downloadedSha -eq $sha) {
    # Create the destination directory if it doesn't exist
    $dstDir = Split-Path -Path $dst -Parent
    if (-not (Test-Path -Path $dstDir)) {
        New-Item -ItemType Directory -Path $dstDir -Force
    }

    # Copy the file to the cache
    Copy-Item -Path $tempFile -Destination $cacheFile -Force

    # Move the file to the destination with the provided filename
    Move-Item -Path $tempFile -Destination $dst -Force
    Write-Output "File downloaded and verified successfully. Moved to $dst and cached at $cacheFile"
} else {
    # Remove the temporary file if the SHA doesn't match
    Remove-Item -Path $tempFile -Force
    Write-Output "SHA512 hash mismatch. Downloaded file removed."
}