Followup: how to deal with automated code coverage checks #443

[amoerie]

Hi, I finally had the time to refactor our Github workflows, using the ideas found in this repository.
If you recall, we have a SonarQube installation that enforces minimum code coverage rules. The problem was that, when we would use incrementalist to only run the tests of code that changed, SonarQube would complain that our code coverage decreased because it wouldn't get full reports.

The solution to this problem is to automatically download the test artifacts from an earlier build (the merge base) and re-upload those as our own test artifacts, as if we just ran the tests after all.

I have written a powershell script that automatically produces a text file "affected-projects/affected-projects.txt" and another file "affected-projects/merge-base.txt" that contains the commit sha of the commit where the current feature branch was started from the main branch.

Initially, this script used incrementalist, but I quickly ran into some issues:

• As far as I could find, this tool does not support providing a specific commit sha, only a branch. This means the "merge base" trick is unusable if the target branch of the PR has advanced since the feature branch was created.
• The behavior of the detected changes seemed broken. It was detecting changes that were recently merged into the main branch, but not changes that were recently added to my feature branch. Maybe this is related to my previous point, maybe not.
• (This is from memory, I can't easily test this again so forgive me if this is wrong) If files are not explicitly included in a .csproj, they would not count as changes. This is problematic for frontend files such as package-lock.json.

I discovered a "competitor" of this package called "dotnet-affected", and simply swapping one tool for another fixed these issues. They work 99% the same, so the transition was a breeze. I hope it is okay to mention that here, I'm just saying all this to share my experience and my lessons learned.

Anyway, here is what my github workflows and scripts look like, I'm quite happy with them!

The flow is now as follows:

If a workflow is triggered outside a PR: nothing changes, everything is built & tested

If a workflow is triggered for a PR:

1. Calculate affected projects
2. Try to download existing test artifacts of an earlier build from the merge base
3. If that succeeds, build & test is skipped and the test artifacts are reused
4. If that fails, the build & test proceeds anyway

In a happy path, this triples our workflow speed!

  Calculate-Affected-Projects:
    outputs:
      affected-projects: ${{ steps.set-affected.outputs.projects }}
      merge-base: ${{ steps.set-affected.outputs.merge-base }}
    steps:
      - name: Checkout
        uses: actions/checkout@v6
      - name: Calculate Affected Projects
        shell: pwsh
        run: ./Scripts/DotNet.CalculateAffectedProjects.ps1 -targetBranch ${{ github.event.pull_request.base.ref }}
      - name: Set Affected Projects Output
        id: set-affected
        run: |
          if (Test-Path "./affected-projects/affected-projects.txt") {
            $affectedProjects = Get-Content "./affected-projects/affected-projects.txt" -Raw
            # Escape for JSON and remove newlines for GitHub Actions output
            $affectedProjects = $affectedProjects -replace "`r`n", "`n" -replace "`n", "||"
            echo "projects=$affectedProjects" >> $env:GITHUB_OUTPUT
          } else {
            echo "projects=" >> $env:GITHUB_OUTPUT
          }
          
          # Set merge base output (empty if not in PR context)
          if (Test-Path "./affected-projects/merge-base.txt") {
            $mergeBase = Get-Content "./affected-projects/merge-base.txt" -Raw
            echo "merge-base=$mergeBase" >> $env:GITHUB_OUTPUT
          } else {
            echo "merge-base=" >> $env:GITHUB_OUTPUT
          }
        shell: pwsh

Now that we have this, our build + test job can look like this:

  MyProject-Build:
    needs: Calculate-Affected-Projects
    steps:
      - name: Checkout
        uses: actions/checkout@v6
      - name: Download Cached Test Artifacts from Merge Base
        id: download-cached-myproject
        uses: dawidd6/action-download-artifact@v11
        if: |
          github.event_name == 'pull_request' &&
          !contains(needs.Calculate-Affected-Projects.outputs.affected-projects, 'MyProject.Tests/MyProject.Tests.csproj')
        continue-on-error: true
        with:
          commit: ${{ needs.Calculate-Affected-Projects.outputs.merge-base }}
          name: MyProject-Test-Artifacts
          path: ./cached-test-artifacts/MyProject.Tests
          check_artifacts: true
          search_artifacts: true
      - name: My Project Build
        if: |
          github.event_name != 'pull_request' ||
          contains(needs.Calculate-Affected-Projects.outputs.affected-projects, 'MyProject.Tests/MyProject.Tests.csproj') ||
          steps.download-cached-myproject.outcome == 'failure'
        run: |
          ./Scripts/DotNet.Build.ps1 -project MyProject
          ./Scripts/DotNet.Test.ps1  -project MyProject.Tests
        shell: pwsh
      - name: Use Cached MyProject Test Artifacts (when build skipped)
        if: |
          github.event_name == 'pull_request' &&
          !contains(needs.Calculate-Affected-Projects.outputs.affected-projects, 'MyProject.Tests/MyProject.Tests.csproj') &&
          steps.download-cached-myproject.outcome == 'success'
        run: |
          ./Scripts/CopyCachedArtifacts.ps1 -sourceDirectory "./cached-test-artifacts/MyProject.Tests" -targetDirectory "./MyProject.Tests/build/test-results"
        shell: pwsh
      - name: Upload My Project Build Artifacts
        if: |
          github.event_name != 'pull_request' ||
          contains(needs.Calculate-Affected-Projects.outputs.affected-projects, 'MyProject/MyProject.csproj')
        uses: actions/upload-artifact@v5
        with:
          name: MyProject-Build-Artifacts
          path: |
            ${{ github.workspace }}/MyProject/nuget
          retention-days: 30
      - name: Upload My Project Test Artifacts
        uses: actions/upload-artifact@v5
        with:
          name: MyProject-Test-Artifacts
          path: |
            ${{ github.workspace }}/MyProject.Tests/build/test-results/*.xml
          retention-days: 30

This is how DotNet.CalculateAffectedProjects.ps1 does its work:

param(
    [Parameter()]
    [string]$targetBranch,
    [Parameter()]
    [string]$outputDir = "affected-projects",
    [Parameter()]
    [string]$solution = "MySolution.sln"
)

$ErrorActionPreference = "Stop"

$root = Resolve-Path(Join-Path $PSScriptRoot "../")
Push-Location $root
try {
    Write-Host ""
    Write-Host "***********************************************" -ForegroundColor Cyan
    Write-Host "[CalculateAffectedProjects] Running dotnet-affected" -ForegroundColor Cyan
    Write-Host "***********************************************" -ForegroundColor Cyan
    Write-Host ""
    Write-Host "targetBranch   = $targetBranch" -ForegroundColor Green
    Write-Host "outputDir      = $outputDir" -ForegroundColor Green
    Write-Host "solution       = $solution" -ForegroundColor Green
    Write-Host ""

    # Skip this optimization if we're not in a pull request
    if ([string]::IsNullOrWhiteSpace($targetBranch)) {
        Write-Host "Not in a pull request context - all projects will be built" -ForegroundColor Yellow
        
        # Create output directory
        if (!(Test-Path $outputDir)) {
            New-Item -Path $outputDir -ItemType Directory -Force | Out-Null
        }
        
        # Create empty affected projects file (will be interpreted as "build all" by GitHub Actions)
        "" | Out-File -FilePath "$outputDir/affected-projects.txt" -Encoding UTF8
        exit 0
    }

    # Unshallowing git branch if necessary
    Write-Host "Checking if repository is shallow..." -ForegroundColor Cyan
    $isShallow = [bool]::Parse($(git rev-parse --is-shallow-repository))

    if($isShallow)
    {
        Write-Host "Repository is shallow; performing git fetch --unshallow" -ForegroundColor Yellow
        git fetch --unshallow
    }
    else
    {
        Write-Host "Repository is not shallow, continuing..." -ForegroundColor Green
    }

    # Calculate the merge base
    Write-Host "Calculating merge base..." -ForegroundColor Cyan
    $currentRef = git rev-parse HEAD
    Write-Host "Current HEAD: $currentRef" -ForegroundColor Gray
    
    # Fetch the target branch to ensure we have the latest
    Write-Host "Fetching target branch: $targetBranch" -ForegroundColor Cyan
    git fetch origin $targetBranch
    if (!$?) {
        Write-Host "Warning: Failed to fetch target branch, continuing anyway..." -ForegroundColor Yellow
    }
    
    $mergeBase = git merge-base HEAD "origin/$targetBranch"
    if (!$?) {
        Write-Host "Failed to calculate merge base, falling back to target branch tip" -ForegroundColor Yellow
        $mergeBase = $targetBranch
    }
    else {
        Write-Host "Merge base: $mergeBase" -ForegroundColor Green
    }

    # Create output directory
    if (!(Test-Path $outputDir)) {
        New-Item -Path $outputDir -ItemType Directory -Force | Out-Null
    }

    $affectedProjectsFile = "$outputDir/affected-projects.txt"
    $mergeBaseFile = "$outputDir/merge-base.txt"
    
    # Save merge base to file for use in GitHub Actions
    $mergeBase | Out-File -FilePath $mergeBaseFile -Encoding UTF8 -NoNewline
    
    # Ensure dotnet-affected is installed
    Write-Host "Ensuring dotnet-affected is installed..." -ForegroundColor Cyan
    dotnet tool restore
    if (!$?) {
        Write-Host "Failed to restore tools" -ForegroundColor Red
        throw "Failed to restore dotnet tools"
    }

    # Run dotnet-affected to get affected projects using the merge base
    Write-Host "Running dotnet-affected to detect affected projects..." -ForegroundColor Cyan
    Write-Host "Command: dotnet affected -v --from $mergeBase --solution-path $solution --format text --output-dir $outputDir --output-name affected-projects" -ForegroundColor Gray
    Write-Host ""
    
    dotnet affected -v --from $mergeBase --solution-path $solution --format text --output-dir $outputDir --output-name affected-projects

    $exitCode = $LASTEXITCODE
    
    # Exit code 166 means nothing changed - this is expected and not an error
    if ($exitCode -eq 166) {
        Write-Host "No projects changed - creating empty affected projects file" -ForegroundColor Yellow
        "" | Out-File -FilePath $affectedProjectsFile -Encoding UTF8
        exit 0
    }
    
    if ($exitCode -ne 0) {
        Write-Host "dotnet-affected failed with exit code: $exitCode" -ForegroundColor Red
        throw "dotnet-affected execution failed"
    }

    # Verify the file was created and show results
    if (Test-Path $affectedProjectsFile) {
        # Read the content
        $affectedProjectsList = Get-Content $affectedProjectsFile
        
        # Get the repository root (current directory)
        $repoRoot = (Get-Location).Path
        
        # Convert absolute paths to relative paths and normalize slashes
        $relativeProjects = $affectedProjectsList | ForEach-Object {
            $absolutePath = $_
            # Convert absolute path to relative using .NET method
            $relativePath = [System.IO.Path]::GetRelativePath($repoRoot, $absolutePath)
            # Convert backslashes to forward slashes for consistency
            $relativePath -replace '\\', '/'
        }
        
        # Join and write back the normalized content
        $affectedProjects = $relativeProjects -join "`n"
        $affectedProjects | Out-File -FilePath $affectedProjectsFile -Encoding UTF8 -NoNewline
        
        Write-Host ""
        Write-Host "***********************************************" -ForegroundColor Green
        Write-Host "[CalculateAffectedProjects] Results" -ForegroundColor Green
        Write-Host "***********************************************" -ForegroundColor Green
        Write-Host ""
        Write-Host "Affected projects:" -ForegroundColor Cyan
        Write-Host $affectedProjects -ForegroundColor Gray
        Write-Host ""
        Write-Host "Results saved to: $affectedProjectsFile" -ForegroundColor Green
    } else {
        Write-Host "dotnet-affected did not produce any output" -ForegroundColor Red
        throw "dotnet-affected execution failed"
    }
}
catch {
    Write-Host ""
    Write-Host "Error: $($_.Exception.Message)" -ForegroundColor Red
    throw
}
finally {
    Pop-Location
}

[Aaronontheweb]

That's fine posting here - glad you were able to resolve it.

As far as I could find, this tool does not support providing a specific commit sha, only a branch. This means the "merge base" trick is unusable if the target branch of the PR has advanced since the feature branch was created.
The behavior of the detected changes seemed broken. It was detecting changes that were recently merged into the main branch, but not changes that were recently added to my feature branch. Maybe this is related to my previous point, maybe not.
(This is from memory, I can't easily test this again so forgive me if this is wrong) If files are not explicitly included in a .csproj, they would not count as changes. This is problematic for frontend files such as package-lock.json.

Incrementalist can do these things based on our own testing of it AFAIK, so this might have been a configuration issue on your end.

So help me understand - what did Affected do that helped you OOTB here?