Add quality metrics for dotnetup #52792
Open
nagilson wants to merge 30 commits intodotnet:release/dnupfrom
Open
Add quality metrics for dotnetup #52792nagilson wants to merge 30 commits intodotnet:release/dnupfrom
dotnetup #52792nagilson wants to merge 30 commits intodotnet:release/dnupfrom
Conversation
we should investigate if the error mapping can be outsourced as it seems silly we need to implement this ourselves
… is wrong failures some of the categories may be incorrect, but this is a good starting point
I also initially included the sha but I want to be able to sort by error an dont have to parse out the sha which should be mappable to /from the version. Still, I kept the outsource of the build sha to a separate file bc I liked that isolated shareable pattern.
- Add llm detection - first run disable env var - stderr over stdout
consolidate error logic code
Read more at dotnet#52789
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces comprehensive telemetry infrastructure for the dotnetup CLI tool using OpenTelemetry with Azure Monitor. The implementation provides actionable insights into installation success rates, user behavior patterns, and error analysis while maintaining strict PII protection rules. The PR includes telemetry collection, error categorization, PII sanitization, test coverage, and Azure Workbook dashboards.
Changes:
- Added OpenTelemetry telemetry infrastructure with Azure Monitor integration, activity-based tracing, and configurable opt-out
- Implemented error categorization system with 17 specific error codes distinguishing product errors from user errors
- Created PII protection mechanisms including version string sanitization, URL domain sanitization, and install path classification
Reviewed changes
Copilot reviewed 71 out of 71 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| test/dotnetup.Tests/dotnetup.Tests.csproj | Excludes TelemetryIntegrationDemo from test compilation |
| test/dotnetup.Tests/VersionSanitizerTests.cs | Comprehensive tests for version string sanitization with 241 test cases covering valid/invalid patterns |
| test/dotnetup.Tests/TestAssets/TelemetryIntegrationDemo/* | Demo project showing how library consumers can integrate with telemetry ActivitySource |
| test/dotnetup.Tests/TelemetryTests.cs | Tests for telemetry common properties, version sanitizer, URL sanitizer, error code mapper, and ActivitySource integration |
| test/dotnetup.Tests/Properties/AssemblyInfo.cs | Module initializer to mark test runs as dev builds via environment variable |
| test/dotnetup.Tests/ParserTests.cs | Tests for version option and Parser.Version property |
| test/dotnetup.Tests/ListCommandTests.cs | Tests for new list command functionality |
| test/dotnetup.Tests/InfoCommandTests.cs | Tests for new --info command functionality |
| test/dotnetup.Tests/ErrorCodeMapperTests.cs | Tests for error code mapping and categorization with 17 error codes |
| test/dotnetup.Tests/ChannelVersionResolverTests.cs | Tests for channel format validation |
| src/Installer/dotnetup/xlf/*.xlf | Localization files updated with new strings for info/list commands and telemetry notice |
| src/Installer/dotnetup/dotnetup.csproj | Added telemetry packages and versioning properties |
| src/Installer/dotnetup/docs/telemetry-notice.txt | Documentation explaining telemetry data collection and opt-out |
| src/Installer/dotnetup/Telemetry/*.cs | Core telemetry infrastructure classes including DotnetupTelemetry, ErrorCodeMapper, VersionSanitizer, etc. |
| src/Installer/dotnetup/Strings.resx | New resource strings for info/list commands and telemetry notice |
| src/Installer/dotnetup/*.cs | Updated progress targets, orchestrator, and commands to integrate telemetry |
| src/Installer/dotnetup/Commands/Sdk/Install/SdkInstallCommand.cs | Comprehensive install behavior tracking with telemetry tags |
| src/Installer/dotnetup/Commands/List/*.cs | New list command implementation |
| src/Installer/dotnetup/Commands/Info/*.cs | New --info command implementation |
| src/Installer/dotnetup/CommandBase.cs | Template method pattern for automatic telemetry in all commands |
| src/Installer/Microsoft.Dotnet.Installation/*.cs | Updated library to throw DotnetInstallException with proper error codes and telemetry support |
| documentation/general/dotnetup/*.md | Documentation for new list and info commands |
| Directory.Packages.props | Added OpenTelemetry package versions |
| .github/copilot-instructions.md | Build/test instructions for dotnetup |
This was referenced Feb 3, 2026
Open
Member
Author
|
Failing test: |
This was referenced Feb 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #50609
(allow #52717 to be merged first - this pr needed code from that branch so we need to merge it back in)
Add OpenTel Data for dotnetup
Summary
This PR introduces telemetry for the
dotnetupCLI tool using OpenTelemetry with Azure Monitor. The telemetry provides actionable insights into installation success rates, user behavior patterns, and error analysis while maintaining PII rules.The current connection key uses my own local app insights because we'll switch to the SDK CLI one once that work has been done by someone else in the team. That shouldn't be used in production, but since this is a dev branch we are not yet releasing, seems like that's ok. I've filed an issue to track: #52785
https://aka.ms/dotnetup-telemetry points to the release/dnup branch for the doc for now, but eventually it'd go on ms learn.
#52784
Key Features
Telemetry Infrastructure
Azure.Monitor.OpenTelemetry.Exporterfor telemetry collectionDOTNET_CLI_TELEMETRY_OPTOUTenvironment variableError Categorization System
VersionNotFound,ManifestFetchFailed,HashMismatch,ArchiveCorrupted, etc.Success Rate Metrics
install.result(installed vs already_installed) for accurate installation countsUser Behavior Tracking (PII-Safe)
install.path_source: Where install path came from (explicit, global_json, default, etc.)install.path_type: Classification of path (system_programfiles, user_profile, local_appdata) - not actual pathsinstall.has_global_json: Whether project has global.jsoninstall.existing_install_type: Admin/User/none for existing installationssdk.request_source: How SDK version was specified (explicit, default-latest, default-globaljson)sdk.requested: Sanitized version stringPII Protection (Critical)
SetStatus()uses error type, notex.MessageRecordException(): Full exception objects not recorded (contain paths/PII)win32_error_5) instead of messages that may contain pathsAzure Workbook Dashboard
Example Data:
^ note some of this data is wrong / when the code was incorrect
Files Changed
New Files
Telemetry/DotnetupTelemetry.cs- Main telemetry singletonTelemetry/ErrorCodeMapper.cs- Exception to error info mapping with categorizationTelemetry/VersionSanitizer.cs- PII-safe version string sanitizationTelemetry/dotnetup-workbook.json- Azure Workbook dashboard definitionModified Files
CommandBase.cs- Template method for command telemetrySdkInstallCommand.cs- Comprehensive install behavior trackingInstallerOrchestratorSingleton.cs- ReturnsInstallResult, proper error throwingNonUpdatingProgressTarget.cs/SpectreProgressTarget.cs- Operation-level telemetryDotnetInstallException.cs- Extended error codesDotnetArchiveExtractor.cs,DotnetArchiveDownloader.cs,ReleaseManifest.cs- Error code usageFollow-up Items
aka.ms/dotnetup-telemetry(or similar) pointing to the telemetry documentation in therelease/dnupbranch of dotnet/sdk repositoryTesting
Telemetry Notice
Users will see a telemetry notice on first run. The notice should link to documentation (via aka.ms redirect) that explains:
DOTNET_CLI_TELEMETRY_OPTOUT=1)