TRG 11 - Testing and Quality (TRG Suggestion)

docs/release/trg-11/_category_.json

@@ -0,0 +1,3 @@
+{
+  "label": "TRG 11 - Testing and Quality"
+}

docs/release/trg-11/trg-11-01.mdx

@@ -0,0 +1,134 @@
+---
+title: TRG 11.01 - Code Coverage
+sidebar_position: 1
+---
+
+| Status | Created     | Post-History            |
+|--------|-------------|-------------------------|
+| Draft  | 21-Mar-2025 | Initial version created |
+| Draft  | 24-Mar-2025 | Update with SonarQube Cloud reference implementation for GitHub Workflow |
+| Draft  | 02-Apr-2025 | Updated workflow integration example |
+| Draft  | 05-Dec-2025 | Moved position to TRG-11. Adapted and reworded suggestions and removed threshold requirement |
+
+
+## Why
+
+Goal: To ensure that all released software components measure test coverage of their code, as a common metric to track and improve quality and reliability.
+
+This guideline applies to all software components and projects that are part of the Eclipse Tractus-X release process.
+
+## Description
+
+### 1. Minimum Requirements
+
+1.1. **Code Coverage**
+
+- All Eclipse Tractus-X software projects should measure and report code coverage as a quality metric.
+- Code coverage measurement and reporting should be automated and integrated into the applicable CI/CD pipelines as part of the release process checks.
+- Eclipse Tractus-X Test and Release Management recommend using 80.0% line coverage for new code as an orientation value, based on the default Eclipse Tractus-X SonarQube [Quality Gate conditions](https://sonarcloud.io/organizations/eclipse-tractusx/quality_gates/show/AWBzEoq-FTEFvoJcI01C).
+- However no fixed minimum code coverage threshold (percentage of line coverage) for new code or the existing code base is currently enforced that would automatically block builds or releases if it falls below such a value.
+- Thus project leads respectively the committers of a project define, document and enforce acceptable coverage levels and any project-specific Quality Gates themselves.
+- Per default, code coverage should be calculated using both unit tests and integration tests combined, unless a project follows a different documented approach.
+
+1.2. **Exceptions**
+
+Certain code sections may be excluded from counting towards the code coverage percentage. For example, this may apply to sections of code that:
+
+- Is itself test code (no "test of tests").
+- Is autogenerated code, for example Swagger-generated API descriptions/clients.
+- Consists of configuration files with no functional logic to test.
+- Is experimental or prototype code, for example incomplete implementations hidden behind feature toggles.
+- Is code only used for logging, metrics, or monitoring.
+- Depend on platform-specifics, for example hardware that cannot be simulated in a test.
+
+Extensive or critical deliberate exclusions from the code coverage measurement should be reviewed, documented and approved by the project's committers.
+
+### 2. Analysis and Reporting
+
+2.1. **Tools for Code Coverage Reporting**
+
+- The recommended reporting tool is **SonarQube Cloud**, as provided by Elipse Tractus-X [at this link](https://sonarcloud.io/organizations/eclipse-tractusx/projects).
+- SonarQube Cloud does not calculate the code coverage itself. Projects must set up a third-party coverage analysis tool and configure SonarQube Cloud to import its analysis results.
+- SonarQube Cloud supports many programming language- and tool-specific formats as well as a generic import format.
+- See section [4. Resources and Examples](#resources-and-examples) for reference implementation guidance.
+- Each project should document which tools it uses to measure and generate coverage reports.
+
+2.2. **Regular Review**
+
+- Code coverage measurement and results reporting should be produced automatically by integration into the CI/CD pipelines, and reviewed at regular intervals (e.g. per merge request or at least per release).
+
+### 3. Quality Assurance
+
+3.1. **Code Review Requirements**
+
+- As part of the code review committers must check that new or modified code is adequately covered by test cases.
+- Projects may define their own code coverage thresholds for a build respectively a release.
+- During review, the project's committers may also accept changes that reduce code coverage below this defined threshold value if they explicitly decide this.
+- As a general guidance, as outlined above new code is recommended to reach at least 80.0% coverage (SonarQube default Quality Gate). This is not mandatory.
+
+3.2. **Risk Analysis**
+
+- If a project’s coverage level falls below its own (self-)defined target threshold for an upcoming release and this cannot be improved anymore in time for the release, this potential risk should be accepted and documented by the project's lead respectively group of committers.
+
+### 4. Resources and Examples {#resources-and-examples}
+
+#### 4.1 **SonarQube Cloud Implementation Guides** {#sonarqube-implementation-guides}
+
+See the following SonarQube Cloud documentation for importing test coverage reports, with detailed guides for several programming languages also widely used by most Eclipse Tractus-X projects as a [reference](https://docs.sonarsource.com/sonarqube-cloud/enriching/test-coverage/overview/).
+
+#### 4.2. **GitHub Workflow Integration Code Example** {#github-workflow-example}
+
+An example of a GitHub workflow integration using Java and Apache Maven can be found below.
+
+```yaml
+name: "Sonar Check"
+
+on:
+  pull_request:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+
+env:
+  JAVA_VERSION: 17
+
+jobs:
+  Test-and-Sonar:
+    permissions:
+      contents: read
+      checks: write
+      pull-requests: write
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-java@v4
+        with:
+          java-version: '${{ env.JAVA_VERSION }}'
+          distribution: 'temurin'
+          cache: 'maven'
+
+      - name: Run unit & integration tests
+        run: |
+          mvn -B verify
+
+      - name: Clean working directories
+        run: |
+          rm -rf .scannerwork
+          rm -rf .sonar
+
+      - name: Cache SonarCloud packages
+        uses: actions/cache@v4
+        with:
+          path: ~/.sonar/cache
+          key: ${{ runner.os }}-sonar
+          restore-keys: ${{ runner.os }}-sonar
+
+      - name: Verify Sonar Scan
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN_BACKEND  }}
+          SONAR_ORGANIZATION: ${{ vars.SONAR_ORGANIZATION }}
+          SONAR_PROJECT_KEY: ${{ vars.SONAR_PROJECT_KEY_BACKEND }}
+        run: mvn --batch-mode sonar:sonar -Dsonar.coverage.jacoco.xmlReportPaths=${{ github.workspace }}/target/site/jacoco-aggregate/jacoco.xml -Dsonar.projectKey=${{ vars.SONAR_PROJECT_KEY_BACKEND }} -Dsonar.organization=${{ vars.SONAR_ORGANIZATION }}
+```