[Testmanagement] TRG Code Coverage Proposal

[ds-hzimmer]

Overview

Providing a Tractus-X Release Guideline (TRG) for testing and quality - code coverage

Explain the topic in 2 sentences

Providing a Tractus-X Release Guideline (TRG) for static code analysis code coverage measurements and target thresholds, using Open Source methods and tools, in order to give product teams a template and best-practice example to start from or improve.

What's the benefit?

In increased quality for testing using an efficient established method, potentially avoiding issues that would then only be found in more costly tests or not discovered at all.

What are the Risks/Dependencies ?

• Potentially overlaps with already existing TRG requirements? For other static code analysis measurements regarding security, dependencies, etc. TRGs are already published. See under https://eclipse-tractusx.github.io/docs/release TRG-8 Security
• TRG would be defined within 25.03 release cycle, but presumably only go into effect for subsequent releases.
• Quality Gate: Code coverage thresholds as a quality gate should initially be set fairly low (e.g. at level 50% - 70%) and then potentially increased in subsequent releases as presumably not all product teams would immediately have e.g. a coverage of > 80%, in order to not break / prevent deployments for releases
  https://sonarcloud.io/organizations/eclipse-tractusx/quality_gates/show/9
• Exceptions have to be defined, e.g. criteria if a product is not sufficiently testable with automated static code analysis

Detailed explanation

Current implementation

A Sonarcloud solution for Tractus-X is already available https://sonarcloud.io/organizations/eclipse-tractusx/projects
But not all Tractus-X product teams have implemented or activated the code coverage functionality, and of those not all have an adequately high coverage.

Proposed improvements

See Pull Request: eclipse-tractusx/eclipse-tractusx.github.io#1149 (closed as outdated and some initial reviewers with their objections currently no longer actively involved in Eclipse Tractus-X projects)
Edit: Updated to eclipse-tractusx/eclipse-tractusx.github.io#1396

Feature Team

Catena-X e.V. test management (doubleSlash, on order of the association)

Involvement of additional product teams for implementation

Contributor

• ds-hzimmer
• ds-asmierzchalski

Additional contributors to be determined

Committer

Edited:

• ds-jhartmann

Additional committers to be determined

User Stories

• Issue 1, linked to specific repository
• Issue 2, linked to another specific repository

Acceptance Criteria

• TRG for Code Coverage proposal created
• TRG for Code Coverage discussed with the respective product teams, expert groups and committees
• Quality Gate criteria for code coverage is discussed with representative product teams, release management and other experts, and adapted as needed
• TRG for Code Coverage published

Test Cases

Note: Not directly applicable as this feature suggestion describes a testing improvement itself.

Test Case 1

Steps

1. Product team(s) implement Sonarcloud Code Coverage check in the components they are responsible for
2. Once this integration has been completed:
   Both the Product Teams and Test Management/Release Management can review the Coverage quality gate for each release

Expected Result

1. Product shows up in list of Eclipse Tractus-X projects
   https://sonarcloud.io/organizations/eclipse-tractusx/projects
2. The "Coverage" value for the respective product is displayed (percentage)
3. For a new release, "Coverage" is within/above the expected quality gate percentage threshold.
   If not, the team will implement new test cases to achieve code coverage (tbd if initially already within release or until subsequent release).

Architectural Relevance

Note: Some checks potentially not directly applicable as this feature suggestion describes a testing/test management improvement and not a feature change in the functionality of a product itself.

The following items are ensured (answer: yes) after this issue is implemented:

• This feature aligns with our current architectural guidelines
  • Data Sovereignty: All data sharing activities across company boundaries follow the Catena-X Regulatory Framework, in particular the Data Exchange Governance, and the Dataspace Protocol via a compliant Connector (like the tractusx-edc or similar, see Connector KIT)
    • CX-0010 Business Partner Number
    • CX-0013 Identity of Member Companies
    • CX-0018 Data Space Connectivity (EDC)
    • CX-0049 DID Document Schema
    • CX-0050 Framework Agreement Credential
    • CX-0149 Verified Company Identity
  • Interoperability: Digital Twins are used (compliant to the Digital Twin KIT and the Industry Core KIT)
    • CX-0001 EDC Discovery API
    • CX-0002 Digital Twins in Catena-X
    • CX-0018 Data Space Connectivity (EDC)
  • Data Format:
    • The data model is based on a published Semantic Model
• The impact on the overall system architecture has been assessed. The Feature does not require changes to the architecture or any existing standard? Please have a look here on the overarching architecture
• Potential risks or conflicts with existing architecture has been assessed

Justification: (Fill this out, if at least one of the checkboxes above cannot be ticked. Contact the Architecture Management Committee to get an approval for the justification)

Additional information

• I am aware that my request may not be developed if no developer can be found for it. I'll try to contribute a developer (bring your own developer)

[stephanbcbauer]

Some hints from Release Management (@ther3sa) and Tractus-X Project Lead (@stephanbcbauer)

• Status currently in Inbox. ⇾ Only features with status backlog are considered in open planning
• Please add missing sections from the feature template, or fill them out

[ds-hzimmer]

Shift of release to 25.06 due to pending improvements from feedback to TRG suggestion before pull request can be approved and merged.