🔁 maven plugin for Conformance index

[adriens]

Reason/Context

Actually we are using ``maven to build our (quarkus`) projects and use `jacoco` plugin to make the build fail if the coverage falls below a given rate :

 
<plugin>
	<groupId>org.jacoco</groupId>
	<artifactId>jacoco-maven-plugin</artifactId>
	<version>0.8.12</version>
	<executions>
		<execution>
			<id>jacoco-check</id>
			<goals>
				<goal>check</goal>
			</goals>
			<phase>post-integration-test</phase>
			<configuration>
				<dataFile>${project.build.directory}/jacoco-quarkus.exec</dataFile>
				<rules>
					<rule>
						<element>BUNDLE</element>
						<limits>
							<limit>
								<counter>LINE</counter>
								<value>COVEREDRATIO</value>
								<minimum>0.85</minimum>
							</limit>
						</limits>
					</rule>
				</rules>
			</configuration>
		</execution>
	</executions>
</plugin>

I would love to get a microcks plugin that would do the same kind of stuff ie. force a minimal Conformance Index, for example to B or A :

Description

Deliver a maven plugin that forces a minimal conformance index that will cause the build to fail when not compliant

Implementation ideas

No idea 😅

[lbroudoux]

Converting this issue into a discussion, I think it's worth opening the topic.
I thought this was a good idea at first read, but I'm not that certain. Let me give some details.

First, about the people who may have an interest in Conformance Index and Conformance Score. I think we're targeting here at least 3 different personas:

• Designers may be interested in the Conformance Index - it gives them feedback on how comprehensive the set of artifacts they're delivering (OpenAPI files, APIExamples, Postman Collection, etc.) is.
• Developers implementing the API may be interested in Conformance Score - it provides them with feedback on how they're covering the test of their implementation - given the provided artifacts
• API Owners may be interested in both Index and Score - they ought to have the better coverage by the specification and the better score by the implementing solution.

As a consequence, I think measuring the Conformance Index and using it as a blocking metric during development is way too late. This measure is not tied to the implementing technology: it's just artifact analysis. Moreover, ideally, this should be done upfront of the development—maybe in a quality gate when designers push their artifacts on a Git repo.

As for the Conformance Score, I'm not sure developers should focus on this but better stick with the raw coverage metric as you're doing with jacoco. Unit test coverage comes earlier in the development process (before validating API endpoint), is more precise (white-box testing methodology) and more representative of code quality (we may have a high code coverage but a very low API conformance score if input artifacts are weak).

Recently, someone highlighted the integration of those metrics into a solution like SonarQube (see this discussion on Discord). I think it would make sense, as in this case, the Conformance Index and Score are kept as high-level metrics that may be shared among the project stakeholders.

What do you think?