Add XUnit Formatting Output to Scorecard

[theishshah]

Description of the change:

Provide XML formatting option for scorecard users. Additionally transforms scorecard result types to xunit testsuite/testcase layout.

Motivation for the change:

Allow automation users to leverage existing xunit pipelines to process scorecard outputs.

Ref: #4616

[theishshah]

Enabling support for Unique ID will require an update to the version of API, and support for timestamp will require a API change to merge and be released in order to adopt, which is why those two lines of the implementation are currently commented out. Many fields that are supported by xunit are not necessarily tracked in the scorecard results, and for that reason many are currently left empty, however support for that data can be added to the back end of scorecard as shown with the timestamp feature with relative ease. (Apologies, mouse slipped to close this PR, it has since been reopened)

[jmrodri]

A couple comments. nit consider moving structs to xunit package (follow on). Paste an actual output xunit xml file as a comment to the PR for QE to review.

[theishshah]

Structs have been moved to a small package in order to keep the CLI clean, comments have been added describing exported structs.

[jmrodri]

You could've just done:

resultSuite := xunit.TestSuites{Name: "scorecard"}

[jmrodri]

/lgtm

[jmrodri]

/hold

[jmrodri]

@theishshah something isn't right. I ran the scorecard with this patch on my test bundle. The default and json output had a ton of information, but xunit was basically empty. See the following comments.

[jmrodri]

JSON output

{
  "kind": "TestList",
  "apiVersion": "scorecard.operatorframework.io/v1alpha3",
  "items": [
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.3.0",
        "entrypoint": [
          "scorecard-test",
          "olm-status-descriptors"
        ],
        "labels": {
          "suite": "olm",
          "test": "olm-status-descriptors-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "olm-status-descriptors",
            "log": "Loaded ClusterServiceVersion: demo-zeus-operator.v0.0.1\nLoaded 1 Custom Resources from alm-examples\n",
            "state": "fail",
            "errors": [
              "tests.test.zeusville.com does not have a status descriptor"
            ]
          }
        ]
      }
    },
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.3.0",
        "entrypoint": [
          "scorecard-test",
          "olm-spec-descriptors"
        ],
        "labels": {
          "suite": "olm",
          "test": "olm-spec-descriptors-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "olm-spec-descriptors",
            "log": "Loaded ClusterServiceVersion: demo-zeus-operator.v0.0.1\nLoaded 1 Custom Resources from alm-examples\n",
            "state": "fail",
            "errors": [
              "foo does not have a spec descriptor"
            ],
            "suggestions": [
              "Add a spec descriptor for foo"
            ]
          }
        ]
      }
    },
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.3.0",
        "entrypoint": [
          "scorecard-test",
          "olm-bundle-validation"
        ],
        "labels": {
          "suite": "olm",
          "test": "olm-bundle-validation-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "olm-bundle-validation",
            "log": "time=\"2021-07-09T18:51:51Z\" level=debug msg=\"Found manifests directory\" name=bundle-test\ntime=\"2021-07-09T18:51:51Z\" level=debug msg=\"Found metadata directory\" name=bundle-test\ntime=\"2021-07-09T18:51:51Z\" level=debug msg=\"Getting mediaType info from manifests directory\" name=bundle-test\ntime=\"2021-07-09T18:51:51Z\" level=info msg=\"Found annotations file\" name=bundle-test\ntime=\"2021-07-09T18:51:51Z\" level=info msg=\"Could not find optional dependencies file\" name=bundle-test\n",
            "state": "pass"
          }
        ]
      }
    },
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.3.0",
        "entrypoint": [
          "scorecard-test",
          "olm-crds-have-resources"
        ],
        "labels": {
          "suite": "olm",
          "test": "olm-crds-have-resources-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "olm-crds-have-resources",
            "log": "Loaded ClusterServiceVersion: demo-zeus-operator.v0.0.1\n",
            "state": "fail",
            "errors": [
              "Owned CRDs do not have resources specified"
            ]
          }
        ]
      }
    },
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.3.0",
        "entrypoint": [
          "scorecard-test",
          "olm-crds-have-validation"
        ],
        "labels": {
          "suite": "olm",
          "test": "olm-crds-have-validation-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "olm-crds-have-validation",
            "log": "Loaded 1 Custom Resources from alm-examples\nLoaded CustomresourceDefinitions: [\u0026CustomResourceDefinition{ObjectMeta:{tests.test.zeusville.com      0 0001-01-01 00:00:00 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[] map[controller-gen.kubebuilder.io/version:v0.4.1] [] []  []},Spec:CustomResourceDefinitionSpec{Group:test.zeusville.com,Names:CustomResourceDefinitionNames{Plural:tests,Singular:test,ShortNames:[],Kind:Test,ListKind:TestList,Categories:[],},Scope:Namespaced,Versions:[]CustomResourceDefinitionVersion{CustomResourceDefinitionVersion{Name:v1,Served:true,Storage:true,Schema:\u0026CustomResourceValidation{OpenAPIV3Schema:\u0026JSONSchemaProps{ID:,Schema:,Ref:nil,Description:Test is the Schema for the tests API,Type:object,Format:,Title:,Default:nil,Maximum:nil,ExclusiveMaximum:false,Minimum:nil,ExclusiveMinimum:false,MaxLength:nil,MinLength:nil,Pattern:,MaxItems:nil,MinItems:nil,UniqueItems:false,MultipleOf:nil,Enum:[]JSON{},MaxProperties:nil,MinProperties:nil,Required:[],Items:nil,AllOf:[]JSONSchemaProps{},OneOf:[]JSONSchemaProps{},AnyOf:[]JSONSchemaProps{},Not:nil,Properties:map[string]JSONSchemaProps{apiVersion: {  \u003cnil\u003e APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources string   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e},kind: {  \u003cnil\u003e Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds string   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e},metadata: {  \u003cnil\u003e  object   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e},spec: {  \u003cnil\u003e TestSpec defines the desired state of Test object   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[foo:{  \u003cnil\u003e Foo is an example field of Test. Edit Test_types.go to remove/update string   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e}] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e},status: {  \u003cnil\u003e TestStatus defines the observed state of Test object   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e},},AdditionalProperties:nil,PatternProperties:map[string]JSONSchemaProps{},Dependencies:JSONSchemaDependencies{},AdditionalItems:nil,Definitions:JSONSchemaDefinitions{},ExternalDocs:nil,Example:nil,Nullable:false,XPreserveUnknownFields:nil,XEmbeddedResource:false,XIntOrString:false,XListMapKeys:[],XListType:nil,XMapType:nil,},},Subresources:\u0026CustomResourceSubresources{Status:\u0026CustomResourceSubresourceStatus{},Scale:nil,},AdditionalPrinterColumns:[]CustomResourceColumnDefinition{},Deprecated:false,DeprecationWarning:nil,},},Conversion:nil,PreserveUnknownFields:false,},Status:CustomResourceDefinitionStatus{Conditions:[]CustomResourceDefinitionCondition{},AcceptedNames:CustomResourceDefinitionNames{Plural:,Singular:,ShortNames:[],Kind:,ListKind:,Categories:[],},StoredVersions:[],},}]\n",
            "state": "pass"
          }
        ]
      }
    },
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.3.0",
        "entrypoint": [
          "scorecard-test",
          "basic-check-spec"
        ],
        "labels": {
          "suite": "basic",
          "test": "basic-check-spec-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "basic-check-spec",
            "state": "pass"
          }
        ]
      }
    }
  ]
}

[jmrodri]

Default text output

--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test:v1.3.0
Entrypoint: [scorecard-test basic-check-spec]
Labels:
	"suite":"basic"
	"test":"basic-check-spec-test"
Results:
	Name: basic-check-spec
	State: pass



--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test:v1.3.0
Entrypoint: [scorecard-test olm-crds-have-validation]
Labels:
	"suite":"olm"
	"test":"olm-crds-have-validation-test"
Results:
	Name: olm-crds-have-validation
	State: pass

	Log:
		Loaded 1 Custom Resources from alm-examples
		Loaded CustomresourceDefinitions: [&CustomResourceDefinition{ObjectMeta:{tests.test.zeusville.com      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[controller-gen.kubebuilder.io/version:v0.4.1] [] []  []},Spec:CustomResourceDefinitionSpec{Group:test.zeusville.com,Names:CustomResourceDefinitionNames{Plural:tests,Singular:test,ShortNames:[],Kind:Test,ListKind:TestList,Categories:[],},Scope:Namespaced,Versions:[]CustomResourceDefinitionVersion{CustomResourceDefinitionVersion{Name:v1,Served:true,Storage:true,Schema:&CustomResourceValidation{OpenAPIV3Schema:&JSONSchemaProps{ID:,Schema:,Ref:nil,Description:Test is the Schema for the tests API,Type:object,Format:,Title:,Default:nil,Maximum:nil,ExclusiveMaximum:false,Minimum:nil,ExclusiveMinimum:false,MaxLength:nil,MinLength:nil,Pattern:,MaxItems:nil,MinItems:nil,UniqueItems:false,MultipleOf:nil,Enum:[]JSON{},MaxProperties:nil,MinProperties:nil,Required:[],Items:nil,AllOf:[]JSONSchemaProps{},OneOf:[]JSONSchemaProps{},AnyOf:[]JSONSchemaProps{},Not:nil,Properties:map[string]JSONSchemaProps{apiVersion: {  <nil> APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources string   nil <nil> false <nil> false <nil> <nil>  <nil> <nil> false <nil> [] <nil> <nil> [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false <nil> false false [] <nil> <nil>},kind: {  <nil> Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds string   nil <nil> false <nil> false <nil> <nil>  <nil> <nil> false <nil> [] <nil> <nil> [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false <nil> false false [] <nil> <nil>},metadata: {  <nil>  object   nil <nil> false <nil> false <nil> <nil>  <nil> <nil> false <nil> [] <nil> <nil> [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false <nil> false false [] <nil> <nil>},spec: {  <nil> TestSpec defines the desired state of Test object   nil <nil> false <nil> false <nil> <nil>  <nil> <nil> false <nil> [] <nil> <nil> [] nil [] [] [] nil map[foo:{  <nil> Foo is an example field of Test. Edit Test_types.go to remove/update string   nil <nil> false <nil> false <nil> <nil>  <nil> <nil> false <nil> [] <nil> <nil> [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false <nil> false false [] <nil> <nil>}] nil map[] map[] nil map[] nil nil false <nil> false false [] <nil> <nil>},status: {  <nil> TestStatus defines the observed state of Test object   nil <nil> false <nil> false <nil> <nil>  <nil> <nil> false <nil> [] <nil> <nil> [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false <nil> false false [] <nil> <nil>},},AdditionalProperties:nil,PatternProperties:map[string]JSONSchemaProps{},Dependencies:JSONSchemaDependencies{},AdditionalItems:nil,Definitions:JSONSchemaDefinitions{},ExternalDocs:nil,Example:nil,Nullable:false,XPreserveUnknownFields:nil,XEmbeddedResource:false,XIntOrString:false,XListMapKeys:[],XListType:nil,XMapType:nil,},},Subresources:&CustomResourceSubresources{Status:&CustomResourceSubresourceStatus{},Scale:nil,},AdditionalPrinterColumns:[]CustomResourceColumnDefinition{},Deprecated:false,DeprecationWarning:nil,},},Conversion:nil,PreserveUnknownFields:false,},Status:CustomResourceDefinitionStatus{Conditions:[]CustomResourceDefinitionCondition{},AcceptedNames:CustomResourceDefinitionNames{Plural:,Singular:,ShortNames:[],Kind:,ListKind:,Categories:[],},StoredVersions:[],},}]


--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test:v1.3.0
Entrypoint: [scorecard-test olm-status-descriptors]
Labels:
	"suite":"olm"
	"test":"olm-status-descriptors-test"
Results:
	Name: olm-status-descriptors
	State: fail

	Errors:
		tests.test.zeusville.com does not have a status descriptor
	Log:
		Loaded ClusterServiceVersion: demo-zeus-operator.v0.0.1
		Loaded 1 Custom Resources from alm-examples


--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test:v1.3.0
Entrypoint: [scorecard-test olm-spec-descriptors]
Labels:
	"suite":"olm"
	"test":"olm-spec-descriptors-test"
Results:
	Name: olm-spec-descriptors
	State: fail

	Suggestions:
		Add a spec descriptor for foo
	Errors:
		foo does not have a spec descriptor
	Log:
		Loaded ClusterServiceVersion: demo-zeus-operator.v0.0.1
		Loaded 1 Custom Resources from alm-examples


--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test:v1.3.0
Entrypoint: [scorecard-test olm-crds-have-resources]
Labels:
	"test":"olm-crds-have-resources-test"
	"suite":"olm"
Results:
	Name: olm-crds-have-resources
	State: fail

	Errors:
		Owned CRDs do not have resources specified
	Log:
		Loaded ClusterServiceVersion: demo-zeus-operator.v0.0.1


--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test:v1.3.0
Entrypoint: [scorecard-test olm-bundle-validation]
Labels:
	"suite":"olm"
	"test":"olm-bundle-validation-test"
Results:
	Name: olm-bundle-validation
	State: pass

	Log:
		time="2021-07-09T18:52:07Z" level=debug msg="Found manifests directory" name=bundle-test
		time="2021-07-09T18:52:07Z" level=debug msg="Found metadata directory" name=bundle-test
		time="2021-07-09T18:52:07Z" level=debug msg="Getting mediaType info from manifests directory" name=bundle-test
		time="2021-07-09T18:52:07Z" level=info msg="Found annotations file" name=bundle-test
		time="2021-07-09T18:52:07Z" level=info msg="Could not find optional dependencies file" name=bundle-test

[jmrodri]

Xunit output

<TestSuites>
  <Name>scorecard</Name>
  <Tests></Tests>
  <Failures></Failures>
  <Errors></Errors>
</TestSuites>

[jmrodri]

/lgtm cancel

[theishshah]

Sample Failure Output

<TestSuite>
    <Name>olm-spec-descriptors</Name>
    <Tests></Tests>
    <Failures>Loaded ClusterServiceVersion: memcached-operator.v0.0.1&#xA;Loaded 1 Custom Resources from alm-examples&#xA;</Failures>
    <Errors></Errors>
    <Group></Group>
    <Skipped></Skipped>
    <Timestamp></Timestamp>
    <Hostname></Hostname>
    <ID></ID>
    <Package></Package>
    <File></File>
    <Log></Log>
    <URL>quay.io/operator-framework/scorecard-test:v1.9.0</URL>
    <Version></Version>
    <TestCases>
      <Name>olm-spec-descriptors</Name>
      <Time></Time>
      <Classname></Classname>
      <Group></Group>
      <Failures>
        <Type>Failure</Type>
        <Message>Loaded ClusterServiceVersion: memcached-operator.v0.0.1&#xA;Loaded 1 Custom Resources from alm-examples&#xA;</Message>
      </Failures>
    </TestCases>
  </TestSuite>

Sample Passing Output

<TestSuite>
    <Name>olm-bundle-validation</Name>
    <Tests></Tests>
    <Failures></Failures>
    <Errors></Errors>
    <Group></Group>
    <Skipped></Skipped>
    <Timestamp></Timestamp>
    <Hostname></Hostname>
    <ID></ID>
    <Package></Package>
    <File></File>
    <Log></Log>
    <URL>quay.io/operator-framework/scorecard-test:v1.9.0</URL>
    <Version></Version>
    <TestCases>
      <Name>olm-bundle-validation</Name>
      <Time></Time>
      <Classname></Classname>
      <Group></Group>
    </TestCases>
  </TestSuite>

[jmrodri]

/lgtm

[jmrodri]

/hold cancel

[apodhrad]

Hi @jmrodri @theishshah which schema did you use for validating the output, please? To be honest the format looks different from what used so far. Moreover, it doesn't provide counts of executed tests, failed tests etc.

[apodhrad]

Hi @jmrodri @theishshah I have raised a new issue about the invalid xunit output at #5457