Incorrect licenseConcluded (OR instead of AND) for pkg:golang/github.com/klauspost/compress@1.17.8

[nodet]

For package pkg:golang/github.com/klauspost/compress@1.17.8, Parlay incorrectly returns

"licenseConcluded": "(Apache-2.0 OR BSD-3-Clause OR MIT)",

where it should be

"licenseConcluded": "(Apache-2.0 AND BSD-3-Clause AND MIT)",

The license text for this package is at https://github.com/klauspost/compress/blob/v1.17.8/LICENSE. This file is organised as such:

[default copyright and license]

------------------

Files: gzhttp/*

[Another copyright and license]

------------------

Files: s2/cmd/internal/readahead/*

[Yet another copyright and license]

etc.

The Parlay output is incorrect, as it should be an 'AND' clause because we have to comply with all of the licenses, not an 'OR' clause that lets us choose. See https://spdx.github.io/spdx-spec/v2.3/SPDX-license-expressions/#d4-composite-license-expressions.

Here's how to reproduce.

$ cat klauspost-compress-1.17.8.json
{
  "spdxVersion": "SPDX-2.3",
  "dataLicense": "CC0-1.0",
  "SPDXID": "SPDXRef-DOCUMENT",
  "creationInfo": {
    "licenseListVersion": "3.19"
  },
  "packages": [
    {
        "name": "github.com/klauspost/compress",
        "SPDXID": "SPDXRef-295-github.com-klauspost-compress-1.17.8",
        "versionInfo": "1.17.8",
        "externalRefs": [
          {
            "referenceCategory": "PACKAGE-MANAGER",
            "referenceType": "purl",
            "referenceLocator": "pkg:golang/github.com/klauspost/compress@1.17.8"
          }
        ]
      }
  ]
}
$ parlay ecosystems enrich klauspost-compress-1.17.8.json | jq
{
  "spdxVersion": "SPDX-2.3",
  "dataLicense": "CC0-1.0",
  "SPDXID": "SPDXRef-DOCUMENT",
  "name": "",
  "documentNamespace": "",
  "creationInfo": {
    "licenseListVersion": "3.19",
    "creators": null,
    "created": ""
  },
  "packages": [
    {
      "name": "github.com/klauspost/compress",
      "SPDXID": "SPDXRef-295-github.com-klauspost-compress-1.17.8",
      "versionInfo": "1.17.8",
      "downloadLocation": "",
      "filesAnalyzed": true,
      "homepage": "https://github.com/klauspost/compress",
      "licenseConcluded": "(Apache-2.0 OR BSD-3-Clause OR MIT)",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceType": "purl",
          "referenceLocator": "pkg:golang/github.com/klauspost/compress@1.17.8"
        }
      ]
    }
  ]
}

[mcombuechen]

Thanks @nodet for raising this issue.

I looked into our data source for this case, which is ecosyste.ms. You can check their response for the given module here: https://packages.ecosyste.ms/api/v1/registries/proxy.golang.org/packages/github.com/klauspost/compress

If you can tell, it does include the list of licenses (under @.normalized_licenses) but there is no indicator for the relationship, is it OR or AND. You can see that we currently default to OR as the desired relationship:

parlay/lib/ecosystems/enrich_spdx.go

Line 93 in 327a30a

pkg.PackageLicenseConcluded = fmt.Sprintf("(%s)", strings.Join(data.NormalizedLicenses, " OR "))

I'm inclined to change it to AND, but would that break other use cases?

[nodet]

I'm not sure that changing the default from OR to AND helps much. Looking at the list of our dependencies, I can see a number of cases that would be broken by this change:

• https://github.com/cure53/DOMPurify?tab=License-1-ov-file
• https://github.com/sindresorhus/type-fest#license
• https://github.com/ralphtheninja/expand-template
• https://github.com/dominictarr/rc#license

All of these are modules that are licensed under terms that allow users to choose under which license they use the code. These ones are the reverse: the user needs to comply with multiple licenses. They are currently incorrect, and would be fixed with the proposed change:

• https://github.com/klauspost/compress/blob/v1.17.8/LICENSE (that triggered this issue)
• https://github.com/browserify/sha.js?tab=readme-ov-file#license-mit-and-bsd-3-clause

I don't know that our set of dependencies is representative, and that it is often the case that there are more ORs than ANDs. But whatever default you use, some will be incorrect. The point is that I think the only way to get this right is to have ecosyste.ms store and return the correct information.

[nodet]

Any feedback on the above?
Thanks!

[mcombuechen]

Hello @nodet
please excuse the radio silence. I think we have a few license-related challenges which I would see us improve on incrementally.

Ambivalent license relationships

We currently "make up" a potentially incorrect SPDX license expression by joining licenses with an expression operator. As it is not clear whether licenses have an AND or OR relationship, this approach will always result in buggy behaviour. You already made this point, and I fully agree. The fact is that the data source ecosyste.ms only provides a blank list with no relationships between the items.

As a fix for this I would suggest to move away from license expressions and rather get a bit smarter, use a mix of SPDX identifiers and custom licenses where needed.

Licenses not being valid SPDX identifiers

We currently don't validate/sanitize licenses. There's another open issue for this here #80. I think this is another case where we could improve license enrichment by migrating from SPDX expressions to a mix of SPDX identifiers and custom licenses.

Please allow us to take some time look into these improvements as the year winds down and most of us spend Christmas holidays with their families. 😄