Improve license required phrase generation #4237

pombredanne · 2025-04-14T12:34:24Z

This PR builds on top of:

Support equivalent words in license detection #4190 #4215

Specifically:

Improve feedback and selectivity when generating new required phrase rules, mostly when updating existing rules
Add SPDX license ids to the addition of required phrases to existing rules
Update all rules "required phrase" status with these improvements

Tasks

Reviewed contribution guidelines
PR is descriptively titled 📑 and links the original issue above 🔗
Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
Run tests locally to check for errors.
Commits are in uniquely-named feature branch and has no merge conflicts 📁
Updated documentation pages (if applicable)
Updated CHANGELOG.rst (if applicable)

Put all required phrase validation in one code block Signed-off-by: Philippe Ombredanne <[email protected]>

Signed-off-by: Philippe Ombredanne <[email protected]>

Ensure it works with all rules at once Add filter to run only a few times and limit the minimum size of rules to gen. Use SPDX keys for rules too. Signed-off-by: Philippe Ombredanne <[email protected]>

Signed-off-by: Philippe Ombredanne <[email protected]>

Do not generate rules for "license key". Skip short rules that would contain stopwords as they cannot be matched accurately Validate short required phrase rules for stopwords Add tests that highlight the stop word issue Reference: #4238 Signed-off-by: Philippe Ombredanne <[email protected]>

Do not generate require phrase rules for these rules. Reference: #4238 Signed-off-by: Philippe Ombredanne <[email protected]>

…ords

Signed-off-by: Philippe Ombredanne <[email protected]>

…ired

pombredanne · 2025-04-22T15:00:29Z

plugin_license2.python

Oops... should not be committed. junk local file!

AyanSinhaMahapatra

@pombredanne thanks++ for handling these. LGTM!
I will make the updates for a few nits and merge this after we merge #4215

AyanSinhaMahapatra · 2025-04-22T15:07:28Z

src/licensedcode/data/rules/bsd-new_1426.RULE

@@ -2,6 +2,7 @@
 license_expression: bsd-new
 is_license_text: yes
 is_continuous: yes
+skip_for_required_phrase_generation: yes


Why don't we set the minimum_coverage instead here?
This is better for large parts of texts in rules that we want to detect exactly, more so when we want to match the entire rule exactly.

AyanSinhaMahapatra · 2025-04-22T15:17:48Z

src/licensedcode/data/rules/false-positive_822.RULE

+notes: Note a license
+---
+
+Some units are dual licensed and some are specifically Artistic-only.


This should not be a false positive, we should instead make this a license clue for artistic, see example at https://github.com/natias/arnon/blob/e1233fdd840601e202d3bf1d3e9e5aff9a2e13d5/docker_with_perl_dbd/xzbnl#L245.

AyanSinhaMahapatra · 2025-04-22T15:40:21Z

src/licensedcode/data/rules/mpl-1.1_or_gpl-2.0-plus_or_lgpl-2.1-plus_16.RULE

-}}
+
+ 
+<licenses><license><name>MPL</name><url>http://www.mozilla.org/MPL/2.0/index.txt</url><distribution>repo</distribution></license><license><name>LGPL</name><url>http://www.gnu.org/licenses/lgpl-2.1.txt</url><distribution>repo</distribution></license><license><name>GPL</name><url>http://www.gnu.org/licenses/gpl-2.0.txt</url><distribution>repo</distribution></license></licenses>


We should not have this entire rule as a required phrase, instead we should hane the name/urls as required phrases.

AyanSinhaMahapatra · 2025-04-22T15:40:45Z

src/licensedcode/data/rules/mpl-1.1_or_gpl-2.0-plus_or_lgpl-2.1-plus_30.RULE

-}}
+
+ 
+<licenses><license><name>MPL</name><url>http://www.mozilla.org/MPL/2.0/index.txt</url><distribution>repo</distribution></license><license><name>LGPL</name><url>https://www.gnu.org/licenses/lgpl-2.1.txt</url><distribution>repo</distribution></license><license><name>GPL</name><url>https://www.gnu.org/licenses/gpl-2.0.txt</url><distribution>repo</distribution></license></licenses>


We should not have this entire rule as a required phrase, instead we should hane the name/urls as required phrases.

AyanSinhaMahapatra · 2025-04-22T15:41:25Z

src/licensedcode/data/rules/mpl-1.1_or_gpl-3.0_or_lgpl-3.0_1.RULE

 ignorable_urls:
    - http://www.gnu.org/licenses/gpl.txt
    - http://www.gnu.org/licenses/lgpl.txt
    - https://www.mozilla.org/en-US/MPL/1.1/
 ---
-{{
+
+


We should not have this entire rule as a required phrase, instead we should hane the name/urls as required phrases.

AyanSinhaMahapatra · 2025-04-22T16:09:18Z

src/licensedcode/required_phrases.py

-        Return True if this phrase is a minimally suitable to use as a required phrase
+        Return True if this phrase is a minimally suitable to use as a required phrase.
+        Use the original rule to ensure we skip when referenced_filenames could be damaged.
+        Also skip short rules that would contain stopwords as they could not be detected correctly.


We should probably also check here for required phrase texts which are quite large as a refinement, as these can be better handled either by minimum coverage or by skipping required phrase generation from the source rules.

Signed-off-by: Philippe Ombredanne <[email protected]>

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

pombredanne added 10 commits April 13, 2025 11:07

Streamline rule validation

5fc22cf

Put all required phrase validation in one code block Signed-off-by: Philippe Ombredanne <[email protected]>

Tag existing rules as required phrases

57f6ed5

Signed-off-by: Philippe Ombredanne <[email protected]>

Update required phrase statuses for rules

e9fe204

Signed-off-by: Philippe Ombredanne <[email protected]>

Update required phrase statuses for rules

c532af7

Signed-off-by: Philippe Ombredanne <[email protected]>

Improve required phrase rules gen

e641691

Ensure it works with all rules at once Add filter to run only a few times and limit the minimum size of rules to gen. Use SPDX keys for rules too. Signed-off-by: Philippe Ombredanne <[email protected]>

Generate more required phrase rules

933ccfc

Signed-off-by: Philippe Ombredanne <[email protected]>

Remove too verbose output

047e34b

Signed-off-by: Philippe Ombredanne <[email protected]>

Update/generate more required phrase rules

bd7eac4

Signed-off-by: Philippe Ombredanne <[email protected]>

Do not fail on license

7a6d03e

Signed-off-by: Philippe Ombredanne <[email protected]>

Correct tests after mass rule update

cd8389c

Signed-off-by: Philippe Ombredanne <[email protected]>

pombredanne requested a review from AyanSinhaMahapatra April 14, 2025 12:34

pombredanne added 5 commits April 14, 2025 22:42

Improve license rules

a779a13

Signed-off-by: Philippe Ombredanne <[email protected]>

Remove stopword from rule

23317ed

Signed-off-by: Philippe Ombredanne <[email protected]>

Convert rule to required phrase

36bf8b8

Signed-off-by: Philippe Ombredanne <[email protected]>

Update rules for stopwords

8766095

Do not generate require phrase rules for these rules. Reference: #4238 Signed-off-by: Philippe Ombredanne <[email protected]>

pombredanne mentioned this pull request Apr 16, 2025

Refine required phrases with stopwords #4238 #4241

Merged

6 tasks

pombredanne added 5 commits April 21, 2025 21:32

Merge branch '4190-license-licence' into improve-required

2ec6013

Merge branch 'improve-required' into 4238-required-phrases-with-stopw…

1f789a2

…ords

Rename modified rule files

dc22bc2

Signed-off-by: Philippe Ombredanne <[email protected]>

Merge latest 4190-license-licence branch

c54321a

Signed-off-by: Philippe Ombredanne <[email protected]>

Merge branch '4238-required-phrases-with-stopwords' into improve-requ…

494723a

…ired

pombredanne mentioned this pull request Apr 22, 2025

Support equivalent words in license detection #4190 #4215

Merged

6 tasks

AyanSinhaMahapatra changed the base branch from develop to 4190-license-licence April 22, 2025 14:23

pombredanne commented Apr 22, 2025

View reviewed changes

AyanSinhaMahapatra approved these changes Apr 22, 2025

View reviewed changes

Remove temp file committed by mistake.

e0a8c42

Signed-off-by: Philippe Ombredanne <[email protected]>

AyanSinhaMahapatra changed the base branch from 4190-license-licence to develop April 22, 2025 19:17

AyanSinhaMahapatra force-pushed the improve-required branch from 0227a0a to 76f8559 Compare April 22, 2025 19:23

Update required phrases, rules and fix minor issues

a5116ff

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra force-pushed the improve-required branch from 76f8559 to a5116ff Compare April 22, 2025 20:29

AyanSinhaMahapatra merged commit e4abe78 into develop Apr 22, 2025
7 of 42 checks passed

pombredanne deleted the improve-required branch April 23, 2025 06:25

AyanSinhaMahapatra mentioned this pull request Apr 28, 2025

Wrong license detected (GPL-2.0 instead of GPL-3.0-or-later) for component Linux::binutils-common:2.40 #4281

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve license required phrase generation #4237

Improve license required phrase generation #4237

pombredanne commented Apr 14, 2025 •

edited

Loading

pombredanne Apr 22, 2025

AyanSinhaMahapatra left a comment

AyanSinhaMahapatra Apr 22, 2025

AyanSinhaMahapatra Apr 22, 2025

AyanSinhaMahapatra Apr 22, 2025

AyanSinhaMahapatra Apr 22, 2025

AyanSinhaMahapatra Apr 22, 2025

AyanSinhaMahapatra Apr 22, 2025 •

edited

Loading

		}}


		<licenses><license><name>MPL</name><url>http://www.mozilla.org/MPL/2.0/index.txt</url><distribution>repo</distribution></license><license><name>LGPL</name><url>http://www.gnu.org/licenses/lgpl-2.1.txt</url><distribution>repo</distribution></license><license><name>GPL</name><url>http://www.gnu.org/licenses/gpl-2.0.txt</url><distribution>repo</distribution></license></licenses>

		}}


		<licenses><license><name>MPL</name><url>http://www.mozilla.org/MPL/2.0/index.txt</url><distribution>repo</distribution></license><license><name>LGPL</name><url>https://www.gnu.org/licenses/lgpl-2.1.txt</url><distribution>repo</distribution></license><license><name>GPL</name><url>https://www.gnu.org/licenses/gpl-2.0.txt</url><distribution>repo</distribution></license></licenses>

Improve license required phrase generation #4237

Improve license required phrase generation #4237

Conversation

pombredanne commented Apr 14, 2025 • edited Loading

Tasks

pombredanne Apr 22, 2025

Choose a reason for hiding this comment

AyanSinhaMahapatra left a comment

Choose a reason for hiding this comment

AyanSinhaMahapatra Apr 22, 2025

Choose a reason for hiding this comment

AyanSinhaMahapatra Apr 22, 2025

Choose a reason for hiding this comment

AyanSinhaMahapatra Apr 22, 2025

Choose a reason for hiding this comment

AyanSinhaMahapatra Apr 22, 2025

Choose a reason for hiding this comment

AyanSinhaMahapatra Apr 22, 2025

Choose a reason for hiding this comment

AyanSinhaMahapatra Apr 22, 2025 • edited Loading

Choose a reason for hiding this comment

pombredanne commented Apr 14, 2025 •

edited

Loading

AyanSinhaMahapatra Apr 22, 2025 •

edited

Loading