Skip to content

Commit b156af0

Browse files
johnmhoranTG1999
andauthored
Add initial fixed-affected-matching work #1228 (#1249)
* Add initial fixed-affected-matching work #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Add Prefetch and univers-based version comparison #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Update affected-fixed package matching #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Improve matching and reporting code and UI #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Add univers version, revise sort and related code, update and add new tests #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Move weakness test #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Modify UI, update dictionary and tests #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Begin replacing strings with objects in package details dictionary #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Clean current package details template and related model code #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Begin work on major-version issue #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Complete first round of major-version vetting #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Remove major-version code, clean comments etc. #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Begin test refactoring #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Finish package details code and template, refactor/create package-related tests #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Refactor package details-related code #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Update Package details UI and Package API #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Fix 1 of 4 failing API tests #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Add initial fixed-affected-matching work #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Explore context and Package class approaches for affected-fixed package matching #1228 Reference: #1228 Note that my updated code is still in testing/dev stage and has not yet been completed or cleaned. Signed-off-by: John M. Horan <[email protected]> * Add Prefetch and univers-based version comparison #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Update affected-fixed package matching #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Improve matching and reporting code and UI #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Add univers version, revise sort and related code, update and add new tests #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Begin work on major-version issue #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Complete first round of major-version vetting #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Remove major-version code, clean comments etc. #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Begin test refactoring #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Finish package details code and template, refactor/create package-related tests #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Commit the initial refactoring changes from last week #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Refactor package details-related code #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Update Package details UI and Package API #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Save test experiments including commented-out variations #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Fix 1 of 4 failing API tests #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Update API including "lesser" fixed by versions, fix and update failing tests #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Update APITestCasePackage() class #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Test lack of "vulnerability" property #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Update get_affected_vulnerabilities() and test #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Update MinimalPackageSerializer() and missing-vulnerability-key test #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Append inside the if condition #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Update get_vulnerability() method #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Enable test_models.py and fix failing tests #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Update per PR comments #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Convert Package method to PackageQuerySet method, clean code and tests #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> * Fix failing tests Signed-off-by: Tushar Goel <[email protected]> * Add property on functions in models Signed-off-by: Tushar Goel <[email protected]> * Add and fix tests, address other comments #1228 Reference: #1228 Signed-off-by: John M. Horan <[email protected]> --------- Signed-off-by: John M. Horan <[email protected]> Signed-off-by: John M. Horan [email protected] Signed-off-by: Tushar Goel <[email protected]> Co-authored-by: Tushar Goel <[email protected]>
1 parent d3f314d commit b156af0

File tree

11 files changed

+1335
-204
lines changed

11 files changed

+1335
-204
lines changed

vulnerabilities/api.py

Lines changed: 54 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -48,11 +48,30 @@ class MinimalPackageSerializer(serializers.HyperlinkedModelSerializer):
4848
Used for nesting inside vulnerability focused APIs.
4949
"""
5050

51+
def get_affected_vulnerabilities(self, package):
52+
parent_affected_vulnerabilities = package.fixed_package_details.get("vulnerabilities") or []
53+
54+
affected_vulnerabilities = [
55+
self.get_vulnerability(vuln) for vuln in parent_affected_vulnerabilities
56+
]
57+
58+
return affected_vulnerabilities
59+
60+
def get_vulnerability(self, vuln):
61+
affected_vulnerability = {}
62+
63+
vulnerability = vuln.get("vulnerability")
64+
if vulnerability:
65+
affected_vulnerability["vulnerability"] = vulnerability.vulnerability_id
66+
return affected_vulnerability
67+
68+
affected_by_vulnerabilities = serializers.SerializerMethodField("get_affected_vulnerabilities")
69+
5170
purl = serializers.CharField(source="package_url")
5271

5372
class Meta:
5473
model = Package
55-
fields = ["url", "purl", "is_vulnerable"]
74+
fields = ["url", "purl", "is_vulnerable", "affected_by_vulnerabilities"]
5675

5776

5877
class MinimalVulnerabilitySerializer(serializers.HyperlinkedModelSerializer):
@@ -99,7 +118,6 @@ class Meta:
99118

100119

101120
class VulnerabilitySerializer(serializers.HyperlinkedModelSerializer):
102-
103121
fixed_packages = MinimalPackageSerializer(
104122
many=True, source="filtered_fixed_packages", read_only=True
105123
)
@@ -126,6 +144,20 @@ class PackageSerializer(serializers.HyperlinkedModelSerializer):
126144
Lookup software package using Package URLs
127145
"""
128146

147+
next_non_vulnerable_version = serializers.SerializerMethodField("get_next_non_vulnerable")
148+
149+
def get_next_non_vulnerable(self, package):
150+
next_non_vulnerable = package.fixed_package_details.get("next_non_vulnerable", None)
151+
if next_non_vulnerable:
152+
return next_non_vulnerable.version
153+
154+
latest_non_vulnerable_version = serializers.SerializerMethodField("get_latest_non_vulnerable")
155+
156+
def get_latest_non_vulnerable(self, package):
157+
latest_non_vulnerable = package.fixed_package_details.get("latest_non_vulnerable", None)
158+
if latest_non_vulnerable:
159+
return latest_non_vulnerable.version
160+
129161
purl = serializers.CharField(source="package_url")
130162

131163
affected_by_vulnerabilities = serializers.SerializerMethodField("get_affected_vulnerabilities")
@@ -134,7 +166,7 @@ class PackageSerializer(serializers.HyperlinkedModelSerializer):
134166

135167
def get_fixed_packages(self, package):
136168
"""
137-
Return a queryset of all packages that fixes a vulnerability with
169+
Return a queryset of all packages that fix a vulnerability with
138170
same type, namespace, name, subpath and qualifiers of the `package`
139171
"""
140172
return Package.objects.filter(
@@ -149,7 +181,7 @@ def get_fixed_packages(self, package):
149181
def get_vulnerabilities_for_a_package(self, package, fix) -> dict:
150182
"""
151183
Return a mapping of vulnerabilities data related to the given `package`.
152-
Return vulnerabilities that affects the `package` if given `fix` flag is False,
184+
Return vulnerabilities that affect the `package` if given `fix` flag is False,
153185
otherwise return vulnerabilities fixed by the `package`.
154186
"""
155187
fixed_packages = self.get_fixed_packages(package=package)
@@ -175,9 +207,23 @@ def get_fixed_vulnerabilities(self, package) -> dict:
175207

176208
def get_affected_vulnerabilities(self, package) -> dict:
177209
"""
178-
Return a mapping of vulnerabilities that affects the given `package`.
210+
Return a mapping of vulnerabilities that affect the given `package` (including packages that
211+
fix each vulnerability and whose version is greater than the `package` version).
179212
"""
180-
return self.get_vulnerabilities_for_a_package(package=package, fix=False)
213+
excluded_purls = []
214+
package_vulnerabilities = self.get_vulnerabilities_for_a_package(package=package, fix=False)
215+
216+
for vuln in package_vulnerabilities:
217+
for pkg in vuln["fixed_packages"]:
218+
real_purl = PackageURL.from_string(pkg["purl"])
219+
if package.version_class(real_purl.version) <= package.current_version:
220+
excluded_purls.append(pkg)
221+
222+
vuln["fixed_packages"] = [
223+
pkg for pkg in vuln["fixed_packages"] if pkg not in excluded_purls
224+
]
225+
226+
return package_vulnerabilities
181227

182228
class Meta:
183229
model = Package
@@ -190,6 +236,8 @@ class Meta:
190236
"version",
191237
"qualifiers",
192238
"subpath",
239+
"next_non_vulnerable_version",
240+
"latest_non_vulnerable_version",
193241
"affected_by_vulnerabilities",
194242
"fixing_vulnerabilities",
195243
]

vulnerabilities/models.py

Lines changed: 165 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
from django.core.validators import MinValueValidator
2424
from django.db import models
2525
from django.db.models import Count
26+
from django.db.models import Prefetch
2627
from django.db.models import Q
2728
from django.db.models.functions import Length
2829
from django.db.models.functions import Trim
@@ -32,6 +33,8 @@
3233
from packageurl.contrib.django.models import PackageURLQuerySet
3334
from packageurl.contrib.django.models import without_empty_values
3435
from rest_framework.authtoken.models import Token
36+
from univers import versions
37+
from univers.version_range import RANGE_CLASS_BY_SCHEMES
3538

3639
from vulnerabilities.severity_systems import SCORING_SYSTEMS
3740
from vulnerabilities.utils import build_vcid
@@ -67,6 +70,12 @@ def paginated(self, per_page=5000):
6770

6871

6972
class VulnerabilityQuerySet(BaseQuerySet):
73+
def affecting_vulnerabilities(self):
74+
"""
75+
Return a queryset of Vulnerability that affect a package.
76+
"""
77+
return self.filter(packagerelatedvulnerability__fix=False)
78+
7079
def with_cpes(self):
7180
"""
7281
Return a queryset of Vulnerability that have one or more NVD CPE references.
@@ -404,6 +413,24 @@ def purl_to_dict(purl: PackageURL):
404413

405414

406415
class PackageQuerySet(BaseQuerySet, PackageURLQuerySet):
416+
def get_fixed_by_package_versions(self, purl: PackageURL, fix=True):
417+
"""
418+
Return a queryset of all the package versions of this `package` that fix any vulnerability.
419+
If `fix` is False, return all package versions whether or not they fix a vulnerability.
420+
"""
421+
filter_dict = {
422+
"name": purl.name,
423+
"namespace": purl.namespace,
424+
"type": purl.type,
425+
"qualifiers": purl.qualifiers,
426+
"subpath": purl.subpath,
427+
}
428+
429+
if fix:
430+
filter_dict["packagerelatedvulnerability__fix"] = True
431+
432+
return Package.objects.filter(**filter_dict).distinct()
433+
407434
def get_or_create_from_purl(self, purl: PackageURL):
408435
"""
409436
Return an existing or new Package (created if neeed) given a
@@ -601,7 +628,6 @@ def __str__(self):
601628
return self.package_url
602629

603630
@property
604-
# TODO: consider renaming to "affected_by"
605631
def affected_by(self):
606632
"""
607633
Return a queryset of vulnerabilities affecting this package.
@@ -642,6 +668,144 @@ def get_absolute_url(self):
642668
"""
643669
return reverse("package_details", args=[self.purl])
644670

671+
def sort_by_version(self, packages):
672+
"""
673+
Return a list of `packages` sorted by version.
674+
"""
675+
if not packages:
676+
return []
677+
678+
return sorted(
679+
packages,
680+
key=lambda x: self.version_class(x.version),
681+
)
682+
683+
@property
684+
def version_class(self):
685+
return RANGE_CLASS_BY_SCHEMES[self.type].version_class
686+
687+
@property
688+
def current_version(self):
689+
return self.version_class(self.version)
690+
691+
@property
692+
def fixed_package_details(self):
693+
"""
694+
Return a mapping of vulnerabilities that affect this package and the next and
695+
latest non-vulnerable versions.
696+
"""
697+
package_details = {}
698+
package_details["purl"] = PackageURL.from_string(self.purl)
699+
700+
next_non_vulnerable, latest_non_vulnerable = self.get_non_vulnerable_versions()
701+
package_details["next_non_vulnerable"] = next_non_vulnerable
702+
package_details["latest_non_vulnerable"] = latest_non_vulnerable
703+
704+
package_details["vulnerabilities"] = self.get_affecting_vulnerabilities()
705+
706+
return package_details
707+
708+
def get_non_vulnerable_versions(self):
709+
"""
710+
Return a tuple of the next and latest non-vulnerable versions as PackageURLs. Return a tuple of
711+
(None, None) if there is no non-vulnerable version.
712+
"""
713+
package_versions = Package.objects.get_fixed_by_package_versions(self, fix=False)
714+
715+
non_vulnerable_versions = []
716+
for version in package_versions:
717+
if not version.is_vulnerable:
718+
non_vulnerable_versions.append(version)
719+
720+
later_non_vulnerable_versions = []
721+
for non_vuln_ver in non_vulnerable_versions:
722+
if self.version_class(non_vuln_ver.version) > self.current_version:
723+
later_non_vulnerable_versions.append(non_vuln_ver)
724+
725+
if later_non_vulnerable_versions:
726+
sorted_versions = self.sort_by_version(later_non_vulnerable_versions)
727+
next_non_vulnerable_version = sorted_versions[0]
728+
latest_non_vulnerable_version = sorted_versions[-1]
729+
730+
next_non_vulnerable = PackageURL.from_string(next_non_vulnerable_version.purl)
731+
latest_non_vulnerable = PackageURL.from_string(latest_non_vulnerable_version.purl)
732+
733+
return next_non_vulnerable, latest_non_vulnerable
734+
735+
return None, None
736+
737+
def get_affecting_vulnerabilities(self):
738+
"""
739+
Return a list of vulnerabilities that affect this package together with information regarding
740+
the versions that fix the vulnerabilities.
741+
"""
742+
package_details_vulns = []
743+
744+
fixed_by_packages = Package.objects.get_fixed_by_package_versions(self, fix=True)
745+
746+
package_vulnerabilities = self.vulnerabilities.affecting_vulnerabilities().prefetch_related(
747+
Prefetch(
748+
"packages",
749+
queryset=fixed_by_packages,
750+
to_attr="fixed_packages",
751+
)
752+
)
753+
754+
for vuln in package_vulnerabilities:
755+
package_details_vulns.append({"vulnerability": vuln})
756+
later_fixed_packages = []
757+
758+
for fixed_pkg in vuln.fixed_packages:
759+
if fixed_pkg not in fixed_by_packages:
760+
continue
761+
fixed_version = self.version_class(fixed_pkg.version)
762+
if fixed_version > self.current_version:
763+
later_fixed_packages.append(fixed_pkg)
764+
765+
next_fixed_package = None
766+
next_fixed_package_vulns = []
767+
768+
sort_fixed_by_packages_by_version = []
769+
if later_fixed_packages:
770+
sort_fixed_by_packages_by_version = self.sort_by_version(later_fixed_packages)
771+
772+
fixed_by_pkgs = []
773+
774+
for vuln_details in package_details_vulns:
775+
if vuln_details["vulnerability"] != vuln:
776+
continue
777+
vuln_details["fixed_by_purl"] = []
778+
vuln_details["fixed_by_purl_vulnerabilities"] = []
779+
780+
for fixed_by_pkg in sort_fixed_by_packages_by_version:
781+
fixed_by_package_details = {}
782+
fixed_by_purl = PackageURL.from_string(fixed_by_pkg.purl)
783+
next_fixed_package_vulns = list(fixed_by_pkg.affected_by)
784+
785+
fixed_by_package_details["fixed_by_purl"] = fixed_by_purl
786+
fixed_by_package_details[
787+
"fixed_by_purl_vulnerabilities"
788+
] = next_fixed_package_vulns
789+
fixed_by_pkgs.append(fixed_by_package_details)
790+
791+
vuln_details["fixed_by_package_details"] = fixed_by_pkgs
792+
793+
return package_details_vulns
794+
795+
@property
796+
def fixing_vulnerabilities(self):
797+
"""
798+
Return a queryset of Vulnerabilities that are fixed by this `package`.
799+
"""
800+
return self.vulnerabilities.filter(packagerelatedvulnerability__fix=True)
801+
802+
@property
803+
def affecting_vulnerabilities(self):
804+
"""
805+
Return a queryset of Vulnerabilities that affect this `package`.
806+
"""
807+
return self.vulnerabilities.filter(packagerelatedvulnerability__fix=False)
808+
645809

646810
class PackageRelatedVulnerability(models.Model):
647811
"""

0 commit comments

Comments
 (0)