This repository provides an overview and high-level instructions about Image scanning results inside a namespace using IBM Cloud Container Registry Vulnerability Advisor inside a Bluemix environment.
These findings and instructions provided in this repository assume that the user has a Bluemix account, has completed the setup instructions and all required CLIs as well as other prerequisites, already configured a namespace and is running cluster(s) inside the Bluemix account.
Supported operating system images in a Bluemix repository are scanned by the IBM Cloud Container Scanner which uses a microservice that scans the containers in a Kubernetes cluster for risks, missing patches and other potential vulnerabilities. "This microservice runs in the kube-system namespace". The output of the results are provided by the Vulnerability Advisor reporting detailed information about the vulnerabilities of an image.
Once an image scan is complete the Vulnerability Advisor provides a detailed report whether the image has passed a scan or whether the image is vulnerable and requires updates/patches. "The possible vulnerabilities are updated daily from published security notices for the Docker image types". The updated image can be downloaded to the local machine and pushed back to the repository using docker commands to meet security requirements detected by the Vulnerability Advisor. Though, in some unique cases some images require manual update on a localhost for the reported security issues before they can be pushed back to the repository.
A playground environment was created in Bluemix using the instruction provided in IBM Cloud Docs to configure and to test the functionalities of the IBM Cloud Container Scanner and to view the report provided by the Vulnerability Advisor. In this particular environment, accessed using the CLI method, a cluster figure 1 and a namespace figure 2 were configured and implemented.
Using the docker commands from the terminal, docker images were pushed to the namespace in the Bluemix environment figure 3. A sample report result provided by the Vulnerability Advisor below, figure 3, displays httpd image has security issues once it was scanned by the microservice. In this example, only one image, httpd, has a security issue while the other images have passed the security scan.
The security reports are accessible via the GUI from the Bluemix console which provides a detailed, user-friendly dashboard about the vulnerabilities of an image residing in the namespace. Furthermore, by clicking on an image in the GUI, information about specific vulnerabilities and how to resolve the listed security issues becomes visible the specific image.
For example, figure 6, httpd image has 6 vulnerabilities reported by the Vulnerability Advisor. In figure 7, the same Docker image, details 1 of 6 vulnerabilities further providing an explanation about the specific vulnerability.
As mentioned earlier in this document, "the possible vulnerabilities are updated daily from published security notices for the Docker image types". The updated image can be downloaded to the local machine and pushed back to the repository using docker commands to meet security requirements detected by the Vulnerability Advisor. Though, in some unique cases, some images require manual update on a localhost for the reported security issues before they can be pushed back to the repository.
For example, in figure 8 below, some Docker commands are provided to push or pull the particular image. Unless pushing the same image to the repository is intended for a different reason, it's important to execute the Docker push command once the particular image is updated.
Image containers can be verified for security issues before they are deployed into a cluster in IBM® Cloud Kubernetes Service. The verification is handled by policy or policies that are in place in the environment. The images can be controlled from where they are deployed, and the policies on them can ensure that they are also coming from a trusted source and meet security compliance. For example, if a particular pod does not meet a specific security policy that is in place, then the cluster will not be modified and the pod will be prevented from being deployed.
"IBM Container Image Security Enforcement gets the information about image content trust and vulnerabilities from IBM® Cloud Container Registry. You can choose whether to block or allow images from other registries in your policies, but you cannot use vulnerability or trust enforcement for these images".
Figure 1 below is a sample policy, Default cluster-wide policy, that can be implemented in a cluster environment. The focus on this particular policy are the components trust and va:
policy:
trust:
enabled: true
va:
enabled: true
In summary, these two specifications in this policy determine whether an image meet the security requirements inside the specific cluster. The trust section looks for a trusted, signed image where the key is pulled from a trusted server. The va section determines if the image has passed or failed a security scan performed by the Vulnerability Advisor. When both trust and va are set to true it indicates that they are both active or as listed in the policy enabled. When either one of these settings are set to false it means that the policy will bypass that particular component and the image will pass regardless of any existing security issue it may have.
In figures 2 - 4 there are live demonstration screenshots that display the results of a pod deployment affected by these policy settings discussed thus far.
There are other policies similar to the Default Cluster-Wide policy, for example: Default kube-system policy, Default ibm-system policy, and others that are deployed by default in a cluster. A custom policy can be created to meet the needs of a specific cluster environment and are further explained with more details here: Enforcing container image security (beta).
Livescan with Vulnerability Advisor provides security management for IBM® Cloud Kubernetes Service. The Vulnerability Advisor generates a security status report, suggests fixes and best practices, and provides management to restrict non-secure images from running inside of a cluster.
According to Container Service Development member, Kristin Kronstain Brown, this feature Image Security Live-scan is currently not available with the Vulnerability Advisor in the production environment at this time. Though, staging documentation can be found here.
-
A cluster with Kubernetes version 1.9 or higher. Tutorial: Creating Kubernetes clusters can be found here.
-
Helm. Setting up Helm and installing default policies can be found here
-
A namespace. How to set up a namespace can be found here
$ git clone https://github.com/ibm-client-success/iks-container-registry.git
$ cd iks-container-registry
The focus of this demonstration is to secure Docker images before entering a namespace or a cluster using custom IBM Container Image Security Enforcement custom policies. This article introduces an overview, considerations, and recommendations using custom policies for identifying and preventing vulnerable Docker containers from being deployed in the IBM Cloud environment.
IBM Container Image Security Enforcement can be configured to verify container images before they are deployed to a cluster on IBM® Cloud Kubernetes Service. Images are controlled where they are deployed from, enforce Vulnerability Advisor policies, and certify that content trust is suitably applied to an image before deployment.
IBM Container Image Security Enforcement automatically installs policies to provide a starting point for building security policy in a cluster. Additional information and how to install default policies can be found here. These policies will apply the default security policies onto all Kubernetes namespaces in the cluster. The default policies that will install are:
1. Default Cluster-wide policy – Enforces that all images in all registries are from a trusted source and Vulnerability Advisor has no security reported issues on the image.
2. Default Kube-system policy – A namespace-wide policy installed specifically for the kube-system namespace. It allows all images for any container registry to be deployed into the kube-system namespace without restriction.
3. Default IBM-system policy – A namespace-wide policy installed by default. This policy permits all images from any container registry to be deployed into the ibm-system without restrictions. Additionally, repositories that are used will be permitted to configure the cluster and to install or upgrade Image Security Enforcement.
This blog will focus on the default cluster-wide policy and its two important flag components: “va” and “trust” as shown in Figure 1. Before moving forward, it is important to explain that “va” stands for Vulnerability Advisor and its primary function is to scan images for any security vulnerabilities. The main function of the “trust” flag is to ensure the image’s integrity, to confirm that it’s signed, and be sure that it originated from a trusted source. Signing an image and verifying its integrity is explained in detail here.
Figure 1 – An Example of a Default Cluster-Wide Policy.yaml File
Figure 2 features a high-level overview of an image that’s been screened by the default cluster-wide policy. Depending on the policy’s flag configuration in a cluster, the image will be validated for the next step in its deployment process differently.
Figure 2 – High-Level Overview of a Custom Policy
The figure below shows the four possible combinations of the component flags in a cluster-wide policy and the corresponding result of attempting to deploy an untrustworthy image with vulnerabilities.
Figure 3 – High-Level Overview of the Custom Policy Possibilities Against an Image
In this section, we’ll see the outcomes of different custom policies applied against an image.
The image list in Figure 4 displays information that can differ when the default cluster-wide policy flags “trust” and “va” are modified.
$ bx cr images
Figure 4 – IBM Cloud Image Repository
Figure 4 displays the list of images in a namespace. As shown, there are few images with security issues and others with no security issues. We can see how the deployment of these images will be affected when we enforce image security using a default Cluster-Wide policy.
Below, we will see the outcomes of four different cases of custom policies by changing the “trust” and “va” flags from DefaultCluster-wide.yaml , applied against the images.
Case 1
For example, when both the “trust” and “va” components in the default cluster-wide policy, exhibited in the defaultcluster-wide.yaml file below, are set to “false” the policy will not prevent a POD deployment regardless of the integrity or any potential vulnerabilities in an image. This means that image deployment and POD creation is allowed without any issues, such as issues reported by Vulnerability Advisor and/or the trustworthiness of an image.
$ vi defaultcluster-wide.yaml
apiVersion: securityenforcement.admission.cloud.ibm.com/v1beta1
kind: ClusterImagePolicy
metadata:
name: ibmcloud-default-cluster-image-policy
spec:
repositories:
# This enforces that all images deployed to this cluster pass trust and VA
# To override, set an ImagePolicy for a specific Kubernetes namespace or # modify this policy
- name: "*"
policy:
trust:
enabled: false
va:
enabled: false
$ kubectl apply -f defaultcluster-wide.yaml
Results:
Attempting to deploy the tomcat:latest image using the tomcat.yaml file example:
Example of tomcat.yaml File
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-deployment
labels:
app: tomcat
spec:
replicas: 2
selector:
matchLabels:
app: tomcat
template:
metadata:
labels:
app: tomcat
spec:
containers:
- name: tomcat
image: YOUR_REGISTRY_ADDRESS_LINK/YOUR_SPACE/tomcat:latest
ports:
- containerPort: 80
Note:
Make sure you have the correct image path in the yaml file associated with image you want to run or deploy
$ kubectl apply -f tomcat.yaml
Results:
$ kubectl get deployments
Results:
$ kubectl get pods
Results:
Validate that your image is deployed and is working as expect.
Clean up
$ kubectl delete deployment tomcat-deployment
Results:
Case 2
Now, let’s leave the default cluster-wide policy “trust” to “false” and change the “va” to “true” and see the results. To make this change you have two options:
Option 1: Note: First navigate to the directory where you have cloned the repo, than execute the following commands:
sed -i -e '15 s/enabled: false/enabled: true/' contents/defaultcluster-wide.yaml
Option 2:
$ vi defaultcluster-wide.yaml
apiVersion: securityenforcement.admission.cloud.ibm.com/v1beta1
kind: ClusterImagePolicy
metadata:
name: ibmcloud-default-cluster-image-policy
spec:
repositories:
# This enforces that all images deployed to this cluster pass trust and VA
# To override, set an ImagePolicy for a specific Kubernetes namespace or # modify this policy
- name: "*"
policy:
trust:
enabled: false
va:
enabled: true
$ kubectl apply -f defaultcluster-wide.yaml
Results:
$ kubectl apply -f tomcat.yaml
Results:
As seen in Figure 4, the image tomcat:latest has one issue reported by the Vulnerability Advisor so POD deployment for this particular image is prevented as expected, since “va” flag is set to “true”.
Case 3:
To demonstrate the use of “trust” flag, we consider taking an image with no vulnerability issues and try to deploy it. The tomcat:v2 image as show in Figure 4, is a secure image in the IBM Cloud image repository without any vulnerability issues. Let’s modify the default cluster-wide policy “trust” to “true” and “va” to “false”, then execute the tomcat-v2.yaml file and analyze the outcome:
Example of tomcat-v2.yaml File
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcatv2-deployment
labels:
app: tomcat
spec:
replicas: 3
selector:
matchLabels:
app: tomcat
template:
metadata:
labels:
app: tomcat
spec:
containers:
- name: tomcat
image: YOUR_REGISTRY_ADDRESS_LINK/YOUR_SPACE/tomcat:v2
ports:
- containerPort: 80
To modify the defaultcluster-wide.yaml you have two options:
Option 1: Note: First navigate to the directory where you have cloned the repo, than execute the following commands:
sed -i -e '13 s/enabled: false/enabled: true/' contents/defaultcluster-wide.yaml
sed -i -e '15 s/enabled: true/enabled: false/' contents/defaultcluster-wide.yaml
Option 2:
$ vi defaultcluster-wide.yaml
apiVersion: securityenforcement.admission.cloud.ibm.com/v1beta1
kind: ClusterImagePolicy
metadata:
name: ibmcloud-default-cluster-image-policy
spec:
repositories:
# This enforces that all images deployed to this cluster pass trust and VA
# To override, set an ImagePolicy for a specific Kubernetes namespace or modify this policy
- name: "*"
policy:
trust:
enabled: true
va:
enabled: false
$ kubectl apply -f defaultcluster-wide.yaml
Results:
$ kubectl apply -f tomcat-v2.yaml
Results:
Deployment of the tomcat:v2 image has been denied because the system has determined that this particular image was not signed when it was pulled from the source and is untrustworthy.
Case 4:
When both flags are set to “true”, as shown in the discoveries so far, the first flag of the default cluster-wide policy is applied first. Therefore, if the image is not signed and it is not from a trusted source, then the deployment will be denied, and the second flag of the policy will not be applied. Deployment will cease at this point.
Depending on the settings of the default cluster-wide policy components, “trust” and “va”, there are several ways to remedy image issues in order to successfully deploy in a cluster:
- Pull a more up-to-date image and verify that it is signed from a trusted source. This is the most common method.
- The user can manually update an existing image and sign an image in a repository to ensure the integrity of images in your registry namespace.
- Create an exception in a repository to bypass an image’s Vulnerability Advisor scan report for a particular reported issue(s).
IBM Container Image Security Enforcement custom policies are easy to be configured and most importantly contribute in pushing secure images into a namespace before the image is deployed in a cluster. Ultimately, these custom policies are put in place to safeguard a cloud environment by validating the images before allowing them to be deployed into a Kubernetes cluster.