Security and Vulnerability Scanning of Container Images


In this article we will focus on security and vulnerability strategies for scanning container images. I know, in the past security was always viewed upon as an impedance to the speed of production but hopefully these days are behind us. Having a security breach, as you probably know, is one of the most costly things an organization can endure. It takes years to build up a reputation and only seconds to tear it down completely. I still see today, many organizations ignoring container images completely because it is often misunderstood. Exactly what is inside a container image? Who should be responsible for it? How does it map to what we have done on servers? Security teams often don’t understand containers or even know what questions to ask. We need to help them and it is our duty to do so. Unfortunately there are not very many tools that can help in broad sense. Containers are new and evolving at breakneck speed. That coupled with the fact that security can negatively impact the speed of a DevOps team (if not done right), it is no wonder we are at square one, in many cases. Before we dive into more detail, let us review important security aspects of containers.
  • Containers can have various packaging formats, Docker is the most popular today
  • Containers are immutable and as such are image based
  • Container are never updated, any change always results in a new container
  • Container images consist of layers (base, runtime, application)
  • Container images require shared responsibility between dev and ops
  • Containers don’t contain, they are in fact, just processes
Container images contain OS, runtime and application (everything but the Linux kernel basically). If a container is breached, there is not much in the way preventing further containers from being breached unless you have SELinux, limit kernel capabilities, run as non-priviledged user and follow many other best practices. OpenShift and using trusted, signed images by a vendor like Red Hat does all of this by default. Even still the attack surface can be large. You still think it is OK to not worry about container images and what is inside? Thankfully there are at least several solutions we will talk about from Red Hat for the container Platform OpenShift.
  • Container Catalog grades images provided by Red Hat and provides history of image as security patches are applied.
  • Atomic CLI scans images and uses OpenSCAP to determine security vulnerabilities.
  • CloudForms scans images using OpenSCAP (same as atomic) and also adds capabilities like taking action when container images are vulnerable, automatically scanning new images and even reporting.
All of these solutions are included with OpenShift.

Container Catalog

Red Hat offers a container catalog for all images that it provides. Images are maintained by Red Hat meaning they are frequently scanned and updated. The container catalog also provides the history of an image, each time it is updated a new tag is created. The image is graded according to a scale A to F.
Health IndexSecurity Errata Conditions
Grade A This image does not have any unapplied Critical or Important security errata
Grade B This image is affected by Critical (no older than 7 days) or Important (no older than 30 days) security errata
Grade C This image is affected by Critical (no older than 30 days) or Important (no older than 90 days) security errata
Grade D This image is affected by Critical (no older than 90 days) or Important (no older than 12 months) security errata
Grade E This image is affected by Critical or Important security errata no older than 12 months
Grade F This image is affected by Critical or Important security errata older than 12 months
UnknownThis image is missing metadata required to calculate a grade and cannot be scanned
Check image (example nodejs-6). cc_1 We see the image has a health index of A and is also signed. We also see it runs as unprivileged user. View Image History Under tags, the history of the image can be viewed. cc_2 Image Details By clicking on the tag name we are able to get more detail about a given image. cc_3 Any CVEs that affect the image are shown in addition to RPM advisory.

Atomic CLI Image Scanning

Atomic Cli provides image scanning using OpenSCAP. This tool can be used via Ansible or other automation in CI/CD pipeline to automatically scan images. Install Atomic cli If you aren’t running RHEL Atomic you need to install the atomic cli. In this case we are doing so on the OpenShift master (master0) but it can be any node that has access to the container images. If image doesn’t exist locally (in this case on master0) it must be pulled down.
[master0 ~]$ yum install -y atomic
Get the image that should be scanned
[master0 ~]$ oc get image |grep nodejs

Scan image. Using the image location we perform a scan.
[master0 ~]$ sudo atomic scan (ae9be2ffb565659)

The following issues were found:

RHSA-2018:0260: systemd security update (Moderate)
 Severity: Moderate
 RHSA ID: RHSA-2018:0260-01
 Associated CVEs:
 CVE ID: CVE-2018-1049

RHSA-2018:0180: kernel-alt security and bug fix update (Important)
 Severity: Important
 RHSA ID: RHSA-2018:0180-01
 Associated CVEs:
 CVE ID: CVE-2017-1000405

RHSA-2017:0372: kernel-aarch64 security and bug fix update (Important)
 Severity: Important
 RHSA ID: RHSA-2017:0372-01
 Associated CVEs:
 CVE ID: CVE-2016-5195
 CVE ID: CVE-2016-7039
 CVE ID: CVE-2016-8666
Here we notice several important or high severity CVEs and a medium. Using atomic cli may be enough if images going into registry are tightly controlled. This could work well if additional tooling such as Artifactory is used to persist container images. They can easily be scanned before being added to Artifactory as part of CI/CD process.

CloudForms Image Scanning

CloudForms provides additional capabilities for security and vulnerability scanning. You can configure policies to take action based on an vulnerable image. For example, not allowing vulnerable images to run or notifying security team. In addition, as soon as an image is created and pushed to OpenShift registry via CI/CD, it can be scanned. Reporting allows for understanding impact of vulnerable images across projects in OpenShift. Image users can be easily notified and it allows roles between development and operations to have clear delineation.

Configure CloudForms Provider for OpenShift

The first step is to configure CloudForms provider to access OpenShift environment. Get management-admin token for management-infra project.
[master0 ~]$ oc get sa management-admin -o yaml -n management-infra
apiVersion: v1
- name: management-admin-dockercfg-4dfcr
kind: ServiceAccount
 creationTimestamp: 2018-02-09T21:01:32Z
 name: management-admin
 namespace: management-infra
 resourceVersion: "1092"
 selfLink: /api/v1/namespaces/management-infra/serviceaccounts/management-admin
 uid: 6860deab-0ddc-11e8-ab6a-fa163e0b5deb
- name: management-admin-token-q9clk
- name: management-admin-dockercfg-4dfcr
Get Secret for management-admin token.
[master0 ~]$ oc describe secret management-admin-token-q9clk -n management-infra
Name: management-admin-token-q9clk
Namespace: management-infra


token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtYW5hZ2VtZW50LWluZnJhIiwia3ViZXJuZXRlcy32ay9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im1hbmFnZW1lbnQtYWRtaW4tdG9rZW4tcTljbGsiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYW55ddssaVudC5uYW1lIjoibWFuYWdlbWVudC1hZG1pbiIsImt1YmVybmV0ZXMuaW
ca.crt: 1070 bytes
namespace: 16 bytes
service-ca.crt: 2186 bytes
Add OpenShift Provider to CloudForms. OpenShift by default creates a management-infra project with a management-admin SA and token. This project is used by CloudForms for access and image scanning. cfme_sec_1 cfme_sec_2 Set Node Selector for management-infra project. In this example we will set the region to infra. This means any nodes with region infra will run image scanning container. Infra nodes also generally run other shared platform services like router, registry, metrics and logging.
[master0 ~]$ oc edit namespace management-infra
 annotations: region=infra

Configure Security Scanning Policies

CloudForms ships with compliance and control policies that will automatically scan containers. If vulnerabilities are detected, containers with those vulnerabilities will be prevented from running. This is not enabled by default but I find it a bit aggressive and recommend scanning images and reporting them as non-compliant instead. Login to CloudForms and navigate to control->explorer. Copy OpenScap Compliance Policy. cfme_policy_1 Edit the copied OpenScap compliance policy. Here we will remove the action to prevent container images that are non-compliant from running. cfme_policy_2 Add new container image condition for sti builder. cfme_policy_5 Under conditions accordion, create a new condition. We will want to ignore scanning the sti builder. cfme_policy_3 Add new container image condition for deployer. Under conditions accordion, create a new condition. We will want to ignore scanning the deployer. cfme_policy_4 Copy control policy Analyse incoming container images and edit condition assignments. Add the two newly created conditions. cfme_policy_6 Add new profile policy. Under “all profile policies”, add a new profile policy. Add the following policies:
  • Copy of OpenScap (compliance)
  • Copy of Analyze incoming container images (control)
  • Schedule compliance after smart state analysis (control)
cfme_policy_7 Enable policy profile on OpenShift provider. Navigate to Compute->Containers->Providers. cfme_policy_8 Our policy profile will ensure the following:
  • All images that change are scanned immediately
  • The deployer, STI builder and image inspector are ignored
  • Images with High vulnerability are marked as non-compliant

Perform Container Image Scanning

Each build creates a new image. As soon as build is pushed the image is automatically scanned. In this case a change was made which kicked off CI/CD, resulting in a build and a new image being pushed to registry. ose_nodejs_1 ose_nodejs_2 Check pods under management-infra project. Each image will trigger a scan. The image scanner container will mount the image and scan it using openscap.
[master0 ~]$ oc get pods -o wide -n management-infra
manageiq-img-scan-ea955 0/1 Running 0 2m infra0
Check image scanner container logs.
[master0 ~]$ oc logs manageiq-img-scan-ea955 -n management-infra
2018/02/15 16:26:56 Pulling image
2018/02/15 16:26:56 Authentication with Default Empty Authentication failed: unauthorized: authentication required
2018/02/15 16:27:07 Downloading Image (17Kb downloaded)
2018/02/15 16:27:09 Finished Downloading Image (17Kb downloaded)
2018/02/15 16:27:24 Extracting image to /var/tmp/image-inspector-545571508
2018/02/15 16:32:49 OpenSCAP scanning /var/tmp/image-inspector-545571508. Placing results in /var/tmp/image-inspector-scan-results-428093571
2018/02/15 16:33:43 Serving image content /var/tmp/image-inspector-545571508 on webdav://
Find image. Once the image scanner runs we can navigate to the image in CloudForms under Compute->Containers->Container Images. We can use the search field to filter and find image by it’s name. In this case it is of course the latest image. cfme_image_1 View Image Details. Here we can see that smart state analysis in CloudForms (container image scan) was run. Notice compliance is not-compliant. Two High severity and a medium severity rule failed. This is exactly what we also saw when running atomic image scan. Finally notice in addition to the OpenScap results, we also have inventory of all the packages and corresponding package versions, installed in the image. cfme_image_2

Container Security and Vulnerability Reporting

Now that we are able to scan images and flag ones that have high security vulnerabilities, it is time to look into reporting. You could easily have 1000s of images so reporting becomes increasingly important to identify projects using high vulnerability images. This allows us to nicely segregate roles and responsibilities. The platform team can scan images and notify devops teams about vulnerabilities, who can in turn fix them. There are of course other models, just an idea. Create Container Image Vulnerability Report. Under CloudIntel->Reports->Custom add a new report. Select Fields. cfme_report_1 Set Consolidation. cfme_report_3 Configure Filters. There are two types of filters: primary and secondary. Primary is used when doing select on database while secondary filters after records are returned from database. Here we configure primary filter to return high severity rules that failed. A secondary filter is used to only show images which are part of OpenShift projects. Note: At this time in CloudForms 5.8 the primary filter seems to have some issues so keep that in mind if you are using that version. cfme_fitter_3 Run Report. Under Reports->Custom select the report “OpenScap High Security Vulnerabilities” and select queue from configuration dropdown. The report will show projects that have images with a high severity rule that failed. cfme_report_5 Once projects and images are identified more detail may be obtained by looking at the OpenScap report. Under Compute->Containers->Container Images you can find the image and open it for more details. The OpenScap html report will contain details on specific rules and security violations. cfme_report_6 Clicking the OpenScap HTML report will download it via browser. The report shows all rules and if they passed or failed. cfme_report_7 You can dril into the rule and get more information. The summary shows the relevant CVEs. cfme_report_9 Violations that caused the rule to fail are also shown. cfme_report_8


In this article we looked into the topic of security and vulnerability scanning of container images. Unfortunately this is a topic that doesn’t get nearly enough attention in my view. We discussed the importance of why you want to keep container images updated, signed and get them from only trusted sources. We looked into several solutions provided by Red Hat. The Container Catalog, Atomic Cli and CloudForms. A guide was provided to explore each of these solutions individually, not only to understand their value, but also how to use them. In the end I think all three tools provide valuable information to ensure security standards are upheld. Choosing one of them or all of them depends on the processes an organization has defined. Happy Secure Containers! (c) 2018 Keith Tenzer

Leave a Comment

%d bloggers like this: