Skip to content

reversinglabs/rl-scan-artifactory

Repository files navigation

rl-scan-artifactory

ReversingLabs provides the official Spectra Assure integration for JFrog Artifactory to help software producers protect their organization, their development and build environments, and their customers by continuously monitoring and scanning software artifacts in Artifactory repositories. Only self-hosted Artifactory instances are supported.

The integration is called rl-scan-artifactory and provided as a Python package that can be installed directly from PyPI. The integration's GitHub repository also contains a separate Artifactory plugin called rlBlock which can be used to prevent access to specific artifacts based on their scanning results from rl-scan-artifactory.

The rl-scan-artifactory integration is most suitable for existing Spectra Assure users who want to protect their current or future Artifactory repositories in an automated, reliable way.

Depending on your use-case, the integration can rely on the Spectra Assure CLI (rl-secure or Docker image) or the Spectra Assure Portal when scanning artifacts.

It also allows for flexible workflows where you can scan some repositories with the Portal while scanning others with the CLI (for example, if a repository contains artifacts you don't want to upload to the Portal). Such combined workflows require an active license for both Spectra Assure products.

Note: This documentation assumes that you already have experience with JFrog Artifactory and with Python scripts.

What is Spectra Assure?

The Spectra Assure platform is a set of ReversingLabs products primarily designed for software assurance and software supply chain security use-cases.

It helps users protect their software supply chains by analyzing compiled software packages, their components and third-party dependencies to detect exposures, reduce vulnerabilities, and eliminate threats before reaching production.

Users can choose to work with Spectra Assure as an on-premises CLI tool, a ReversingLabs-hosted SaaS solution called Portal, or use Spectra Assure Docker images and integrations in their CI/CD pipelines.

How this integration works

This integration relies on user-specified parameters to:

  • check for supported artifact types in the specified Artifactory repositories
  • scan the supported artifacts with the specified Spectra Assure product
  • save analysis reports to Artifactory and update artifact metadata with scan results and status information

SVG rl-scan-artifactory

The integration should be installed on a machine that can access and interact with a self-hosted JFrog Artifactory instance. After it's installed, the integration can be started manually or scheduled to run in custom intervals.

When the integration is started for the first time, scanning may take many hours or even days depending on the size and the number of artifacts in the specified Artifactory repositories.

The integration downloads the artifacts to a temporary directory and scans them one by one using one of the selected Spectra Assure products.

The following products are supported:

  • Spectra Assure CLI (rl-secure). In this workflow, the integration scans artifacts locally without uploading them to ReversingLabs. To scan the artifacts, rl-secure must be installed on the machine and a permanent package store must be initialized. Analysis reports are automatically uploaded to Artifactory as a compressed file. This workflow is ideal for users already familiar with the Spectra Assure CLI, and for those who need to scan their artifacts exclusively on-premises.

  • Spectra Assure CLI Docker image reversinglabs/rl-scanner. In this workflow, the integration scans artifacts locally inside the Docker container, without uploading them to ReversingLabs. To scan the artifacts, the rl-scanner Docker image is used to run rl-secure in a container. A package store must be initialized. The sync option is not supported. This workflow is ideal for users who want to scan their artifacts in ephemeral environments.

  • Spectra Assure Portal. In this workflow, the integration uploads artifacts to a Portal instance for scanning. The Portal account used for scanning must have valid credentials, and enough analysis capacity configured on the Portal. The analysis report is accessible on the Portal and linked in the artifact metadata properties. This workflow is ideal for users already familiar with the Spectra Assure Portal, and for those who want a quick way to share analysis reports with internal and external stakeholders.

After the scan has finished, the integration applies metadata properties on the scanned artifacts in Artifactory.

For each artifact, the properties indicate that it has been scanned with Spectra Assure and record its overall scan status (pass or fail). This allows the integration to skip already scanned artifacts the next time it starts, and to scan only new artifacts in the repository. It also allows users to prevent downloading artifacts that have the fail status with the rlBlock Artifactory plugin.

Requirements and dependencies

Supported Artifactory repository types

The following repository types can be provided to the integration for scanning:

  • Local
  • Remote

Virtual repository types are currently not supported.

Cache repositories are implicitly used for all remote repository types.

Supported Artifactory package types

The following package types will be detected by the integration as supported artifacts for scanning:

  • Debian - files ending in .deb
  • Docker - manifest.json files. More details in the Working with Docker images section
  • RubyGems - files ending in .gem
  • Generic - files ending in .rl_meta when scanning with the Portal. When scanning with the CLI or CLI Docker image, all files are scanned and assigned a placeholder package URL, but it's not possible to use the sync option. More details in the Working with generic repositories section
  • Maven - files ending in .jar
  • npm - files ending in .tgz
  • PyPI - files with properties .name and .version
  • RPM - files ending in .rpm.

All files ending in -reports.zip are skipped regardless of the package type.

All files ending in .rl_meta are skipped.

Installation

Before installing the integration, we recommend creating a virtual environment according to your preferences.

Install the latest version of the integration from PyPI with pip:

pip3 install rl-scan-artifactory

The installation makes the rl-scan-artifactory command available in your virtual environment.

You can then check the installed version with:

rl-scan-artifactory --version

Follow the quick start guide for instructions on how to get started with the integration.

Configuration

Environment variables

All environment variables except LOG_LEVEL can also be passed to the rl-scan-artifactory command as parameters. Their parameter names are listed in the Equivalent parameter column.

Environment variable Equivalent parameter Description
ARTIFACTORY_HOST --artifactory-host Required. Fully qualified domain name (without https://) of your Artifactory server.
ARTIFACTORY_USER --artifactory-user Required. Name of the Artifactory user configured to add properties.
ARTIFACTORY_TOKEN --artifactory-token Required. Access token for the specified Artifactory user.
PROXY_SERVER --proxy-server Server name for proxy configuration (IP address or DNS name).
PROXY_PORT --proxy-port Network port on the proxy server for proxy configuration. Required if PROXY_SERVER is used.
PROXY_USER --proxy-user User name for proxy authentication.
PROXY_PASS --proxy-pass Password for proxy authentication. Required if PROXY_USER is used.
LOG_LEVEL Not supported as parameter Allows controlling the type of output messages displayed in the logs. Set to one of the following: CRITICAL, WARNING, INFO. Default is WARNING

CLI only

Environment variable Equivalent parameter Description
RLSECURE_SITE_KEY --rlsecure-site-key Required only for the CLI Docker workflow. The rl-secure license site key. The site key is a string generated by ReversingLabs and sent to users with the license file.
RLSECURE_ENCODED_LICENSE --rlsecure-encoded-license Required only for the CLI Docker workflow. The rl-secure license file as a Base64-encoded string. Users must encode the contents of their license file, and provide the resulting string with this variable..

Portal only

Environment variable Equivalent parameter Description
RLPORTAL_SERVER --rlportal-server Required only for the Portal workflow. Name of the Portal instance to use for the scan. The Portal instance name usually matches the subdirectory of my.secure.software in your Portal URL. For example, if your Portal URL is my.secure.software/demo, the instance name to use with this parameter is demo.
RLPORTAL_GROUP --rlportal-group Required only for the Portal workflow. The name of a Portal group to use for the scan. The group must exist in the specified Portal organization. Group names are case-sensitive.
RLPORTAL_ORG --rlportal-org Required only for the Portal workflow. The name of a Portal organization to use for the scan. The organization must exist on the specified Portal instance. The user account authenticated with the token must be a member of the specified organization and have the appropriate permissions to upload and scan a file. Organization names are case-sensitive.
RLPORTAL_ACCESS_TOKEN --rlportal-access-token Required only for the Portal workflow. A Personal Access Token for authenticating requests to the Portal. To use the Portal workflow with this integration, you must create the token in your Portal settings. Tokens can expire and be revoked, in which case you'll have to update the value of this environment variable. It's strongly recommended to treat this token as a secret and manage it according to your organization's security best practices.

Parameters

One of the following parameters must always be specified to choose the Spectra Assure product that will be used for scanning: --portal, --cli, --cli-docker.

Depending on the product choice, some parameters may be ignored because they are incompatible with the specified product.

Parameter Description
--repo, -r Required. Specify one or more repository names to scan and monitor for artifacts. At least one repository name must be specified. To specify multiple repository names, separate them by space or comma, or repeat the parameter for each repository name. Specified repositories must have the supported repository type and contain artifacts that are among the supported package types.
--cli, -C Use the Spectra Assure CLI for artifact scanning and generating analysis reports. Requires rl-deploy or rl-secure to be installed. Mutually exclusive with --portal and --cli-docker.
--cli-docker Use the Spectra Assure CLI Docker image (reversinglabs/rl-scanner:latest) for artifact scanning and generating analysis reports. Requires setting the environment variables RLSECURE_ENCODED_LICENSE and RLSECURE_SITE_KEY. Mutually exclusive with --cli and --portal.
--portal, -P Use the Spectra Assure Portal for artifact scanning and generating analysis reports. Mutually exclusive with --cli and --cli-docker.
--cli-rlstore-path Required when using --cli or --cli-docker. Path to an existing package store that the integration can use.
--cli-rlsecure-path Required when using --cli. Path to the locally installed rl-secure executable.
--sync, -S Enables reanalyzing previously scanned artifacts. If a package URL associated with an artifact already exists in the Portal or in the specified package store, this parameter instructs the integration to use the sync action instead of scan. Not supported for --cli-docker.
If using the rlBlock plugin, you won't be able to sync artifacts with status fail if you don't allow the server where rl-scan-artifactory is running from. Check the plugin README for instructions.
--pack-safe Include the RL-SAFE archive in the compressed file with analysis reports. Incompatible with --portal
--cli-reports-repo Compatibility parameter for storing reports in remote repositories. By default, Artifactory repositories of type remote cannot be used to store reports. The integration needs a custom local generic repository to store the reports (e.g Spectra-Assure-Reports), and it should be specified with this parameter. If not specified, all remote repositories will be skipped.
--download, -d Path to an existing directory that the integration can use for temporary artifact downloads from Artifactory. If not specified, Python tempfile.gettempdir() will be used.
--ignore-cert-errors Allow working with invalid or self-signed certificates. Default: false
--ignore-artifactory-properties, -I If specified, the integration will ignore any existing properties set for the scanned artifacts in Artifactory.
--verbose, -v Display more detailed progress messages and scan results on stdout.
--version, -V Show currently installed version of the integration and exit.

Artifactory properties

After scanning the artifacts, the integration adds metadata properties to each top-level item in the repository. For Docker images, the properties are set on the manifest.json file. The properties are also visible in the Artifactory GUI.

All properties have the RL prefix to indicate they are custom properties set by a ReversingLabs integration.

Property Description
RL.progress Indicates the progress of the artifact scan. Can be one of the following values: 'upload_to_portal_ok', 'scanned'
RL.scan-status Indicates the final status of the artifact scan. Can be one of the following values: 'pass', 'fail'
RL.package-url The package URL assigned to the artifact during upload to the Portal. The package URL is in the format project/package@version. You can use this information to identify and find the artifact in your Portal Projects.
RL.scan-report Direct URL to the analysis report for the scanned artifact.
RL.group Name of the Portal group associated with your account. Applies only if using the Portal.
RL.organization Name of the Portal organization associated with your account. Applies only if using the Portal.
RL.noscan Used to skip scanning report files uploaded by the integration when using the CLI. Applies only if using the CLI or CLI Docker. Default value is 'true'
RL.timestamp UTC timestamp of the last operation performed on the artifact. Value: YYYY-MM-DDTHH:MM:SSZ

Working with Docker images

When Docker images are uploaded to Artifactory, they typically come with manifest.json files that describe each Docker image and its layers. However, the file path to manifest.json may not have a version. In that case, the integration will extract a version (if present) from the config or an associated list.manifest.json.

For Docker images, the integration currently sets metadata properties only on the manifest.json file. As a result, the rlBlock Artifactory plugin does not support Docker images because it only blocks the manifest.json download.

Marker files

Under certain circumstances (when a layer was already downloaded outside of Artifactory), a Docker image layer is represented by a marker file with zero (0) length. This will make the Spectra Assure scan invalid.

To resolve this issue, you can use the fix_marker.sh script to properly populate all marker files.

Working with generic repositories

Scanning repositories with the package type 'generic' requires a custom .rl_meta file for each artifact, and is only supported when scanning with the Spectra Assure Portal.

The file must be in ini format with a mandatory section called rl_meta like in the following example.

File name: mypackagename-windows-arm_v8-1.0.99.rl_meta

File contents:

[rl_meta]
namespace = windows
name = mypackagename
version = v1.0.99
architecture = arm_v8
path = mypackagename-windows-arm_v8-v1.0.99.msi
sha256 = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

The rl_meta section supports the following fields:

Name Required Description
name Yes The name of the file identified by the path value.
version Yes The version of the file identified by the path value.
path Yes The relative path to the file (relative to the meta file).
namespace No Use if the name is not sufficiently unique on its own.
architecture No Should be used for binary packages (e.g. i386, amd64, arm_v8...). Can be set to 'none' for neutral source code, or can be left empty.
sha256 No The SHA256 hash of the file identified by the path value.

Optional fields can be empty.

The information from the rl_meta file is used to construct a unique package URL in the format <project>/<package>@<version> that will be used by the Spectra Assure Portal. Specifically, the following components from the rl_meta file map to the parts of the package URL:

  • project: <artifactory repository name>
  • package: [ <architecture>. ] [ <namespace>. ] <name>
  • version: <version>

The resulting package URL should not contain @ or / or any HTML-conflicting data. If any of its parts do contain restricted substrings, they will be automatically converted to safe values so that a valid package URL can be constructed.

When using --cli or --cli-docker to scan a generic repository, the rl_meta file is not supported, and all files in the repository will be scanned. In this case, a placeholder ("dummy") package URL is used for both --cli and --cli-docker. Because generic items without a proper package URL cannot be reanalyzed, the sync option is not supported.

Blocking artifact downloads

Optionally, an Artifactory plugin called rlBlock can be used together with the rl-scan-artifactory integration to prevent download requests for artifacts that received the fail status after analysis.

Download blocking must be enabled in the plugin's .properties file. If enabled, the plugin checks the artifact properties whenever a download request is received by Artifactory. If a Spectra Assure scan status property exists for an artifact and its value is "fail", the plugin immediately returns HTTP 403 and a JSON body with a message stating that the download was blocked due to a failing ReversingLabs Spectra Assure scan. If the property doesn't exist for an artifact or its value is "pass", the plugin allows the download to continue as normal.

To use the plugin, download it directly from the rl-scan-artifactory GitHub repository. Find more detailed configuration instructions in the plugin README.

Useful resources

About

Scan JFrog Artifactory artifacts with Spectra Assure

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages