Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add sigstore-policy-tester extension #597

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

BobyMCbobs
Copy link
Contributor

@BobyMCbobs BobyMCbobs commented Feb 5, 2025

To enable verifying of every container image run in containerd using Sigstore.

containerd has an ImageVerifier plugin which allows images to be allowed or denied depending on whether they return 0 or 1.

Most of the images delivered through Talos are signed, though problematically some are not.

Using a ClusterImagePolicy (from https://github.com/sigstore/policy-controller), at a system level, things can be defined for allow or deny.

@BobyMCbobs BobyMCbobs force-pushed the add-sigstore-policy-tester branch from fbd7fd9 to ea1c078 Compare February 5, 2025 21:53
## Usage

```yaml
machine:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say to put these files as part of extension, so adding this extension enables it. ClusterImagePolicy can be an extension config (though /etc/containers/sigstore needs to be an allowed extension path. possibly create a Talos discussion first)

Copy link
Contributor Author

@BobyMCbobs BobyMCbobs Feb 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@frezbo, thank you for taking a look!

I would say to put these files as part of extension, so adding this extension enables it

That's valid, though in Talos' current state, Kubernetes will fail to come up due to several images not being signed and even after that were resolved their workloads won't run without modifying this policy to include their expected keys or OIDC ID/issuer, see:

talosctl images default | xargs -I{} sh -c 'echo {}; cosign verify --certificate-oidc-issuer-regexp='.*' --certificate-identity-regexp='.*' -o text {}' sh

ClusterImagePolicy can be an extension config (though /etc/containers/sigstore needs to be an allowed extension path

as in something like this?

apiVersion: v1alpha1
kind: ExtensionServiceConfig
name: sigstore-policy-tester
configFiles:
    - content: |
        apiVersion: policy.sigstore.dev/v1alpha1
        kind: ClusterImagePolicy
        metadata:
          name: system
        spec:
          images:
          - glob: "**"
          authorities:
          - keyless:
              url: https://fulcio.sigstore.dev
              identities:
              - issuer: https://accounts.google.com
                subject: k8s-infra-gcr-promoter@k8s-artifacts-prod.iam.gserviceaccount.com
            ctlog:
              url: https://rekor.sigstore.dev
          - keyless:
              identities:
              - issuer: https://accounts.google.com
                subjectRegExp: "@siderolabs\.com$"
      mountPath: /usr/local/etc/containers/sigstore/policy.yaml # The mount path of the extension service config file.
# The environment for the extension service.
environment:
    - CONTAINERD_IMAGE_VERIFIER_SIGSTORE_POLICY_PATH=/usr/local/etc/containers/sigstore/policy.yaml

possibly create a Talos discussion first

I'll create one now

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To enable verifying of every container image run through containerd
@BobyMCbobs BobyMCbobs force-pushed the add-sigstore-policy-tester branch from ea1c078 to 0a6e49f Compare February 6, 2025 02:34
@BobyMCbobs
Copy link
Contributor Author

BobyMCbobs commented Feb 6, 2025

Testing locally building and running with roughly the following

cd ~/github.com/siderolabs/extensions
make docker-sigstore-policy-tester PLATFORM=linux/arm64 TARGET_ARGS="--load"

podman run -it --rm -v "$PWD:$PWD" --workdir "$PWD" \
  ghcr.io/siderolabs/imager:v1.9.3 \
    installer \
    --arch arm64 \
    --system-extension-image ghcr.io/bobymcbobs/siderolabs-extensions/sigstore-policy-tester:latest@sha256:1f0dc82955f4e4861aa13df6fdcd3de6fc5a826a81068148c5d9aa76c146eb36 \
    --output _out

docker image load --input _out/installer-arm64.tar
podman tag ghcr.io/siderolabs/installer:v1.9.3 ghcr.io/bobymcbobs/siderolabs/talos/installer:v1.9.3-with-sigstore-policy-tester
podman push ghcr.io/bobymcbobs/siderolabs/talos/installer:v1.9.3-with-sigstore-policy-tester

podman run -it --rm -v "$PWD:$PWD" --workdir "$PWD" \
  ghcr.io/siderolabs/imager:v1.9.3 \
    iso \
    --arch arm64 \
    --system-extension-image ghcr.io/bobymcbobs/siderolabs-extensions/sigstore-policy-tester:latest@sha256:1f0dc82955f4e4861aa13df6fdcd3de6fc5a826a81068148c5d9aa76c146eb36 \
    --output _out

Then for the config, using a patch like so

machine:
  env:
    CONTAINERD_IMAGE_VERIFIER_SIGSTORE_POLICY_PATH: /var/local/etc/containers/sigstore/policy.yaml
  files:
    - content: |
        [plugins]
          [plugins."io.containerd.image-verifier.v1.bindir"]
            bin_dir = "/usr/local/bin/containerd-image-verifier"
            max_verifiers = 10
            per_verifier_timeout = "10s"
      path: /etc/cri/conf.d/20-customization.part
      op: create
    - content: |
        apiVersion: policy.sigstore.dev/v1alpha1
        kind: ClusterImagePolicy
        metadata:
          name: system
        spec:
          images:
          - glob: "**"
          authorities:
          - keyless:
              url: https://fulcio.sigstore.dev
              identities:
              - issuer: https://accounts.google.com
                subject: k8s-infra-gcr-promoter@k8s-artifacts-prod.iam.gserviceaccount.com
            ctlog:
              url: https://rekor.sigstore.dev
          - keyless:
              identities:
              - issuer: https://accounts.google.com
                subjectRegExp: "@siderolabs\.com$"
      path: /var/local/etc/containers/sigstore/policy.yaml
      op: create

booting up a VM and generating config like so

talosctl gen secrets

talosctl gen config talossigstore https://192.168.64.5:6443 --config-patch @./patch.yaml --install-disk /dev/vda --with-secrets ./secrets.yaml --force --install-image ghcr.io/bobymcbobs/siderolabs/talos/installer:v1.9.3-with-sigstore-policy-tester
talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig apply-config -i --file ./controlplane.yaml

validating the environment with

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig ls /var/local/etc/containers/sigstore/policy.yaml
NODE           NAME
192.168.64.5   policy.yaml

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig ls /etc/cri/conf.d/20-customization.part
NODE           NAME
192.168.64.5   20-customization.part

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig ls --depth 2 /usr/local/bin/
NODE           NAME
192.168.64.5   .
192.168.64.5   containerd-image-verifier
192.168.64.5   containerd-image-verifier/containerd-image-verifier-sigstore
192.168.64.5   sigstore-policy-tester

so far, I can't see that there's any affect from the config. Diving on debugging

@BobyMCbobs
Copy link
Contributor Author

BobyMCbobs commented Feb 6, 2025

logs should appear like the following: https://github.com/containerd/containerd/blob/main/pkg/imageverifier/bindir/bindir.go#L58

I see nothing relevant under the containerd logs

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig logs containerd

except that the plugin has loaded

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig logs containerd | grep -i image-verifier
192.168.64.5: {"id":"io.containerd.image-verifier.v1.bindir","level":"info","msg":"loading plugin","time":"2025-02-06T04:08:03.899137666Z","type":"io.containerd.image-verifier.v1"}

the containerd config looks as expected

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig cat /etc/cri/conf.d/cri.toml
## /etc/cri/conf.d/00-base.part
## /etc/cri/conf.d/01-registries.part
## /etc/cri/conf.d/20-customization.part

version = 3

[debug]
  level = 'debug'

[plugins]
  [plugins.'io.containerd.cri.v1.images']
    discard_unpacked_layers = true

    [plugins.'io.containerd.cri.v1.images'.registry]
      config_path = '/etc/cri/conf.d/hosts'

      [plugins.'io.containerd.cri.v1.images'.registry.configs]

  [plugins.'io.containerd.cri.v1.runtime']
    [plugins.'io.containerd.cri.v1.runtime'.containerd]
      [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
          base_runtime_spec = '/etc/cri/conf.d/base-spec.json'

  [plugins.'io.containerd.image-verifier.v1.bindir']
    bin_dir = '/usr/local/bin/containerd-image-verifier'
    max_verifiers = 10
    per_verifier_timeout = '10s'

  [plugins.'io.containerd.nri.v1.nri']
    disable = true

the program is executable

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig ls --long /usr/local/bin/containerd-image-verifier/containerd-image-verifier-sigstore
NODE           MODE         UID   GID   SIZE(B)   LASTMOD           LABEL                                              NAME
192.168.64.5   -rwxr-xr-x   0     0     2650580   Feb  6 16:38:45   system_u:object_r:container_file_t:s0:c325,c705   containerd-image-verifier-sigstore

@frezbo
Copy link
Member

frezbo commented Feb 6, 2025

logs should appear like the following: https://github.com/containerd/containerd/blob/main/pkg/imageverifier/bindir/bindir.go#L58

I see nothing relevant under the containerd logs

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig logs containerd

except that the plugin has loaded

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig logs containerd | grep -i image-verifier
192.168.64.5: {"id":"io.containerd.image-verifier.v1.bindir","level":"info","msg":"loading plugin","time":"2025-02-06T04:08:03.899137666Z","type":"io.containerd.image-verifier.v1"}

the containerd config looks as expected

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig cat /etc/cri/conf.d/cri.toml
## /etc/cri/conf.d/00-base.part
## /etc/cri/conf.d/01-registries.part
## /etc/cri/conf.d/20-customization.part

version = 3

[debug]
  level = 'debug'

[plugins]
  [plugins.'io.containerd.cri.v1.images']
    discard_unpacked_layers = true

    [plugins.'io.containerd.cri.v1.images'.registry]
      config_path = '/etc/cri/conf.d/hosts'

      [plugins.'io.containerd.cri.v1.images'.registry.configs]

  [plugins.'io.containerd.cri.v1.runtime']
    [plugins.'io.containerd.cri.v1.runtime'.containerd]
      [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
          base_runtime_spec = '/etc/cri/conf.d/base-spec.json'

  [plugins.'io.containerd.image-verifier.v1.bindir']
    bin_dir = '/usr/local/bin/containerd-image-verifier'
    max_verifiers = 10
    per_verifier_timeout = '10s'

  [plugins.'io.containerd.nri.v1.nri']
    disable = true

the program is executable

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig ls --long /usr/local/bin/containerd-image-verifier/containerd-image-verifier-sigstore
NODE           MODE         UID   GID   SIZE(B)   LASTMOD           LABEL                                              NAME
192.168.64.5   -rwxr-xr-x   0     0     2650580   Feb  6 16:38:45   system_u:object_r:container_file_t:s0:c325,c705   containerd-image-verifier-sigstore

check the cri logs too, containerd is only used to pull etcd and kubelet

@frezbo
Copy link
Member

frezbo commented Feb 6, 2025

machine:
  env:
    CONTAINERD_IMAGE_VERIFIER_SIGSTORE_POLICY_PATH: /var/local/etc/containers/sigstore/policy.yaml

I wonder if we can avoid global env's

@smira
Copy link
Member

smira commented Feb 6, 2025

I don't quite like the name of the extensions, it should probably be around image-verifier naming, or the fact that it's a plugin.

I'm not personally excited about the imager verifier API in containerd (of course it has nothing to do with this change), but I see the need to have such verification.

files:
- content: |
[plugins]
[plugins."io.containerd.image-verifier.v1.bindir"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also would like to check how this plugs in into the image pull process, there are multiple paths in the CRI/Talos itself before we declare this as a production solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants