Build distroless package for better security, smaller size, speed and more

[hpvd]

As proofed in practice (e.g. see our own example #20253 (comment) )
there are sometimes (often!) security problems in a container/package whose origin is not the software one build, but in the software which is also situated in this container.

In most cases, there is no (little) use case for this additional software.
This is where the idea of distroless containers comes in and "free" your software:

1. for better security
2. fewer bugs
3. smaller packages
4. a faster build process
5. a faster check process (e.g. security scans for CVEs and CWEs)
6. faster, cheaper and less annoying development process, because of less noise to understand and fix
7. faster spin-up / faster dynamic scaling on load
8. less demanding for needed infrastructure = less cost for infrastructure to run on
9. ...

Traditional, this approach is somehow strenuous to implement and associated with restrictions.

But it looks like 2 new tools makes it pretty easy and straight forward:

good overview on distroless containers
https://dev.to/dansiviter/distroless-alpine-ci8
and
https://blog.chainguard.dev/minimal-container-images-towards-a-more-secure-future/
see last paragraph for how it works

the tools:
source to abk:
https://github.com/chainguard-dev/melange

abk to oci:
https://github.com/chainguard-dev/apko

to debug distroless containers:
official: https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/
detail flow: https://iximiuz.com/en/posts/kubernetes-ephemeral-containers/

(this idea/issue was created as follow-up to ongoing progress in distroless for functionmesh see streamnative/function-mesh#448)

[hpvd]

as @michaeljmarshall already said

Note that using a distroless base image is not a viable option for pulsar 2.x because we use shell scripts to configure each component before executing the java command and changing that configuration paradigm would be a breaking change.

see #11269 (comment)

=> maybe this is a high value long term topic for Pulsar 4.0

[michaeljmarshall]

=> maybe this is a high value long term topic for Pulsar 4.0

Based on our current versioning scheme, this kind of change could be introduced in a minor release, like 3.1.

[michaeljmarshall]

Also, pulsar's configuration could be ready for an overhaul. There is an ongoing discussion about configuration here https://lists.apache.org/thread/ok6hnvwsfcckj50471tkfbbhtgo6ng35

[hpvd]

for example regarding security, see sources of vulnerabilities in pulsars helm chart.
Even an update to latest version of everything would not solve the main problem, that included distros always add a huge part of vulnerabilities
=> included distros simply add more code, with more chances for vulnerabilities to be introduced.

see
https://artifacthub.io/packages/helm/apache/pulsar?modal=security-report&section=vulnerabilities

[dave2wave]

About CVEs in a dependency. There can be many types of false positives.

1. The use of the package may already workaround the vulnerability. This happened a lot in projects like POI where XMLBeans was worked around until it was taken out of the Attic and fixed. In the meantime those who did not need Entity Expansion could turn it off.
2. The vulnerable part of the dependency may not be used.

[dave2wave]

We should be discussing this with the report for 2.10.4 release since many CVEs have been fixed. I think that this would give us a good basis for comparison.

[hpvd]

sure you are right about false positives!

..but even if the absolute numbers are lower e.g. with a new release of pulsar and an included distro,
the mechanism keeps always alive:
the more code within a package, the more chances for vulnerabilities (plus the influence on the other 7 points noted in #20253 (comment) )

And if the absolute number of vulnerabilities is low on release day, it will always be higher next week...

[hpvd]

hmm maybe distroless is not the only suitable approach to pay-in on goals named above for distroless
coming from related #20095 (comment)

Example from the other side:
Kafka broker (and Zookeeper) compiled to native using Quarkus and GraalVM
https://github.com/ozangunalp/kafka-native

[michaeljmarshall]

This would be a great feature to add!

[hpvd]

thanks for positive feedback and adding hints for possibilities and classification!
-> clicking the up vote button may help to gain further visibility for this topic :-)

[hpvd]

maybe 2 videos from a third party for this topic would help...

Which Base Container Images Should We Use?
Why Debian, Ubuntu etc is a terrible choice...
https://www.youtube.com/watch?v=82ZCJw9poxM

How to Debug Kubernetes Applications With Ephemeral Containers (the right way)
https://www.youtube.com/watch?v=qKb6loAEPV0