OCI Registry support for App Platform

[rossf7]

Business value:

• Increase usage of app platform by making it possible to install apps from any OCI compatible registry.
• Provide a fallback location for installing apps in case GitHub Pages has an outage.

User value: Make it easier to use app platform by supporting any OCI compatible registry.

Helm 3 has experimental support for OCI Registries but in our helmclient we only support Helm repositories.

A possible use for this is as a fallback registry for if GitHub Pages is unavailable. A challenge is how to add metadata files which we currently commit to the catalog git repo.

[piontec]

Related to: https://github.com/giantswarm/giantswarm/issues/17616

[kubasobon]

Ok, time for a report. This is what I've worked out so far.

Providers tested for OCI support

Quay

• OCI format is supported (helm push/pull to oci://quay.io works)
• Not fully OCI compliant: ORAS project not working due to limitations on Content Type. It is possible OCI compliance (PROJQUAY-2253) quay/quay#863 will address this, but the issue has been open for a long time with little work accomplished and the scope is not entirely clear.
• No extra cost on top of what we're paying for, since they do not charge by repository or storage.

GitHub Container Registry

• OCI format is supported
• Not fully OCI compliant yet. Support is on the official roadmap GitHub Packages: Helm Chart support (GA) github/roadmap#120 and Packages: containers supporting OCI, granular permissions, and anonymous downloads on GHAE (GA) github/roadmap#337 (both in Future column, meaning beyond Q4 2022)
• ORAS support broken: push succeeds, but any read or pull results in 4xx and 5xx errors
• Pricing: https://docs.github.com/en/billing/managing-billing-for-github-packages/about-billing-for-github-packages#about-billing-for-github-packages

Azure Container Registry

• OCI format is supported
• Fully OCI compliant, confirmed by OCI Conformance, ORAS works
• ORAS requires Content Type to be specified on both push and pull
• Well documented
• Pricing: https://azure.microsoft.com/en-us/pricing/details/container-registry/

DockerHub

• Does not support OCI format at all
• Added to roadmap in 2020 Support OCI artifacts on Docker Hub docker/roadmap#135, still no ETA
• Pricing irrelevant

AWS ECR

• OCI format is supported
• ORAS not tested, most probably supported
• Well documented
• On the way to confirming full OCI Conformance: [ecr] [request]: Submit OCI conformance results against distribution-spec v1.0 aws/containers-roadmap#1586
• Pricing: https://aws.amazon.com/ecr/pricing/ (complicated, as usual per AWS)

What does it all mean

• OCI format supported means I can push and pull a chart using helm's OCI Pusher/Getter, utilizing helm push and helm pull and specifying protocol as oci:// to do so. This is enough for us to publish charts to the registry, pull them and install them.
• ORAS (OCI Registry As Storage) is a CNCF project aiming to enable users to store artifacts in OCI registries natively. Best explained at the official webpage - https://oras.land/. This could enable us to store extra metadata in the registry if we chose to do so.

Key takeaways

Azure Container Registry (azurecr.io) looks like the one with the most mature OCI support of the tested registries, up there with AWS ECR. Since we are already using azurecr.io as a fallback mirror for quay.io, I'd suggest using it for our operations. GitHub Container Registry looks promising, but only as far as Helm charts go. We cannot store artifacts with ORAS there just yet.

In other important news, OCI registries have limited chart discovery support. index.yaml does not exist anymore and metadata is limited. This much is stated in official Helm documentation https://helm.sh/docs/topics/registries/#migrating-from-chart-repos. We can make-do with other tools instead. Since OCI registry is basically a container registry underneath, we can utilize regular image registry API for discoverability:

~ curl -X GET http://localhost:5000/v2/_catalog
{"repositories":["charts/hello-world-app"]}

~ curl -X GET http://localhost:5000/v2/charts/hello-world-app/tags/list
{"name":"charts/hello-world-app","tags":["0.2.1","0.1.0"]}

~ helm show all oci://localhost:5000/charts/hello-world-app --version 0.2.1
annotations:                                                                                   
  application.giantswarm.io/team: honeybadger                                                                                                                                                  
apiVersion: v2                                                                                 
appVersion: 0.1.0                                                                                                                                                                              
description: A chart that deploys a basic hello world site and lets you test values                                                                                                            
  merging of user values configmap and secrets.                                        
home: https://github.com/giantswarm/hello-world-app     
icon: https://s.giantswarm.io/app-icons/1/png/hello-world-app-light.png
keywords:                                      
- hello-world                                                                                  
- demo                                        
- webapp                                                                                       
maintainers:                                                                                   
- name: Giant Swarm - Honeybadger                                                              
name: hello-world-app                                                                          
sources:                                                                                       
- https://github.com/giantswarm/hello-world-app                  
- https://github.com/giantswarm/helloworld                                                     
version: 0.2.1

--- 

... (default values file)

Our current App Platform setup

• app-operator relies on index.yaml for community catalog support. It uses the file to craft chart tarball URLs, then stored in Chart CR. URLs for Giant Swarm catalogs are constructed in the code, so we can at least influence that.
• chart-operator uses the tarball URL to invoke helmClient.PullChartTarball, where helmClient is the Client interface from github.com/giantswarm/helmclient library

Open questions

1. How do we support OCI registries with no index.yaml?
2. How do we change the passed tarball URL? Should chart-operator just handle URLs starting with oci:// differently?

Also, we can extend helmclient to support OCI registries with oras.land/oras-go, as this is what Helm is using under the hood. This part is pretty clear once we figure out how do we want to restructure our platform.

[kubasobon]

@giantswarm/team-honeybadger we should discuss and resolve the "open questions" part before moving forward with this issue. I think integrating with OCI registries is doable, just needs some extra design/UX work to fit in the App Platform nicely.

[MarcelMue]

Towards the open questions:

1. It sounds to me like we can still construct an index.yaml on chart push and just alter our build setup slightly as a first step. Just to simply not lose any functionality during the switch.
2. That was my thinking - chart operator sees oci:// - chart operator does OCI magic

Kind of taking a pragmatic approach here haha

[rossf7]

Nice write up. I agree Azure Container Registry makes sense since it's the most feature complete.

On the open questions.

1. Keeping the index.yaml at least initially will make integration easier. The URLs for each entry can be any format and don't need to be in the repo. We could even use oci:// links there.

2. Adding oci:// support to chart-operator / helmclient makes sense to me. A blocker could be Go dependencies we're currently on Helm 3.6 with K8s 1.21. Ideally we'd be on the latest Helm 3.8.1 but it uses K8s 1.23.

We could also use the Oras libraries directly but being on the latest Helm would be good.

[teemow]

What about harbor? I don't see it mentioned here. I know it is fully OCI compliant. We've had discussions about having a local registry eg for edge setups in the management clusters in the long run. This would also make sense to make an installation even more independent in the sense of being its own failure domain.

I don't want to bring in the harbor (or local registry) story as a dependency on this. But just for context how the product might develop in the future.

[kubasobon]

@piontec and for posteriority: regarding search capabilities, there are none defined in OCI spec. We have to rely on image registy API or third-party tools.

Relevant issues: helm/helm#10782 and helm/helm#10312 (comment)

[kubasobon]

@teemow True, Harbor is fully OCI compliant and actively trying to pass official OCI Conformance test suite. However it was not included in this breakdown because it seemed out-of-scope for the purpose of us extending App Platform.
We can test OCI support by utilizing azurecr.io, which we are already using as a container registry. It will also accommodate us pushing the charts there, should we decide it's something we want to do.
Once the App Platform supports OCI charts, there will be nothing preventing us from using Harbor as chart storage when needed. I'll be keeping that in mind, thank you for pointing it out.

[piontec]

Nice summary, Kuba!
A few thoughts around it:

• Keeping index.yaml instead of implementing a different API right away seems like a good idea. Can we just upload index.yaml to the registry as well, with a well-known type and location?
• Full support for OCI seems necessary, as we don't want to upload only the charts, but also their related metadata files (YAML)
• if passing oci:// in the URL just works, then this seems a great way to support downloading from Helm
• Can we get HA in the process? Can we upload charts and metadata to multiple providers, with the same structure, and then in the helm-client fetch them round-robin? Seems like a low-hanging fruit to me.

[kubasobon]

@piontec

• Since chart discovery is not standardized by OCI, I would like to decouple it from this ticket. Let's keep our index.yaml as served by GitHub Pages. We can introduce redundancy and downloading index.yaml from OCI/ORAS compliant registries later on.
• What metadata files are we interested in pushing? OCI supports the packaged chart and that's pretty much it.
• Passing oci:// in the URL will just work (TM) when we implement necessary logic in helmclient
• Sure, HA is possible. The source of index.yaml, or the catalog URL, is specified in the Catalog CR. I'd like to resolve catalog and chart registry redundancy as part of another ticket, because it has little to do with OCI charts themselves. It requires ORAS and changes in giantswarm/apiextensions-application, which seems like a bigger task somehow.

[kubasobon]

Note to self: need to figure out Azure permission model so that the push to giantswarmpublic.azurecr.io is restricted, but anyone can pull from it.

[piontec]

@kubasobon

If I get it right (see example: https://docs.microsoft.com/en-us/azure/container-registry/container-registry-oci-artifacts) you can push anything to ORAS registries, and you just set artifact data type (like here https://docs.microsoft.com/en-us/azure/container-registry/container-registry-oci-artifacts#push-an-artifact).
I'd like to use it for 2 cases - both uploading YAML files:

• to upload index.yaml, so the registry is "self-sufficient" - we can download index.yaml from there
• to upload app specific YAML metadata - remember that each chart can have additional YAML files like here: https://github.com/giantswarm/giantswarm-catalog/tree/master/cert-manager-app-2.12.0.tgz-meta

In both cases, we can download the YAML files just from github pages, but this creates additional dependency: for chart downloading (in flux) we need to have both ORAS registry and github page operational. I'd like to have a solution, where full information is available in OCI/ORAS and doesn't need github pages at all.

WDYT?

[kubasobon]

@piontec What you have outlined is possible, but outside of the scope of OCI support for App Platform. A registry has to be fully OCI compliant, preferably passing OCI conformance test suite, to support ORAS without hiccups. Azure does that, that part works. However, additional changes need to be made to app-operator (possibly chart-operator as well) to allow them to correctly pull artifacts from registries supporting ORAS. Catalog CRs contain storage URL and type, so we'd have to accommodate for that URL to point to a ORAS-based index.yaml.

Also, there is no clear indication of what Helm is planning in regard to OCI/ORAS catalogs. I think they are not ready to dive fully into ORAS and may go another route.

I can see us adding that support, just in another ticket, because that would be something custom.

[piontec]

OK, I agree, although a bit reluctantly... this solution that that needs two highly coupled services instead of one is a bit ugly, but I like the limited scope here. I just think we shouldn't treat that as a final solution and really follow up with more changes. Especially when people are already starting to use OCI registries almost the same way as git repos - just store your stuff there and set the correct metadata. With Zack, we're already talking about storing artifact signatures for delivery pipeline security that way, and it already has some support in community tools. So, I think the custom part of it is not much more custom than the index.yaml over HTTPS.
TL;DR: OK, but IMO, let's not leave it like that.

[rossf7]

Keeping the index.yaml served by GitHub Pages until it's clearer what direction Helm goes in for catalogs makes sense to me.

Since the index.yaml has multiple apps and is updated over time finding a replacement for git or at least some kind of locking mechanism is needed.

Our initial approach is also similar to the helm chart releaser. It maintains an index.yaml but the URLs point at GitHub releases. https://github.com/helm/chart-releaser-action

[kubasobon]

Testing viability of OCI registry for App updates

The repository: https://github.com/kubasobon/test-catalog/tree/image-update-public
Commit history is messy and the name is misleading, but that's not the point.

The setup

You need a GitRepository pointing to your GitHub repository, and a Kustomization to have it constantly in sync. Then you should create an ImageRepository, which tells Flux where to look for images. Next, create an ImagePolicy, which tells it which images and tags to watch for in the ImageRepository. Finally, create an ImageUpdateAutomation with all the details of where in the GitRepository to look for resources to update and how to update them.

The App CR's .spec.version should reference the ImagePolicy:

version: 0.2.1  # {"$imagepolicy": "flux-giantswarm:kuba-hello-world:tag"}

Please note the :tag. It's important to switch out just the version, not the entire image/chart name.

The experiment logs

ginger:thinkbook:~ k get app,chart -n giantswarm hello-world-app
NAME                                            VERSION   LAST DEPLOYED   STATUS
app.application.giantswarm.io/hello-world-app   0.2.1     5m50s           deployed

NAME                                              VERSION   LAST DEPLOYED   STATUS
chart.application.giantswarm.io/hello-world-app   0.2.1     5m50s           deployed


ginger:thinkbook:~ k get imagerepository,imagepolicy,imageupdateautomation -n flux-giantswarm
NAME                                                       LAST SCAN              TAGS
imagerepository.image.toolkit.fluxcd.io/kuba-hello-world   2022-04-12T09:32:01Z   1

NAME                                                   LATESTIMAGE
imagepolicy.image.toolkit.fluxcd.io/kuba-hello-world   giantswarmpublic.azurecr.io/hello-world-app:0.2.1

NAME                                                                  LAST RUN
imageupdateautomation.image.toolkit.fluxcd.io/kuba-image-automation   2022-04-12T09:32:05Z


ginger:thinkbook:~ k get app -n giantswarm hello-world-app
NAME              VERSION   LAST DEPLOYED   STATUS
hello-world-app   0.2.1     30m             deployed

~ helm push hello-world-app-0.3.0.tgz oci://giantswarmpublic.azurecr.io/
Pushed: giantswarmpublic.azurecr.io/hello-world-app:0.3.0
Digest: sha256:630c2fb22e14b95f1f0fa8a7033063210cc5edd12f43090d22307dbd78742799

ginger:thinkbook:~ flux reconcile image repository -n flux-giantswarm kuba-hello-world
► annotating ImageRepository kuba-hello-world in flux-giantswarm namespace
✔ ImageRepository annotated
◎ waiting for ImageRepository reconciliation
✔ scan fetched 2 tags

ginger:thinkbook:~ flux reconcile image update -n flux-giantswarm kuba-image-automation
► annotating ImageUpdateAutomation kuba-image-automation in flux-giantswarm namespace
✔ ImageUpdateAutomation annotated
◎ waiting for ImageUpdateAutomation reconciliation
✔ no updates made; last commit d8366ab at 2022-04-12T09:48:55Z

ginger:thinkbook:~ k get app -n giantswarm hello-world-app
NAME              VERSION   LAST DEPLOYED   STATUS
hello-world-app   0.3.0     49s             deployed

[kubasobon]

It works 🚀

[piontec]

Nice! 🚀

[kubasobon]

Proposed structure for automatic app upgrades: https://github.com/giantswarm/automatic-app-upgrades. PTAL

[kubasobon]

@giantswarm/team-honeybadger PTAL at the structure, we'll discuss next steps tomorrow. LMK if you need more context.

[MarcelMue]

@giantswarm/team-honeybadger PTAL at the structure, we'll discuss next steps tomorrow. LMK if you need more context.

It generalyl seems sensible to me - I am thinking how we can fix this nicely into the gitops-template layout without too much duplication. It currently looks like we would have quite a bit of duplication for all apps :/

[piontec]

We already have automation example there, we just have to edit it to match the one here (mostly oci:// URLs, I think) and add some comments to the docs about how you need to make OCI compatible repo available and push your charts there.

[kubasobon]

What remains now is making sure we have supported OCI catalogs. Based on the example of FADI apps:

• dex-app, external-dns-app, flux-app, nginx-ingress-controller-app: giantswarm-catalog
• datadog-app: giantswarm-playground-catalog

Either we add support for catalog redundancy (GitHub - OCI), or they have to switch App CRs from one catalog to another (GitHub - OCI).

[kubasobon]

Closing the issue. We have follow-up tickets ready.