HelmRepository (OCI) with insecure: true fails to pull from plain-http registry, attempts HTTPS first

[sanxitang]

Hi,

I am using in cluster harbor as the OCI registry to hold the helmchart which will be used by the HelmRepository. But chart failed to download form the plain http registry.

Environment

K8S: 1.30.11

Harbor Version: 2.7.1

Flux Version:

./flux version
flux: v2.6.4
distribution: flux-v2.6.4
helm-controller: v1.3.0
kustomize-controller: v1.6.1
source-controller: v1.6.2

Because the harbor running in cluster so I am using svc FQDN to access harbor OCI registry

The following are the svc of harbor which harbor-core svc using port 80

harbor-core         ClusterIP   10.104.30.61     <none>        80/TCP              5d23h
harbor-database     ClusterIP   10.110.87.163    <none>        5432/TCP            5d23h
harbor-jobservice   ClusterIP   10.111.232.107   <none>        80/TCP              5d23h
harbor-portal       ClusterIP   10.110.128.101   <none>        80/TCP              5d23h
harbor-redis        ClusterIP   10.100.19.135    <none>        6379/TCP            5d23h
harbor-registry     ClusterIP   10.106.89.164    <none>        5000/TCP,8080/TCP   5d23h

I did a test in a pod within the same cluster which use helm command directly login the harbor svc url and pull the chart which work as expected, the following commands were issued in a pod which installed the helm 3.17.3 (From the go.mod of the source-controller v1.6.2, the required helm version is 3.17.3, so I tested with the same version)

bash-4.2# ./helm version 
version.BuildInfo{Version:"v3.17.3", GitCommit:"e4da49785aa6e6ee2b86efd5dd9e43400318262b", GitTreeState:"clean", GoVersion:"go1.23.7"}

bash-4.2# ./helm registry login harbor-core.cdp-system --insecure
Username: admin
Password: 
Login Succeeded

bash-4.2# ./helm pull oci://harbor-core.cdp-system/cdp-foundation/charts/umami --version 4.2.0-3 --plain-http
Pulled: harbor-core.cdp-system/cdp-foundation/charts/umami:4.2.0-3
Digest: sha256:35b6e137280d6df9568869bec4eaff207c87949e66b59aef924371dc14ef20c3

bash-4.2# ls -lah 
-rw-r--r-- 1 root root 156K Nov 17 11:49 umami-4.2.0-3.tgz

The following are the configuration of the helmrelease, and helmrepository

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: umami
  namespace: cdp-foundation
spec:
  dependsOn:
    - name: sealed-secrets
      namespace: cdp-foundation
  interval: 2m0s
  timeout: 10m0s
  install:
    remediation:
      retries: -1
  chart:
    spec:
      chart: umami
      version: 4.2.0-3
      sourceRef:
        kind: HelmRepository
        name: helm-repository
        namespace: cdp-foundation
      interval: 1m0s
  values:
    image:xxxx
    xxxxxxx

HelmRepository

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: helm-repository
  namespace: cdp-foundation
spec:
  interval: 1m0s
  timeout: 1m0s
  type: oci
  insecure: true
  url: oci://harbor-core.cdp-system/cdp-foundation/charts
  secretRef:
    name: helm-harbor-secret

HelmRelease and HelmChart status

kubectl get hr umami -n cdp-foundation 
NAME    AGE   READY   STATUS
umami   60s   False   HelmChart 'cdp-foundation/cdp-foundation-umami' is not ready: latest generation of object has not been reconciled

kubectl get helmchart cdp-foundation-umami  -n cdp-foundation
cdp-foundation-umami                         umami                         4.2.0-3                         HelmRepository   helm-repository   2m32s   False   unknown build error: failed to login to OCI registry: Get "https://harbor-core.cdp-system/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Our testing shows that even with insecure: true, the source-controller still appears to first attempt a connection to the OCI repository URL using HTTPS (port 443). But the harbo-core svc only expose 80 port which lead to the connection timeout

This contrasts directly with the helm --plain-http command, which correctly forces an HTTP-only connection and works as expected. Because the source-controller attempts communication over both protocols, the configuration must allow it to resolve the hostname and access both port 443 and port 80 for it to eventually succeed over HTTP.

There is another issue which the HelmRepository not showing READY and STATUS

NAMESPACE         NAME              URL                                                  AGE        READY   STATUS
cdp-application   helm-repository   oci://harbor-core/cdp-application/charts             6d1h           
cdp-foundation    helm-repository   oci://harbor-core.cdp-system/cdp-foundation/charts   6d1h           
cdp-platform      helm-repository   oci://harbor-core/cdp-platform/charts                6d1h           
cdp-system        helm-repository   oci://harbor-core/cdp-system/charts                  6d1h           
cdp-xaas          helm-repository   oci://haror-core.cdp-system                          6d1h           

[stefanprodan]

Can you try OCIRepository and see if that works? Example here: https://fluxcd.io/flux/components/helm/helmreleases/#recommended-settings

[sanxitang]

@stefanprodan Yse I have tried OCIRepository, which worked with insecure: true setting.

kubectl get ocirepository -n cdp-xaas
NAME            URL                                                    READY   STATUS                                                                                                       AGE
ocirepository   oci://harbor-core.cdp-system/cdp-xaas/charts/adminer   True    stored artifact for digest '1.0.1@sha256:17de9348ed350089965064fa5c2938984d23e1e3cc3bbf5e264afec949caf3d5'   3d23h

kubectl get hr -n cdp-xaas 
NAME      AGE     READY   STATUS
adminer   3d23h   True    Helm install succeeded for release cdp-xaas/adminer.v1 with chart adminer@1.0.1+17de9348ed35

But for us it's hard to shift from helmrepository to ocirepository because of some legacy design.

[matheuscscp]

Is there a previous version of Flux where this was working? (Is this a regression?)

But for us it's hard to shift from helmrepository to ocirepository because of some legacy design.

FYI new OCI-related features will only be added to OCIRepository. We're trying to make users switch from HelmRepository with type: oci to OCIRepository. Please add that to your tech debt backlog 😁

[sanxitang]

Is there a previous version of Flux where this was working? (Is this a regression?)

Currently our system is using flux 2.1.2, in 2.1.2 we have to do a lot of hacks add self-signed cert to the helmrepository to make it work. We want to remove the hacks, then we realized since the version 2.2.0 , HelmRepository start to support the field .spec.insecure.. Then for the new release we plan to upgrade flux to 2.3.0, but in 2.3.0 we got a lot of issues the for helmchart resource keep getting invalid host error then we decide to try the latest version before 2.7.0 which is the 2.6.4 then I got the issues above. So for the previous version never worked

[matheuscscp]

Cool, if it never worked there's no regression 🙌

I suggest migrating to OCIRepository