[BUG] repeated restarts #2222
Description
Expected behavior and actual behavior:
We deployed Harbor via Helm on our GKE cluster (w ArgoCD) but constantly getting those errors
ERROR: unauthorized: unauthorized to access repository: app-admin/app, action: push: unauthorized to access repository: app-admin/app, action: push
Along with restarts of the "harbor-core" pod.
Here is a common scenario:
`23 pushing app.run/app-admin/app:latest with docker
23 pushing layer c6b6f2839143
....
23 pushing layer e4dc8cd9ecc8 24.8s done
23 pushing layer 6428cc293366 24.8s done
23 pushing layer 2f7436e79a0b 24.8s done
23 ERROR: unauthorized: unauthorized to access repository: app-admin/app, action: push: unauthorized to access repository: app-admin/app, action: push
pushing app.run/app-admin/app:latest with docker:
ERROR: unauthorized: unauthorized to access repository: app-admin/app, action: push: unauthorized to access repository: app-admin/app, action: push
Reference
Check build summary support
Error: buildx failed with: ERROR: unauthorized: unauthorized to access repository: app-admin/app, action: push: unauthorized to access repository: app-admin/app, action: push`
Steps to reproduce the problem:
Deploying Harbor with the attached helm chart values, chart version 1.16.2 from https://helm.goharbor.io
Versions:
Please specify the versions of following systems.
- harbor version: [2.12.2]
- docker engine version: [not used; containerd 1.7.27]
- docker-compose version: [not applicable]
- gke node version: v1.32.6-gke.1013000
Additional context:
We think the problem is with secret value rotation that changes the auth/CSRF token, of the following secret:
`❯ ku get secret
NAME TYPE DATA AGE
harbor-core Opaque 5 161d
harbor-jobservice Opaque 2 161d
harbor-registry Opaque 2 161d
harbor-registry-htpasswd Opaque 1 51d
harbor-registryctl Opaque 0 161d
harbor-trivy Opaque 2 161d
company-harbor-admin-password Opaque 1 161d
company-harbor-db-password Opaque 1 161d
company-harbor-secret-key Opaque 1 160d
company-app-tls kubernetes.io/tls 2 84d
❯ ku get secret harbor-core -o yaml
apiVersion: v1
data:
CSRF_KEY: UWN...Vo=
REGISTRY_CREDENTIAL_PASSWORD: aGF...3Jk
secret: UG...rRw==
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZ...tLS0K
tls.key: LS0tLS1CRUdJTiBSU0EgUFJ...LS0tLQo=
kind: Secret
metadata:
annotations:
argocd.argoproj.io/tracking-id: harbor:/Secret:harbor/harbor-core
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"CSRF_KEY":"UWN...Vo=","REGISTRY_CREDENTIAL_PASSWORD":"aGF...3Jk","secret":"UG...rRw==","tls.crt":"LS0tLS1CR...LS0K","tls.key":"LS0tLS1CRU...LQo="},"kind":"Secret","metadata":{"annotations":{"argocd.argoproj.io/tracking-id":"harbor:/Secret:harbor/harbor-core"},"labels":{"app":"harbor","app.kubernetes.io/instance":"harbor","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"harbor","app.kubernetes.io/part-of":"harbor","app.kubernetes.io/version":"2.12.2","chart":"harbor","heritage":"Helm","release":"harbor"},"name":"harbor-core","namespace":"harbor"},"type":"Opaque"}
creationTimestamp: "2025-02-18T11:27:48Z"
labels:
app: harbor
app.kubernetes.io/instance: harbor
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: harbor
app.kubernetes.io/part-of: harbor
app.kubernetes.io/version: 2.12.2
chart: harbor
heritage: Helm
release: harbor
name: harbor-core
namespace: harbor
resourceVersion: "1753786651979999002"
uid: 7a21d948-766c-4f85-aabc-8a8514cf4fe4
type: Opaque`
When comparing the fields of the last-applied-configuration to the actual secrets "tls.crt" & "tls.key" were different.
You can also see the huge resourceVersion field values (1753786651979999002)
- Harbor config files:
This is the values file we use
- Log files:
Attached
We are trying to fix this issue for a long time and would appriciate every input and help.