Failed to autoscale when there were events due to missing permissions

[justinabrahms]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.10.1

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

I'm unclear. I think it was working for a while then stopped?

Describe the bug

I expected my cluster to have runners. It did not. Upon accessing the logs, I found spew related to insufficient permissions.

Describe the expected behavior

It should continue to yield runners.

Additional Context

The relevant portion of my helmfile.

  - name: github-actions-controller
    installed: {{ eq .Environment.Name "platform-tools" }}
    labels:
      layer: github
    namespace: arc-systems
    createNamespace: true
    chart: oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller
    version: 0.10.1

  - name: gha-k8s # decides what the runs-on: value is for github actions
    installed: {{ eq .Environment.Name "platform-tools" }}
    needs:
      - github-actions-controller
    labels:
      layer: github
    namespace: arc-systems
    createNamespace: true
    chart: oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set
    version: 0.10.1
    values:
      - githubConfigUrl: https://github.com/REDACTED
      - githubConfigSecret: {{ exec "aws" (list "secretsmanager" "get-secret-value" "--secret-id" "REDACTED" "--query" "SecretString" "--output" "text" "--profile" "REDACTED") }}



Service account:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    meta.helm.sh/release-name: github-actions-controller
    meta.helm.sh/release-namespace: arc-systems
  creationTimestamp: "2025-01-08T23:51:10Z"
  labels:
    app.kubernetes.io/instance: github-actions-controller
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: gha-rs-controller
    app.kubernetes.io/namespace: arc-systems
    app.kubernetes.io/part-of: gha-rs-controller
    app.kubernetes.io/version: 0.10.1
    helm.sh/chart: gha-rs-controller-0.10.1
  name: github-actions-controller-gha-rs-controller
  namespace: arc-systems
  resourceVersion: "8518253"
  uid: 87460dfd-1236-4a50-b448-85d61cd1a02f


Rolebindings


apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
  kind: RoleBinding
  metadata:
    creationTimestamp: "2025-01-09T18:07:32Z"
    labels:
      actions.github.com/organization: ThriveMarket
      actions.github.com/scale-set-name: gha-k8s
      actions.github.com/scale-set-namespace: arc-systems
      app.kubernetes.io/component: runner-scale-set-listener
      app.kubernetes.io/instance: gha-k8s
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: gha-k8s
      app.kubernetes.io/part-of: gha-runner-scale-set
      app.kubernetes.io/version: 0.10.1
      auto-scaling-listener-name: gha-k8s-6cd58d58-listener
      auto-scaling-listener-namespace: arc-systems
      helm.sh/chart: gha-rs-0.10.1
      role-binding-role-ref-hash: 78b5dc5754
      role-binding-subject-hash: 85f9d6dcc7
    name: gha-k8s-6cd58d58-listener
    namespace: arc-systems
    resourceVersion: "8986540"
    uid: 88b7fe17-4fea-4ccd-8755-d093474d2e3a
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: Role
    name: gha-k8s-6cd58d58-listener
  subjects:
  - kind: ServiceAccount
    name: gha-k8s-6cd58d58-listener
    namespace: arc-systems
- apiVersion: rbac.authorization.k8s.io/v1
  kind: RoleBinding
  metadata:
    annotations:
      meta.helm.sh/release-name: gha-k8s
      meta.helm.sh/release-namespace: arc-systems
    creationTimestamp: "2025-01-08T23:51:33Z"
    finalizers:
    - actions.github.com/cleanup-protection
    labels:
      actions.github.com/scale-set-name: gha-k8s
      actions.github.com/scale-set-namespace: arc-systems
      app.kubernetes.io/component: manager-role-binding
      app.kubernetes.io/instance: gha-k8s
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: gha-k8s
      app.kubernetes.io/part-of: gha-rs
      app.kubernetes.io/version: 0.10.1
      helm.sh/chart: gha-rs-0.10.1
    name: gha-k8s-gha-rs-manager
    namespace: arc-systems
    resourceVersion: "21534312"
    uid: 5431aa25-d7bd-44d1-a4b1-404735427618
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: Role
    name: gha-k8s-gha-rs-manager
  subjects:
  - kind: ServiceAccount
    name: gha-rs-controller
    namespace: arc-systems
- apiVersion: rbac.authorization.k8s.io/v1
  kind: RoleBinding
  metadata:
    annotations:
      meta.helm.sh/release-name: github-actions-controller
      meta.helm.sh/release-namespace: arc-systems
    creationTimestamp: "2025-01-08T23:51:10Z"
    labels:
      app.kubernetes.io/managed-by: Helm
    name: github-actions-controller-gha-rs-controller-listener
    namespace: arc-systems
    resourceVersion: "8518258"
    uid: dd10536a-9362-4f5a-a8d8-fcb1d3c75deb
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: Role
    name: github-actions-controller-gha-rs-controller-listener
  subjects:
  - kind: ServiceAccount
    name: github-actions-controller-gha-rs-controller
    namespace: arc-systems
kind: List
metadata:
  resourceVersion: ""


Roles

apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    creationTimestamp: "2025-01-09T18:07:32Z"
    labels:
      actions.github.com/organization: ThriveMarket
      actions.github.com/scale-set-name: gha-k8s
      actions.github.com/scale-set-namespace: arc-systems
      app.kubernetes.io/component: runner-scale-set-listener
      app.kubernetes.io/instance: gha-k8s
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: gha-k8s
      app.kubernetes.io/part-of: gha-runner-scale-set
      app.kubernetes.io/version: 0.10.1
      auto-scaling-listener-name: gha-k8s-6cd58d58-listener
      auto-scaling-listener-namespace: arc-systems
      helm.sh/chart: gha-rs-0.10.1
      role-policy-rules-hash: 7cd9c55b7f
    name: gha-k8s-6cd58d58-listener
    namespace: arc-systems
    resourceVersion: "8986539"
    uid: 96bd480f-7beb-4b10-83b4-9aa66e9f3a93
  rules:
  - apiGroups:
    - actions.github.com
    resourceNames:
    - gha-k8s-ptktm
    resources:
    - ephemeralrunnersets
    verbs:
    - patch
  - apiGroups:
    - actions.github.com
    resources:
    - ephemeralrunners
    - ephemeralrunners/status
    verbs:
    - patch
- apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    annotations:
      meta.helm.sh/release-name: gha-k8s
      meta.helm.sh/release-namespace: arc-systems
    creationTimestamp: "2025-01-08T23:51:33Z"
    finalizers:
    - actions.github.com/cleanup-protection
    labels:
      actions.github.com/scale-set-name: gha-k8s
      actions.github.com/scale-set-namespace: arc-systems
      app.kubernetes.io/component: manager-role
      app.kubernetes.io/instance: gha-k8s
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: gha-k8s
      app.kubernetes.io/part-of: gha-rs
      app.kubernetes.io/version: 0.10.1
      helm.sh/chart: gha-rs-0.10.1
    name: gha-k8s-gha-rs-manager
    namespace: arc-systems
    resourceVersion: "8518453"
    uid: 608655aa-f1e3-48e2-9d39-8f192b6afa50
  rules:
  - apiGroups:
    - ""
    resources:
    - pods
    verbs:
    - create
    - delete
    - get
  - apiGroups:
    - ""
    resources:
    - pods/status
    verbs:
    - get
  - apiGroups:
    - ""
    resources:
    - secrets
    verbs:
    - create
    - delete
    - get
    - list
    - patch
    - update
  - apiGroups:
    - ""
    resources:
    - serviceaccounts
    verbs:
    - create
    - delete
    - get
    - list
    - patch
    - update
  - apiGroups:
    - rbac.authorization.k8s.io
    resources:
    - rolebindings
    verbs:
    - create
    - delete
    - get
    - patch
    - update
  - apiGroups:
    - rbac.authorization.k8s.io
    resources:
    - roles
    verbs:
    - create
    - delete
    - get
    - patch
    - update
- apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    annotations:
      meta.helm.sh/release-name: github-actions-controller
      meta.helm.sh/release-namespace: arc-systems
    creationTimestamp: "2025-01-08T23:51:10Z"
    labels:
      app.kubernetes.io/managed-by: Helm
    name: github-actions-controller-gha-rs-controller-listener
    namespace: arc-systems
    resourceVersion: "8518257"
    uid: 9574cbde-f1c1-4557-b274-aaf4b48c5b3b
  rules:
  - apiGroups:
    - ""
    resources:
    - pods
    verbs:
    - create
    - delete
    - get
  - apiGroups:
    - ""
    resources:
    - pods/status
    verbs:
    - get
  - apiGroups:
    - ""
    resources:
    - secrets
    verbs:
    - create
    - delete
    - get
    - patch
    - update
  - apiGroups:
    - ""
    resources:
    - serviceaccounts
    verbs:
    - create
    - delete
    - get
    - patch
    - update
kind: List
metadata:
  resourceVersion: ""

Controller Logs

https://gist.github.com/justinabrahms/aee16f9f0b129b3bb3eba29b94921936

Runner Pod Logs

n/a

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[justinabrahms]

Manually deleting those resources and “upgrading” (with the same version) the helm chart for both things corrected the issue, as best I can tell.Typed with thumbs.On Jan 27, 2025, at 8:42 PM, github-actions[bot] ***@***.***> wrote: Hello! Thank you for filing an issue. The maintainers will triage your issue shortly. In the meantime, please take a look at the troubleshooting guide for bug reports. If this is a feature request, please review our contribution guidelines. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>