Container Actions in K8S ARC

[drew-viles]

I'm just looking for clarification here if possible.

I'm running ARC in Kubernetes with a custom image - cuttiong out the fluff, here is the vaules passed to the helm chart

githubConfigSecret: github-runner-creds
githubConfigUrl: https://github.com/MY_ORG
maxRunners: 10
minRunners: 2
runnerGroup: k8s-runners
runnerScaleSetName: arc-{{ .Values.clusterName }}-runners
controllerServiceAccount:
  namespace: github-runner
  name: github-runner-controller-gha-rs-controller
template:
  spec:
    serviceAccount: arc-runners
    serviceAccountName: arc-runners
    securityContext:
      fsGroup: 123
    imagePullSecrets:
      - name: ghcr
    volumes:
      - name: work
        ephemeral:
          volumeClaimTemplate:
            spec:
              accessModes: [ "ReadWriteOnce" ]
              storageClassName: "{{ .Values.storageClassName }}"
              resources:
                requests:
                  storage: 100Gi
      - name: dind-externals
        emptyDir: {}
    imagePullSecrets:
      - name: ghcr
    initContainers:
      - name: init-dind-externals
        image: ghcr.io/actions/actions-runner:latest
        command: ["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]
        volumeMounts:
          - name: dind-externals
            mountPath: /home/runner/tmpDir
    containers:
      - name: runner
        image: ghcr.io/MY_ORG/gh-arc:v0.3.6-beta1
        imagePullPolicy: Always
        env:
          - name: ACTIONS_RUNNER_CONTAINER_HOOKS
            value: /home/runner/k8s/index.js
          - name: ACTIONS_RUNNER_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
            value: "false"
        command: [ "entrypoint.sh" ]
        securityContext:
          privileged: true # we need dind-rootless so we don't have to use this!
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work

I've spent time getting Docker installed and running as a daemon whilst also installing a bunch of other tools, packages etc in the image used to enable an easy to use, feature rich, self-hosted GitHub runner.

I've pretty much covered everything now and the majority of the developers and engineers can use the runners for what they need. However, there is that small percentage that are experiencing the odd issue and it's down to one thing.

Docker based actions.

Any composite or nodejs based actions work flawlessly including anything that builds and pushes Docker images and even runs Docker containers.

However, when running any action that is Docker based, I get these errors (I've included the action that's failing for an example)

Run bpicode/github-action-fpm@master
Run '/home/runner/k8s/index.js'
  
Error: Error: Building container actions is not currently supported
Error: Process completed with exit code 1.
Error: Executing the custom container implementation failed. Please contact your self hosted runner administrator.

Now, is this simply a limitation of ARC, the hook, a bit of both or some missing configuration on my side?

The suggestion is that it's currently not supported and that's the end of it. But then in the logs for the runner pod itself I'm seeing:

ERR  StepsRunner]  ---> System.Exception: The hook script at '/home/runner/k8s/index.js' running command 'RunContainerStep' did not execute successfully

There isn't much more useful information on top of that in the stack trace that I can see - happy to share if needed, but this suggests maybe because I'm targetting the k8s hook, it will not complete correctly. The roblem is I can't use the Docker one becuase this is a Kubernetes cluster whic, obviously, doesn't have a Docker underlying CRI.

I've been through every issue that contains similar errors to this and tried all the permissions etc that are suggested but I cannot get this to work. I expect it's because it's trying to run a Docker container, inside the pod and maybe GitHub can't track that?

I'm not sure why it wouldn't do becuase the theory would be the same as if I ran it in a VM due to the nature of my setup running Docker as a daemon inside the pod - I'm not mounting the socket as you can see. I mean, yeah there will be some differences to a physical or VM setup with Docker and the runner installed but surely not enough that would prevent this working?
Maybe I'm wrong (it wouldn't be the first time 😆 )!

Any guidence or assistance would be greatly appreciated - and if I've missed an obvious page in the docs saying "DON'T DO THIS" I apologise in advanced - I've gone a bit docs blind over the pas couple days!