workload identity to connect to a remote AKS is not working

[bapatvin]

Hello guy, I am trying to setup workload identity to connect to a remote AKS. I have configured a secret that will give me access to the remote cluster, however I am getting the following error:

Get "https://mycluster:443 getting credentials: exec: executable argocd-k8s-auth failed with exit code 20
The workload identity works just fine as I have already integrated it with my External Secret Operator but I am not sure what is going on with the argocd. We also don't believe it is a network related issue as we've tested the communication between argocd main cluster > AKS remote.

I wonder what should put in:

AZURE_FEDERATED_TOKEN_FILE
caData

I can see the secret created in argocd Settings/Clusters yet it shows as failed. Below the secret that I am using:

apiVersion: v1
kind: Secret
metadata:
  name: mycluster
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: mycluster
  server: mycluster.com
  config: |
    {
      "execProviderConfig": {
        "command": "argocd-k8s-auth",
        "env": {
          "AAD_ENVIRONMENT_NAME": "AzurePublicCloud",
          "AZURE_CLIENT_ID": "my client id",
          "AZURE_TENANT_ID": "my tentant id ",
          "AZURE_FEDERATED_TOKEN_FILE": "/var/run/secrets/azure/tokens/azure-identity-token.json",
          "AZURE_AUTHORITY_HOST": "https://login.microsoftonline.com/",
          "AAD_LOGIN_METHOD": "workloadidentity"
        },
        "args": ["azure"],
        "apiVersion": "client.authentication.k8s.io/v1beta1"
      },
      "tlsClientConfig": {
        "insecure": true,
        "caData": ""
      }
    }

i appreciate any help!

[michielvha]

Hello, I was recently struggling with this same topic, I have some usefull insights that I would like to share.

For this setup to work you need to be sure to also add the umi annotations to the service account for argocd-application-controller just like how you did for the argocd-server service account. Afterwards you need to make sure the correct federated credentials are added to the UMI

azure.workload.identity/client-id: xxx
azure.workload.identity/tenant-id: xxx

afterwards also make sure to put the annotation on the server deployment and controller statefulset.

  labels:
    azure.workload.identity/use: "true"

to answer the question asked in this posted, you need to use the CA data from the cluster you are trying to join, you can find this in your kubeconfig file after you have connected with the cluster atleast once.

also the path to the token is probably already mounted in the container as an env var, for me it was at /var/run/secrets/azure/tokens/azure-identity-token

hope this will help people because I spend about a week figuring out that you need to put those annotations on the application controller aswell and not only on the argocd-server deployment and I stumbled on it by accident because I broke something on the controller and went digging through the logs. Don't think it was mentioned anywhere in the docs or maybe I looked over it :)

[michielvha]

@bapatvin so the UMI you use doesn't really matter I would create a seperate one called "argocd-umi" or whatever. You then need to add 2 federated credentials to this umi, one for each service account used in the deployment of argocd server and the statefulset of the application controller. this UMI needs to have "azure kubernetes service cluster admin" role on the cluster that you want to add, else the join will fail. when you have the ID you need to set the annotations on those service accounts, they already exist you just need to add the annotations with client id and tenant id as explained above.

[bapatvin]

@MKTHEPLUGG - That worked. I was also looking into this article for those exact steps you mentioned. Thanks a lot again.

https://github.com/weinong/azure-federated-identity-samples/blob/main/setup.sh

[michielvha]

@bapatvin I'm using terraform to create the resources in azure, I personally prefer this approach over using scripts like the one you mentioned but it indeed seems to be doing what you need :) very welcome and enjoy !

[ishuar]

@bapatvin and @MKTHEPLUGG , could you please confirm if your argocd control plane is also running on AKS cluster ?

I had the same error message when running argocd control plane outside of azure ( local cluster or any other cloud provider ) and want to add AKS clusters as remote cluster on my argocd instance.

Do you have any idea in this case what would be the OIDC issuer URL for federation. I am not sure if its the OIDC issuer URL from remote cluster as the service accouts will not be available on that cluster.

[michielvha]

When the identities are not within the same cloud infrastructure, you need to set up a trust relationship between the external IdP (in this case, the IdP of the control plane's environment) and Azure AD, which is used by AKS.

For your specific question about the OIDC issuer URL in a cross-cloud setup:

1. OIDC Issuer URL from the Remote Cluster: Normally, when federating identities with AKS, you would use Azure AD as the OIDC issuer. This is because AKS uses Azure AD for identity and access management by default.

2. Service Accounts on the Remote Cluster: The service accounts mentioned (like those used by Argo CD components) will indeed not exist on the remote AKS cluster by default. You would need to create equivalent service accounts in the AKS cluster and configure them with the necessary roles and permissions.

3. Cross-Cloud Federation: To establish a federation between an external identity provider and Azure AD, you would need to register the external IdP with Azure AD and configure a federation trust. This involves setting up claims mapping and ensuring that the tokens issued by the external IdP are accepted and trusted by Azure AD.

4. Implementation: This setup might involve configuring Azure AD to trust tokens from the external IdP and using those tokens to assume roles or identities within AKS. This can be complex and may require custom configuration and possibly some development work to handle token exchange and role assumption.

Given these considerations, if your Argo CD control plane is outside of Azure, and you wish to integrate with AKS, you would need to:

• Establish a federation trust between your external IdP and Azure AD.
• Ensure that Azure AD can issue tokens that are trusted by AKS and can be used to authenticate and authorize actions within AKS.
• Configure Argo CD to use these federated tokens when communicating with the AKS clusters.