Sealed Secrets

[sylus]

The only real feedback I wanted to have a discussion about is the SealedSecrets functionality.

https://github.com/bitnami-labs/sealed-secrets

Encrypts your Secret into a SealedSecret, which is safe to store - even to a public repository. The SealedSecret can be decrypted only by the controller running in the target cluster and nobody else (not even the original author) is able to obtain the original Secret from the SealedSecret.

The only way to unencrypt the secret is via the master key:

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >master.key
kubectl apply -f master.key

How the sealed secret gets created in the first place is via kubeseal on the client:

kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)" --dry-run=client -o yaml | kubeseal | yq eval -P > ${DISTRIBUTION_PATH}/metallb/secret.yaml

I like the ease of this workflow but i'm still thinking we might not want the secrets in the repo or just are parameter replaced environment variables.

However i'd like some opinions on this first on whether we can do this and won't run afoul @zachomedia.

[sylus]

@blairdrummond @zachomedia

[sylus]

I think this also prevents sharing of the specific generated branch since anyone trying to run it will need the controllers secret to properly decrypt rather then just invoking their own secrets.

Would be nice if argo can read from a central key store and then param replace them on the fly.

[davidspek]

This will indeed prevent sharing of deployment branches. If you have a reason to want to export the Sealed Secret controller master key, you might not want to use sealed secrets. In terms of security I think the system is well designed and doesn't pose a risk anymore than another solution. For situations where you don't have access to Vault or something like AWS SSM, I think Sealed Secrets are the only option currently.

However, you depending on the environment there is of course the option to share the master key, and this key itself can be a sealed secret as well. https://github.com/bitnami-labs/sealed-secrets#manual-key-management-advanced

There are advanced use cases that you can address by creative management of the sealing keys. For example, you can share the same sealing key among a few clusters so that you can apply exactly the same sealed secret in multiple clusters. Since sealing keys are just normal k8s secrets you can even use sealed secrets itself and use a GitOps workflow to manage your sealing keys (useful when you want to share the same key among different clusters)!

For the AWS repo, we are using External Secrets almost everywhere, which also has support for Vault. I think this would be the natural alternative to look at.

Another option is the Argo CD Vault Plugin. I know very little about this and only recently came across it. It seems like it allows you to submit regular secret manifests with an added annotation. The plugin then does a replace within Argo CD while deploying the manifest. It's a cool approach, but it might be a bit more fragile.

[zachomedia]

@sylus Thank you for all the amazing work on getting this working in our setup! For the secrets, I think it is best that we don't maintain secrets within the repository itself, whether encrypted or not. I don't have specific controls off the top of my head, but I am positive that it would be contrary to some ITSG-33 controls. I like the concept of it, but because we want the repository to be open and shared I don't think it'll work.

@davidspek Thank you for the detailed write up, and in particular for linking to some useful tools.

---

My understanding @sylus is that this is intended to be used for both local development and deployment into the actual AAW clusters, correct?

If so, I think we need to think about some form of dynamic secret management. I like to idea of looking at something that connects with Hashicorp Vault since we're using it already. For local deployments, we could then deploy Vault and generate some of the necessary secrets. And for dev/prod deployments we could leverage the existing Vault instance.

[sylus]

@davidspek I think we would like to proceed with the https://github.com/IBM/argocd-vault-plugin route so for Kind we would also deploy a vault dev instance, then maybe have a bash script to populate the values.

This would then make it easy for our developers to run their own instance but against a common spec in feat-kind-release without worrying about secrets.

What do you think about this?

[davidspek]

@sylus I believe the setup for the Argo CD plugin and External Secrets will be mostly the same. In both cases you will need a Vault instance setup and some variables to connect to it.

[sylus]

Thanks @davidspek I'll try to take a look at this tonight :D

This is the only thing I think we need then once that is done I can go full on the Azure integration with Kind all done :D