Shorter deploy commands

[ramontayag]

Feature request

• If the maintainers agree with the feature as described here, I intend to submit a Pull Request myself.¹

Proposal: provide shorter deploy commands

Currently, a deploy command looks like:

KUBECONFIG=~/.kube/config ENVIRONMENT=staging REVISION=958567c80b1d288255e77df0fa9bd381dec784ab kubernetes-deploy mynamespace gke_company-project-name_us-central1-a_anothername

This is quite long. There are certain things that probably don't change between deploys. These things can be potentially removed using direnv or similar tools, but there are other things that differ from deploy to deploy.

I propose a command reminiscent of the Capistrano command:

# deploy the latest revision of master branch to a set default (most likely production):
kubernetes-deploy master

# deploy the latest revision of staging_06 branch to a set default (most likely staging)
kubernetes-deploy staging_06

# deploy the latest revision of master branch to staging (override the default)
kubernetes-deploy master staging

# deploy revision 958567c80b1d288255e77df0fa9bd381dec784ab to staging
kubernetes-deploy 958567c80b1d288255e77df0fa9bd381dec784ab staging

# yes, you can override others too
kubernetes-deploy master staging

Below you'll find ways I think we can achieve this, parameter at a time.

KUBECONFIG

This will likely never change per machine, but may be different machine to machine. A solution to this would be to use direnv or a similar format where we have a .envrc.local that is listed in .gitignore that either a) exports the environment variable export KUBECONFIG=/path/to/kubeconfig/on/my/machine. If .envrc.local does not exist, then kubernetes-deploy can require KUBECONFIG to bet set like it is today.

ENVIRONMENT

When running kubernetes-deploy master staging, it's obvious what ENVIRONMENT will be. However, it should have a default depending on the branch that is being deployed.

In config/deploy/config.yml, something like this can be specified:

deploys:
  - branch: master
    environment: production
  - branch: /staging_(\d+)/
    environment: staging

We would need to recognize regex

kubernetes-deploy would be smart enough to figure out that if a branch name is being passed in and no environment is specified, it will default to what is listed in the config.yml. If that file does not exist, then yes it will have to require ENVIRONMENT or the second argument in kubernetes-deploy master staging.

This config is something that will be committed to the project, and will be the same for all developers. If the developer need to customize this, then the setting either does not belong to the config file or there should be a way to override it.

Namespace

There are two solutions I've thought of that would eliminate the need to specify the namespace each deploy:

1. Templates found in config/#{environment}/* can specify the namespace (this isn't the case today though, perhaps because of template reuse?)
2. Add it to the config:

deploys:
  - branch: master
    environment: production
    namespace: prod
  - branch: /staging_(\d+)/
    environment: staging
    namespace: stg

Cluster name

The cluster name may be the differ between environments, but I could not think of a reason it would differ per developer. Therefore, I think this can be saved in the config as well:

deploys:
  - branch: master
    environment: production
    namespace: default
    context: gke_company-project-name_us-central1-prod
  - branch: /staging_(\d+)/
    environment: staging
    namespace: default
    context: gke_company-project-name_us-central1-stg

Closing

There should be a way to still keep the old format and the new format working. If you'd rather not overload kubernetes-deploy that's perfectly understandable. This can be in another command like kdeploy, which underneath calls kubernetes-deploy. In fact, maybe this can be done in a separate gem that uses kubernetes-deploy underneath (what do you think?). 🤔

[dturn]

Hi @ramontayag, thanks for writing this up. We've internally discussed adding a config file and this seems like a good usecase. We have an open issue to make kubernetes-deploy a single command with sub commands, #256. I think this would be great to support after we tackle #256.

A couple of thoughts on the details:

• I think we should be explicit about the revision being deployed: kubernetes-deploy deploy master REVISION. (If you were brave you could use latest but its discouraged for many reasons). Otherwise if someone merges to master before a container has finished building your deploy is going to fail.
• KUBECONFIG: The common thing to do is add export KUBECONFIG=~/.kube/config in ~/.zshrc file (though you'll have to obviously fix it to work with your preferred shell flavor). We could support different ones via the config file if that were valuable.

[ramontayag]

@dturn, thanks for the reply.

• kubernetes-deploy deploy master would deploy the latest revision of local master, so if something was merged in the repo, it would not affect the developer deploying until they updated their local branch. Does this answer that issue?
• KUBECONFIG: you're right: no need for extra work here. direnv does the same as putting that in the ~/.zshrc or .bash_profile file.

[benlangfeld]

I'm not sure I understand why KUBECONFIG=~/.kube/config is necessary. This is the default path to the file; why can kubernetes-deploy not use that default?

[ramontayag]

Maybe my previous comment wasn't clear? I think kubernetes-deploy should use that by default.

[benlangfeld]

@ramontayag af3e236 makes setting KUBECONFIG optional.

[ramontayag]

Hi, everyone. I finally was able to look into the changes that came with the 1.0 release of Krane. Thank you for all the hard work!

Have any of your opinions changed related to this issue? I use erb templates a lot, and the command has gotten longer:

krane render \
  --current-sha=$(git rev-parse --verify staging_12) \
  --filenames=config/deploy/staging | \
  krane deploy br-staging $CLUSTER --stdin \
  --filenames=config/deploy/staging/secrets.ejson

Translating what I wrote above in the proposed commands to krane, it would require deploy (or some other command) to both render, read secrets, and deploy:

krane deploy staging_12 staging-namespace

Perhaps using a different command will make it less confusing:

krane lift staging_12 staging-namespace

[dturn]

One of the things we've learned over the last few years is that coupling rendering and deploying is a dangerous anti-pattern. We very intentionally removed that ability in 1.0 and we're not willing to create a command that does both.

We're still open to the idea of a config file, but we don't have the time to work on it right now, so it will have to be driven by someone outside of Shopify. That being said it seems like a bash script that wraps your invocations of krane might do most of what you're proposing here.

[ramontayag]

Yes, something could be built that just uses krane underneath. Mind pointing me to any discussions where those learnings are laid out?

[dturn]

We haven't written much up about this (yet). We do talk a bit about it in our kubecon talk: https://youtu.be/5ACPxn49HlA?t=1337

[ramontayag]

I'll take a look at this. What I was looking for wasn't to merge the two, but rather a command that calls render and then deploy, which essentially expands to krane render ... | krane deploy. Will close this. Thanks!

[yfxie]

Late to this one, but in case anyone else lands here looking for the same thing: I built a small gem that does the wrapper approach discussed above. It's called kran. Context, namespace and kubeconfig live in config/kran.yml, -d staging layers a second file on top, and kran deploy builds and pushes the image, then runs krane render | krane deploy with the git SHA as the tag. It doesn't change anything in krane, it only calls it.

https://github.com/yfxie/kran