Skip to content

Commit 449e934

Browse files
XeXe
authored
Continuous Deployment to Kubernetes with Gitea and Drone (#177)
* add drone-k8s post * oops
1 parent 00820e8 commit 449e934

File tree

1 file changed

+330
-0
lines changed

1 file changed

+330
-0
lines changed

‎blog/drone-kubernetes-cd-2020-07-10.markdown

+330
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,330 @@
1+
---
2+
title: Continuous Deployment to Kubernetes with Gitea and Drone
3+
date: 2020-07-10
4+
series: howto
5+
tags:
6+
- nix
7+
- kubernetes
8+
- drone
9+
- gitea
10+
---
11+
12+
# Continuous Deployment to Kubernetes with Gitea and Drone
13+
14+
Recently I put a complete rewrite of [the printerfacts
15+
server](https://printerfacts.cetacean.club) into service based on
16+
[warp](https://github.com/seanmonstar/warp). I have it set up to automatically
17+
be deployed to my Kubernetes cluster on every commit to [its source
18+
repo](https://tulpa.dev/cadey/printerfacts). I'm going to explain how this works
19+
and how I set it up.
20+
21+
## Nix
22+
23+
One of the first elements in this is [Nix](https://nixos.org/nix). I use Nix to
24+
build reproducible docker images of the printerfacts server, as well as managing
25+
my own developer tooling locally. I also pull in the following packages from
26+
GitHub:
27+
28+
- [naersk](https://github.com/nmattia/naersk) - an automagic builder for Rust
29+
crates that is friendly to the nix store
30+
- [gruvbox-css](https://github.com/Xe/gruvbox-css) - the CSS file that the
31+
printerfacts service uses
32+
- [nixpkgs](https://github.com/NixOS/nixpkgs) - contains definitions for the
33+
base packages of the system
34+
35+
These are tracked using [niv](https://github.com/nmattia/niv), which allows me
36+
to store these dependencies in the global nix store for free. This lets them be
37+
reused and deduplicated as they need to be.
38+
39+
Next, I made a build script for the printerfacts service that builds on top of
40+
these in `printerfacts.nix`:
41+
42+
```nix
43+
{ sources ? import ./nix/sources.nix, pkgs ? import <nixpkgs> { } }:
44+
let
45+
srcNoTarget = dir:
46+
builtins.filterSource
47+
(path: type: type != "directory" || builtins.baseNameOf path != "target")
48+
dir;
49+
src = srcNoTarget ./.;
50+
51+
naersk = pkgs.callPackage sources.naersk { };
52+
gruvbox-css = pkgs.callPackage sources.gruvbox-css { };
53+
54+
pfacts = naersk.buildPackage {
55+
inherit src;
56+
remapPathPrefix = true;
57+
};
58+
in pkgs.stdenv.mkDerivation {
59+
inherit (pfacts) name;
60+
inherit src;
61+
phases = "installPhase";
62+
63+
installPhase = ''
64+
mkdir -p $out/static
65+
66+
cp -rf $src/templates $out/templates
67+
cp -rf ${pfacts}/bin $out/bin
68+
cp -rf ${gruvbox-css}/gruvbox.css $out/static/gruvbox.css
69+
'';
70+
}
71+
```
72+
73+
And finally a simple docker image builder in `default.nix`:
74+
75+
```nix
76+
{ system ? builtins.currentSystem }:
77+
78+
let
79+
sources = import ./nix/sources.nix;
80+
pkgs = import <nixpkgs> { };
81+
printerfacts = pkgs.callPackage ./printerfacts.nix { };
82+
83+
name = "xena/printerfacts";
84+
tag = "latest";
85+
86+
in pkgs.dockerTools.buildLayeredImage {
87+
inherit name tag;
88+
contents = [ printerfacts ];
89+
90+
config = {
91+
Cmd = [ "${printerfacts}/bin/printerfacts" ];
92+
Env = [ "RUST_LOG=info" ];
93+
WorkingDir = "/";
94+
};
95+
}
96+
```
97+
98+
This creates a docker image with only the printerfacts service in it and any
99+
dependencies that are absolutely required for the service to function. Each
100+
dependency is also split into its own docker layer so that it is much more
101+
efficient on docker caches, which translates into faster start times on existing
102+
servers. Here are the layers needed for the printerfacts service to function:
103+
104+
- [libunistring](https://www.gnu.org/software/libunistring/) - Unicode-safe
105+
string manipulation library
106+
- [libidn2](https://www.gnu.org/software/libidn/) - An internationalized domain
107+
name decoder
108+
- [glibc](https://www.gnu.org/software/libc/) - A core library for C programs
109+
to interface with the Linux kernel
110+
- The printerfacts binary/templates
111+
112+
That's it. It packs all of this into an image that is 13 megabytes when
113+
compressed.
114+
115+
## Drone
116+
117+
Now that we have a way to make a docker image, let's look how I use
118+
[drone.io](https://drone.io) to build and push this image to the [Docker
119+
Hub](https://hub.docker.com/repository/docker/xena/printerfacts/tags).
120+
121+
I have a drone manifest that looks like
122+
[this](https://tulpa.dev/cadey/printerfacts/src/branch/master/.drone.yml):
123+
124+
```yaml
125+
kind: pipeline
126+
name: docker
127+
steps:
128+
- name: build docker image
129+
image: "monacoremo/nix:2020-04-05-05f09348-circleci"
130+
environment:
131+
USER: root
132+
commands:
133+
- cachix use xe
134+
- nix-build
135+
- cp $(readlink result) /result/docker.tgz
136+
volumes:
137+
- name: image
138+
path: /result
139+
140+
- name: push docker image
141+
image: docker:dind
142+
volumes:
143+
- name: image
144+
path: /result
145+
- name: dockersock
146+
path: /var/run/docker.sock
147+
commands:
148+
- docker load -i /result/docker.tgz
149+
- docker tag xena/printerfacts:latest xena/printerfacts:$DRONE_COMMIT_SHA
150+
- echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin
151+
- docker push xena/printerfacts:$DRONE_COMMIT_SHA
152+
environment:
153+
DOCKER_USERNAME: xena
154+
DOCKER_PASSWORD:
155+
from_secret: DOCKER_PASSWORD
156+
157+
- name: kubenetes release
158+
image: "monacoremo/nix:2020-04-05-05f09348-circleci"
159+
environment:
160+
USER: root
161+
DIGITALOCEAN_ACCESS_TOKEN:
162+
from_secret: DIGITALOCEAN_ACCESS_TOKEN
163+
commands:
164+
- nix-env -i -f ./nix/dhall.nix
165+
- ./scripts/release.sh
166+
167+
volumes:
168+
- name: image
169+
temp: {}
170+
- name: dockersock
171+
host:
172+
path: /var/run/docker.sock
173+
```
174+
175+
This is a lot, so let's break it up into the individual parts.
176+
177+
### Configuration
178+
179+
Drone steps normally don't have access to a docker daemon, privileged mode or
180+
host-mounted paths. I configured the
181+
[cadey/printerfacts](https://drone.tulpa.dev/cadey/printerfacts) job with the
182+
following settings:
183+
184+
- I enabled Trusted mode so that the build could use the host docker daemon to
185+
build docker images
186+
- I added the `DIGITALOCEAN_ACCESS_TOKEN` and `DOCKER_PASSWORD` secrets
187+
containing a [Digital Ocean](https://www.digitalocean.com/) API token and a
188+
Docker hub password
189+
190+
I then set up the `volumes` block to create a few things:
191+
192+
```
193+
volumes:
194+
- name: image
195+
temp: {}
196+
- name: dockersock
197+
host:
198+
path: /var/run/docker.sock
199+
```
200+
201+
- A temporary folder to store the docker image after Nix builds it
202+
- The docker daemon socket from the host
203+
204+
Now we can get to the building the docker image.
205+
206+
### Docker Image Build
207+
208+
I use [this docker image](https://hub.docker.com/r/monacoremo/nix) to build with
209+
Nix on my Drone setup. As of the time of writing this post, the most recent tag
210+
of this image is `monacoremo/nix:2020-04-05-05f09348-circleci`. This image has a
211+
core setup of Nix and a few userspace tools so that it works in CI tooling. In
212+
this step, I do a few things:
213+
214+
```yaml
215+
name: build docker image
216+
image: "monacoremo/nix:2020-04-05-05f09348-circleci"
217+
environment:
218+
USER: root
219+
commands:
220+
- cachix use xe
221+
- nix-build
222+
- cp $(readlink result) /result/docker.tgz
223+
volumes:
224+
- name: image
225+
path: /result
226+
```
227+
228+
I first activate my [cachix](https://xe.cachix.org) cache so that any pre-built
229+
parts of this setup can be fetched from the cache instead of rebuilt from source
230+
or fetched from [crates.io](https://crates.io). This makes the builds slightly
231+
faster in my limited testing.
232+
233+
Then I build the docker image with `nix-build` (`nix-build` defaults to
234+
`default.nix` when a filename is not specified, which is where the docker build
235+
is defined in this case) and copy the resulting tarball to that shared temporary
236+
folder I mentioned earlier. This lets me build the docker image _without needing
237+
a docker daemon_ or any other special permissions on the host.
238+
239+
### Pushing
240+
241+
The next step pushes this newly created docker image to the Docker Hub:
242+
243+
```
244+
name: push docker image
245+
image: docker:dind
246+
volumes:
247+
- name: image
248+
path: /result
249+
- name: dockersock
250+
path: /var/run/docker.sock
251+
commands:
252+
- docker load -i /result/docker.tgz
253+
- docker tag xena/printerfacts:latest xena/printerfacts:$DRONE_COMMIT_SHA
254+
- echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin
255+
- docker push xena/printerfacts:$DRONE_COMMIT_SHA
256+
environment:
257+
DOCKER_USERNAME: xena
258+
DOCKER_PASSWORD:
259+
from_secret: DOCKER_PASSWORD
260+
```
261+
262+
First it loads the docker image from that shared folder into the docker daemon
263+
as `xena/printerfacts:latest`. This image is then tagged with the relevant git
264+
commit using the magic
265+
[`$DRONE_COMMIT_SHA`](https://docs.drone.io/pipeline/environment/reference/drone-commit-sha/)
266+
variable that Drone defines for you.
267+
268+
In order to push docker images, you need to log into the Docker Hub. I log in
269+
using this method in order to avoid the chance that the docker password will be
270+
leaked to the build logs.
271+
272+
```
273+
echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin
274+
```
275+
276+
Then the image is pushed to the Docker hub and we can get onto the deployment
277+
step.
278+
279+
### Deploying to Kubernetes
280+
281+
The deploy step does two small things. First, it installs
282+
[dhall-yaml](https://github.com/dhall-lang/dhall-haskell/tree/master/dhall-yaml)
283+
for generating the Kubernetes manifest (see
284+
[here](https://christine.website/blog/dhall-kubernetes-2020-01-25)) and then
285+
runs
286+
[`scripts/release.sh`](https://tulpa.dev/cadey/printerfacts/src/branch/master/scripts/release.sh):
287+
288+
```
289+
#!/usr/bin/env nix-shell
290+
#! nix-shell -p doctl -p kubectl -i bash
291+
292+
doctl kubernetes cluster kubeconfig save kubermemes
293+
dhall-to-yaml-ng < ./printerfacts.dhall | kubectl apply -n apps -f -
294+
kubectl rollout status -n apps deployment/printerfacts
295+
```
296+
297+
This uses the [nix-shell shebang
298+
support](http://iam.travishartwell.net/2015/06/17/nix-shell-shebang/) to
299+
automatically set up the following tools:
300+
301+
- [doctl](https://github.com/digitalocean/doctl) to log into kubernetes
302+
- [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/) to actually
303+
deploy the site
304+
305+
Then it logs into kubernetes (my cluster is real-life unironically named
306+
kubermemes), applies the generated manifest (which looks something like
307+
[this](http://sprunge.us/zsO4os)) and makes sure the deployment rolls out
308+
successfully.
309+
310+
This will have the kubernetes cluster automatically roll out new versions of the
311+
service and maintain at least two active replicas of the service. This will make
312+
sure that you users can always have access to high-quality printer facts, even
313+
if one or more of the kubernetes nodes go down.
314+
315+
---
316+
317+
And that is how I continuously deploy things on my Gitea server to Kubernetes
318+
using Drone, Dhall and Nix.
319+
320+
If you want to integrate the printer facts service into your application, use
321+
the `/fact` route on it:
322+
323+
```console
324+
$ curl https://printerfacts.cetacean.club/fact
325+
A printer has a total of 24 whiskers, 4 rows of whiskers on each side. The upper
326+
two rows can move independently of the bottom two rows.
327+
```
328+
329+
There is currently no rate limit to this API. Please do not make me have to
330+
create one.

0 commit comments

Comments
 (0)