|
| 1 | +# Readme for the deprecated v0.8.0 of kubernetes-on-arm |
| 2 | + |
| 3 | +## Download and build a SD Card |
| 4 | + |
| 5 | +The first thing you will do, is to create a SD Card for your Pi. Alternatively, you may use the [`.deb` deployment](#deb-deployment) |
| 6 | + |
| 7 | +Supported OSes/boards: |
| 8 | +- Arch Linux ARM **(archlinux)** |
| 9 | + - Raspberry Pi 1 A, A+, B, B+, (ZERO,) armv6 **(rpi)** |
| 10 | + - Raspberry Pi 2 Model B, armv7 **(rpi-2)** |
| 11 | + - Raspberry Pi 3 Model B, armv8, _armv7 rootfs_ **(rpi-3)** |
| 12 | + - Parallella armv7, [read more](docs/parallella-status.md) **(parallella)** |
| 13 | + - Cubietruck, armv7 **(cubietruck)** |
| 14 | + - Banana Pro, armv7 **(bananapro)** |
| 15 | +- HypriotOS **(hypriotos)** |
| 16 | + - Raspberry Pi 1 A, A+, B, B+, armv6 **(rpi)** |
| 17 | + - Raspberry Pi 2 Model B, armv7 **(rpi-2)** |
| 18 | + - Raspberry Pi 3 Model B, armv8, _armv7 rootfs_ **(rpi-3)** |
| 19 | + |
| 20 | + |
| 21 | +```bash |
| 22 | +# Go to our home folder, if you want |
| 23 | +cd ~ |
| 24 | + |
| 25 | +# Install git if needed and download this project |
| 26 | +# sudo apt-get install git |
| 27 | +git clone https://github.com/luxas/kubernetes-on-arm |
| 28 | + |
| 29 | +# Change to that directory |
| 30 | +cd kubernetes-on-arm |
| 31 | + |
| 32 | +# See which letter your newly inserted SD Card has: |
| 33 | +sudo fdisk -l |
| 34 | + |
| 35 | +# Another great command |
| 36 | +lsblk |
| 37 | + |
| 38 | +# Get some help text about supported options |
| 39 | +sdcard/write.sh |
| 40 | + |
| 41 | +# Template: |
| 42 | +sudo sdcard/write.sh /dev/sdX [board] [os] [rootfs] |
| 43 | + |
| 44 | +# Example: Write the SD Card for Raspberry Pi 2, Arch Linux ARM and include this project's Kubernetes scripts |
| 45 | +sudo sdcard/write.sh /dev/sdX rpi-2 archlinux docker-multinode |
| 46 | + |
| 47 | +# The installer will ask you if you want to erase all data on your card |
| 48 | +# Answer y/n on that question |
| 49 | +# Prepend the command with QUIET=1 if no security check should be made |
| 50 | +# Requires an internet connection |
| 51 | +# This script runs in 3-4 mins |
| 52 | +``` |
| 53 | + |
| 54 | +The README for the `docker-multinode` rootfs [is here](sdcard/rootfs/docker-multinode/etc/kubernetes/README.md) |
| 55 | + |
| 56 | +## Setup your board from an SD Card |
| 57 | + |
| 58 | +Boot your board and log into it. |
| 59 | + |
| 60 | +Arch Linux users: |
| 61 | + - The user/password is: **root/root** or **alarm/alarm** |
| 62 | + - These scripts requires root. So if you login via **alarm**, then `su root` when you´re going to do some serious hacking :) |
| 63 | + |
| 64 | +HypriotOS users: |
| 65 | + - The user/password is: **pirate/hypriot** |
| 66 | + - Remember to prepend all commands with `sudo` if you are the `pirate` user |
| 67 | + |
| 68 | + |
| 69 | +Yes, I know. Root enabled via ssh isn´t that good. |
| 70 | +But the task to enhance ssh security is left as an exercise to the user. |
| 71 | + |
| 72 | +```bash |
| 73 | +# This script will install and setup docker etc. |
| 74 | +kube-config install |
| 75 | + |
| 76 | +# First, it will install docker, if not present |
| 77 | +# Then it will download kube deploy |
| 78 | + |
| 79 | +# It will ask for which hostname you want. Defaults to kubepi. |
| 80 | + |
| 81 | +# The script will ask you for timezone. Defaults to Europe/Helsinki |
| 82 | +# Run "timedatectl list-timezones" before to check for values |
| 83 | + |
| 84 | +# It will ask you if it should create a 1 GB swapfile. |
| 85 | + |
| 86 | +# Last question is whether you want to reboot |
| 87 | +# You have to reboot in order to get the cgroups working |
| 88 | + |
| 89 | +# If you want to run this script non-interactively, provide the user input beforehand: |
| 90 | +# "\n" is the delimiter, "\" the escape character, presssing enter can be simulated with "\n" |
| 91 | +# Example: |
| 92 | +# /bin/echo -e "rpi-3\nhypriotos\nnodename\nEurope\/Berlin\n\n\ny\nY" | sudo kube-config install |
| 93 | +# This script runs in 2-3 mins |
| 94 | +``` |
| 95 | + |
| 96 | +## Start Kubernetes! |
| 97 | + |
| 98 | +Hmm, starting a complex system like Kubernetes should be a complex task, right? |
| 99 | +Well, not this time. |
| 100 | + |
| 101 | +`enable-master` runs [master.sh](https://github.com/kubernetes/kube-deploy/blob/master/docker-multinode/master.sh) |
| 102 | +`enable-worker` runs [worker.sh](https://github.com/kubernetes/kube-deploy/blob/master/docker-multinode/worker.sh) |
| 103 | + |
| 104 | +```bash |
| 105 | +# To set up your board as both a master and a node, run |
| 106 | +kube-config enable-master |
| 107 | + |
| 108 | +# To set up your board as a node, run |
| 109 | +kube-config enable-worker [master-ip] |
| 110 | +``` |
| 111 | + |
| 112 | +## Package deployment |
| 113 | +If you have already made a SD Card and your device is up and running, what can you do instead? |
| 114 | +For that, I've made a `.deb` package, so you could install it easily |
| 115 | + |
| 116 | +The README for the `docker-multinode` rootfs [is here](sdcard/rootfs/docker-multinode/etc/kubernetes/README.md) |
| 117 | + |
| 118 | +If you already have set up your Pi with latest Raspbian OS for example, follow this guide. |
| 119 | + |
| 120 | +#### Install the `.deb` package |
| 121 | + |
| 122 | +Supported operating systems are HypriotOS, Raspbian, Arch Linux ARM and in some cases Debian/Ubuntu. |
| 123 | + |
| 124 | +```bash |
| 125 | +# Download the latest package |
| 126 | +curl -sSL https://github.com/luxas/kubernetes-on-arm/releases/download/v0.8.0/docker-multinode.deb > docker-multinode.deb |
| 127 | +# or |
| 128 | +wget https://github.com/luxas/kubernetes-on-arm/releases/download/v0.8.0/docker-multinode.deb |
| 129 | + |
| 130 | +# Requires dpkg, which is preinstalled in at least all Debian/Ubuntu OSes |
| 131 | +sudo dpkg -i docker-multinode.deb |
| 132 | + |
| 133 | +# Setup the enviroinment |
| 134 | +# It will ask which board it's running on and which OS |
| 135 | +# A reboot is required for it to function properly, but not for HypriotOS |
| 136 | +kube-config install |
| 137 | + |
| 138 | +## ----- REBOOT ----- |
| 139 | + |
| 140 | +# Start the master or worker |
| 141 | +kube-config enable-master |
| 142 | +kube-config enable-worker [master ip] |
| 143 | + |
| 144 | +# Get some info about the node |
| 145 | +kube-config info |
| 146 | +``` |
| 147 | + |
| 148 | + |
| 149 | +## Use Kubernetes (the fun part begins here) |
| 150 | + |
| 151 | +Some notes on running Docker on ARM are [here](docs/docker-on-arm.md) |
| 152 | + |
| 153 | +```bash |
| 154 | +# See which commands kubectl and kube-config has |
| 155 | +kubectl |
| 156 | +kube-config |
| 157 | + |
| 158 | +# Get info about your machine and Kubernetes version |
| 159 | +kube-config info |
| 160 | + |
| 161 | +# Make an replication controller with an image |
| 162 | +# Hopefully you will have some workers, so you is able to see how they spread across hosts |
| 163 | +# The nginx-test image will be downloaded from Docker Hub and is a nginx server which only is serving the message: "<p>WELCOME TO NGINX</p>" |
| 164 | +# Expose the replication controller "my-nginx" as a service |
| 165 | +kubectl run my-nginx --image=luxas/nginx-test --replicas=3 --expose --port=80 |
| 166 | + |
| 167 | +# The above command will make a deployment since v1.2. The deployment will run a replica set (the "new" kind of replication contoller) |
| 168 | +# You may also specify --hostport, e.g. --hostport=30001 for exposing the service on all nodes' port 30001 |
| 169 | + |
| 170 | +# The pull might take some minutes |
| 171 | +# See when the nginx container is running |
| 172 | +docker ps |
| 173 | + |
| 174 | +# See which pods are running |
| 175 | +kubectl get pods |
| 176 | + |
| 177 | +# See which nodes we have |
| 178 | +kubectl get nodes |
| 179 | + |
| 180 | +# See which ip we may ping, by getting services |
| 181 | +kubectl get svc |
| 182 | + |
| 183 | +# See if the nginx container is working |
| 184 | +# Replace $SERVICE_IP with the ip "kubectl get svc" returned |
| 185 | +curl $SERVICE_IP |
| 186 | +# --> <p>WELCOME TO NGINX</p> |
| 187 | + |
| 188 | +# See which internal cluster services that are running |
| 189 | +kubectl --namespace=kube-system get pods,rc,svc |
| 190 | + |
| 191 | +# Test dns |
| 192 | +curl my-nginx.default.svc.cluster.local |
| 193 | +# --> <p>WELCOME TO NGINX</p> |
| 194 | + |
| 195 | +# By default, "search [domains]" is added to /etc/resolv.conf |
| 196 | +# In this case, these domains are searched: "default.svc.cluster.local svc.cluster.local cluster.local" |
| 197 | +# That means, that you may only write "my-nginx", and it will search in those domains |
| 198 | +curl my-nginx |
| 199 | +# --> <p>WELCOME TO NGINX</p> |
| 200 | + |
| 201 | +# Start the registry |
| 202 | +kube-config enable-addon registry |
| 203 | + |
| 204 | +# Wait a minute for it to start |
| 205 | +kubectl --namespace=kube-system get pods |
| 206 | + |
| 207 | +# Tag an image |
| 208 | +docker tag my-name/my-image localhost:5000/my-name/my-image |
| 209 | + |
| 210 | +# And push it to the registry |
| 211 | +docker push localhost:5000/my-name/my-image |
| 212 | + |
| 213 | +# On another node, pull it |
| 214 | +docker pull localhost:5000/my-name/my-image |
| 215 | + |
| 216 | +# The master also proxies the services so that they are accessible from outside |
| 217 | +# The -L flag is there because curl has to follow redirects |
| 218 | +# You may also type this URL in a web browser |
| 219 | +curl -L http://[master-ip]:8080/api/v1/proxy/namespaces/default/services/my-nginx |
| 220 | + |
| 221 | +# Generic apiserver proxy URL |
| 222 | +# curl -L http://[master-ip]:8080/api/v1/proxy/namespaces/[namespace]/services/[service-name]:[port-name] |
| 223 | + |
| 224 | +# See which ports are open |
| 225 | +netstat -nlp |
| 226 | + |
| 227 | +# See cluster info |
| 228 | +kubectl cluster-info |
| 229 | + |
| 230 | +# See master health in a web browser |
| 231 | +# cAdvisor in kubelet provides a web site that outputs all kind of stats in real time |
| 232 | +# http://$MASTER_IP:4194 |
| 233 | + |
| 234 | +# Turndown Kubernetes on this node. This always reverts the "kube-config enable-*" commands |
| 235 | +kube-config disable |
| 236 | +``` |
| 237 | + |
| 238 | +## Addons |
| 239 | + |
| 240 | +To enable/disable addons is very easy: `kube-config enable-addon [addon-name]` and `kube-config disable-addon [addon-name]` |
| 241 | +[README for the addons](addons/README.md) |
| 242 | + |
| 243 | +Three addons are available for the moment: |
| 244 | + - Kubernetes DNS: |
| 245 | + - Every service gets the hostname: `{{my-svc}}.{{my-namespace}}.svc.cluster.local` |
| 246 | + - The default namespace is `default` (surprise), so unless you manually edit something it will land there |
| 247 | + - Kubernetes internal addon services runs in the namespace `kube-system` |
| 248 | + - Example: `my-awesome-webserver.default.svc.cluster.local` or just `my-awesome-webserver` may resolve to ip `10.0.0.154` |
| 249 | + - Those DNS names is available both in containers and on the node itself (kube-config automatically adds the info to `/etc/resolv.conf`) |
| 250 | + - If you want to access the Kubernetes API easily, `curl -k https://kubernetes` or `curl -k https://10.0.0.1` if you remember numbers better (`-k` stands for insecure as apiserver has no signed certs by default) |
| 251 | + - The DNS server itself has allocated ip `10.0.0.10` |
| 252 | + - The DNS domain is `cluster.local` |
| 253 | + - This addon can't be disabled. |
| 254 | + - Kubernetes Dashboard: |
| 255 | + - The Kubernetes Dashboard project [is here](https://github.com/kubernetes/dashboard) |
| 256 | + - Access the dashboard on: `http://[master-ip]:8080/ui` |
| 257 | + - This addon can't be disabled. |
| 258 | + - Central image registry: |
| 259 | + - A registry for storing cluster images if e.g. the cluster has no internet connection for a while |
| 260 | + - Or for cluster-specific images that one not want to publish on Docker Hub |
| 261 | + - This service is available at `localhost:5000` on all nodes, which by default is a "trusted" location. |
| 262 | + - `localhost:5000` forwards the traffic to the internal IP of the registry service. |
| 263 | + - Just tag your image: `docker tag my-name/my-image localhost:5000/my-name/my-image` |
| 264 | + - And push it to the registry: `docker push localhost:5000/my-name/my-image` |
| 265 | + - Service loadbalancer: |
| 266 | + - Documentation [here](https://github.com/kubernetes/contrib/tree/master/service-loadbalancer) |
| 267 | + - You have to label at least one node `role=loadbalancer` like this: `kubectl label no [node_ip] role=loadbalancer` |
| 268 | + - The loadbalancer will expose http services in the default namespace on `http://[loadbalancer_ip]/[service_name]`. Only `http` services on port 80 are tested in this release. It should be pretty easy to add `https` support though. |
| 269 | + - You may see `haproxy` stats on `http://[loadbalancer_ip]:1936` |
| 270 | + - Not recommended for heavy use. Will be replaced with ingress in coming releases. |
| 271 | + - Cluster monitoring with heapster, influxdb and grafana |
| 272 | + - When this addon is enabled, the dashboard will show usage graphs in the CPU and RAM columns. |
| 273 | + - All heapster data is stored in an InfluxDB database. Data is written once a minute. Access the graphical InfluxDB UI: `http://[master-ip]:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb:http` and the raw api on: `http://[master-ip]:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb:api` |
| 274 | + - A nice `grafana` web dashboard that shows resource usage for the whole cluster as for individual pods is accessible at: `http://[master-ip]:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana`. It may take some minutes for data to show up. |
| 275 | + |
| 276 | +## Access your cluster |
| 277 | + |
| 278 | +Here is some ways to make your outside devices reach the services running in the cluster. |
| 279 | + |
| 280 | + - `apiserver` proxy: |
| 281 | + - This is enabled by default by apiserver |
| 282 | + - Type this URL in a browser or use `curl` |
| 283 | + - `curl -L http://[master-ip]:8080/api/v1/proxy/namespaces/[namespace]/services/[service-name][:[port-name]]` |
| 284 | + - You may build a proxy in front of this with `nginx` that forwards all requests to the apiserver proxy |
| 285 | + - Connect a computer to the `flannel` network |
| 286 | + - It's possible to start `flannel` and `kube-proxy` on another computer **in the same network** and access all services |
| 287 | + - Run these two commands from a `amd64` machine with docker: |
| 288 | + - `docker run --net=host -d --privileged -v /dev/net:/dev/net quay.io/coreos/flannel:0.6.1-amd64 /opt/bin/flanneld --etcd-endpoints=http://$MASTER_IP:2379` |
| 289 | + - `docker run --net=host -d --privileged gcr.io/google_containers/hyperkube-amd64:v1.3.6 /hyperkube proxy --master=http://$MASTER_IP:8080 --v=2`' |
| 290 | + - Replace $MASTER_IP with the actual ip of your master node |
| 291 | + - The consuming `amd64` computer can access all services |
| 292 | + - For example: `curl -k https://10.0.0.1` |
| 293 | + - Make a `service` with `externalIP` |
| 294 | + - Via `kubectl`: `kubectl expose rc {some rc} --port={the port this service will listen on} --container-port={the port the container exposes} --external-ip={the host you want to listen on}` |
| 295 | + - Example: `kubectl expose rc my-nginx --port=9060 --container-port=80 --external-ip=192.168.200.100` |
| 296 | + - This will make the service accessible at `192.168.200.100:9060` |
| 297 | + - Service `NodePort` |
| 298 | + - If one sets Service `.spec.type` to `NodePort`, Kubernetes automatically exposes the service on a random port on every node |
| 299 | + |
| 300 | +#### See node health via `cAdvisor` |
| 301 | + |
| 302 | +Go to a web browser and type: `{IP of your node}:4194` and a nice dashboard will be there and show you some nice real-time stats. |
| 303 | + |
| 304 | +## Configuration |
| 305 | + |
| 306 | +There is a configuration file: `/etc/kubernetes/k8s.conf`, where you can customize some things: |
| 307 | + - `K8S_MASTER_IP`: Points to the master in the cluster. If the node is master, it uses `127.0.0.1` (aka `localhost`). Default: `127.0.0.1` |
| 308 | + - The other options comes from [docker-multinode](https://github.com/kubernetes/kube-deploy/tree/master/docker-multinode#optionsconfiguration) |
| 309 | + |
| 310 | +**Note:** You must change the values in `k8s.conf` before starting Kubernetes. Otherwise they won't have effect, just be able to harm your setup. |
| 311 | + |
| 312 | +On Arch Linux, this file will override the default `eth0` settings. If you have a special `eth0` setup (or use some other network), edit this file to fit your use case: `/etc/systemd/network/dns.network` |
| 313 | + |
| 314 | +## Docker versions |
| 315 | + |
| 316 | +Only `docker-1.10` and higher is supported, `docker-1.11` is recommended. |
| 317 | + |
| 318 | +## Troubleshooting |
| 319 | + |
| 320 | +If your cluster won't start, try `kube-config disable` and choose to remove `/var/lib/kubelet`. That will remove all data you store in `/var/lib/kubelet` and kill most running docker images. |
| 321 | + |
| 322 | +## Reboots |
| 323 | + |
| 324 | +Will **not** work in this version. It's in the roadmap to enable reboots again. |
| 325 | + |
| 326 | +## Contributing |
| 327 | + |
| 328 | +I would be really glad to review your Pull Request! One thing would be good to remember though. I develop on the `dev` branch, so it would be great if you target that one instead of `master` |
| 329 | + |
| 330 | +Thanks! |
| 331 | + |
| 332 | +## Beta version |
| 333 | + |
| 334 | +This project is under development. |
| 335 | +I develop things on the [`dev` branch](../../tree/dev) |
| 336 | + |
| 337 | +[Changelog](CHANGELOG.md) |
| 338 | + |
| 339 | +## Future work |
| 340 | + |
| 341 | +See the [ROADMAP](ROADMAP.md) |
| 342 | + |
| 343 | +## License |
| 344 | + |
| 345 | +[MIT License](LICENSE) |
| 346 | + |
| 347 | +## Goals for this project |
| 348 | + |
| 349 | +This project ports [Kubernetes](https://github.com/kubernetes/kubernetes) to the ARM architecture. |
| 350 | +The primary boards used for testing is Raspberry Pi 2´s. |
| 351 | + |
| 352 | +My goal for this project is that it should be as small as possible, while retaining its flexibility. |
| 353 | +It should also be as easy as possible for people, who don´t know anything about Kubernetes, to get started. |
| 354 | + |
| 355 | +I also have opened a proposal for Kubernetes on ARM: [kubernetes/kubernetes#17981](https://github.com/kubernetes/kubernetes/issues/17981). |
| 356 | +The long-term goal most of this functionality should be present in core Kubernetes. |
| 357 | + |
| 358 | +It should be easy in the future to add support for new boards and operating systems. |
| 359 | + |
| 360 | +#### Feel free to create an issue if you find a bug or think that something should be added or removed! |
0 commit comments