052-k8s-intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Kubernetes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "maybe() {\n",
    "    \"$@\" > .last_maybe 2>&1 || true\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Kubernetes\n",
    "\n",
    "- Docker provides individual containers on a local machine\n",
    "- Kubernetes manages collections of running containers across a cluster/datacenter\n",
    "- also provides networking, storage, monitoring, service discovery"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The Cluster\n",
    "\n",
    "A cluster with 6 CPU nodes and 8 GPU nodes (running on Google GCE)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                                         STATUS   ROLES    AGE   VERSION\n",
      "gke-tmb-cluster-default-pool-bbd84174-14xz   Ready    <none>   47m   v1.13.11-gke.14\n",
      "gke-tmb-cluster-default-pool-bbd84174-7dhh   Ready    <none>   47m   v1.13.11-gke.14\n",
      "gke-tmb-cluster-default-pool-bbd84174-9rq8   Ready    <none>   47m   v1.13.11-gke.14\n",
      "gke-tmb-cluster-default-pool-bbd84174-g34c   Ready    <none>   47m   v1.13.11-gke.14\n",
      "gke-tmb-cluster-default-pool-bbd84174-ss8w   Ready    <none>   47m   v1.13.11-gke.14\n",
      "gke-tmb-cluster-default-pool-bbd84174-x6vg   Ready    <none>   47m   v1.13.11-gke.14\n",
      "gke-tmb-cluster-gpus-7ccf9a63-13q0           Ready    <none>   45m   v1.13.11-gke.14\n",
      "gke-tmb-cluster-gpus-7ccf9a63-3nv8           Ready    <none>   44m   v1.13.11-gke.14\n",
      "gke-tmb-cluster-gpus-7ccf9a63-69jq           Ready    <none>   45m   v1.13.11-gke.14\n",
      "gke-tmb-cluster-gpus-7ccf9a63-9pk7           Ready    <none>   45m   v1.13.11-gke.14\n",
      "gke-tmb-cluster-gpus-7ccf9a63-f3p4           Ready    <none>   44m   v1.13.11-gke.14\n",
      "gke-tmb-cluster-gpus-7ccf9a63-qdpd           Ready    <none>   45m   v1.13.11-gke.14\n",
      "gke-tmb-cluster-gpus-7ccf9a63-qr90           Ready    <none>   45m   v1.13.11-gke.14\n",
      "gke-tmb-cluster-gpus-7ccf9a63-x11p           Ready    <none>   45m   v1.13.11-gke.14\n"
     ]
    }
   ],
   "source": [
    "# make sure we have a running cluster\n",
    "kubectl get nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No resources found\n",
      "No resources found\n"
     ]
    }
   ],
   "source": [
    "kubectl delete jobs --all\n",
    "kubectl delete pods --all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Pods\n",
    "\n",
    "- Kubernetes groups containers into _pods_\n",
    "- (Docker container = whale, Pod = group of whales)\n",
    "- specifications are written in YAML or JSON"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pod/mypod created\n"
     ]
    }
   ],
   "source": [
    "maybe kubectl delete pod/mypod\n",
    "kubectl apply -f - <<'EOF'\n",
    "apiVersion: v1\n",
    "kind: Pod\n",
    "metadata:\n",
    "  name: mypod\n",
    "spec:\n",
    "  containers:\n",
    "  - name: mypod\n",
    "    image: gcr.io/research-191823/bigdata19\n",
    "    command: [\"nvidia-smi\"]\n",
    "    resources:\n",
    "      limits:\n",
    "        nvidia.com/gpu: \"1\"\n",
    "  restartPolicy: Never\n",
    "EOF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Pod Status and Logs\n",
    "\n",
    "The Kubernetes runtime keeps track of pod status and logs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME    READY   STATUS              RESTARTS   AGE\n",
      "mypod   0/1     ContainerCreating   0          0s\n"
     ]
    }
   ],
   "source": [
    "kubectl get pods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "sleep 15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thu Dec 12 17:52:22 2019       \n",
      "+-----------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.1     |\n",
      "|-------------------------------+----------------------+----------------------+\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
      "|===============================+======================+======================|\n",
      "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
      "| N/A   40C    P8    12W /  70W |      0MiB / 15079MiB |      0%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "                                                                               \n",
      "+-----------------------------------------------------------------------------+\n",
      "| Processes:                                                       GPU Memory |\n",
      "|  GPU       PID   Type   Process name                             Usage      |\n",
      "|=============================================================================|\n",
      "|  No running processes found                                                 |\n",
      "+-----------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "kubectl logs pod/mypod"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Debugging Pod Startup Problems\n",
    "\n",
    "Sometimes pods don't get scheduled (never start running). Here are some tricks to debug this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name:               mypod\n",
      "Namespace:          default\n",
      "Priority:           0\n",
      "PriorityClassName:  <none>\n",
      "Node:               gke-tmb-cluster-gpus-7ccf9a63-69jq/10.138.0.57\n",
      "Start Time:         Thu, 12 Dec 2019 09:50:00 -0800\n",
      "Labels:             <none>\n",
      "Annotations:        kubectl.kubernetes.io/last-applied-configuration:\n",
      "                      {\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"name\":\"mypod\",\"namespace\":\"default\"},\"spec\":{\"containers\":[{\"command\":[\"nvid...\n",
      "                    kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container mypod\n",
      "... 60 ...\n"
     ]
    }
   ],
   "source": [
    "# sometimes pods don't schedule; there is tons of info\n",
    "kubectl describe pod/mypod | sed 10q\n",
    "kubectl describe pod/mypod | echo ... $(wc -l) ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Events:\n",
      "  Type    Reason     Age    From                                         Message\n",
      "  ----    ------     ----   ----                                         -------\n",
      "  Normal  Scheduled  2m49s  default-scheduler                            Successfully assigned default/mypod to gke-tmb-cluster-gpus-7ccf9a63-69jq\n",
      "  Normal  Pulling    2m48s  kubelet, gke-tmb-cluster-gpus-7ccf9a63-69jq  pulling image \"gcr.io/research-191823/bigdata19\"\n",
      "  Normal  Pulled     49s    kubelet, gke-tmb-cluster-gpus-7ccf9a63-69jq  Successfully pulled image \"gcr.io/research-191823/bigdata19\"\n",
      "  Normal  Created    27s    kubelet, gke-tmb-cluster-gpus-7ccf9a63-69jq  Created container\n",
      "  Normal  Started    27s    kubelet, gke-tmb-cluster-gpus-7ccf9a63-69jq  Started container\n"
     ]
    }
   ],
   "source": [
    "# the Events: section usually tells you why a job didn't get assigned to a node\n",
    "\n",
    "kubectl describe pod/mypod | grep -A100 Events:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name:               gke-tmb-cluster-gpus-7ccf9a63-13q0\n",
      "Roles:              <none>\n",
      "Labels:             beta.kubernetes.io/arch=amd64\n",
      "                    beta.kubernetes.io/fluentd-ds-ready=true\n",
      "                    beta.kubernetes.io/instance-type=n1-standard-16\n",
      "                    beta.kubernetes.io/os=linux\n",
      "                    cloud.google.com/gke-accelerator=nvidia-tesla-t4\n",
      "                    cloud.google.com/gke-nodepool=gpus\n",
      "                    cloud.google.com/gke-os-distribution=cos\n",
      "                    failure-domain.beta.kubernetes.io/region=us-west1\n",
      "... 94 ...\n"
     ]
    }
   ],
   "source": [
    "# nodes also have descriptions (even longer)\n",
    "\n",
    "node=$(kubectl get nodes | awk '/gpus/{print $1; exit}')\n",
    "kubectl describe node/$node | sed 10q\n",
    "kubectl describe node/$node | echo ... $(wc -l) ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Allocatable:\n",
      " attachable-volumes-gce-pd:  127\n",
      " cpu:                        15890m\n",
      " ephemeral-storage:          47093746742\n",
      " hugepages-2Mi:              0\n",
      " memory:                     56288600Ki\n",
      " nvidia.com/gpu:             1\n",
      " pods:                       110\n",
      "System Info:\n",
      " Machine ID:                 2d2db73e820f61ce6b21fdf0cbe7d3f7\n",
      " System UUID:                2D2DB73E-820F-61CE-6B21-FDF0CBE7D3F7\n"
     ]
    }
   ],
   "source": [
    "# you want to make sure that nodes have the right allocatable resources\n",
    "kubectl describe node/$node | grep -A10 Allocatable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Allocated resources:\n",
      "  (Total limits may be over 100 percent, i.e., overcommitted.)\n",
      "  Resource                   Requests    Limits\n",
      "  --------                   --------    ------\n",
      "  cpu                        400m (2%)   1050m (6%)\n",
      "  memory                     210Mi (0%)  510Mi (0%)\n",
      "  ephemeral-storage          0 (0%)      0 (0%)\n",
      "  attachable-volumes-gce-pd  0           0\n",
      "  nvidia.com/gpu             0           0\n",
      "Events:\n",
      "  Type    Reason                   Age                From                                            Message\n"
     ]
    }
   ],
   "source": [
    "# also make sure there are resources available\n",
    "kubectl describe node/$node | grep -A10 Allocated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Taints:             nvidia.com/gpu=present:NoSchedule\n",
      "Unschedulable:      false\n",
      "Conditions:\n"
     ]
    }
   ],
   "source": [
    "# nodes can be prevented from scheduling jobs by \"taints\"\n",
    "kubectl describe node/$node | grep -A2 Taints:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s\n",
      "                 node.kubernetes.io/unreachable:NoExecute for 300s\n",
      "                 nvidia.com/gpu:NoSchedule\n",
      "Events:\n",
      "  Type    Reason     Age    From                                         Message\n",
      "  ----    ------     ----   ----                                         -------\n"
     ]
    }
   ],
   "source": [
    "# only pods that tolerate the taints are scheduled\n",
    "kubectl describe pod/mypod | grep -A5 Tolerations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pod \"mypod\" deleted\n"
     ]
    }
   ],
   "source": [
    "kubectl delete pods --all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Jobs\n",
    "\n",
    "Jobs are like batch queuing. Job specs are a wrapper around pod specs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "job.batch/myjob created\n"
     ]
    }
   ],
   "source": [
    "maybe kubectl delete job/myjob\n",
    "kubectl apply -f - <<'EOF'\n",
    "apiVersion: batch/v1\n",
    "kind: Job\n",
    "metadata:\n",
    "  name: myjob\n",
    "  labels:\n",
    "    app: bigdata19\n",
    "spec:\n",
    "  backoffLimit: 0\n",
    "  template:\n",
    "    # below is a regular Pod spec\n",
    "    spec:\n",
    "      containers:\n",
    "        - name: myjob\n",
    "          image: gcr.io/research-191823/bigdata19\n",
    "          command:\n",
    "            - \"/bin/bash\"\n",
    "            - \"-c\"\n",
    "            - |\n",
    "              nvidia-smi\n",
    "          stdin: true\n",
    "          tty: true\n",
    "          resources:\n",
    "            limits:\n",
    "              nvidia.com/gpu: \"1\"\n",
    "      restartPolicy: Never\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "sleep 15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thu Dec 12 17:54:44 2019       \n",
      "+-----------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.2     |\n",
      "|-------------------------------+----------------------+----------------------+\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
      "|===============================+======================+======================|\n",
      "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
      "| N/A   41C    P0    27W /  70W |      0MiB / 15079MiB |      5%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "                                                                               \n",
      "+-----------------------------------------------------------------------------+\n",
      "| Processes:                                                       GPU Memory |\n",
      "|  GPU       PID   Type   Process name                             Usage      |\n",
      "|=============================================================================|\n",
      "|  No running processes found                                                 |\n",
      "+-----------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "kubectl logs job.batch/myjob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME    COMPLETIONS   DURATION   AGE\n",
      "myjob   1/1           2s         22s\n"
     ]
    }
   ],
   "source": [
    "kubectl get jobs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "job.batch \"myjob\" deleted\n"
     ]
    }
   ],
   "source": [
    "kubectl delete jobs --all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Configmaps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "configmap/files created\n"
     ]
    }
   ],
   "source": [
    "# configmaps are little mountable file systems, for config information and scripts\n",
    "# we put our Python scripts there\n",
    "maybe kubectl delete configmap files\n",
    "kubectl create configmap files \\\n",
    "--from-file=training.py=training.py \\\n",
    "--from-file=helpers.py=helpers.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Running a Training Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "job.batch/myjob created\n"
     ]
    }
   ],
   "source": [
    "# with the scripts transferred, let's run actual training\n",
    "# note the use of multi-line quoting for the shell script\n",
    "maybe kubectl delete job/myjob\n",
    "kubectl apply -f - <<'EOF'\n",
    "apiVersion: batch/v1\n",
    "kind: Job\n",
    "metadata:\n",
    "  name: myjob\n",
    "  labels:\n",
    "    app: bigdata19\n",
    "spec:\n",
    "  backoffLimit: 0\n",
    "  template:\n",
    "    spec:\n",
    "      containers:\n",
    "        - name: myjob\n",
    "          image: gcr.io/research-191823/bigdata19\n",
    "          command:\n",
    "            - \"/bin/bash\"\n",
    "            - \"-c\"\n",
    "            - |\n",
    "              cp /files/*.py .\n",
    "              python3 training.py\n",
    "          stdin: true\n",
    "          tty: true\n",
    "          resources:\n",
    "            limits:\n",
    "              nvidia.com/gpu: \"1\"\n",
    "          volumeMounts:\n",
    "            - mountPath: /files\n",
    "              name: files\n",
    "      restartPolicy: Never\n",
    "      volumes:\n",
    "        - configMap:\n",
    "            name: files\n",
    "          name: files\n",
    "EOF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Training Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME    COMPLETIONS   DURATION   AGE\n",
      "myjob   0/1           11s        11s\n"
     ]
    }
   ],
   "source": [
    "kubectl get jobs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "sleep 30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/torchvision/io/_video_opt.py:17: UserWarning: video reader based on ffmpeg c++ ops not available\n",
      "  warnings.warn(\"video reader based on ffmpeg c++ ops not available\")\n",
      "Thu Dec 12 17:55:27 UTC 2019; myjob-j2pfp; root; /workspace; GPU 0: Tesla T4 (UUID: GPU-e3b63d8c-056b-140d-43e1-de274722818d); \n",
      "creating resnet50\n",
      "        0 bs   128 per sample loss 5.53e-02 loading 7.77e-03 training 1.45e-02\n",
      "     1024 bs   128 per sample loss 5.53e-02 loading 4.61e-03 training 7.99e-03\n",
      "     1920 bs   128 per sample loss 5.52e-02 loading 3.35e-03 training 5.45e-03\n",
      "     2816 bs   128 per sample loss 5.50e-02 loading 2.74e-03 training 4.25e-03\n",
      "     3712 bs   128 per sample loss 5.49e-02 loading 2.46e-03 training 3.70e-03\n",
      "     4608 bs   128 per sample loss 5.48e-02 loading 2.38e-03 training 3.45e-03\n",
      "     5504 bs   128 per sample loss 5.46e-02 loading 2.28e-03 training 3.37e-03\n",
      "     6400 bs   128 per sample loss 5.45e-02 loading 2.28e-03 training 3.36e-03\n",
      "     7296 bs   128 per sample loss 5.44e-02 loading 2.22e-03 training 3.39e-03\n"
     ]
    }
   ],
   "source": [
    "kubectl logs job/myjob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "job.batch \"myjob\" deleted\n",
      "pod \"myjob-j2pfp\" deleted\n",
      "pod \"mypod\" deleted\n"
     ]
    }
   ],
   "source": [
    "kubectl delete jobs --all || true\n",
    "kubectl delete pods --all || true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Kubernetes\n",
    "\n",
    "- a way of running services and jobs on a cluster of machines\n",
    "- configurations are given as JSON or YAML files (or via APIs)\n",
    "- both CPUs and GPUs supported"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}