chore: add docs (#3)

Signed-off-by: Moritz Johner <[email protected]>

Author: moritzjohner-form3

Diff for: .github/actions/mkdocs/Dockerfile

@@ -0,0 +1,8 @@
+FROM squidfunk/mkdocs-material:9.4.5
+
+COPY action.sh /action.sh
+
+RUN apk add --no-cache bash \
+  && chmod +x /action.sh
+
+ENTRYPOINT ["/action.sh"]

Diff for: .github/actions/mkdocs/action.sh

@@ -0,0 +1,51 @@
+#!/bin/bash
+
+# Copyright 2020 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e
+
+REQUIREMENTS="${GITHUB_WORKSPACE}/docs/requirements.txt"
+
+if [ -f "${REQUIREMENTS}" ]; then
+    pip install -r "${REQUIREMENTS}"
+fi
+
+if [ -n "${GITHUB_TOKEN}" ]; then
+    remote_repo="https://x-access-token:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"
+elif [ -n "${PERSONAL_TOKEN}" ]; then
+    remote_repo="https://x-access-token:${PERSONAL_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"
+fi
+
+git config --global user.name "$GITHUB_ACTOR"
+git config --global user.email "$GITHUB_ACTOR@users.noreply.github.com"
+
+mkdocs build --config-file "${GITHUB_WORKSPACE}/docs/mkdocs.yml"
+
+git clone --branch=gh-pages --depth=1 "${remote_repo}" gh-pages
+cd gh-pages
+
+# copy current index file index.yaml before any change
+temp_worktree=$(mktemp -d)
+cp --force "index.yaml" "$temp_worktree/index.yaml"
+# remove current content in branch gh-pages
+git rm -r .
+# copy new doc.
+cp -r ../site/* .
+# restore chart index
+cp "$temp_worktree/index.yaml" .
+# commit changes
+git add .
+git commit -m "Deploy GitHub Pages"
+git push --force --quiet "${remote_repo}" gh-pages > /dev/null 2>&1

Diff for: .github/actions/mkdocs/action.yml

@@ -0,0 +1,9 @@
+# action.yml
+name: 'Deploy MkDocs'
+description: 'Deploys MkDocs site'
+branding:
+  icon: 'arrow-up-circle'
+  color: 'orange'
+runs:
+  using: 'docker'
+  image: 'Dockerfile'

Diff for: README.md

@@ -5,54 +5,7 @@ X-PDB allows you to define multi-cluster PodDisruptionBudgets and blocks evictio
 This allows you to operate stateful workloads spanning multiple clusters and limit disruptions to a necessary minimum.
 This is needed, because each cluster acts individually (evictions, rolling out updates etc) which may cause service disruptions simultaneously across clusters.
 
-## Eviction Process
-
-X-PDB hooks into the DELETE `pods` or CREATE `pods/eviction` API calls using a `ValidatingWebhookConfiguration`.
-It will acquire locks on all clusters to prevent race conditions and read the state (expected pod count & healthy pod count) from all clusters and compute if a eviction/deletion is allowed.
-
-X-PDB need to talk to the other X-PDB pods the other clusters to read the remote state.
-
-### Locking mechanism
-
-X-PDB acquires a lock on the remote clusters using a HTTP API.
-The lock is valid for a specific `namespace/selector` combination and it has a `leaseHolderIdentity`. This is the owner of the given lock.
-
-The lock is **valid for 5 seconds**. After that it can be re-acquired or taken over by a different holder.
-The lock prevents a race condition which occur if multiple evictions happen simultaneously across clusters which would lead to inconsistent data and wrong decisions. E.g. a read can happen while a eviction is being processed which would lead to multiple evictions happen at the same time that could break the pod disruption budget.
-
-We leave the lock as it is and DO NOT unlock it after the admission webhook have finished processing.
-Once it expires it can be re-acquired or taken over. We rely on the caller to retry the eviction or deletion of a Pod.
-
-
-**Why 5 seconds? - Why leave it locked when we've finished processing?**
-
-We need to lock evictions for a period of time **after** we have returned the admission result to allow kube-apiserver to update the Pod (set the `deletionTimestamp`).
-
-The lease duration should be higher than the sum of the following duration:
-  - the round-trip latency across all clusters (1-2 seconds)
-  - the processing time of the x-pdb http server (<1 second)
-  - the time kube-apiserver needs to process the x-pdb admission control response
-    and the time it takes until the desired action (evict/delete pod) is observable through the kube-apiserver (1-2 seconds)
-  - a generous surcharge (1-... seconds)
-
-### State server
-
-In order for x-pdb servers to communicate between each other they expose a [gRPC server](./protos/state/state.proto) which has the following API:
-
-```proto
-service State {
-
-  rpc Lock(LockRequest) returns (LockResponse) {}
-  rpc Unlock(UnlockRequest) returns (UnlockResponse) {}
-  rpc GetState(GetStateRequest) returns (GetStateResponse) {}
-}
-```
-
-## Threat Modelling
-
-- [Threat Modelling inventory](/threat-modelling/TMinventory.md)
-
-## Observability & monitoring
+Please refer to the documentation at https://form3tech-oss.github.com/x-pdb
 
 ## Development
 ### Running tests

Diff for: docs/.gitignore

@@ -0,0 +1 @@
+site/

Diff for: docs/Makefile

@@ -0,0 +1,39 @@
+.PHONY: check_dead_links
+check_dead_links: ## Check if the documentation contains dead links.
+	@docker run -t \
+	  -w /tmp \
+	  -v $$PWD:/tmp dkhamsing/awesome_bot:1.20.0 \
+	  --allow-dupe \
+	  --allow-redirect $(shell find $$PWD -mindepth 1 -name vendor -prune -o -name .modcache -prune -o -iname Changelog.md -prune -o -name "*.md" | sed -e "s#$$PWD/##")
+
+.PHONY: build-image
+build-image:
+	@docker build \
+		--no-cache \
+		-t xpdb-docs ../.github/actions/mkdocs
+
+.PHONY: build-docs
+build-docs: build-image
+	@docker run --rm -it \
+		-p 8000:8000 \
+		-v ${PWD}:/docs \
+		--entrypoint /bin/bash   \
+		xpdb-docs \
+		-c "pip install -r /docs/requirements.txt && mkdocs build --config-file mkdocs.yml"
+
+.PHONY: live-docs
+live-docs: build-image ## Build and launch a local copy of the documentation website in http://localhost:8000
+	@docker run --rm -it \
+		-p 8000:8000 \
+		-v ${PWD}:/docs \
+		--entrypoint /bin/bash   \
+		xpdb-docs \
+		-c "pip install -r /docs/requirements.txt && mkdocs serve --dev-addr=0.0.0.0:8000"
+
+.PHONY: misspell
+misspell:  ## Check for spelling errors.
+	@go install github.com/client9/misspell/cmd/misspell@latest
+	misspell \
+		-locale US \
+		-error \
+		.

Diff for: docs/extra.css

@@ -0,0 +1,9 @@
+.md-typeset__table {
+	min-width: 100%;
+}
+
+@media only screen and (min-width: 768px) {
+	td:nth-child(1) {
+		white-space: nowrap;
+	}
+}

Diff for: docs/mkdocs.yml

@@ -0,0 +1,77 @@
+site_name: X-PDB
+repo_name: "form3tech-oss/x-pdb"
+repo_url: https://github.com/form3tech-oss/x-pdb
+site_url: https://form3tech-oss.github.io/x-pdb
+edit_uri: edit/main/docs/src
+docs_dir: "/docs/src"
+
+# Extensions
+markdown_extensions:
+  - admonition
+  - abbr
+  - attr_list
+  - def_list
+  - footnotes
+  - meta
+  - md_in_html
+  - pymdownx.blocks.caption
+  - toc:
+      # insert a blank space before the character
+      permalink: " ¶"
+  - pymdownx.arithmatex:
+      generic: true
+  - pymdownx.betterem:
+      smart_enable: all
+  - pymdownx.caret
+  - pymdownx.critic
+  - pymdownx.details
+  - pymdownx.emoji:
+      emoji_index: !!python/name:materialx.emoji.twemoji
+      emoji_generator: !!python/name:materialx.emoji.to_svg
+  - pymdownx.highlight
+  - pymdownx.inlinehilite
+  - pymdownx.keys
+  - pymdownx.mark
+  - pymdownx.smartsymbols
+  - pymdownx.snippets:
+      check_paths: true
+  - pymdownx.superfences:
+      custom_fences:
+        - name: mermaid
+          class: mermaid
+          format: !!python/name:pymdownx.superfences.fence_code_format
+  - pymdownx.tabbed
+  - pymdownx.tasklist:
+      custom_checkbox: true
+  - pymdownx.tilde
+
+theme:
+  name: material
+  features:
+    - navigation.instant
+    - navigation.sections
+
+  palette:
+    primary: "deep orange"
+    accent: "indigo"
+
+  include_sidebar: true
+
+plugins:
+  - search
+  - awesome-pages
+  - minify:
+      minify_html: true
+
+extra_css: [extra.css]
+
+nav:
+  - Introduction:
+      - Overview: "index.md"
+      - Getting Started: "getting-started.md"
+  - Configuring x-pdb:
+      - XPodDisruptionBudget: "configuring-xpdb.md"
+      - Disruption Probes: "configuring-disruption-probes.md"
+  - Operating x-pdb:
+      - Failure Scenarios: "failure-scenarios.md"
+      - Metrics & SLOs: "metrics-slos.md"

Diff for: docs/requirements.txt

@@ -0,0 +1,5 @@
+mkdocs-material==9.4.5
+mkdocs-awesome-pages-plugin==2.9.2
+mkdocs-minify-plugin==0.7.1
+mkdocs-redirects==1.2.1
+pymdown-extensions==10.12

Diff for: docs/src/configuring-disruption-probes.md

@@ -0,0 +1,66 @@
+# Configuring Disruption Probes
+
+x-pdb allows workload owners to define a disruption probe endpoint.
+
+This endpoint is used by x-pdb to make a decision whether or not a disruption is allowed. Without the probe endpoint, x-pdb considers only pod readiness as a indicator of pod healthiness.
+
+With the probe endpoint configured, x-pdb will ask the probe endpoint whether or not the disruption is allowed.
+
+The probe endpoint is just a gRPC server, see details below.
+
+It might be helpful to probe internal state of some workloads like databases to verify wether an eviction can happen or not.
+Database Raft groups might become unavailable if a given pod is disrupted. In these cases workload owners might want to
+block disruptions to happen, even if all pods are ready.
+
+Example use-cases:
+
+- assess health of a database cluster as a whole before allowing the deletion of a single pod, as this disruption may add more pressure on the database.
+- assess the state of raft group leaders in a cluster before allowing an eviction
+- assess the replication lag of database clusters
+- assess if any long-running queries/jobs or backup are running, where a deletion can cause a problem
+
+```yaml
+apiVersion: x-pdb.form3.tech/v1alpha1
+kind: XPodDisruptionBudget
+metadata:
+  name: kube-dns
+  namespace: kube-system
+spec:
+  minAvailable: 80%
+  selector:
+    matchLabels:
+      k8s-app: kube-dns
+  probe:
+    endpoint: opensearch-disruption-probe.opensearch.svc.cluster.local:8080
+```
+
+## TLS and authentication
+
+At this point, the communication between x-pdb and the probe server does not use mutual TLS and only validates the server certificate presented by the probe endpoint and verifies it has been issued by the CA defined in `--controller-certs-dir`.
+
+#### DisruptionProbe Server
+
+The DisruptionProbe server allows the client to ask if a disruption for a given pod and XPDB resource is allowed.
+
+```proto
+service DisruptionProbe {
+  // Sends a IsDisruptionAllowed request
+  rpc IsDisruptionAllowed(IsDisruptionAllowedRequest) returns (IsDisruptionAllowedResponse) {}
+}
+
+// The request message containing the user's name.
+message IsDisruptionAllowedRequest {
+  string pod_name = 1;
+  string pod_namespace = 2;
+  string xpdb_name = 3;
+  string xpdb_namespace = 4;
+}
+
+// The response message containing the greetings
+message IsDisruptionAllowedResponse {
+  bool is_allowed = 1;
+  string error = 2;
+}
+
+```
+

Diff for: docs/src/configuring-xpdb.md

@@ -0,0 +1,66 @@
+# Configuring XPodDisruption Resource
+
+## Nomenclature
+
+| term            | description                                                                         |
+| --------------- | ----------------------------------------------------------------------------------- |
+| local cluster   | This is the Kubernetes cluster where a pod deletion or eviction is being requested. |
+| remote clusters | These are the clusters which are not handling the pod eviction / deletion.          |
+
+## Configuration and Behavior
+
+X-PDB works under the assumption that your workloads are structured in a similar way across clusters, i.e. that pods that you want to protect sit in the same namespace no matter which cluster you look at.
+
+The `XPodDisruptionBudget` resources looks and feels just like `PodDisruptionBudget` resources:
+
+- A label selector `.spec.selector` to specify the set of pods to which it applies. The `selector` applies to all clusters for decision making, not just the local one where pod eviction/deletion is happening.
+
+- `.spec.minAvailable` which is a description of the number of pods from that set that must still be available after the eviction, even in the absence of the evicted pod. minAvailable can be either an absolute number or a percentage.
+- `.spec.maxUnavailable` which is a description of the number of pods from that set that can be unavailable after the eviction. It can be either an absolute number or a percentage.
+
+You can specify only one of `maxUnavailable` and `minAvailable` in a single PodDisruptionBudget. `maxUnavailable` can only be used to control the eviction of pods that have an associated controller managing them.
+
+In addition to that, `XPodDisruptionBudget` has the following fields:
+
+- `.spec.suspend` which allows you to disable the XPDB resource. This allows all pod deletions/evictions. It is intended to be used as a break-glass procedure to allow engineers to take manual action. The suspension is configured on a per-cluster basis and affects only local pods. I.e. other clusters that run x-pdb will not be able to evict pods if there isn't enough disruption budget available globally.
+- `.spec.probe` that allows workload owners to define a [disruption probe](./configuring-disruption-probes.md) endpoint. Without a probe, x-pdb will only consider pod readiness as an indicator of healthiness and compute the disruption verdict based on that. With `.spec.probe`, x-pdb considers the response of the probe endpoint as well.
+
+It is irrelevant for `x-pdb` if the remote cluster has a `XPodDisruptionBudget` resource and whether or not the configuration match.
+
+The user is supposed to deploy the `XPodDisruptionBudget` to all clusters. It may lead to unexpected disruptions when the resource is missing.
+
+```yaml
+apiVersion: x-pdb.form3.tech/v1alpha1
+kind: XPodDisruptionBudget
+metadata:
+  name: kube-dns
+  namespace: kube-system
+spec:
+  # Specify either `minAvailable` or `maxUnavailable`
+  # Both percentages and numbers are supported
+  minAvailable: 80%
+  selector:
+    matchLabels:
+      k8s-app: kube-dns
+```
+
+## gRPC State Server
+
+In order for x-pdb servers to communicate between each other they expose a gRPC state server interface with the following APIs. It allows x-pdb to asses the health of pods on remote clusters.
+
+The communication between x-pdb servers is secured using mutual TLS. The certificate directory can be configured with `--controller-certs-dir` which is supposed to contain `ca.crt`, `tls.crt` and `tls.key` files.
+
+```proto
+// State is the service that allows x-pdb servers to talk with
+// each other.
+service State {
+  // Acquires a lock on the local cluster using the specified leaseHolderIdentity.
+  rpc Lock(LockRequest) returns (LockResponse) {}
+
+  // Frees a lock on the local cluster if the lease identity matches.
+  rpc Unlock(UnlockRequest) returns (UnlockResponse) {}
+
+  // Calculates the expected count based off the Deployment/StatefulSet/ReplicaSet number of replicas or - if implemented - a `scale` sub resource.
+  rpc GetState(GetStateRequest) returns (GetStateResponse) {}
+}
+```