Add ingress class docs (Kong#811)

Add documentation for the Ingress class annotation, which the controller uses to filter out which Kubernetes resources it needs to translate into Kong configuration. This includes an overview document describing the annotation's use and providing an example configuration demonstrating it, updates to existing reference documentation that account for the new behavior, and updates to existing guides to incorporate the annotation where needed.

Author: Travis Raines

docs/README.md

@@ -7,6 +7,7 @@
   - [Custom Resources](#custom-resources)
   - [Deployment methods](#deployment-methods)
   - [High-availability and scaling](#high-availability-and-scaling)
+  - [Resource classes](#resource-classes)
   - [Security](#security)
 - [Guides and Tutorials](#guides-and-tutorials)
 - [Configuration reference](#configuration-reference)
@@ -48,6 +49,12 @@ and infrastructure.
 Please refer to [this document](concepts/ha-and-scaling.md) to understand
 failures scenarios, recovery methods, as well as scaling considerations.
 
+### Ingress classes
+
+[Ingress classes](concepts/ingress-classes.md) filter which resources the
+controller loads. They ensure that Kong Ingress Controller instances do not
+load configuration intended for other instances or other ingress controllers.
+
 ### Security
 
 Please refer to [this document](concepts/security.md) to understand the

docs/concepts/custom-resources.md

@@ -64,7 +64,7 @@ Once this resource is created, the resource needs to be associated with an
 `Ingress`, `Service`, or `KongConsumer` resource in Kubernetes.
 For more details, please read the reference documentation on `KongPlugin`.
 
-The below diagram shows how the `KongPlugin` resource can be linked to an
+The below diagram shows how you can link `KongPlugin` resource to an
 `Ingress`, `Service`, or `KongConsumer`:
 
 |  |  |
@@ -73,6 +73,8 @@ The below diagram shows how the `KongPlugin` resource can be linked to an
 
 ## KongClusterPlugin
 
+_This resource requires the [`kubernetes.io/ingress.class` annotation](../README.md#resource-classes)._
+
 KongClusterPlugin resource is exactly same as KongPlugin, except that it is a
 Kubernetes cluster-level resources instead of being a namespaced resource.
 This can help when the configuration of the plugin needs to be centralized
@@ -87,12 +89,20 @@ KongClusterPlugin with the same name.
 
 ## KongConsumer
 
+_This resource requires the `kubernetes.io/ingress.class` annotation. Its value
+must match the value of the controller's `--ingress-class` argument, which is
+"kong" by default._
+
 This custom resource configures `Consumers` in Kong.
 Every `KongConsumer` resource in Kubernetes directly translates to a
 [Consumer][kong-consumer] object in Kong.
 
 ## TCPIngress
 
+_This resource requires the `kubernetes.io/ingress.class` annotation. Its value
+must match the value of the controller's `--ingress-class` argument, which is
+"kong" by default._
+
 This Custom Resource is used for exposing non-HTTP
 and non-GRPC services running inside Kubernetes to
 the outside world via Kong. This proves to be useful when

docs/concepts/ingress-classes.md

@@ -0,0 +1,183 @@
+# Kong Ingress Controller and ingress class
+
+
+## Table of Contents
+
+- [**Introduction**](#introduction)
+- [**Configuring the controller class**](#configuring-the-controller-ingress-class)
+- [**Loading resources by class**](#loading-resouces-by-class)
+- [**When to use a custom class**](#when-to-use-a-custom-class)
+- [**Legacy behavior**](#legacy-behavior)
+- [**Examples**](#examples)
+
+## Introduction
+
+The Kong Ingress Controller uses ingress classes to filter Kubernetes Ingress
+objects and other resources before converting them into Kong configuration.
+This allows it to coexist with other ingress controllers and/or other
+deployments of the Kong Ingress Controller in the same cluster: a Kong Ingress
+Controller will only process configuration marked for its use.
+
+## Configuring the controller ingress class
+
+The `--ingress-class` flag (or `CONTROLLER_INGRESS_CLASS` environment variable)
+specify the ingress class expected by the Kong Ingress Controller. By default,
+it expects the `kong` class.
+
+## Loading resources by class
+
+The Kong Ingress Controller translates a variety of Kubernetes resources into
+Kong configuration. Broadly speaking, we can separate these resources into two
+categories:
+
+- Resources that the controller translates directly into Kong configuration.
+- Resources referenced by some other resource, where the other resource is
+  directly translated into Kong configuration.
+
+For example, an Ingress is translated directly into a Kong route, and a
+KongConsumer is translated directly into a [Kong consumer](https://docs.konghq.com/latest/admin-api/#consumer-object). A Secret containing
+an authentication plugin credential is _not_ translated directly: it is only
+translated into Kong configuration if a KongConsumer resource references it.
+
+Because they create Kong configuration indenpendent of any other resources,
+directly-translated resources require an ingress class, and their class must
+match the class configured for the controller. Referenced resources do not
+require a class, but must be referenced by a directly-translated resource
+that matches the controller.
+
+### Adding class information to resources
+
+Most resources use a [kubernetes.io/ingress-class annotation][class-annotation]
+to indicate their class. There are several exceptions:
+
+- v1 Ingress resources have a dedicated `class` field.
+- Knative Services [use the class specifed][knative-class] by the
+  `ingress.class` key of the Knative installation's `config-network` ConfigMap.
+  You can optionally [override this on a per-Service basis][knative-override]
+  by adding a `networking.knative.dev/ingress.class` annotation to the Service.
+
+### Enabling support for classless resources
+
+Specifying a class is optional for some resources. Although specifying a class
+is recommended, you can instruct the controller to process resources without a
+class annotation using flags:
+
+- `--process-classless-ingress-v1beta1` instructs the controller to translate
+  v1beta1 Ingress resources with no class annotation.
+- `--process-classless-kong-consumer` instructs the controller to translate
+  KongConsumer resources with no class annotation.
+
+These flags are primarily intended for compatibility with older configuration
+(Kong Ingress Controller before 0.10 had less strict class
+requirements, and it was common to omit class annotations). If you are creating
+new configuration and do not have older configuration without class
+annotations, recommended best practice is to add class information to Ingress
+and KongConsumer resources and not set the above flags. Doing so avoids
+accidentally creating duplicate configuration in other ingress controller
+instances.
+
+These flags do not _ignore_ `ingress.class` annotations: they allow resources
+with no such annotation, but will not allow resource that have a non-matching
+`ingress.class` annotation.
+
+## When to use a custom class
+
+Using the default `kong` class is fine for simpler deployments, where only one
+Kong Ingress Controller instance is running in a cluster. Changing the class is
+typical when:
+
+- You install multiple Kong environments in one Kubernetes cluster to handle
+  different types of ingress traffic, e.g. when using separate Kong instances
+  to handle traffic on internal and external load balancers, or deploying
+  different types of non-production environments in a single test cluster.
+- You install multiple controller instances alongside a single Kong cluster to
+  separate configuration into different Kong workspaces (using the
+  `--kong-workspace` flag) or to restrict which Kubernetes namespaces any one
+  controller instance has access to.
+
+## Legacy behavior
+
+This overview covers behavior in Kong Ingress Controller version 0.10.0 onward.
+Earlier versions had a special case for the default class and a bug affecting
+custom classes:
+
+- When using the default `kong` class, the controller would always process
+  classless resources in addition to `kong`-class resources. When using a
+  non-default controller class, the controller would only process resources
+  with that class, not classless resources. Although this was by design, it was
+  a source of user confusion.
+- When using a custom controller class, some resources that should not have
+  required a class (because they were referenced by other resources)
+  effectively did require a class: while these resources were loaded initially,
+  the controller would not track updates to them unless they had a class
+  annotation.
+
+In versions 0.10.0+ you must instruct the controller to load classless
+resources, which is allowed (but not recommended) for either the default or
+custom classes. Resources referenced by another resource are always loaded and
+updated correctly regardless of which class you set on the controller; you do
+not need to add class annotations to these resources when using a custom class.
+
+## Examples
+
+Typical configurations will include a mix of resources that have class
+information and resources that are referenced by them. For example, consider
+the following configuration for authenticating a request, using a KongConsumer,
+credential Secret, Ingress, and KongPlugin (a Service is implied, but not
+shown):
+
+```yaml
+kind: KongConsumer
+metadata:
+  name: dyadya-styopa
+  annotations:
+    kubernetes.io/ingress.class: "kong"
+username: styopa
+credentials:
+- styopa-key
+
+---
+
+kind: Secret
+apiVersion: v1
+stringData:
+  key: bylkogdatomoryakom
+  kongCredType: key-auth
+metadata:
+  name: styopa-key
+
+---
+
+kind: Ingress
+apiVersion: extensions/v1beta1
+metadata:
+  name: ktonezhnaet
+  annotations:
+    kubernetes.io/ingress.class: "kong"
+    konghq.com/plugins: "key-auth-example"
+spec:
+  rules:
+  - http:
+      paths:
+      - path: /vsemznakom
+        backend:
+          serviceName: httpbin
+          servicePort: 80
+
+---
+
+apiVersion: configuration.konghq.com/v1
+kind: KongPlugin
+metadata:
+  name: key-auth-example
+plugin: key-auth
+```
+
+The KongConsumer and Ingress resources both have class annotations, as they are
+resources that the controller uses as a basis for building Kong configuration.
+The Secret and KongPlugin _do not_ have class annotations, as they are
+referenced by other resources that do.
+
+[class-annotation]: ../references/annotations.md#kubernetesioingressclass
+[knative-class]: ../guides/using-kong-with-knative.md#ingress-class
+[knative-override]: https://knative.tips/networking/ingress-override/

docs/deployment/admission-webhook.md

@@ -142,6 +142,8 @@ $ echo "apiVersion: configuration.konghq.com/v1
 kind: KongConsumer
 metadata:
   name: harry
+  annotations:
+    kubernetes.io/ingress.class: kong
 username: harry" | kubectl apply -f -
 kongconsumer.configuration.konghq.com/harry created
 ```
@@ -153,6 +155,8 @@ $ echo "apiVersion: configuration.konghq.com/v1
 kind: KongConsumer
 metadata:
   name: harry2
+  annotations:
+    kubernetes.io/ingress.class: kong
 username: harry" | kubectl apply -f -
 Error from server: error when creating "STDIN": admission webhook "validations.kong.konghq.com" denied the request: consumer already exists
 ```

docs/guides/cert-manager.md

@@ -124,6 +124,8 @@ apiVersion: extensions/v1beta1
 kind: Ingress
 metadata:
   name: demo-yolo42-com
+  annotations:
+    kubernetes.io/ingress.class: kong
 spec:
   rules:
   - host: demo.yolo42.com
@@ -190,6 +192,7 @@ metadata:
   annotations:
     kubernetes.io/tls-acme: "true"
     cert-manager.io/cluster-issuer: letsencrypt-prod
+    kubernetes.io/ingress.class: kong
 spec:
   tls:
   - secretName: demo-yolo42-com

docs/guides/configure-acl-plugin.md

@@ -54,6 +54,7 @@ metadata:
   name: demo-get
   annotations:
     konghq.com/strip-path: "false"
+    kubernetes.io/ingress.class: kong
 spec:
   rules:
   - http:
@@ -71,6 +72,7 @@ metadata:
   name: demo-post
   annotations:
     konghq.com/strip-path: "false"
+    kubernetes.io/ingress.class: kong
 spec:
   rules:
   - http:
@@ -161,6 +163,7 @@ metadata:
   annotations:
     plugins.konghq.com: app-jwt
     konghq.com/strip-path: "false"
+    kubernetes.io/ingress.class: kong
 spec:
   rules:
   - http:
@@ -179,6 +182,7 @@ metadata:
   annotations:
     plugins.konghq.com: app-jwt
     konghq.com/strip-path: "false"
+    kubernetes.io/ingress.class: kong
 spec:
   rules:
   - http:
@@ -234,6 +238,8 @@ apiVersion: configuration.konghq.com/v1
 kind: KongConsumer
 metadata:
   name: admin
+  annotations:
+    kubernetes.io/ingress.class: kong
 username: admin
 " | kubectl apply -f -
 
@@ -242,6 +248,8 @@ apiVersion: configuration.konghq.com/v1
 kind: KongConsumer
 metadata:
   name: plain-user
+  annotations:
+    kubernetes.io/ingress.class: kong
 username: plain-user
 " | kubectl apply -f -
 ```
@@ -299,6 +307,8 @@ apiVersion: configuration.konghq.com/v1
 kind: KongConsumer
 metadata:
   name: admin
+  annotations:
+    kubernetes.io/ingress.class: kong
 username: admin
 credentials:
   - app-admin-jwt
@@ -309,6 +319,8 @@ apiVersion: configuration.konghq.com/v1
 kind: KongConsumer
 metadata:
   name: plain-user
+  annotations:
+    kubernetes.io/ingress.class: kong
 username: plain-user
 credentials:
   - app-user-jwt
@@ -536,6 +548,8 @@ apiVersion: configuration.konghq.com/v1
 kind: KongConsumer
 metadata:
   name: admin
+  annotations:
+    kubernetes.io/ingress.class: kong
 username: admin
 credentials:
   - app-admin-jwt
@@ -547,6 +561,8 @@ apiVersion: configuration.konghq.com/v1
 kind: KongConsumer
 metadata:
   name: plain-user
+  annotations:
+    kubernetes.io/ingress.class: kong
 username: plain-user
 credentials:
   - app-user-jwt
@@ -567,6 +583,7 @@ metadata:
   annotations:
     plugins.konghq.com: app-jwt,plain-user-acl
     konghq.com/strip-path: "false"
+    kubernetes.io/ingress.class: kong
 spec:
   rules:
   - http:
@@ -585,6 +602,7 @@ metadata:
   annotations:
     plugins.konghq.com: app-jwt,admin-acl
     konghq.com/strip-path: "false"
+    kubernetes.io/ingress.class: kong
 spec:
   rules:
   - http:

docs/guides/configuring-fallback-service.md

@@ -56,6 +56,7 @@ metadata:
   name: demo
   annotations:
     konghq.com/strip-path: "true"
+    kubernetes.io/ingress.class: kong
 spec:
   rules:
   - http:
@@ -104,6 +105,8 @@ apiVersion: extensions/v1beta1
 kind: Ingress
 metadata:
   name: fallback
+  annotations:
+    kubernetes.io/ingress.class: kong
 spec:
   backend:
     serviceName: fallback-svc