Add OIDC configuration support

robo-cap · hyder · commit df6134a0eba5 · 2025-03-07T23:05:18.000+11:00
diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md
@@ -12,6 +12,8 @@
     - [Subnets](./guide/network_subnets.md)
     - [Network Security Groups](./guide/network_nsgs.md)    
   - [Cluster](./guide/cluster.md)
+    - [Cluster OIDC Authentication](./guide/cluster_oidc_authentication.md)
+    - [Cluster OIDC Discovery](./guide/cluster_oidc_discovery.md)
     - [Cluster Add-ons](./guide/cluster_addons.md)
   - [Workers](./guide/workers.md)
     - [Mode](./guide/workers_mode.md)
diff --git a/docs/src/guide/cluster_addons.md b/docs/src/guide/cluster_addons.md
@@ -1,6 +1,6 @@
 # Cluster Add-ons
 
-With this module to manage both essential and optional add-ons on **enhanced** OKE clusters. 
+With this module you can manage both essential and optional add-ons on **enhanced** OKE clusters. 
 
 This module provides the option to remove [Essential addons](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengintroducingclusteraddons.htm#contengintroducingclusteraddons__section-essential-addons) and to manage, both essential & [optional addons](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengintroducingclusteraddons.htm#contengintroducingclusteraddons__section-optional-addons).
 
diff --git a/docs/src/guide/cluster_oidc_authentication.md b/docs/src/guide/cluster_oidc_authentication.md
@@ -0,0 +1,179 @@
+# OpenID Connect Authentication
+
+By default, OKE clusters are set up to authenticate individuals (human users, groups or service principals) accessing the API endpoint using OCI Identity and Access Management (IAM). 
+
+Using OKE OIDC Authentication, we can authenticate OKE API Endpoint requests (from human users or service principals) using tokens issued by third-party Identity Providers without the need for federation with OCI IAM.
+
+## Prerequisites
+
+Note the following prerequisites for enabling a cluster for OIDC authentication:
+
+- The cluster must be an enhanced cluster. OIDC authentication is not supported for basic clusters.
+- The cluster must be running Kubernetes version 1.21 (or later) -- for single external OIDC IdP setup.
+- The cluster must be running Kubernetes version 1.30 (or later) -- for multiple external OIDC IdPs setup.
+
+## Configuration
+
+In addition to the implicit OCI IAM, you can configure the OKE cluster to authenticate the cluster API endpoint requests using a **single** external OIDC (OpenID Connect) Identity Provider (IdP).
+
+For this is necessary to set the following variables:
+
+```
+oidc_discovery_enabled = true
+oidc_token_authentication_config = {
+  client_id          = ...,
+  issuer_url         = ...,
+  username_claim     = ...,
+  username_prefix    = ...,
+  groups_claim       = ...,
+  groups_prefix      = ...,
+  required_claims    = [
+    {
+      key = ...,
+      value = ...
+    },
+    {
+      key = ...,
+      value = ...
+    }
+  ],
+  ca_certificate     = ...,
+  signing_algorithms = []
+}
+```
+
+In case you're looking to authenticate the cluster API endpoint requests with **multiple** OIDC IdPs, you can take advantage of the [authentication configuration via file Kubernetes feature](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#using-authentication-configuration).
+
+```
+oidc_discovery_enabled = true
+oidc_token_authentication_config = {
+  configuration_file = base64encode(yamlencode(
+    {
+      "apiVersion" = "apiserver.config.k8s.io/v1beta1"
+      "kind"       = "AuthenticationConfiguration"
+      "jwt"        = [
+        {
+          "issuer"= {
+            "url"       = "...",
+            "audiences" = [
+              "..."
+            ],
+            "audienceMatchPolicy" = "MatchAny"
+          }
+          "claimMappings" = {
+            "username" = {
+              "claim" = "..."
+              "prefix" = ""
+            }
+          }
+          "claimValidationRules" = [
+            {
+              "claim" = "..."
+              "requiredValue" = "..."
+            }
+          ]
+        }
+      ]
+    }
+  ))
+}
+```
+
+The authenticated users are mapped to a `User` resource in Kubernetes and you have to setup the desired RBAC polices to provide access.
+
+E.g. for Github Action workflow:
+
+```
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  namespace: default
+  name: actions-oidc-role
+rules:
+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["get", "watch", "list"]
+  - apiGroups: ["apps"]
+    resources: ["deployments"]
+    verbs: ["get", "watch", "list", "create", "update", "delete"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: actions-oidc-binding
+  namespace: default
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: actions-oidc-role
+subjects:
+  - apiGroup: rbac.authorization.k8s.io
+    kind: User
+    name: actions-oidc:repo:GH-ACCOUNT/GH-REPO:ref:refs/heads/main
+```
+
+**Note:** 
+1. You need to make sure the OKE Control Plane endpoint is allowed to connect to the IdP.
+
+```
+allow_rules_cp = {
+  "Allow egress to anywhere HTTPS from OKE CP" : {
+    protocol = "6", port=443, destination = "0.0.0.0/0", destination_type = "CIDR_BLOCK",
+  }
+}
+```
+2. You cannot configure cluster OIDC Authentication using the arguments of the `oidc_token_authentication_config` (`client_id`, `issuer_url`, etc..) **and** the `configuration_file` at the same time.
+
+## OpenID Connect Discovery
+
+### Prerequisites
+
+Note the following points when using OIDC Discovery:
+
+- The cluster must be an enhanced cluster. OIDC Discovery is not supported for basic clusters.
+- The cluster must be running Kubernetes version 1.21 (or later).
+
+### Configuration
+
+OKE already supports Workload Identity to enable Kubernetes pods to access OCI resources, such as a secret or cloud storage bucket without storing access credentials in your Kubernetes cluster.
+
+If you are looking to authorize Kubernetes pods to access non-OCI resources you can enable OKE OIDC Discovery.
+
+When you enable OIDC discovery for an OKE cluster, OKE provides an OpenID Connect issuer endpoint. This endpoint serves the OIDC discovery document and the JSON web key set (JWKS), which contain the public key necessary for token validation. These resources enable third-party IdP to validate tokens issued for pods in the OKE cluster, allowing those pods to access non-OCI resources.
+
+[ ![](../images/oidc-discovery.png) ](../images/oidc-discovery.png)
+*Figure 1: OIDC Discovery*
+
+To enable the OKE OIDC Discovery, you have to set the following variable:
+
+```
+open_id_connect_discovery_enabled = true
+```
+
+The OpenID Connect issuer endpoint is available in the output:
+
+```
+cluster_oidc_discovery_endpoint
+```
+
+## Example usage
+
+OIDC Authentication setup using Kubernetes API server flags
+
+```javascript
+{{#include ../../../examples/cluster-addons/vars-cluster-oidc-auth-single.auto.tfvars:4:}}
+```
+
+OIDC Authentication setup using Kubernetes API server configuration file
+
+```javascript
+{{#include ../../../examples/cluster-addons/vars-cluster-oidc-auth-multiple.auto.tfvars:4:}}
+```
+
+## Reference
+* [OKE OpenID Authetication](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengOpenIDConnect-Authentication.htm)
+* [OKE Cluster Terraform resource](https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/containerengine_cluster)
+* [Github workflow OKE OIDC authentication](https://docs.oracle.com/en/learn/gaw-oke-odic/index.html#introduction)
+* [Kubernetes OIDC Authentication setup using Kubernetes API server configuration file](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#using-authentication-configuration)
+* [Kubernetes OIDC Authentication setup using Kubernetes API server flags](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#using-flags)
diff --git a/docs/src/guide/cluster_oidc_discovery.md b/docs/src/guide/cluster_oidc_discovery.md
@@ -0,0 +1,42 @@
+# OpenID Connect Discovery
+
+With OKE OIDC Discovery, it is possible to validate Kubernetes pods running on OKE clusters with third-party STS (Security Token Service) issuers, whether on-premises or in cloud service providers (CSPs) such as Amazon Web Services (AWS) and Google Cloud Platform (GCP), and authorize them to access non-OCI resources. OKE OIDC Discovery enables this integration.
+
+## Prerequisites
+
+Note the following points when using OIDC Discovery:
+
+- The cluster must be an enhanced cluster. OIDC Discovery is not supported for basic clusters.
+- The cluster must be running Kubernetes version 1.21 (or later).
+
+## Configuration
+
+When you enable OIDC discovery for an OKE cluster, OKE provides an OpenID Connect issuer endpoint. This endpoint serves the OIDC discovery document and the JSON web key set (JWKS), which contain the public key necessary for token validation. These resources enable third-party IdP to validate tokens issued for pods in the OKE cluster, allowing those pods to access non-OCI resources.
+
+[ ![](../images/oidc-discovery.png) ](../images/oidc-discovery.png)
+*Figure 1: OIDC Discovery*
+
+To enable the OKE OIDC Discovery, you have to set the following variable:
+
+```
+open_id_connect_discovery_enabled = true
+```
+
+The OpenID Connect issuer endpoint is available in the output:
+
+```
+cluster_oidc_discovery_endpoint
+```
+
+## Example usage
+
+OIDC Discovery setup using Kubernetes API server flags
+
+```javascript
+{{#include ../../../examples/cluster-addons/vars-cluster-oidc-discovery.auto.tfvars:4:}}
+```
+
+## Reference
+* [OKE OpenID Discovery](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengOpenIDConnect-Discovery.htm)
+* [OKE Cluster Terraform resource](https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/containerengine_cluster)
+* [OKE Pods access AWS resources](https://umashankar-s.medium.com/multicloud-use-case-oke-apps-pods-accessing-aws-resources-using-openid-disovery-8e147500656f)
diff --git a/docs/src/images/oidc-discovery.png b/docs/src/images/oidc-discovery.png
diff --git a/docs/src/outputs.md b/docs/src/outputs.md
@@ -52,7 +52,10 @@
 <!-- BEGIN_TF_CLUSTER -->
 
 * **`cluster_id`**&nbsp;&nbsp; 
-* **`endpoints`**&nbsp;&nbsp; 
+* **`cluster_endpoints`**&nbsp;&nbsp; 
+* **`cluster_oidc_discovery_endpoint`**&nbsp;&nbsp; 
+* **`cluster_kubeconfig`**&nbsp;&nbsp; 
+* **`cluster_ca_cert`**&nbsp;&nbsp; 
 
 <!-- END_TF_CLUSTER -->
 
diff --git a/examples/cluster/vars-cluster-basic.auto.tfvars b/examples/cluster/vars-cluster-basic.auto.tfvars
@@ -2,4 +2,4 @@
 # Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl
 
 cluster_name       = "oke-example"
-kubernetes_version = "v1.26.2"
+kubernetes_version = "v1.31.1"
diff --git a/examples/cluster/vars-cluster-enhanced.auto.tfvars b/examples/cluster/vars-cluster-enhanced.auto.tfvars
@@ -9,7 +9,7 @@ cluster_type                      = "enhanced" // *basic/enhanced
 cni_type                          = "flannel"  // *flannel/npn
 assign_public_ip_to_control_plane = true // true/*false
 image_signing_keys                = []
-kubernetes_version                = "v1.26.2"
+kubernetes_version                = "v1.31.1"
 pods_cidr                         = "10.244.0.0/16"
 services_cidr                     = "10.96.0.0/16"
 use_signed_images                 = false // true/*false
diff --git a/module-cluster.tf b/module-cluster.tf
@@ -7,22 +7,22 @@ data "oci_containerengine_clusters" "existing_cluster" {
   count          = var.cluster_id != null ? 1 : 0
   compartment_id = local.compartment_id
 
-  state = ["ACTIVE","UPDATING"]
+  state = ["ACTIVE", "UPDATING"]
   filter {
-    name = "id"
+    name   = "id"
     values = [var.cluster_id]
   }
 }
 
 data "oci_containerengine_cluster_kube_config" "public" {
-  count      = local.cluster_enabled && local.public_endpoint_available ? 1 : 0
+  count = local.cluster_enabled && local.public_endpoint_available ? 1 : 0
 
   cluster_id = local.cluster_id
   endpoint   = "PUBLIC_ENDPOINT"
 }
 
 data "oci_containerengine_cluster_kube_config" "private" {
-  count      = local.cluster_enabled && local.private_endpoint_available ? 1 : 0
+  count = local.cluster_enabled && local.private_endpoint_available ? 1 : 0
 
   cluster_id = local.cluster_id
   endpoint   = "PRIVATE_ENDPOINT"
@@ -35,11 +35,11 @@ locals {
 
   cluster-context = try(format("context-%s", substr(local.cluster_id, -11, -1)), "")
 
-  existing_cluster_endpoints  = coalesce(one(flatten(data.oci_containerengine_clusters.existing_cluster[*].clusters[*].endpoints)), tomap({}))
-  public_endpoint_available   = var.cluster_id != null ? length(lookup(local.existing_cluster_endpoints, "public_endpoint", "")) > 0 : var.control_plane_is_public && var.assign_public_ip_to_control_plane
-  private_endpoint_available  = var.cluster_id != null ? length(lookup(local.existing_cluster_endpoints, "private_endpoint", "")) > 0 : true
-  kubeconfig_public           = var.control_plane_is_public ? try(yamldecode(replace(lookup(one(data.oci_containerengine_cluster_kube_config.public), "content", ""), local.cluster-context, var.cluster_name)), tomap({})) : null
-  kubeconfig_private          = try(yamldecode(replace(lookup(one(data.oci_containerengine_cluster_kube_config.private), "content", ""), local.cluster-context, var.cluster_name)), tomap({}))
+  existing_cluster_endpoints = coalesce(one(flatten(data.oci_containerengine_clusters.existing_cluster[*].clusters[*].endpoints)), tomap({}))
+  public_endpoint_available  = var.cluster_id != null ? length(lookup(local.existing_cluster_endpoints, "public_endpoint", "")) > 0 : var.control_plane_is_public && var.assign_public_ip_to_control_plane
+  private_endpoint_available = var.cluster_id != null ? length(lookup(local.existing_cluster_endpoints, "private_endpoint", "")) > 0 : true
+  kubeconfig_public          = var.control_plane_is_public ? try(yamldecode(replace(lookup(one(data.oci_containerengine_cluster_kube_config.public), "content", ""), local.cluster-context, var.cluster_name)), tomap({})) : null
+  kubeconfig_private         = try(yamldecode(replace(lookup(one(data.oci_containerengine_cluster_kube_config.private), "content", ""), local.cluster-context, var.cluster_name)), tomap({}))
 
   kubeconfig_clusters = try(lookup(local.kubeconfig_private, "clusters", []), [])
   apiserver_private_host = (var.create_cluster
@@ -131,6 +131,11 @@ module "cluster" {
     },
     local.service_lb_freeform_tags,
   )
+
+  oidc_discovery_enabled           = var.oidc_discovery_enabled
+  oidc_token_auth_enabled          = var.oidc_token_auth_enabled
+  oidc_token_authentication_config = var.oidc_token_authentication_config
+
   depends_on = [
     module.iam_cluster_prerequisites,
   ]
@@ -146,6 +151,11 @@ output "cluster_endpoints" {
   value       = var.create_cluster ? one(module.cluster[*].endpoints) : local.existing_cluster_endpoints
 }
 
+output "cluster_oidc_discovery_endpoint" {
+  description = "OIDC discovery endpoint for the OKE cluster"
+  value       = var.create_cluster && var.oidc_discovery_enabled ? one(module.cluster[*].oidc_discovery_endpoint) : null
+}
+
 output "cluster_kubeconfig" {
   description = "OKE kubeconfig"
   value = var.output_detail ? (
diff --git a/modules/cluster/cluster.tf b/modules/cluster/cluster.tf
@@ -39,6 +39,40 @@ resource "oci_containerengine_cluster" "k8s_cluster" {
   }
 
   options {
+
+    dynamic "open_id_connect_token_authentication_config" {
+      for_each = var.oidc_token_auth_enabled ? [1] : []
+
+      content {
+        is_open_id_connect_auth_enabled = var.oidc_token_auth_enabled
+
+
+        issuer_url         = lookup(var.oidc_token_authentication_config, "issuer_url", null)
+        ca_certificate     = lookup(var.oidc_token_authentication_config, "ca_certificate", null)
+        client_id          = lookup(var.oidc_token_authentication_config, "client_id", null)
+        signing_algorithms = lookup(var.oidc_token_authentication_config, "signing_algorithms", null)
+
+        groups_claim  = lookup(var.oidc_token_authentication_config, "groups_claim", null)
+        groups_prefix = lookup(var.oidc_token_authentication_config, "groups_prefix", null)
+
+        username_claim  = lookup(var.oidc_token_authentication_config, "username_claim", null)
+        username_prefix = lookup(var.oidc_token_authentication_config, "username_prefix", null)
+
+        dynamic "required_claims" {
+          for_each = lookup(var.oidc_token_authentication_config, "required_claims", [])
+          content {
+            key    = lookup(required_claims.value, "key")
+            value  = lookup(required_claims.value, "value")
+          }
+        }
+        configuration_file = lookup(var.oidc_token_authentication_config, "configuration_file", null)
+      }
+    }
+
+    open_id_connect_discovery {
+      is_open_id_connect_discovery_enabled = var.oidc_discovery_enabled
+    }
+
     kubernetes_network_config {
       pods_cidr     = var.pods_cidr
       services_cidr = var.services_cidr
diff --git a/modules/cluster/outputs.tf b/modules/cluster/outputs.tf
@@ -8,3 +8,7 @@ output "cluster_id" {
 output "endpoints" {
   value = one(oci_containerengine_cluster.k8s_cluster.endpoints)
 }
+
+output "oidc_discovery_endpoint" {
+  value = oci_containerengine_cluster.k8s_cluster.open_id_connect_discovery_endpoint
+}
diff --git a/modules/cluster/variables.tf b/modules/cluster/variables.tf
@@ -31,3 +31,8 @@ variable "persistent_volume_defined_tags" { type = map(string) }
 variable "persistent_volume_freeform_tags" { type = map(string) }
 variable "service_lb_defined_tags" { type = map(string) }
 variable "service_lb_freeform_tags" { type = map(string) }
+
+# OIDC
+variable "oidc_discovery_enabled" { type = bool }
+variable "oidc_token_auth_enabled" { type = bool }
+variable "oidc_token_authentication_config" { type = any }
diff --git a/modules/cluster/versions.tf b/modules/cluster/versions.tf
diff --git a/modules/workers/instance.tf b/modules/workers/instance.tf
diff --git a/variables-cluster.tf b/variables-cluster.tf

Original file line number	Diff line number	Diff line change
`@@ -8,3 +8,7 @@ output "cluster_id" {`
`8`	`8`	`output "endpoints" {`
`9`	`9`	`value = one(oci_containerengine_cluster.k8s_cluster.endpoints)`
`10`	`10`	`}`
	`11`	`+`
	`12`	`+output "oidc_discovery_endpoint" {`
	`13`	`+ value = oci_containerengine_cluster.k8s_cluster.open_id_connect_discovery_endpoint`
	`14`	`+}`