This Chart deploys the API Gateway v10.x onward with the following optional subcharts: hazelcast, mysql, influxdb, grafana.
The included MySQL subChart is enabled by default to make trying this chart out easier. It is not supported or recommended for production. Layer7 assumes that you are deploying a Gateway solution to a Kubernetes environment with an external MySQL database.
- Kubernetes 1.24.x
- Refer to techdocs for the latest version support
- Helm v3.7.x
- Gateway v10.x License
- Persistent Volume Provisioner (if using PVC for the Demo MySQL Database or Service Metrics example with Grafana or InfluxDb)
- Metrics Server if you would like to enable autoscaling.
If you are using a previous version of this Chart please read the updates section before you upgrade.
The Layer7 API Gateway is now running with Java 11 with the release of the v10.1.00. The Gateway chart's version has been incremented to 2.0.2.
Things to note and be aware of are the deprecation of TLSv1.0/TLSv1.1 and the JAVA_HOME dir has gone through some changes as well.
- Updated bootstrap script
- 'find' replaced with 'du'
- Added pod labels and annotations to the otk-install job.
- otk.job.podLabels
- otk.job.podAnnotations
- The OTK Install job now uses podSecurity and containerSecurity contexts if set.
- Updated how pod labels and annotations are templated in deployment.yaml
Traffic Policies for Gateway Services are now configurable. The Kubernetes default for these options is Cluster if left unset.
Updates to Gateway Container Lifecycle.
- A new preStop script has been added for graceful termination
- terminationGracePeriodSeconds must be greater than preStopScript.timeoutSeconds
- Container Lifecycle can be overridden for custom exec/http calls
Custom labels and annotations have been extended to all objects the Gateway Chart deploys. Pod Labels and annotations have been added to the Gateway and PM-Tagger deployments.
- Additional Labels/Annotations apply to everything in this Chart's templates
# Additional Annotations apply to all deployed objects
additionalAnnotations: {}
# Additional Labels apply to all deployed objects
additionalLabels: {}
- Pod Labels/Annotations at the base level apply to the Gateway Pod
## Pod Labels for the Gateway Pod
## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
podLabels: {}
# Pod Annotations apply to the Gateway Pod
## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
podAnnotations: {}
- PM-Tagger pod labels/annotations are separate
pmtagger:
...
## Pod Labels for the PM Tagger Pod
## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
podLabels: {}
# Pod Annotations apply to the PM Tagger Pod
## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
podAnnotations: {}
PM tagger has following additional configuration options
- Topology Spread Constraints
- Pod Security Context
- Container Security Context
- All PM-Tagger Configuration
The default image tag in values.yaml is updated to 5.2.1 and xsd version in configmap.yaml to 5.2. The updates are due to vulnerability from CVE-2022-36437. The updates are applied to both the gateway and gateway-otk chart.
The bootstrap script has been updated to reflect changes to the Container Gateway's filesystem. The updates are currently limited to 10.1.00_CR3. Please see the InitContainer Examples for more info .
The PM Tagger image default version tag been updated to 1.0.1.
The default image tag in values.yaml and production-values.yaml for OTK updated to 4.6.1. Support for liveness and readiness probes using OTK health check service.
The default image tag in values.yaml and production-values.yaml, and the appVersion in Chart.yaml have been updated to 11.0.00.
Before upgrading existing deployments, please see the Container Gateway 11.0 Release Notes for important information regarding the procedure.
OTK installation and upgrade is now supported as part of Gateway charts. Please refer to OTK Install or Upgrade for more details. Gateway-OTK is now deprecated.
The default image tag in values.yaml and production-values.yaml now points at specific GA or CR versions of the API Gateway. The appVersion in Chart.yaml has also been updated to reflect that. As of this release, that is 10.1.00_CR2
To reduce reliance on requiring a custom/derived gateway image for custom and modular assertions, scripts and restman bundles a bootstrap script has been introduced. The script works with the /opt/docker/custom folder.
The best way to populate this folder is with an initContainer where files can be copied directly across or dynamically loaded from an external source.
- InitContainer Examples - this repository also contains examples for custom health checks and configuration files.
The following configuration options have been added
- Custom Health Checks
- Custom Configuration Files
- Topology Spread Constraints
- Tolerations
- Pod Security Context
- Container Security Context
- Http headers can also now be added to the liveness and readiness probes
- Ingress and HPA API Version validation has been updated to check for available APIs vs. KubeVersion
- SubCharts now show image repository and tags
Please see the 3.0.0 updates, this release brings significant updates and breaking changes if you are using an external Hazelcast 3.x server. Services and Ingress configuration have also changed. Read the 3.0.0 Updates below and check out the additional guides for more info.
Hazelcast 4.x/5.x servers are now supported this represents a breaking change if you have configured an external Hazelcast 3.x server.
- If you are using Gateway v10.1 and below you will either need to set hazelcast.legacy.enabled=true and use the following gateway image docker.io/caapim/gateway:10.1.00_20220802 or update your external Hazelcast server.
- The included Hazelcast subChart has been updated to reflect this change
Ingress configuration has been updated to include multiple hosts, please see Ingress Configuration for more detail. You will need to update your values.yaml to reflect the changes.
-
You can now configure Gateway Ports. This does not cover Kubernetes Service level configuration which will need to be updated to reflect your changes.
-
New Management Service
- Provides separation of concerns for external/management traffic. This was previously a manual step.
-
- PM (Policy Manager) tagger is a lightweight go application that works with the new management service.
- RBAC Role Required if using PM Tagger.
-
Default values.yaml restructure
- configuration items more closely aligned
-
Added production-values.yaml
- Includes a baseline for production configuration
- Resources are set to minimum recommended values
- Application ports are hardened
- 8080 (disabled)
- 8443 (management features disabled - service is ClusterIP)
- 9443 (configured with management service)
- Autoscaling is enabled
- Ingress is enabled
- Rules are configured for 8443
- Database is not created - you will need to supply a JDBC Url
- MySQL Stable Chart is deprecated - the demo database subChart has been changed to Bitnami MySQL - if your database is NOT externalised you will lose any policy/configuration you have there.
- tls.customKey ==> tls.useSignedCertificates tls.key tls.pass tls.existingSecretName
- Fixing bitnami repository dependency issue.
- Internal only.
- Added support for the Kubernetes CSI Driver for gateway bundles. This does not currently extend to environment variables or the Gateway license.
- The CSI functionality is optional
- Added support for sidecars and initContainers
- volumeMounts are automatically configured with emptyDir
- Updated default values update to reflect empty objects/arrays for optional fields.
- Load the Gateway Deployment's ServiceAccountToken as a stored password for querying the Kubernetes API.
- management.kubernetes.loadServiceAccountToken
- You can now specify existing secrets for Gateway Configuration, DefaultSSLKey (tls) and bundles
- Ingress Definition updated to reflect the new API Version, additional configuration added.
- HostAliases applies to /etc/hosts for dns names that aren't available on a dns server.
- System.properties is now mounted to the Gateway Container, default values have been applied.
- You can now reference existing bundles stored in configMaps/Secrets
- NodeSelector and Affinity settings for the Gateway Deployment
- Resources values updated to reflect minimum recommended configuration
If you are using the demo database in a previous version of this Chart this upgrade will remove it. If you wish to keep your data you will need to perform a backup.
$ helm repo update
$ helm show values layer7/gateway > gateway-values.yaml
Inspect and update the new gateway-values.yaml
$ helm upgrade my-ssg --set-file "license.value=path/to/license.xml" --set "license.accept=true" -f ./gateway-values.yaml layer7/gateway
Check out this guide for more in-depth instruction
$ helm repo add layer7 https://caapim.github.io/apim-charts/
$ helm repo update
$ helm install my-ssg --set-file "license.value=path/to/license.xml" --set "license.accept=true" layer7/gateway
To upgrade your Gateway Release
$ helm upgrade my-ssg --set-file "license.value=path/to/license.xml" --set "license.accept=true" layer7/gateway
To uninstall the Gateway Chart
$ helm uninstall <release name> -n <release namespace>
To make sure that your custom values don't get overwritten by a pull, create your own values.yaml (myvalues.yaml..) then specify -f myvalues.yaml when deploying/upgrading
You only need to include the values you wish to change in your myvalues.yaml
For example, you wish to deploy the Gateway Chart as is without a database. Your myvalues.yaml would then contain the following
database:
enabled: false
create: false
- Service Configuration
- Gateway Application Ports
- Ingress Configuration
- PM Tagger Configuration
- OTK Install or Upgrade
- Database Configuration
- Cluster-Wide Properties
- Java Args
- System Properties
- Gateway Bundles
- Bootstrap Script
- Custom Health Checks
- Custom Configuration Files
- Logs & Audit Configuration
- Graceful Termination
- Autoscaling
- RBAC Parameters
- Service Metrics Demo
- SubChart Configuration
The following table lists the configurable parameters of the Gateway chart and their default values. See values.yaml for additional parameters and info
| Parameter | Description | Default |
|---|---|---|
nameOverride |
Name override | nil |
fullnameOverride |
Full name override | nil |
global.schedulerName |
Override the default scheduler | nil |
license.value |
Gateway license file | nil |
license.accept |
Accept Gateway license EULA | false |
image.registry |
Image Registry | docker.io |
image.repository |
Image Repository | caapim/gateway |
image.tag |
Image tag | 11.0.00 |
image.pullPolicy |
Image Pull Policy | IfNotPresent |
imagePullSecret.enabled |
Configures Gateway Deployment to use imagePullSecret, you can also leave this disabled and associate an image pull secret with the Gateway's Service Account | false |
imagePullSecret.existingSecretName |
Point to an existing Image Pull Secret | commented out |
imagePullSecret.username |
Registry Username | nil |
imagePullSecret.password |
Registry Password | nil |
additionalAnnotations |
Additional Annotations apply to all deployed objects | {} |
additionalLabels |
Additional Labels apply to all deployed objects | {} |
podLabels |
Pod Labels for the Gateway Pod | {} |
podAnnotations |
Pod Annotations apply to the Gateway Pod | {} |
replicas |
Number of Gateway replicas | 1 |
updateStrategy.type |
Deployment Strategy | RollingUpdate |
updateStrategy.rollingUpdate.maxSurge |
Rolling Update Max Surge | 1 |
updateStrategy.rollingUpdate.maxUnavailable |
Rolling Update Max Unavailable | 0 |
clusterHostname |
Gateway Cluster Hostname | my.localdomain |
existingGatewaySecretName |
Existing Secret that contains management credentials, see values.yaml for what must be included | commented out |
clusterPassword |
Cluster Password, used if db backed | mypassword |
management.enabled |
Enable/Disable Policy Manager access | true |
management.restman.enabled |
Enable/Disable the Rest Management API (Restman) | false |
management.username |
Policy Manager Username | admin |
management.password |
Policy Manager Password | mypassword |
management.kubernetes.loadServiceAccountToken |
Automatically load the Gateway Deployment's ServiceAccount Token for querying the Kubernetes API | false |
database.enabled |
Run in DB Backed or Ephemeral Mode | true |
database.create |
Deploy the MySQL stable deployment as part of this release | true |
database.username |
Database Username | gateway |
database.password |
Database Password | mypassword |
database.name |
Database name | ssg |
tls.useSignedCertificates |
Enable/Disable use of your own TLS Certificate, this ovverides the Gateway's defaultSSLKey | false |
tls.existingSecretName |
Existing Secret that contains TLS p12 container and pass, see values.yaml for what must be included | commented out |
tls.key |
p12 container - this can be set with --set-file tls.key=/path/to/tls.p12 | nil |
tls.pass |
p12 container password - this cannot be empty | nil |
config.heapSize |
Java Heap Size | 2g |
config.javaArgs |
Additional Java Args to pass to the SSG process | see values.yaml |
config.log.override |
Override the standard log configuration | true |
config.log.properties |
Custom logging properties | see values.yaml |
config.cwp.enabled |
Enable/Disable settable cluster-wide properties | false |
config.cwp.properties |
Set name/value pairs of cluster-wide properties | see values.yaml |
config.sytemProperties |
Configure the Gateway's system.properties file | see values.yaml |
additionalEnv |
Additional environment variables you wish to pass to the Gateway Configmap | see values.yaml |
additionalSecret |
Additional secret variables you wish to pass to the Gateway Secret | see values.yaml |
bundle.enabled |
Creates a configmap with bundles from the ./bundles folder | false |
bundle.path |
Specify the path to the bundle files. The bundles folder in this repo has some example bundle files | "bundles/*.bundle" |
existingBundle.enabled |
Enable mounting existing configMaps/Secrets that contain Layer7 Gateway Bundles - see values.yaml for more info | false |
existingBundle.configMaps |
Array of configMaps that will be mounted to the Gateway's bootstrap folder | see values.yaml |
existingBundle.secrets |
Array of Secrets that will be mounted to the Gateway's bootstrap folder | see values.yaml |
customHosts.enabled |
Enable customHosts on the Gateway, this overrides /etc/hosts. | see values.yaml |
customHosts.hostAliases |
Array of hostAliases to add to the Container Gateway | see values.yaml |
service.type |
Service Type | LoadBalancer |
service.loadbalancer |
Additional Loadbalancer Configuration | see https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service |
service.ports |
List of http external port mappings | https: 8443 -> 8443, management: 9443->9443 |
service.annotations |
Additional annotations to add to the service | {} |
service.internalTrafficPolicy |
Internal Traffic Policy | Cluster |
service.externalTrafficPolicy |
External Traffic Policy | Cluster |
ingress.enabled |
Enable/Disable an ingress record being created | false |
ingress.annotations |
Additional ingress annotations | {} |
ingress.hostname |
Sets Ingress Hostname | nil |
ingress.port |
The Gateway Port number/name to route to | 8443 |
ingress.tlsHostnames |
Register additional Hostnames for the TLS Certificate | see values.yaml |
ingress.secretName |
The name of an existing Cert secret, setting this does not auto-create the secret | tls-secret |
ingress.additionalHostnamesAndPorts |
key/value pairs of hostname:port that will be added to the ingress object | see values.yaml |
livenessProbe.enabled |
Enable/Disable | true |
livenessProbe.initialDelaySeconds |
Initial delay | 60 |
livenessProbe.timeoutSeconds |
Timeout | 1 |
livenessProbe.periodSeconds |
Frequency | 10 |
livenessProbe.successThreshold |
Success Threshold | 1 |
livenessProbe.failureThreshold |
Failure Threshold | 10 |
readinessProbe.enabled |
Enable/Disable | true |
readinessProbe.initialDelaySeconds |
Initial delay | 60 |
readinessProbe.timeoutSeconds |
Timeout | 1 |
readinessProbe.periodSeconds |
Frequency | 10 |
readinessProbe.successThreshold |
Success Threshold | 1 |
readinessProbe.failureThreshold |
Failure Threshold | 10 |
resources.limits |
Resource Limits | {} |
resources.requests |
Resource Requests | {} |
nodeSelector |
Node Selector | {} |
affinity |
Affinity | {} |
topologySpreadConstraints |
Topology Spread Constraints | [] |
tolerations |
Tolerations | [] |
podSecurityContext |
Pod Security Context | [] |
containerSecurityContext |
Container Security Context | {} |
bootstrap.script.enabled |
Enable the bootstrap script | false |
There are two types of port configuration available in the Gateway Helm Chart that are configured in the following ways
Sample entry that exposes 8443 which is one of the default TLS port on the API Gateway using service type LoadBalancer.
service:
type: LoadBalancer
annotations: {}
ports:
- name: https
internal: 8443
external: 8443
protocol: TCP
Sample entry that exposes 8443 which is one of the default TLS ports on the API Gateway Note that the service type is ClusterIP which does not receive an external IP address. We can make this service accessible by configuring an ingress resource.
service:
type: ClusterIP
annotations: {}
ports:
- name: https
internal: 8443
external: 8443
protocol: TCP
The Gateway Management Service is primarily used to expose Gateway Ports that you wish to use for Internal Management Access for tools like Policy Manager. This Service requires the PM Tagger to function correctly.
management:
...
service:
enabled: true
type: LoadBalancer
annotations: {}
# cloud.google.com/load-balancer-type: "Internal"
# service.beta.kubernetes.io/azure-load-balancer-internal: "true"
ports:
- name: management
internal: 9443
external: 9443
protocol: TCP
OTK job is used to install or upgrade otk on gateway. It supports Single, Internal and DMZ type of OTK installations.
NOTE: In dual gateway installation, restart the pods after OTK install or upgrade is required.
Prerequisites:
- Create or upgrade the OTK Database https://techdocs.broadcom.com/us/en/ca-enterprise-software/layer7-api-management/api-management-oauth-toolkit/4-6/installation-workflow/create-or-upgrade-the-otk-database.html
- Configure cluster wide property for otk.port pointing to gateway ingress port.
config:
cwp:
enabled: true
properties:
- name: otk.port
value: 443
- Restman is enabled. Can be disabled once the install/upgrage is complete.
management:
restman:
enabled: true
- Management is enabled with restman (management.enabled: true, management.restman.enabled: true)
Limitations:
- OTK Instance modifiers are not supported.
- OTK not supported on ephemeral gateway.
database:
# DB Backed or ephemeral
enabled: true
| Parameter | Description | Default |
|---|---|---|
otk.enabled |
Enable/Disable OTK installation or upgrade | false |
otk.type |
OTK installation type - SINGLE, DMZ or INTERNAL | SINGLE |
otk.forceInstallOrUpgrade |
Force install or upgrade by uninstalling existing otk soluction kit and install. | false |
otk.enablePortalIngeration |
Not applicable for DMZ and INTERNAL OTK types | false |
otk.skipPostInstallationTasks |
Skip post installation tasks for OTK type INTERNAL and DMZ Intrenal Gateway: - #OTK Client Context Variables - #OTK id_token configuration - Import SSL Certificate of DMZ gateway DMZ Gareway: - #OTK OVP Configuration - #OTK Storage Configuration - Import SSL Certificate of Internal gateway |
false |
otk.internalGatewayHost |
Internal gateway host for OTK type DMZ | |
otk.internalGatewayPort |
Internal gateway post for OTK type DMZ | |
otk.dmzGatewayHost |
DMZ gateway host for OTK type INTERNAL | |
otk.dmzGatewayPort |
DMZ gateway port for OTK type INTERNAL | |
otk.subSolutionKitNames |
List of comma seperated sub soluction Kits to install or upgrade. | |
otk.job.image.repository |
Image Repositor | caapim/otk-install |
otk.job.image.tag |
Image Tag. (OTK version) | 4.6 |
otk.job.image.pullPolicy |
Image Pull Policy | IfNotPresent |
otk.job.image.labels |
Job lables | {} |
otk.job.image.nodeSelector |
Job Node selector | {} |
otk.job.image.tolerations |
Job tolerations | [] |
otk.job.podLabels |
OTK Job podLabels | {} |
otk.job.podAnnotations |
OTK Job podAnnotations | {} |
otk.database.type |
OTK database type - mysql/oracle/cassandra | mysql |
otk.database.connectionName |
OTK database connection name | OAuth |
otk.database.existingSecretName |
Point to an existing OTK database Secret | |
otk.database.username |
OTK database user name | |
otk.database.password |
OTK database password | |
otk.database.properties |
OTK database additional properties | {} |
otk.database.sql.type |
OTK database type (mysql/oracle/cassandra) | mysql |
otk.database.sql.jdbcURL |
OTK database sql jdbc URL (oracle/mysql) | |
otk.database.sql.jdbcDriverClass |
OTK database sql driver class name (oracle/mysql) | |
otk.database.sql.databaseName |
OTK database Oracle database name | |
otk.database.cassandra.connectionPoints |
OTK database cassandra connection points (comma seperated) | |
otk.database.cassandra.port |
OTK database cassandra connection port | |
otk.database.cassandra.keyspace |
OTK database cassandra keyspace | |
otk.database.cassandra.driverConfig |
OTK database cassandra driver config (Gateway 11+) | {} |
otk.healthCheckBundle.enabled |
Enable/Disable installation of OTK health check service bundle | true |
otk.healthCheckBundle.useExisting |
Use exising OTK health check service bundle | false |
otk.healthCheckBundle.name |
OTK health check service bundle name | otk-health-check-bundle-config |
otk.livenessProbe.enabled |
Enable/Disable. Requires otk.healthCheckBundle.enabled set to true and OTK version >= 4.6.1. Valid only for SINGLE and INTERNAL OTK type installation. | true |
otk.livenessProbe.type |
httpGet |
|
otk.livenessProbe.httpGet.path |
/auth/oauth/health |
|
otk.livenessProbe.httpGet.port |
8443 |
|
otk.readinessProbe.enabled |
Enable/Disable. Requires otk.healthCheckBundle.enabled set to true and OTK version >= 4.6.1. Valid only for SINGLE and INTERNAL OTK type installation. | true |
otk.readinessProbe.type |
httpGet |
|
otk.readinessProbe.httpGet.path |
/auth/oauth/health |
|
otk.readinessProbe.httpGet.port |
8443 |
Once you have decided on which container ports you would like to expose, you need to create the corresponding ports on the API Gateway. These will need match the corresponding service and management service ports above.
This configuration creates and mounts a gateway bundle, if you wish to maintain Gateway ports outside of the Gateway Chart you can either use Policy Manager or create and mount your own bundle in the existingBundle section.
By default the following ports are created
- 8080 (HTTP - disabled)
- 8443 (HTTPS - Published service message input only)
- 9443 (HTTPS - Published service message input only, Administrative access, Browser-based administration, Built-in services)
Things to note before you get started:
- If you are deploying the Gateway with a fresh MySQL database
- This port configuration will replace the defaults.
- If you are using an existing database
- Named ports will be updated. Ports not configured in this list (like 2124 - internode communication) will remain.
config:
...
listenPorts:
custom:
enabled: true
ports:
- name: Default HTTPS (8443)
port: 8443
enabled: true
protocol: HTTPS
managementFeatures:
- Published service message input
# - Administrative access
# - Browser-based administration
# - Built-in services
properties:
- name: server
value: A
tls:
enabled: true
# privateKey: 00000000000000000000000000000002:ssl
clientAuthentication: Optional
versions:
#- TLSv1.0
#- TLSv1.1
- TLSv1.2
- TLSv1.3
useCipherSuitesOrder: true
cipherSuites:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
- TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
- TLS_DHE_RSA_WITH_AES_256_CBC_SHA
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
- TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
- TLS_DHE_RSA_WITH_AES_128_CBC_SHA
- TLS_AES_256_GCM_SHA384
- TLS_AES_128_GCM_SHA256
# - TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384
# - TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384
# - TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
# - TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
# - TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
# - TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
# - TLS_RSA_WITH_AES_256_GCM_SHA384
# - TLS_RSA_WITH_AES_256_CBC_SHA256
# - TLS_RSA_WITH_AES_256_CBC_SHA
# - TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256
# - TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256
# - TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
# - TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
# - TLS_ECDH_RSA_WITH_AES_128_CBC_SHA
# - TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
# - TLS_RSA_WITH_AES_128_GCM_SHA256
# - TLS_RSA_WITH_AES_128_CBC_SHA256
# - TLS_RSA_WITH_AES_128_CBC_SHA
The Gateway Helm Chart allows you to configure an Ingress Resource that your central Ingress Controller can manage. You can find more information on Ingress Controllers here.
This represents the ingress configuration for Gateway Chart < 3.0.0 you need to configure an Ingress Resource for the API Gateway
ingress:
enabled: true
annotations:
# Ingress class
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
# nginx.ingress.kubernetes.io/ssl-passthrough: "true"
secretName: tls-secret
hostname: dev.ca.com
tlsHostnames: []
# - dev.ca.com
# - dev1.ca.com
## The port that you want to route to via ingress. This needs to be available via service.ports.
port: 8443
## Define additional hostnames and ports as key-value pairs.
additionalHostnamesAndPorts: {}
New Ingress Configuration Gateway Chart >= 3.0.0
ingress:
enabled: true
# Ingress Class Name
ingressClassName: nginx
# Ingress annotations
annotations:
# Ingress class
# kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
# nginx.ingress.kubernetes.io/ssl-passthrough: "true"
# When the ingress is enabled, a host pointing to this will be created
# By default clusterHostname is used, only set this if you want to use a different host
## Enable TLS configuration for the hostname defined at ingress.hostname/clusterHostname parameter
tls:
- hosts:
- dev.ca.com
secretName: tls-secret-1
# - hosts:
# - dev1.ca.com
# secretName: tls-secret-2
rules:
- host: dev.ca.com
path: "/"
service:
port:
name: https
#number:
# - host: dev1.ca.com
# path: "/"
# service:
# port:
# name: anotherport
# #number:
PM (Policy Manager) Tagger is a lightweight go application that works in conjunction with the management service to provide a stable connection to your container gateway via Policy Manager.
| Parameter | Description | Default |
|---|---|---|
pmtagger.enabled |
Enable pm-tagger | false |
pmtagger.replicas |
Replicas (you should never need more than one | 1 |
pmtagger.image.registry |
Image Registry | docker.io |
pmtagger.image.repository |
Image Repository | layer7api/pm-tagger |
pmtagger.image.tag |
Image Tag | 1.0.1 |
pmtagger.image.pullPolicy |
Image Pull Policy | IfNotPresent |
pmtagger.image.imagePullSecret.enabled |
Use Image Pull secret - this uses the image pull secret configured for the API Gateway | false |
pmtagger.resources |
Resources | see values.yaml |
pmtagger.podLabels |
Pod Labels for the Gateway Pod | {} |
pmtagger.podAnnotations |
Pod Annotations apply to the Gateway Pod | {} |
pmtagger.nodeSelector |
Node Selector | {} |
pmtagger.affinity |
Affinity | {} |
pmtagger.topologySpreadConstraints |
Topology Spread Constraints | [] |
pmtagger.tolerations |
Tolerations | [] |
pmtagger.podSecurityContext |
Pod Security Context | [] |
pmtagger.containerSecurityContext |
Container Security Context | {} |
You can configure the deployment to use an external database (this is the recommended approach - the included MySQL SubChart is not supported). In the values.yaml file, set the create field in the database section to false, and set jdbcURL to use your own database server:
database:
enabled: true
create: false
jdbcURL: jdbc:mysql://myprimaryserver:3306,mysecondaryserver:3306/ssg?failOverReadOnly=false
username: myuser
password: mypassword
name: ssg
In the above example, two MySQL database servers are specified with myprimaryserver acting as the primary server and mysecondaryserver acting as the secondary server. The failOverReadOnly property is also set to false meaning that the secondary server db is also writable.
More info on the JDBC URL:
- Connection URL syntax: https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-url-format.html
- Failover config: https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-config-failover.html
- Configuration properties: https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-configuration-properties.html
Configuring SSL/TLS: the following parameters can be added to enable secure communication between the Gateway and an external MySQL Database
- useSSL=true
- requireSSL=true
- verifyServerCertificate=false
jdbcURL: jdbc:mysql://myprimaryserver:3306,mysecondaryserver:3306/ssg?useSSL=true&requireSSL=true&verifyServerCertificate=false
In order the create the database on the remote server, the provided user in the username field must have write privilege on the database. See GRANT statement usage: https://dev.mysql.com/doc/refman/8.0/en/grant.html#grant-database-privileges
You can specify cluster-wide properties in values.yaml, you can also use the bundle to load your own Gateway Bundles.
| Parameter | Description | Default |
|---|---|---|
config.cwp.enabled |
Enable the CWP functionality (mounts a volume) | false |
config.cwp.properties |
Array of Key/Value pairs of cluster-wide properties | see values.yaml |
The default cluster-wide properties are as follows
config:
...
cwp:
enabled: true
properties:
- name: io.httpsHostAllowWildcard
value: true
- name: log.levels
value: |
com.l7tech.level = CONFIG
com.l7tech.server.policy.variable.ServerVariables.level = SEVERE
com.l7tech.external.assertions.odata.server.producer.jdbc.GenerateSqlQuery.level = SEVERE
com.l7tech.server.policy.assertion.ServerSetVariableAssertion.level = SEVERE
com.l7tech.external.assertions.comparison.server.ServerComparisonAssertion = SEVERE
- name: audit.setDetailLevel.FINE
value: 152 7101 7103 9648 9645 7026 7027 4155 150 4716 4114 6306 4100 9655 150 151 11000 4104
Additional Java Arguments as may be recommended by support can be configured in values.yaml
| Parameter | Description | Default |
|---|---|---|
config.heapSize |
Java Heap Size - this should be a percentage of the memory configured in resources.limits and should be updated together. The default assumes 50%, going above 75% is not recommended | 2G |
config.javaArgs |
Additional Java Args to pass to the SSG process | see values.yaml |
The default Java Args are as follows
config:
heapSize: "2g"
javaArgs:
- -Dcom.l7tech.bootstrap.autoTrustSslKey=trustAnchor,TrustedFor.SSL,TrustedFor.SAML_ISSUER
- -Dcom.l7tech.server.audit.message.saveToInternal=false
- -Dcom.l7tech.server.audit.admin.saveToInternal=false
- -Dcom.l7tech.server.audit.system.saveToInternal=false
- -Dcom.l7tech.server.audit.log.format=json
- -Djava.util.logging.config.file=/opt/SecureSpan/Gateway/node/default/etc/conf/log-override.properties
- -Dcom.l7tech.server.pkix.useDefaultTrustAnchors=true
- -Dcom.l7tech.security.ssl.hostAllowWildcard=true
Additional System Properties as may be recommended by support can be configured in values.yaml
| Parameter | Description | Default |
|---|---|---|
config.systemProperties |
Gateway System Properties | see values.yaml |
The default systemProperties represent what is currently in the base Gateway image with one added flag
Period of time before the Gateway removes inactive nodes.
com.l7tech.server.clusterStaleNodeCleanupTimeoutSeconds=86400
The full default is this
systemProperties: |-
# Default Gateway system properties
# Configuration properties for shared state extensions.
com.l7tech.server.extension.sharedKeyValueStoreProvider=embeddedhazelcast
com.l7tech.server.extension.sharedCounterProvider=ssgdb
com.l7tech.server.extension.sharedClusterInfoProvider=ssgdb
# By default, FIPS module will block an RSA modulus from being used for encryption if it has been used for
# signing, or visa-versa. Set true to disable this default behaviour and remain backwards compatible.
com.safelogic.cryptocomply.rsa.allow_multi_use=true
# Specifies the type of Trust Store (JKS/PKCS12) provided by AdoptOpenJDK that is used by Gateway.
# Must be set correctly when Gateway is running in FIPS mode. If not specified it will default to PKCS12.
javax.net.ssl.trustStoreType=jks
# Period of time before the Gateway removes inactive nodes.
com.l7tech.server.clusterStaleNodeCleanupTimeoutSeconds=86400
# Additional properties go here
There are a variety of ways to mount Gateway (Restman format) Bundles to the Gateway Container. The best option is making use of existingBundles where the bundle has been created ahead of deployment as a configMap or secret. This allows for purpose built Gateways with a guaranteed set of configuration, apis/services.
| Parameter | Description | Default |
|---|---|---|
existingBundle.enabled |
Enable bundle mounts | false |
existingBundle.configMaps |
Array of configmap configurations | see values.yaml |
existingBundle.secrets |
Array of secret configurations | see values.yaml |
This example shows 1 secret and 1 configmap configured - you can also use the secrets-store.csi.k8s.io driver for bundles that contain sensitive information.
# Bundles that contain sensitive information can be mounted using the Kubernetes CSI Driver
existingBundle:
enabled: true
configMaps:
- name: mybundle1
configMap:
defaultMode: 420
optional: false
name: mybundle1
secrets:
- name: mysecretbundle1
# csi:
# driver: secrets-store.csi.k8s.io
# readOnly: true
# volumeAttributes:
# secretProviderClass: "secret-provider-class-name"
To reduce reliance on requiring a custom gateway image for custom and modular assertions, scripts and restman bundles a bootstrap script has been introduced. The script works with the /opt/docker/custom folder. The best way to populate this folder is with an initContainer where files can be copied directly across or dynamically loaded from an external source.
The following configuration enables the script
bootstrap:
script:
enabled: true
cleanup: false <== set this to true if you'd like to clear the /opt/docker/custom folder after it has run.
The bootstrap script scans files in /opt/docker/custom. This folder is populated by an initContainer.
The following folder stucture must be maintained
- Restman Bundles (.bundle)
- Source
/opt/docker/custom/bundles - Target
/opt/SecureSpan/Gateway/node/default/etc/bootstrap/bundle
- Source
- Custom Assertions (.jar)
- Source
/opt/docker/custom/custom-assertions - Target
/opt/SecureSpan/Gateway/runtime/modules/lib/
- Source
- Modular Assertions (.aar)
- Source
/opt/docker/custom/modular-assertions - Target
/opt/SecureSpan/Gateway/runtime/modules/assertions
- Source
- Properties (.properties)
- Source
/opt/docker/custom/properties - Target
/opt/SecureSpan/Gateway/node/default/etc/conf/
- Source
More information on how to use initContainers with examples can be found on the Layer7 Community Github Utilities Repository.
You can now specify a configMap or Secret that contains healthcheck scripts. These are mounted to /opt/docker/rc.d/diagnostic/health_check where they are run by /opt/docker/rc.d/diagnostic/health_check.sh.
- Limited to a single configmap or secret.
- ConfigMaps and Secrets can hold multiple scripts.
- See this example
NOTE: if you set a configMap and a Secret only one of them will be applied to your API Gateway.
existingHealthCheck:
enabled: false
configMap: {}
# name: healthcheck-scripts-configmap
# defaultMode: 292
# optional: false
secret: {}
# name: healthcheck-scripts-secret
# csi:
# driver: secrets-store.csi.k8s.io
# readOnly: true
# volumeAttributes:
# secretProviderClass: "vault-database"
Certain folders on the Container Gateway are not writeable by design. This configuration allows you to mount existing configMap/Secret keys to specific paths on the Gateway without the need for a root user or a custom/derived image.
customConfig:
enabled: false
# mounts:
# - name: sampletrafficloggerca-override
# mountPath: /opt/SecureSpan/Gateway/node/default/etc/conf/sampletrafficloggerca.properties
# subPath: sampletrafficloggerca.properties
# secret:
# name: config-override-secret
# item:
# key: sampletrafficloggerca.properties
# path: sampletrafficloggerca.properties
During upgrades and other events where Gateway pods are replaced you may have APIs/Services that have long running connections open.
This functionality delays Kubernetes sending a SIGTERM to the container gateway while connections remain open. This works in conjunction with terminationGracePeriodSeconds which should always be higher than preStopScript.timeoutSeconds. If preStopScript.timeoutSeconds is exceeded, the script will exit 0 and normal pod termination will resume.
The preStop script will monitor connections to inbound (not outbound) Gateway Application TCP ports (i.e. inbound listener ports opened by the Gateway Application and not some other process) except those that are explicitly excluded.
The following ports are excluded from monitoring by default.
- 8777 (Hazelcast) - Embedded Hazelcast.
- 2124 (Internode-Communication) - not utilised by the Container Gateway.
If there are no open connections, the preStop script will exit immediately ignoring preStopScript.timeoutSeconds to avoid unnecessary resource utilisation (pod stuck in terminating state) during upgrades.
While there aren't any explicit limits on preStopScript.timeoutSeconds and terminationGracePeriodSeconds running these for extended periods of time (i.e. more than 5 minutes) may be less reliable where other Kubernetes processes may remove the pod before terminationGracePeriodSeconds is reached. If you do run services like this we recommend testing before any real life implementation or better, creating a dedicated workload without autoscaling enabled (HPA) where you have more control over when/how pods are replaced.
The graceful termination (preStop script) is disabled by default.
| Parameter | Description | Default |
|---|---|---|
lifecycleHooks |
Custom lifecycle hooks, takes precedence over the preStopScript | {} |
preStopScript.enabled |
Enable the preStop script | false |
preStopScript.periodSeconds |
The time in seconds between checks | 3 |
preStopScript.timeoutSeconds |
Timeout - must be lower than terminationGracePeriodSeconds | 60 |
preStopScript.excludedPorts |
Array of ports that should be excluded from the preStop script check | [8777, 2124] |
terminationGracePeriodSeconds |
Default duration in seconds kubernetes waits for container to exit before sending kill signal. | see values.yaml |
Autoscaling is disabled by default, you will need metrics server in conjunction with the configuration below. In order for Kubernetes to determine when to scale, you will also need to configure resources
We do not recommend setting MaxReplicas for a MySQL backed API Gateway above 8.
| Parameter | Description | Default |
|---|---|---|
autoscaling.enabled |
Enable autoscaling | false |
autoscaling.hpa.minReplicas |
Minimum replicas that should be available | 1 |
autoscaling.hpa.maxReplicas |
Maximum replicas that should be available | 3 |
autoscaling.hpa.metrics |
Metrics to scale on | see values.yaml |
autoscaling.hpa.behaviour |
Scale up/down behaviour | see values.yaml |
Here is an example of a configured autoscaling section.
autoscaling:
enabled: true
hpa:
minReplicas: 1
maxReplicas: 3
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60
behavior:
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Pods
value: 1
periodSeconds: 60
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 15
PM Tagger requires access to pods in the current namespace, it uses the Gateway Configured service account.
| Parameter | Description | Default |
|---|---|---|
serviceAccount.create |
Create a service account for the Gateway | true |
serviceAccount.name |
Use an existing service account or specify the name of the service account that you would like to be created | nil |
rbac.create |
Create Role and Rolebinding for Gateway Service Account | true |
If you would like to create and use your own service account the Gateway with PM Tagger will require the following role to function correctly. This should NOT be a cluster role
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["list", "patch"]
The API Gateway containers are configured to output logs and audits as JSON events, and to never write audits to the in-memory Derby database:
- System properties in the default template for the
config.javaArgsvalue configure the log and audit behaviour:- Auditing to the database is disabled:
-Dcom.l7tech.server.audit.message.saveToInternal=false -Dcom.l7tech.server.audit.admin.saveToInternal=false -Dcom.l7tech.server.audit.system.saveToInternal=false - JSON formatting is enabled:
-Dcom.l7tech.server.audit.log.format=json - Default log output configuration is overridden by specifying an alternative configuration properties file:
-Djava.util.logging.config.file=/opt/SecureSpan/Gateway/node/default/etc/conf/log-override.properties
- Auditing to the database is disabled:
- The alternative log configuration properties file
log-override.propertiesis mounted on the container, via ConfigMap. - System property to include well known Certificate Authorities Trust Anchors
- API Gateway does not implicitly trust certificates without importing it but If you want to avoid import step then configure Gateway to accept any certificate signed by well known CA's (Certificate Authorities) configure following property to true - Set '-Dcom.l7tech.server.pkix.useDefaultTrustAnchors=true' for well known Certificate Authorities be included as Trust Anchors (true/false)
- Allow wildcards when verifying hostnames (true/false)
- Set '-Dcom.l7tech.security.ssl.hostAllowWildcard=true' to allow wildcards when verifying hostnames (true/false)
To deploy the service metrics example you will need to enable serviceMetrics, influxdb and grafana.
| Parameter | Description | Default |
|---|---|---|
serviceMetrics.enabled |
Enable the background metrics processing task | false |
serviceMetrics.external |
Point to an external influx database. Set influxDbUrl if true | false |
serviceMetrics.influxDbUrl |
InfluxDB URL | http://influxdb |
serviceMetrics.influxDbPort |
InfluxDB port | 8086 |
serviceMetrics.influxDbDatabase |
InfluxDB Database Name | serviceMetricsDb |
serviceMetrics.tags |
InfluxDB tags | env=dev |
influxdb.enabled |
Enable/Disable deployment of InfluxDb | false |
grafana.enabled |
Enable/Disable deployment of Grafana | false |
these do not represent production configurations
For Production implementations, please see the Chart links for recommended settings. The best approach would be deploying each independently MySQL doesn't have a tried and tested K8's production deployment so it's best to use an external service.
The following table lists the configured parameters of the MySQL Bitnami chart - https://github.com/bitnami/charts/tree/master/bitnami/mysql (DO NOT USE IN PRODUCTION!!)
| Parameter | Description | Default |
|---|---|---|
mysql.image.tag |
MySQL Image to use | 8 |
mysql.auth.username |
MySQL Username | gateway |
mysql.auth.password |
MySQL User Password | mypassword |
mysql.auth.database |
Database to create | ssg |
mysql.auth.rootPassword |
MySQL root password, not set by default | mypassword |
mysql.primary.persistence.enabled |
Enable/Disable Persistence | true |
mysql.primary.persistence.size |
Persistent Volume Size | 8Gi |
mysql.primary.persistence.storageClass |
Storage class to use | nil |
mysql.primary.configuration |
MySQL Configuration | see values.yaml |
The following table lists the configured parameters of the Hazelcast Subchart - see the following for more detail https://github.com/hazelcast/charts/blob/master/stable/hazelcast/values.yaml
| Parameter | Description | Default |
|---|---|---|
hazelcast.enabled |
Enable/Disable deployment of Hazelcast | false |
hazelcast.external |
Point to an external Hazelcast - set enabled to false and configure the url | false |
hazelcast.image.tag |
The Gateway currently supports Hazelcast 4.x/5.x servers. | 5.2.1 |
hazelcast.url |
External Hazelcast Url | hazelcast.example.com:5701 |
hazelcast.cluster.memberCount |
Number of Hazelcast Replicas you wish to deploy | see values.yaml |
hazelcast.hazelcast.yaml |
Hazelcast configuration | see the documentation link |
The following table lists the configured parameters of the InfluxDb Subchart - see the following for more detail https://github.com/influxdata/helm-charts/tree/master/charts/influxdb
| Parameter | Description | Default |
|---|---|---|
influxdb.enabled |
Enable/Disable deployment of InfluxDb | false |
influxdb.service.port |
Service Port | 8086 |
influxdb.persistence.enabled |
Enable Persistence for InfluxDb | true |
influxdb.persistence.storageClass |
Persistence Storage Class | nil |
influxdb.persistence.size |
Persistent Volume Claim Size | 8Gi |
influxdb.env |
Array of additional environment variables | see values.yaml |
The following table lists the configured parameters of the Grafana Subchart - see the following for more detail https://github.com/bitnami/charts/tree/master/bitnami/grafana
| Parameter | Description | Default |
|---|---|---|
grafana.enabled |
Enable/Disable deployment of Grafana | false |
grafana.admin.user |
admin username | admin |
grafana.admin.password |
admin password, leave blank to auto-generate | password |
grafana.dashboardsProvider.enabled |
Allows dashboards to be preloaded | true |
grafana.dashboardsConfigMaps |
Configmaps that contain grafana dashboards. You can create your own and set here. | see values.yaml |
grafana.datasources.secretName |
Configures an InfluxDb Datasource. | see values.yaml |
- Hazelcast (default: disabled) ==> https://github.com/helm/charts/tree/master/stable/hazelcast
- MySQL (default: enabled) ==> https://github.com/bitnami/charts/tree/master/bitnami/mysql
- InfluxDb (default: disabled) ==> https://github.com/influxdata/helm-charts/tree/master/charts/influxdb
- Grafana (default: disabled) ==> https://github.com/bitnami/charts/tree/master/bitnami/grafana