Skip to content

Commit dddbc55

Browse files
weltekiwelteki
committed
Update kafka-connector chart for async invocations with backpressure
Signed-off-by: Han Verstraete (OpenFaaS Ltd) <[email protected]>
1 parent a382352 commit dddbc55

File tree

5 files changed

+192
-37
lines changed

5 files changed

+192
-37
lines changed

‎chart/kafka-connector/README.md

Lines changed: 94 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -47,9 +47,9 @@ $ helm repo add openfaas https://openfaas.github.io/faas-netes/
4747

4848
Prepare a custom [values.yaml](values.yaml) with:
4949

50-
* brokerHosts - comma separted list of host:port
51-
* topics - the topics to subscribe to
52-
* replicas - this should match the partition size, so if the size is 3, set this to 3
50+
- brokerHosts - comma separted list of host:port
51+
- topics - the topics to subscribe to
52+
- replicas - this should match the partition size, so if the size is 3, set this to 3
5353

5454
Then you will need to read up on the encryption and authentication options and update the settings accordingly.
5555

@@ -65,46 +65,105 @@ $ helm repo update && \
6565
6666
## Encryption options
6767

68-
1) TLS off (default)
69-
2) TLS on
68+
1. TLS off (default)
69+
2. TLS on
7070

7171
## Authentication options
7272

73-
1) TLS with SASL using CA from the default trust store
74-
3) TLS with SASL using a custom CA
75-
4) TLS with client certificates
73+
1. TLS with SASL using CA from the default trust store
74+
2. TLS with SASL using a custom CA
75+
3. TLS with client certificates
76+
77+
## Async invocations
78+
79+
The connector can be configured to invoke function asynchronously. This lets you use [OpenFaaS async](https://docs.openfaas.com/reference/async/) features like retries.
80+
To prevent the connector from consuming all Kafka messages at once and submitting them to the OpenFaaS async queue a limit on the number of inflight async invocations can be configured.
81+
82+
Configure the connector for async invocations:
83+
84+
```yaml
85+
# Invoke functions asynchronously.
86+
asyncInvocation: true
87+
88+
async:
89+
# Limit the number of inflight async invocations for the connector.
90+
# A value of 0 indicates no concurrency limit.
91+
maxInflight: 0
92+
93+
# Configure an externally-managed NATS server.
94+
# NATS is used for async invocations and is required when
95+
# setting the 'async.maxInflight' parameter to a value other than 0.
96+
# By default the OpenFaaS embedded nats deployment is used.
97+
# These values should be identical to the configuration in your OpenFaaS deployment values.yaml file
98+
# when external nats is enabled.
99+
nats:
100+
external:
101+
enabled: false
102+
host: ""
103+
port: ""
104+
```
105+
106+
### Reset the inflight concurrency counter
107+
108+
If the inflight counter gets out if sync for some reason, e.g a misconfiguration, network issues, it can be forcefully reset.
109+
The connecter checks if a Lease object exists on startup and resets the counter if the Lease does not exist.
110+
111+
Remove the lease and restart the connector to reset the counter.
112+
113+
1. Remove the lease
114+
115+
```sh
116+
$ kubectl get lease -n openfaas
117+
118+
NAME HOLDER AGE
119+
kafka-connector 18m
120+
```
121+
122+
```sh
123+
kubectl delete lease kafka-connector -n openfaas
124+
```
125+
126+
2. Restart the connector
127+
128+
```sh
129+
kubectl rollout restart deploy/kafka-connector -n openfaas
130+
```
76131

77132
## Configuration
78133

79134
Additional kafka-connector options in `values.yaml`.
80135

81-
| Parameter | Description | Default |
82-
|------------------------|------------------------------------------------------------------------------------------------------------------------------------|--------------------------------|
83-
| `topics` | A single topic or list of comma separated topics to consume. | `faas-request` |
84-
| `replicas` | The number of replicas of this connector, should be set to the size of the partition for the given topic, or a higher lower value. | `1` |
85-
| `brokerHosts` | Host and port for the Kafka bootstrap server, multiple servers can be specified as a comma-separated list. | `kafka:9092` |
86-
| `asyncInvocation` | For long running or slow functions, offload to asychronous function invocations and carry on processing the stream | `false` |
87-
| `upstreamTimeout` | Maximum timeout for upstream function call, must be a Go formatted duration string. | `2m` |
88-
| `rebuildInterval` | Interval for rebuilding function to topic map, must be a Go formatted duration string. | `30s` |
89-
| `gatewayURL` | The URL for the API gateway. | `http://gateway.openfaas:8080` |
90-
| `printResponse` | Output the response of calling a function in the logs. | `true` |
91-
| `printResponseBody` | Output to the logs the response body when calling a function. | `false` |
92-
| `printRequestBody` | Output to the logs the request body when calling a function. | `false` |
93-
| `fullnameOverride` | Override the name value used for the Connector Deployment object. | `` |
94-
| `tls` | Connect to the broker server(s) using TLS encryption | `true` |
95-
| `sasl` | Enable auth with a SASL username/password | `false` |
96-
| `brokerPasswordSecret` | Name of secret for SASL password | `kafka-broker-password` |
97-
| `brokerUsernameSecret` | Name of secret for SASL username | `kafka-broker-username` |
98-
| `caSecret` | Name secret for TLS CA - leave empty to disable | `kafka-broker-ca` |
99-
| `certSecret` | Name secret for TLS client certificate cert - leave empty to disable | `kafka-broker-cert` |
100-
| `keySecret` | Name secret for TLS client certificate private key - leave empty to disable | `kafka-broker-key` |
101-
| `contentType` | Set a HTTP Content Type during function invocation. | `""` |
102-
| `group` | Set the Kafka consumer group name. | `""` |
103-
| `maxBytes` | Set the maximum size of messages from the Kafka broker. | `1024*1024` |
104-
| `sessionLogging` | Enable detailed logging from the consumer group. | `"false"` |
105-
| `initialOffset` | Either newest or oldest. | `"oldest"` |
106-
| `logs.debug` | Print debug logs | `false` |
107-
| `logs.format` | The log encoding format. Supported values: `json` or `console` | `console` |
136+
| Parameter | Description | Default |
137+
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
138+
| `topics` | A single topic or list of comma separated topics to consume. | `faas-request` |
139+
| `replicas` | The number of replicas of this connector, should be set to the size of the partition for the given topic, or a higher lower value. | `1` |
140+
| `brokerHosts` | Host and port for the Kafka bootstrap server, multiple servers can be specified as a comma-separated list. | `kafka:9092` |
141+
| `asyncInvocation` | Invoke function asychronously and carry on processing the stream | `false` |
142+
| `async.maxInflight` | Limit the number of inflight async invocations for the connector. A value of 0 indicates no concurrency limit. | `0` |
143+
| `nats.external.enabled` | Whether to use an externally-managed NATS server. | `false` |
144+
| `nats.external.host` | The host at which the externally-managed NATS server can be reached | `""` |
145+
| `nats.external.port` | The port at which the externally-managed NATS server can be reached | `""` |
146+
| `upstreamTimeout` | Maximum timeout for upstream function call, must be a Go formatted duration string. | `2m` |
147+
| `rebuildInterval` | Interval for rebuilding function to topic map, must be a Go formatted duration string. | `30s` |
148+
| `gatewayURL` | The URL for the API gateway. | `http://gateway.openfaas:8080` |
149+
| `printResponse` | Output the response of calling a function in the logs. | `true` |
150+
| `printResponseBody` | Output to the logs the response body when calling a function. | `false` |
151+
| `printRequestBody` | Output to the logs the request body when calling a function. | `false` |
152+
| `fullnameOverride` | Override the name value used for the Connector Deployment object. | `""` |
153+
| `tls` | Connect to the broker server(s) using TLS encryption | `true` |
154+
| `sasl` | Enable auth with a SASL username/password | `false` |
155+
| `brokerPasswordSecret` | Name of secret for SASL password | `kafka-broker-password` |
156+
| `brokerUsernameSecret` | Name of secret for SASL username | `kafka-broker-username` |
157+
| `caSecret` | Name secret for TLS CA - leave empty to disable | `kafka-broker-ca` |
158+
| `certSecret` | Name secret for TLS client certificate cert - leave empty to disable | `kafka-broker-cert` |
159+
| `keySecret` | Name secret for TLS client certificate private key - leave empty to disable | `kafka-broker-key` |
160+
| `contentType` | Set a HTTP Content Type during function invocation. | `""` |
161+
| `group` | Set the Kafka consumer group name. | `""` |
162+
| `maxBytes` | Set the maximum size of messages from the Kafka broker. | `1024*1024` |
163+
| `sessionLogging` | Enable detailed logging from the consumer group. | `"false"` |
164+
| `initialOffset` | Either newest or oldest. | `"oldest"` |
165+
| `logs.debug` | Print debug logs | `false` |
166+
| `logs.format` | The log encoding format. Supported values: `json` or `console` | `console` |
108167

109168
Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. See `values.yaml` for the default configuration.
110169

‎chart/kafka-connector/templates/deployment.yml

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,9 @@ spec:
3232
app: {{ template "connector.name" . }}
3333
component: kafka-connector
3434
spec:
35+
{{- if and .Values.asyncInvocation (gt (int .Values.async.maxInflight) 0) }}
36+
serviceAccountName: {{ template "connector.fullname" . }}
37+
{{- end }}
3538
volumes:
3639
- name: openfaas-license
3740
secret:
@@ -87,6 +90,12 @@ spec:
8790
- "-key-file=/var/secrets/broker-key/broker-key"
8891
{{- end }}
8992
env:
93+
- name: connector_id
94+
value: "{{template "connector.fullname" . }}"
95+
- name: namespace
96+
valueFrom:
97+
fieldRef:
98+
fieldPath: metadata.namespace
9099
- name: gateway_url
91100
value: {{ .Values.gatewayURL | quote }}
92101
- name: topics
@@ -99,6 +108,16 @@ spec:
99108
value: {{ .Values.printRequestBody | quote }}
100109
- name: asynchronous_invocation
101110
value: {{ .Values.asyncInvocation | quote }}
111+
- name: async_max_inflight
112+
value: {{ .Values.async.maxInflight | quote }}
113+
- name: async_callback_url
114+
value: "http://{{ template "connector.fullname" . }}.{{ .Release.Namespace }}:8080/api/v1/callback"
115+
- name: nats_url
116+
{{- if .Values.nats.external.enabled }}
117+
value: "nats://{{ .Values.nats.external.host}}:{{ .Values.nats.external.port }}"
118+
{{- else }}
119+
value: "nats://nats.openfaas:4222"
120+
{{- end }}
102121
{{- if .Values.basic_auth }}
103122
- name: basic_auth
104123
value: "true"

‎chart/kafka-connector/templates/rbac.yaml

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
{{- if and .Values.asyncInvocation (gt (int .Values.async.maxInflight) 0) }}
2+
apiVersion: v1
3+
kind: ServiceAccount
4+
metadata:
5+
name: {{ template "connector.fullname" . }}
6+
namespace: {{ .Release.Namespace | quote }}
7+
labels:
8+
app: {{ template "connector.fullname" . }}
9+
component: kafka-connector
10+
---
11+
apiVersion: rbac.authorization.k8s.io/v1
12+
kind: Role
13+
metadata:
14+
name: {{ template "connector.fullname" . }}
15+
namespace: {{ .Release.Namespace | quote }}
16+
labels:
17+
app: {{ template "connector.name" . }}
18+
component: kafka-connector
19+
rules:
20+
- apiGroups: ["coordination.k8s.io"]
21+
resources: ["leases"]
22+
verbs: ["get", "create"]
23+
---
24+
apiVersion: rbac.authorization.k8s.io/v1
25+
kind: RoleBinding
26+
metadata:
27+
name: {{ template "connector.fullname" . }}
28+
namespace: {{ .Release.Namespace | quote }}
29+
labels:
30+
app: {{ template "connector.fullname" . }}
31+
component: kafka-connector
32+
subjects:
33+
- kind: ServiceAccount
34+
name: {{ template "connector.fullname" . }}
35+
namespace: {{ .Release.Namespace | quote }}
36+
roleRef:
37+
kind: Role
38+
name: {{ template "connector.fullname" . }}
39+
apiGroup: rbac.authorization.k8s.io
40+
{{- end }}

‎chart/kafka-connector/templates/service.yaml

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
apiVersion: v1
2+
kind: Service
3+
metadata:
4+
labels:
5+
app: {{ template "connector.name" . }}
6+
component: kafka-connector
7+
chart: {{ .Chart.Name }}-{{ .Chart.Version }}
8+
heritage: {{ .Release.Service }}
9+
release: {{ .Release.Name }}
10+
name: {{ template "connector.fullname" . }}
11+
namespace: {{ .Release.Namespace | quote }}
12+
spec:
13+
type: ClusterIP
14+
ports:
15+
- name: http
16+
port: 8080
17+
protocol: TCP
18+
targetPort: 8080
19+
selector:
20+
app: kafka-connector

‎chart/kafka-connector/values.yaml

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,9 +30,14 @@ upstreamTimeout: 2m
3030
# interval for rebuilding the map of functions and topics
3131
rebuildInterval: 30s
3232

33-
# Use with slow consumers or long running functions
33+
# Invoke functions asynchronously.
3434
asyncInvocation: false
3535

36+
async:
37+
# Limit the number of inflight async invocations for the connector.
38+
# A value of 0 indicates no concurrency limit.
39+
maxInflight: 0
40+
3641
# 1MB = 1024 bytes * 1024
3742
maxBytes: "1048576"
3843

@@ -76,6 +81,18 @@ gatewayURL: http://gateway.openfaas:8080
7681
# Basic auth for the gateway
7782
basic_auth: true
7883

84+
# NATS is used for async invocations and is required when
85+
# setting the 'async.maxInflight' parameter to a value other than 0.
86+
nats:
87+
# Configure an externally-managed NATS server.
88+
# When disabled the OpenFaaS embedded nats deployment is used.
89+
# These values should be identical to the configuration in your OpenFaaS deployment values.yaml file
90+
# when external nats is enabled.
91+
external:
92+
enabled: false
93+
host: ""
94+
port: ""
95+
7996
nodeSelector: {}
8097

8198
tolerations: []
@@ -138,4 +155,4 @@ keySecret: ""
138155

139156
# caSecret: kafka-broker-ca
140157
# certSecret: kafka-broker-cert
141-
# keySecret: kafka-broker-key
158+
# keySecret: kafka-broker-key

0 commit comments

Comments
 (0)