Skip to content

Commit 754f128

Browse files
weltekiwelteki
committed
Update kafka-connector chart for async invocations with backpressure
Signed-off-by: Han Verstraete (OpenFaaS Ltd) <[email protected]>
1 parent a382352 commit 754f128

File tree

5 files changed

+126
-36
lines changed

5 files changed

+126
-36
lines changed

‎chart/kafka-connector/README.md

Lines changed: 38 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -47,9 +47,9 @@ $ helm repo add openfaas https://openfaas.github.io/faas-netes/
4747

4848
Prepare a custom [values.yaml](values.yaml) with:
4949

50-
* brokerHosts - comma separted list of host:port
51-
* topics - the topics to subscribe to
52-
* replicas - this should match the partition size, so if the size is 3, set this to 3
50+
- brokerHosts - comma separted list of host:port
51+
- topics - the topics to subscribe to
52+
- replicas - this should match the partition size, so if the size is 3, set this to 3
5353

5454
Then you will need to read up on the encryption and authentication options and update the settings accordingly.
5555

@@ -65,46 +65,49 @@ $ helm repo update && \
6565
6666
## Encryption options
6767

68-
1) TLS off (default)
69-
2) TLS on
68+
1. TLS off (default)
69+
2. TLS on
7070

7171
## Authentication options
7272

73-
1) TLS with SASL using CA from the default trust store
74-
3) TLS with SASL using a custom CA
75-
4) TLS with client certificates
73+
1. TLS with SASL using CA from the default trust store
74+
2. TLS with SASL using a custom CA
75+
3. TLS with client certificates
7676

7777
## Configuration
7878

7979
Additional kafka-connector options in `values.yaml`.
8080

81-
| Parameter | Description | Default |
82-
|------------------------|------------------------------------------------------------------------------------------------------------------------------------|--------------------------------|
83-
| `topics` | A single topic or list of comma separated topics to consume. | `faas-request` |
84-
| `replicas` | The number of replicas of this connector, should be set to the size of the partition for the given topic, or a higher lower value. | `1` |
85-
| `brokerHosts` | Host and port for the Kafka bootstrap server, multiple servers can be specified as a comma-separated list. | `kafka:9092` |
86-
| `asyncInvocation` | For long running or slow functions, offload to asychronous function invocations and carry on processing the stream | `false` |
87-
| `upstreamTimeout` | Maximum timeout for upstream function call, must be a Go formatted duration string. | `2m` |
88-
| `rebuildInterval` | Interval for rebuilding function to topic map, must be a Go formatted duration string. | `30s` |
89-
| `gatewayURL` | The URL for the API gateway. | `http://gateway.openfaas:8080` |
90-
| `printResponse` | Output the response of calling a function in the logs. | `true` |
91-
| `printResponseBody` | Output to the logs the response body when calling a function. | `false` |
92-
| `printRequestBody` | Output to the logs the request body when calling a function. | `false` |
93-
| `fullnameOverride` | Override the name value used for the Connector Deployment object. | `` |
94-
| `tls` | Connect to the broker server(s) using TLS encryption | `true` |
95-
| `sasl` | Enable auth with a SASL username/password | `false` |
96-
| `brokerPasswordSecret` | Name of secret for SASL password | `kafka-broker-password` |
97-
| `brokerUsernameSecret` | Name of secret for SASL username | `kafka-broker-username` |
98-
| `caSecret` | Name secret for TLS CA - leave empty to disable | `kafka-broker-ca` |
99-
| `certSecret` | Name secret for TLS client certificate cert - leave empty to disable | `kafka-broker-cert` |
100-
| `keySecret` | Name secret for TLS client certificate private key - leave empty to disable | `kafka-broker-key` |
101-
| `contentType` | Set a HTTP Content Type during function invocation. | `""` |
102-
| `group` | Set the Kafka consumer group name. | `""` |
103-
| `maxBytes` | Set the maximum size of messages from the Kafka broker. | `1024*1024` |
104-
| `sessionLogging` | Enable detailed logging from the consumer group. | `"false"` |
105-
| `initialOffset` | Either newest or oldest. | `"oldest"` |
106-
| `logs.debug` | Print debug logs | `false` |
107-
| `logs.format` | The log encoding format. Supported values: `json` or `console` | `console` |
81+
| Parameter | Description | Default |
82+
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
83+
| `topics` | A single topic or list of comma separated topics to consume. | `faas-request` |
84+
| `replicas` | The number of replicas of this connector, should be set to the size of the partition for the given topic, or a higher lower value. | `1` |
85+
| `brokerHosts` | Host and port for the Kafka bootstrap server, multiple servers can be specified as a comma-separated list. | `kafka:9092` |
86+
| `asyncInvocation` | Invoke function asychronously and carry on processing the stream | `false` |
87+
| `asyncMaxInflight` | Limit the number of inflight async invocations for the connector. A value of 0 indicates no concurrency limit. | `0` |
88+
| `asyncCallbackURL` | Override the callback url passed as the X-Callback-URl header for async invocations to notify the connector an async invocation has completed. | `""` |
89+
| `natsURL` | URL used to connect to nats to count the async concurrency. Override this value if you are using external nats. | `nats://nats.openfaas:4222` |
90+
| `upstreamTimeout` | Maximum timeout for upstream function call, must be a Go formatted duration string. | `2m` |
91+
| `rebuildInterval` | Interval for rebuilding function to topic map, must be a Go formatted duration string. | `30s` |
92+
| `gatewayURL` | The URL for the API gateway. | `http://gateway.openfaas:8080` |
93+
| `printResponse` | Output the response of calling a function in the logs. | `true` |
94+
| `printResponseBody` | Output to the logs the response body when calling a function. | `false` |
95+
| `printRequestBody` | Output to the logs the request body when calling a function. | `false` |
96+
| `fullnameOverride` | Override the name value used for the Connector Deployment object. | `` |
97+
| `tls` | Connect to the broker server(s) using TLS encryption | `true` |
98+
| `sasl` | Enable auth with a SASL username/password | `false` |
99+
| `brokerPasswordSecret` | Name of secret for SASL password | `kafka-broker-password` |
100+
| `brokerUsernameSecret` | Name of secret for SASL username | `kafka-broker-username` |
101+
| `caSecret` | Name secret for TLS CA - leave empty to disable | `kafka-broker-ca` |
102+
| `certSecret` | Name secret for TLS client certificate cert - leave empty to disable | `kafka-broker-cert` |
103+
| `keySecret` | Name secret for TLS client certificate private key - leave empty to disable | `kafka-broker-key` |
104+
| `contentType` | Set a HTTP Content Type during function invocation. | `""` |
105+
| `group` | Set the Kafka consumer group name. | `""` |
106+
| `maxBytes` | Set the maximum size of messages from the Kafka broker. | `1024*1024` |
107+
| `sessionLogging` | Enable detailed logging from the consumer group. | `"false"` |
108+
| `initialOffset` | Either newest or oldest. | `"oldest"` |
109+
| `logs.debug` | Print debug logs | `false` |
110+
| `logs.format` | The log encoding format. Supported values: `json` or `console` | `console` |
108111

109112
Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. See `values.yaml` for the default configuration.
110113

‎chart/kafka-connector/templates/deployment.yml

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ spec:
3232
app: {{ template "connector.name" . }}
3333
component: kafka-connector
3434
spec:
35+
serviceAccountName: {{ template "connector.fullname" . }}
3536
volumes:
3637
- name: openfaas-license
3738
secret:
@@ -87,6 +88,12 @@ spec:
8788
- "-key-file=/var/secrets/broker-key/broker-key"
8889
{{- end }}
8990
env:
91+
- name: connector_id
92+
value: "{{template "connector.fullname" . }}"
93+
- name: namespace
94+
valueFrom:
95+
fieldRef:
96+
fieldPath: metadata.namespace
9097
- name: gateway_url
9198
value: {{ .Values.gatewayURL | quote }}
9299
- name: topics
@@ -99,6 +106,16 @@ spec:
99106
value: {{ .Values.printRequestBody | quote }}
100107
- name: asynchronous_invocation
101108
value: {{ .Values.asyncInvocation | quote }}
109+
- name: async_max_inflight
110+
value: {{ .Values.asyncMaxInflight | quote }}
111+
- name: async_callback_url
112+
{{- if .Values.asyncCallbackURL }}
113+
value: {{ .Values.asyncCallbackURL | quote }}
114+
{{- else}}
115+
value: "http://{{ template "connector.fullname" . }}.{{ .Release.Namespace }}:8080/api/v1/callback"
116+
{{- end}}
117+
- name: nats_url
118+
value: {{ .Values.natsURL | quote }}
102119
{{- if .Values.basic_auth }}
103120
- name: basic_auth
104121
value: "true"

‎chart/kafka-connector/templates/rbac.yaml

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
apiVersion: v1
2+
kind: ServiceAccount
3+
metadata:
4+
name: {{ template "connector.fullname" . }}
5+
namespace: {{ .Release.Namespace | quote }}
6+
labels:
7+
app: {{ template "connector.fullname" . }}
8+
component: kafka-connector
9+
---
10+
apiVersion: rbac.authorization.k8s.io/v1
11+
kind: Role
12+
metadata:
13+
name: {{ template "connector.fullname" . }}
14+
namespace: {{ .Release.Namespace | quote }}
15+
labels:
16+
app: {{ template "connector.name" . }}
17+
component: kafka-connector
18+
rules:
19+
- apiGroups: ["coordination.k8s.io"]
20+
resources: ["leases"]
21+
verbs: ["get", "create"]
22+
---
23+
apiVersion: rbac.authorization.k8s.io/v1
24+
kind: RoleBinding
25+
metadata:
26+
name: {{ template "connector.fullname" . }}
27+
namespace: {{ .Release.Namespace | quote }}
28+
labels:
29+
app: {{ template "connector.fullname" . }}
30+
component: kafka-connector
31+
subjects:
32+
- kind: ServiceAccount
33+
name: {{ template "connector.fullname" . }}
34+
namespace: {{ .Release.Namespace | quote }}
35+
roleRef:
36+
kind: Role
37+
name: {{ template "connector.fullname" . }}
38+
apiGroup: rbac.authorization.k8s.io

‎chart/kafka-connector/templates/service.yaml

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
apiVersion: v1
2+
kind: Service
3+
metadata:
4+
labels:
5+
app: {{ template "connector.name" . }}
6+
component: kafka-connector
7+
chart: {{ .Chart.Name }}-{{ .Chart.Version }}
8+
heritage: {{ .Release.Service }}
9+
release: {{ .Release.Name }}
10+
name: {{ template "connector.fullname" . }}
11+
namespace: {{ .Release.Namespace | quote }}
12+
spec:
13+
type: ClusterIP
14+
ports:
15+
- name: http
16+
port: 8080
17+
protocol: TCP
18+
targetPort: 8080
19+
selector:
20+
app: kafka-connector

‎chart/kafka-connector/values.yaml

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,9 +30,21 @@ upstreamTimeout: 2m
3030
# interval for rebuilding the map of functions and topics
3131
rebuildInterval: 30s
3232

33-
# Use with slow consumers or long running functions
33+
# Invoke functions asynchronously.
3434
asyncInvocation: false
3535

36+
# Limit the number of inflight async invocations for the connector.
37+
# A value of 0 indicates no concurrency limit.
38+
asyncMaxInflight: 0
39+
40+
# Override the callback url passed as the X-Callback-URl header for async invocations
41+
# to notify the connector an async invocation has completed.
42+
asyncCallbackURL: ""
43+
44+
# URL used to connect to nats to count the async concurrency.
45+
# Override this value if you are using external nats.
46+
natsURL: "nats://nats.openfaas:4222"
47+
3648
# 1MB = 1024 bytes * 1024
3749
maxBytes: "1048576"
3850

0 commit comments

Comments
 (0)