Skip to content

Commit aea4a32

Browse files
yhl25vigith
andauthored
doc: Fallback Sink (numaproj#1691)
Signed-off-by: Yashash H L <[email protected]> Signed-off-by: Vigith Maurice <[email protected]> Co-authored-by: Vigith Maurice <[email protected]>
1 parent 2f854b0 commit aea4a32

File tree

2 files changed

+86
-0
lines changed

2 files changed

+86
-0
lines changed

docs/user-guide/sinks/fallback.md

+51
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# Fallback Sink
2+
3+
A `Fallback` Sink functions as a `Dead Letter Queue (DLQ)` Sink and can be configured to serve as a backup when the primary sink is down,
4+
unavailable, or under maintenance. This is particularly useful when multiple sinks are in a pipeline; if a sink fails, the resulting
5+
back-pressure will back-propagate and stop the source vertex from reading more data. A `Fallback` Sink can beset up to prevent this from happening.
6+
This backup sink stores data while the primary sink is offline. The stored data can be replayed once the primary sink is back online.
7+
8+
Note: The `fallback` field is optional.
9+
10+
Users are required to return a fallback response from the [user-defined sink](https://numaflow.numaproj.io/user-guide/sinks/user-defined-sinks/) when the primary sink fails; only
11+
then the messages will be directed to the fallback sink.
12+
13+
Example of a fallback response in a user-defined sink: [here](https://github.com/numaproj/numaflow-go/blob/main/pkg/sinker/examples/fallback/main.go)
14+
15+
## CAVEATs
16+
The `fallback` field can only be utilized when the primary sink is a `User Defined Sink.`
17+
18+
19+
## Example
20+
21+
### Builtin Kafka
22+
An example using builtin kafka as fallback sink:
23+
24+
```yaml
25+
- name: out
26+
sink:
27+
udsink:
28+
container:
29+
image: my-sink:latest
30+
fallback:
31+
kafka:
32+
brokers:
33+
- my-broker1:19700
34+
- my-broker2:19700
35+
topic: my-topic
36+
```
37+
### UD Sink
38+
An example using custom user-defined sink as fallback sink.
39+
40+
User Defined Sink as a fallback sink:
41+
```yaml
42+
- name: out
43+
sink:
44+
udsink:
45+
container:
46+
image: my-sink:latest
47+
fallback:
48+
udsink:
49+
container:
50+
image: my-sink:latest
51+
```
+35
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
apiVersion: numaflow.numaproj.io/v1alpha1
2+
kind: Pipeline
3+
metadata:
4+
name: simple-fallback
5+
spec:
6+
vertices:
7+
- name: in
8+
source:
9+
# A self data generating source
10+
generator:
11+
rpu: 5
12+
duration: 1s
13+
- name: cat
14+
scale:
15+
min: 1
16+
udf:
17+
builtin:
18+
name: cat
19+
- name: output
20+
scale:
21+
min: 1
22+
sink:
23+
udsink:
24+
container:
25+
image: quay.io/numaio/numaflow-go/fb-sink-log:v0.7.0-rc2
26+
imagePullPolicy: Always
27+
fallback:
28+
udsink:
29+
container:
30+
image: quay.io/numaio/numaflow-go/fb-sink-log:v0.7.0-rc2
31+
edges:
32+
- from: in
33+
to: cat
34+
- from: cat
35+
to: output

0 commit comments

Comments
 (0)