Skip to content

Commit 5b32cc8

Browse files
authored
Manage Kafka topics on Confluent Cloud (#149)
* Confluent Cloud topic creation * updated the format * minor formatting changes
1 parent f43c68c commit 5b32cc8

File tree

3 files changed

+108
-0
lines changed

3 files changed

+108
-0
lines changed

hybrid/ccloud-topic/README.md

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Create and manage topic on Confluent Cloud
2+
3+
You can use Confluent for Kubernetes to create, edit, and delete the topic on Confluent Cloud.
4+
5+
Before continuing with the scenario, ensure that you have set up the
6+
[prerequisites](https://github.com/confluentinc/confluent-kubernetes-examples/blob/master/README.md#prerequisites).
7+
8+
## Set the current tutorial directory
9+
10+
Set the tutorial directory for this tutorial under the directory you downloaded
11+
the tutorial files:
12+
```
13+
export TUTORIAL_HOME=<Tutorial directory>/hybrid/ccloud-topic
14+
```
15+
16+
## Deploy Confluent for Kubernetes
17+
18+
Set up the Helm Chart:
19+
```
20+
helm repo add confluentinc https://packages.confluent.io/helm
21+
```
22+
23+
Install Confluent for Kubernetes using Helm:
24+
```
25+
helm upgrade --install operator confluentinc/confluent-for-kubernetes --namespace confluent
26+
```
27+
28+
Check that the Confluent for Kubernetes pod comes up and is running:
29+
```
30+
kubectl get pods --namespace confluent
31+
```
32+
33+
## Create authentication credentials
34+
35+
Confluent Cloud provides an API key/secret for Kafka. Create a Kubernetes secret object for Confluent Cloud Kafka access.
36+
This secret object contains file based properties.
37+
```
38+
kubectl create secret generic cloud-rest-access \
39+
--from-file=basic.txt=$TUTORIAL_HOME/creds-kafka-sasl-user.txt --namespace confluent
40+
```
41+
42+
## Create Kafka Topic
43+
44+
Edit the ``topic.yaml`` custom resource file, and update the details in the following places:
45+
46+
- ``<rest-endpoint>`` : Admin REST APIs endpoint.
47+
- ``<cluster-id>`` : ID of the Confluent Cloud Kafka cluster
48+
49+
Create the Kafka topic:
50+
```
51+
kubectl apply -f $TUTORIAL_HOME/topic.yaml
52+
```
53+
54+
## Update Kafka Topic
55+
56+
You can update the editable settings of the topic in the topic CR, `topic.yaml`. To update the kafka topic config, add the config under `spec.configs` in the kafka topic custom resource file, apply the changes using the `kubectl apply -f topic.yaml` command.
57+
58+
Limitations:
59+
60+
- [Custom topic settings for all cluster types](https://docs.confluent.io/cloud/current/clusters/broker-config.html#custom-topic-settings-for-all-cluster-types)
61+
- You cannot update topics with the `_` character in the topic name.
62+
- `spec.replicas` and `spec.partitionCount` cannot be updated using KafkaTopic CR.
63+
64+
## Validate
65+
66+
### Validate in Confluent Cloud Console
67+
68+
- Sign in to your Confluent account.
69+
- If you have more than one environment, select an environment.
70+
- Select a cluster from the navigation bar and click the `Topics` menu.
71+
- The Topics page displays the created topic
72+
- Select the created topic, and view the `Configuration` of the topic from the Confluent Cloud Console.
73+
74+
## Tear down
75+
76+
- This command will delete the topic:
77+
```
78+
kubectl delete -f $TUTORIAL_HOME/test.yaml
79+
```
80+
81+
- Delete the secret:
82+
```
83+
kubectl delete secrets cloud-rest-access --namespace confluent
84+
```
85+
86+
- Uninstall the operator
87+
```
88+
helm delete operator --namespace confluent
89+
```
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
username=<api-key>
2+
password=<api-secret>

hybrid/ccloud-topic/topic.yaml

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
apiVersion: platform.confluent.io/v1beta1
2+
kind: KafkaTopic
3+
metadata:
4+
name: cloud-demo-topic
5+
namespace: confluent
6+
spec:
7+
replicas: 3
8+
partitionCount: 4
9+
configs:
10+
message.timestamp.type: "LogAppendTime"
11+
kafkaRest:
12+
endpoint: <rest-endpoint>
13+
kafkaClusterID: <cluster-id>
14+
authentication:
15+
type: basic
16+
basic:
17+
secretRef: cloud-rest-access

0 commit comments

Comments
 (0)