Skip to content

Commit 6fd955f

Browse files
Maksim DavydovMaksim Davydov
Maksim Davydov
authored and
Maksim Davydov
committed
* doc update
1 parent b9889ce commit 6fd955f

File tree

1 file changed

+79
-9
lines changed

1 file changed

+79
-9
lines changed

docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc

Lines changed: 79 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ Add link:https://github.com/apache/ignite-extensions/blob/master/modules/cdc-ext
8080
</property>
8181

8282
<property name="onlyPrimary" value="false"/>
83-
<property name="maxBatchSize" value="256"/>
83+
<property name="maxBatchSize" value="1024"/>
8484
</bean>
8585
```
8686

@@ -142,7 +142,7 @@ Add link:https://github.com/apache/ignite-extensions/blob/master/modules/cdc-ext
142142
</property>
143143

144144
<property name="onlyPrimary" value="false"/>
145-
<property name="maxBatchSize" value="256"/>
145+
<property name="maxBatchSize" value="1024"/>
146146
</bean>
147147

148148
<!--Destination TcpDiscoverySpi for CDC streamer-->
@@ -221,7 +221,7 @@ Add link:https://github.com/apache/ignite-extensions/blob/master/modules/cdc-ext
221221
<value>terminator</value>
222222
</list>
223223
</property>
224-
<property name="maxBatchSize" value="256"/>
224+
<property name="maxBatchSize" value="1024"/>
225225
<property name="onlyPrimary" value="false"/>
226226
<property name="kafkaProperties" ref="kafkaProperties"/>
227227
</bean>
@@ -513,26 +513,96 @@ NOTE: Please, enable `ignite-rest-http` and `ignite-json` to be able to run `cdc
513513

514514
Examples for reference:
515515

516+
* Help message
517+
518+
```
519+
./cdc-start-up.sh --help
520+
```
521+
516522
* Start Ignite node with specified properties:
523+
524+
To start an Ignite cluster node, use `--ignite` or `-i` command with `cdc-start-up.sh`. You also need to specify properties holder directory.
525+
526+
There are currently 2 examples for 2 clusters, that you can run sumalteniously. You can find them under `$IGNITE_HOME/examples/config/cdc-start-up/cluster-1` and `$IGNITE_HOME/examples/config/cdc-start-up/cluster-2` as `ignite-cdc.properties`. These files contains all independent settings that you can tinker for your needs.
527+
528+
NOTE: All properties files are preconfigured to work out of the box.
529+
517530
```
518531
./cdc-start-up.sh --ignite ../examples/config/cdc-start-up/cluster-1
532+
./cdc-start-up.sh --ignite ../examples/config/cdc-start-up/cluster-2
519533
```
520-
* Start IgniteToIgnite CDC thin client with specified properties:
534+
535+
* Start CDC clients with specified properties:
536+
537+
** To start any CDC client node, use `--cdc-client` or `-c` command with `cdc-start-up.sh`. In addition, you have to specify CDC client mode and properties holder directory for the source cluster (as in the previous example).
538+
539+
** There are 5 options you can specify CDC client mode from. Take a look at `--help` command output to learn about them.
540+
541+
** You can optionaly activate both clusters at CDC client start-up with `--activate-cluster`. You need this for steady data replication. You can also use `control.sh` capabilities to activate clusters manually.
542+
543+
NOTE: Start both clusters (as in previous example with Ignite nodes) before starting CDC client
544+
545+
Here is an example on how to start Active-Passive inter-cluster communication with 2 separate nodes and one thin CDC client for Ignite-to-Ignite replication from cluster 1 to cluster 2 (Run the commands independently):
521546
```
522-
./cdc-start-up.sh --cdc-client --ignite-to-ignite-thin ../examples/config/cdc-start-up/cluster-1
547+
./cdc-start-up.sh --ignite ../examples/config/cdc-start-up/cluster-1
548+
./cdc-start-up.sh --ignite ../examples/config/cdc-start-up/cluster-2
549+
./cdc-start-up.sh --cdc-client --ignite-to-ignite-thin --activate-cluster ../examples/config/cdc-start-up/cluster-1
523550
```
524-
* Start Active-Active replication with specified properties for both clusters (CDC uses thin clients):
551+
552+
NOTE: To start CDC with Kafka you need to start topics beforehand.
553+
554+
We use the following topics naming for our examples:
555+
556+
`cluster 1 -> cluster 2`
557+
525558
```
526-
./cdc-start-up.sh --active-active --thin ../examples/config/cdc-start-up/cluster-1 ../examples/config/cdc-start-up/cluster-2
559+
./kafka-topics.sh --create --partitions 16 --replication-factor 1 --topic dc1_to_dc2 --bootstrap-server localhost:9092
560+
./kafka-topics.sh --create --partitions 1 --replication-factor 1 --topic metadata_from_dc1 --bootstrap-server localhost:9092
527561
```
562+
563+
`cluster 2 -> cluster 1`
564+
565+
```
566+
./kafka-topics.sh --create --partitions 16 --replication-factor 1 --topic dc2_to_dc1 --bootstrap-server localhost:9092
567+
./kafka-topics.sh --create --partitions 1 --replication-factor 1 --topic metadata_from_dc2 --bootstrap-server localhost:9092
568+
```
569+
570+
* Start a whole replication system with specified strategy and properties for both clusters:
571+
572+
** Use `--active-active` for Active-Active replication and `--active-passive` for Active-Passive replication.
573+
574+
** You should provide properties files paths for both cluster nodes in the end of the command
575+
576+
** By default, this script uses Ignite clients (thick) for replication. You can optionally use `--with-kafka` to transfer data through Kafka, and `--thin` to use thin clients for connections to the destination clusters.
577+
578+
NOTE: There is no need to preactivate clusters, as the script will do it for you.
579+
580+
To repeat the replication example from the previous point, you can use the following:
581+
582+
```
583+
./cdc-start-up.sh --active-passive --thin ../examples/config/cdc-start-up/cluster-1 ../examples/config/cdc-start-up/cluster-2
584+
```
585+
586+
Active-Active with thin clients and Kafka as data transport:
587+
588+
```
589+
./cdc-start-up.sh --active-active --with-kafka --thin ../examples/config/cdc-start-up/cluster-1 ../examples/config/cdc-start-up/cluster-2
590+
```
591+
528592
* You can check CDC replication with `--check-cdc`. Use it in parallel with Active-Passive/Active-Active replication. To start CDC check for proposed entry:
529593
```
530594
./cdc-start-up.sh --check-cdc --key 11006 --value 1 --version 1 --cluster 1
531595
```
532596

533-
The last command basically puts the entry to the chosen cluster and shows the difference between the two clusters until the data fully transfered.
597+
The command basically puts the entry to the chosen cluster and shows the difference between the two clusters until the data fully transfered.
598+
599+
NOTE: To use it with Active-Passive, push entries only in the Active part of the CDC inter-cluster system. See the example below
600+
601+
```
602+
./cdc-start-up.sh --active-passive ../examples/config/cdc-start-up/cluster-1 ../examples/config/cdc-start-up/cluster-2
603+
```
534604

535-
NOTE: With Active-Passive strategy there is no point to put data in the 2nd cluster, as in these examples the data is replicated from the first only
605+
`cluster-1` is an active part since its properties' directory goes first in the command. In this case there is no reason to push data to the second cluster during the check step, as the data is not replicated from the second cluster.
536606

537607
NOTE: Try to play with version value to see how the conflict resolver works. We propose the following sequence of operations with Active-Active:
538608

0 commit comments

Comments
 (0)