Skip to content

README.md

Latest commit

 

History

History
476 lines (388 loc) · 19.1 KB

README.md

File metadata and controls

476 lines (388 loc) · 19.1 KB
Raw

Demo Outline

✅ Setup and start Oracle database (Source)
✅ Setup and start Redis Enterprise database (Target)
✅ Setup and start Redis Connect
✅ Perform Initial load and CDC with Redis Connect
✅ Setup Grafana dashboard for Redis Connect metrics

Prerequisites

  • Docker compatible *nix OS and Docker installed.
  • Please have 8 vCPU*, 8GB RAM and 50GB storage for this demo to function properly. Adjust the resources based on your requirements. For HA, at least have 2 Redis Connect Connector instances deployed on separate hosts.

Execute the following commands (copy & paste) to download and setup Redis Connect and demo scripts. i.e.

wget -c https://github.com/redis-field-engineering/redis-connect-dist/archive/main.zip && \
mkdir -p redis-connect/demo && \
unzip main.zip "redis-connect-dist-main/examples/oracle/*" -d redis-connect && \
cp -R redis-connect/redis-connect-dist-main/examples/oracle/demo/* redis-connect/demo && \
rm -rf main.zip redis-connect/redis-connect-dist-main && \
cd redis-connect && \
chmod a+x demo/*.sh && \
cd demo

Expected output:

demo$ ls
README.md  delete.sql  emp.ctl                  employees1k_insert.sql  load_c##rcuser_schema.sh  setup_logminer.sh  setup_re.sh
config     emp.csv     employees10k_insert.sql  extlib                  load_sql.sh               setup_oracle.sh    update.sql

Setup Oracle database in docker (Source)


Execute setup_oracle.sh
Oracle 12c and 18c:

redis-connect-oracle$ cd demo
demo$ ./setup_oracle.sh 12.2.0.1-ee 1521 logminer

Oracle 19c:

demo$ ./setup_oracle.sh 19.3.0-ee 1522 logminer
Expected output: 

Status: Downloaded newer image for virag/oracle-12.2.0.1-ee:latest
ae728fa6e001c2f67e7a783ae2db9bd1999b0d4d6d9f72888a1b0b4473216db1
nc: connect to 172.17.0.9 port 1521 (tcp) failed: Connection refused
Wed Nov 17 19:39:02 UTC 2021 - still trying
Connection to 172.17.0.9 1521 port [tcp/*] succeeded!
Wed Nov 17 19:39:04 UTC 2021 - connected successfully
Wed Nov 17 19:39:04 UTC 2021 - Waiting for oracle database to be up (attempt: 1)...
....
....
SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
The Oracle base remains unchanged with value /opt/oracle
#########################
DATABASE IS READY TO USE!
#########################
....
....
Completed:   ALTER DATABASE ADD SUPPLEMENTAL LOG DATA
2021-11-17T19:46:34.230847+00:00
===========================================================
....
....
SQL> SQL> Connected.
SQL>
  COUNT(*)
----------
       107

1 row selected.

SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

Validate Oracle database is running as expected: 

demo$ docker ps -a | grep oracle
f31d84987694        virag/oracle-19.3.0-ee                       "/bin/sh -c 'exec $O…"   8 days ago          Up 8 days (healthy)   0.0.0.0:1522->1521/tcp                                                                                                                                                                                                                                                                                          oracle-19.3.0-ee-virag-cdc-1522

demo$ docker exec -it oracle-19.3.0-ee-$(hostname)-1522 bash -c "sqlplus c##rcuser/rcpwd@ORCLPDB1"

SQL*Plus: Release 19.0.0.0.0 - Production on Thu May 26 02:00:21 2022
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Last Successful login time: Tue May 17 2022 03:21:34 +00:00

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> select 1 from dual;

	 1
----------
	 1

SQL> select count(*) from c##rcuser.emp;

  COUNT(*)
----------
	 0

Setup Redis Enterprise cluster, databases and RedisInsight in docker (Target)


Execute setup_re.sh

demo$ ./setup_re.sh

NOTE

The above script will create a 1-node Redis Enterprise cluster in a docker container, Create a target database with RediSearch module, Create a job management and metrics database with RedisTimeSeries module, Create a RediSearch index for emp Hash, Start a docker instance of grafana with Redis Data Source and Start an instance of RedisInsight.

Start Redis Connect

Review options by running Redis Connect docker container 

demo$ docker run \
-it --rm --privileged=true \
--name redis-connect-$(hostname) \
-v $(pwd)/config:/opt/redislabs/redis-connect/config \
-v $(pwd)/config/samples/credentials:/opt/redislabs/redis-connect/config/samples/credentials \
--net host \
redislabs/redis-connect

Expected output: 

-------------------------------
Redis Connect startup script.
*******************************
Please ensure that these environment variables are correctly mapped before executing start and cli options. They can also be found in /opt/redislabs/redis-connect/bin/redisconnect.conf
Example environment variables and volume mapping for docker based deployments
-e REDISCONNECT_JOB_MANAGER_CONFIG_PATH=/opt/redislabs/redis-connect/config/jobmanager.properties [OPTIONAL]
-e REDISCONNECT_LOGBACK_CONFIG=/opt/redislabs/redis-connect/config/logback.xml [OPTIONAL]
-e REDISCONNECT_JAVA_OPTIONS=-Xms1g -Xmx2g [OPTIONAL]
-e REDISCONNECT_EXTLIB_DIR=/opt/redislabs/redis-connect/extlib [OPTIONAL]
-v <HOST_PATH_TO_JOB_MANAGER_PROPERTIES>:/opt/redislabs/redis-connect/config
-v <HOST_PATH_TO_CREDENTIALS>:/opt/redislabs/redis-connect/config/samples/credentials
-v <HOST_PATH_TO_EXTLIB>:/opt/redislabs/redis-connect/extlib [OPTIONAL]
-p 8282:8282

Usage: [-h|cli|start]
options:
-h: Print this help message and exit.
-v: Print version.
cli: init Redis Connect CLI
start: init Redis Connect Instance (Cluster Member)
-------------------------------

Start Redis Connect Instance 

docker run \
-it --rm --privileged=true \
--name redis-connect-$(hostname) \
-v $(pwd)/config:/opt/redislabs/redis-connect/config \
-v $(pwd)/extlib:/opt/redislabs/redis-connect/extlib \
--net host \
redislabs/redis-connect start

Expected output: 

-------------------------------
Starting redis-connect v0.10.1.5 Instance using JAVA 11.0.19 on redis-connect started by root in /opt/redislabs/redis-connect/bin
Loading redis-connect Instance configuration from /opt/redislabs/redis-connect/config/jobmanager.properties
Instance classpath /opt/redislabs/redis-connect/lib/*:/opt/redislabs/redis-connect/extlib/*
Check redis-connect-manager-<PID>.log for cluster-level information, redis-connect-heartbeat-<PID>.log for heartbeat-lease renewals, and redis-connect-<PID>.log for the job-level information
06:32:33.750 [main] INFO  redis-connect-manager - ----------------------------------------------------------------------------------------------------------------------------
  /#######                  /## /##          	  /######                                                      /##
 | ##__  ##                | ## |__/          	 /##__  ##                                                    | ##
 | ##  \ ##  /######   /####### /##  /#######	| ##  \__/  /######  /#######  /#######   /######   /####### /######
 | #######/ /##__  ## /##__  ##| ## /##_____/	| ##       /##__  ##| ##__  ##| ##__  ## /##__  ## /##_____/|_  ##_/
 | ##__  ##| ########| ##  | ##| ##|  ###### 	| ##      | ##  \ ##| ##  \ ##| ##  \ ##| ########| ##        | ##
 | ##  \ ##| ##_____/| ##  | ##| ## \____  ##	| ##    ##| ##  | ##| ##  | ##| ##  | ##| ##_____/| ##        | ## /##
 | ##  | ##|  #######|  #######| ## /#######/	|  ######/|  ######/| ##  | ##| ##  | ##|  #######|  #######  |  ####/
 |__/  |__/ \_______/ \_______/|__/|_______/ 	 \______/  \______/ |__/  |__/|__/  |__/ \_______/ \_______/   \___/   v0.10.1
Powered by Redis Enterprise
06:32:38.758 [main] INFO  redis-connect-manager - ----------------------------------------------------------------------------------------------------------------------------
06:32:40.351 [main] INFO  redis-connect-manager - Instance: 30@redis-connect successfully established Redis connection with ClientId: JobManager ConnectionId: JobManager
06:32:40.360 [main] INFO  redis-connect-manager - Instance: 30@redis-connect successfully established Redis connection with ClientId: JobManager ConnectionId: JobReaper
06:32:40.369 [main] INFO  redis-connect-manager - Instance: 30@redis-connect successfully established Redis connection with ClientId: JobManager ConnectionId: JobClaimer
06:32:40.377 [main] INFO  redis-connect-manager - Instance: 30@redis-connect successfully established Redis connection with ClientId: JobManager ConnectionId: JobOrchestrator
06:32:40.385 [main] INFO  redis-connect-manager - Instance: 30@redis-connect successfully established Redis connection with ClientId: JobManager ConnectionId: HeartbeatManager
06:32:40.392 [main] INFO  redis-connect-manager - Instance: 30@redis-connect successfully established Redis connection with ClientId: JobManager ConnectionId: MetricsReporter
06:32:40.399 [main] INFO  redis-connect-manager - Instance: 30@redis-connect successfully established Redis connection with ClientId: JobManager ConnectionId: CredentialsRotationEventListener
06:32:40.406 [main] INFO  redis-connect-manager - Instance: 30@redis-connect successfully established Redis connection with ClientId: JobManager ConnectionId: ChangeEventQueue
06:32:40.487 [main] INFO  redis-connect-manager - Instance: 30@redis-connect skipped creating Job Claim Assignment Consumer Group since it already exists
06:32:40.491 [main] INFO  redis-connect-manager - Instance: 30@redis-connect successfully started JobManager service
06:32:40.493 [main] INFO  redis-connect-manager - Instance: 30@redis-connect successfully started JobReaper service
06:32:40.495 [main] INFO  redis-connect-manager - Instance: 30@redis-connect successfully started JobClaimer service
06:32:40.497 [main] INFO  redis-connect-manager - Instance: 30@redis-connect successfully started MetricsReporter service
06:32:44.370 [main] INFO  redis-connect-manager - Instance: 30@redis-connect started Redis Connect REST API listening on ["http-nio-8282"]
06:32:44.371 [main] INFO  redis-connect-manager - ----------------------------------------------------------------------------------------------------------------------------
06:32:44.371 [main] INFO  redis-connect-manager -
06:32:44.372 [main] INFO  redis-connect-manager - Started Redis Connect Instance v0.10.1
06:32:44.372 [main] INFO  redis-connect-manager -
06:32:44.372 [main] INFO  redis-connect-manager - ----------------------------------------------------------------------------------------------------------------------------
06:32:50.492 [JOB_MANAGER_THREADPOOL-2] INFO  redis-connect-manager - Instance: 30@redis-connect was successfully elected Redis Connect cluster leader

Open browser to access Swagger UI - http://localhost:8282/swagger-ui/index.html
For quick start, use 'cdc_job' as jobName

Create Job Configuration - /connect/api/vi/job/config/{jobName}
For quick start, use the sample cdc-job.json configuration: Oracle


Or Use curl to create the cdc-job configuration 

demo$ curl -v -X POST "http://localhost:8282/connect/api/v1/job/config/cdc-job" -H "accept: */*" -H "Content-Type: multipart/form-data" -F "file=@config/samples/payloads/cdc-job.json;type=application/json"

SUCCESS

Initial Loader Steps

INSERT few records into emp table (source)

You can also use redis-connect-crud-loader to load large amount of data using a csv or sql file. 

demo$ docker exec -it oracle-19.3.0-ee-$(hostname)-1522 bash -c "/tmp/load_sql.sh insert1k_emp"

-------------------------------

SQL*Loader: Release 19.0.0.0.0 - Production on Thu May 26 02:30:57 2022
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Path used:      Conventional
Commit point reached - logical record count 50
Commit point reached - logical record count 100
Commit point reached - logical record count 150
Commit point reached - logical record count 200
Commit point reached - logical record count 250
Commit point reached - logical record count 300
Commit point reached - logical record count 350
Commit point reached - logical record count 400
Commit point reached - logical record count 450
Commit point reached - logical record count 500
Commit point reached - logical record count 550
Commit point reached - logical record count 600
Commit point reached - logical record count 650
Commit point reached - logical record count 700
Commit point reached - logical record count 750
Commit point reached - logical record count 800
Commit point reached - logical record count 850
Commit point reached - logical record count 900
Commit point reached - logical record count 950
Commit point reached - logical record count 1000

Table C##RCUSER.EMP:
  1000 Rows successfully loaded.

Check the log file:
  emp.log
for more information about the load.
-------------------------------

Start Job - /connect/api/vi/job/transition/start/{jobName}/{jobType}
Use 'load' as jobType

Or Use curl to start the initial load for cdc-job 

demo$ curl -X POST "http://localhost:8282/connect/api/v1/job/transition/start/cdc-job/load" -H "accept: */*"

SUCCESS - Transition has been scheduled
Query for the above inserted record in Redis (target) 

demo$ sudo docker exec -it re-node1 bash -c 'redis-cli -p 12000 ft.search idx_emp "@EMPNO:[1 2]"'
1) (integer) 2
2) "EMP:1"
3)  1) "COMM"
    2) "123517.13"
    3) "LNAME"
    4) "McGarvie"
    5) "FirstName"
    6) "Chlo"
    7) "EmployeeNumber"
    8) "1"
    9) "MGR"
   10) "19"
   11) "HireDate"
   12) "2016-08-05 04:07:50.0"
   13) "DEPT"
   14) "96"
   15) "JOB"
   16) "General Manager"
   17) "SAL"
   18) "167105.34"
   19) "EMPNO"
   20) "1"
   21) "FNAME"
   22) "Chlo"
   23) "HIREDATE"
   24) "2016-08-05 04:07:50.0"
4) "EMP:2"
5)  1) "COMM"
    2) "165687.45"
    3) "LNAME"
    4) "Humm"
    5) "FirstName"
    6) "Alex"
    7) "EmployeeNumber"
    8) "2"
    9) "MGR"
   10) "70"
   11) "HireDate"
   12) "2019-08-14 04:01:21.0"
   13) "DEPT"
   14) "51"
   15) "JOB"
   16) "Assistant Media Planner"
   17) "SAL"
   18) "162370.71"
   19) "EMPNO"
   20) "2"
   21) "FNAME"
   22) "Alex"
   23) "HIREDATE"
   24) "2019-08-14 04:01:21.0"

CDC Steps

Start Job - /connect/api/vi/job/transition/start/{jobName}/{jobType}
Use 'stream' as jobType

Or Use curl to start the stream for cdc-job
demo$ curl -X POST "http://localhost:8282/connect/api/v1/job/transition/start/cdc-job/stream" -H "accept: */*"

Confirm Job Claim - /connect/api/vi/jobs/claim/{jobStatus}
For quick start, use 'all' as jobStatus

Or Use curl to query the cdc-job status
demo$ curl -X GET "http://localhost:8282/connect/api/v1/cluster/jobs/claim/all" -H "accept: */*"

Expected output: [{"jobId":"{connect}:job:cdc-job","jobName":"cdc-job","jobStatus":"CLAIMED","jobOwner":"30@virag-cdc","jobType":"STREAM"}]

INSERT a record into emp table (source) 

demo$ docker exec -it oracle-19.3.0-ee-$(hostname)-1522 bash -c "sqlplus c##rcuser/rcpwd@ORCLPDB1"
SQL*Plus: Release 19.0.0.0.0 - Production on Thu May 26 03:01:01 2022
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Last Successful login time: Thu May 26 2022 02:58:29 +00:00

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> insert into C##RCUSER.emp values (1001, 'Allen', 'Terleto', 'FieldCTO', 19, (TO_DATE('2018-08-05 04:07:50', 'yyyy-MM-dd HH:mi:ss')), 167105.34, 123517.13, 96);

1 row created.

Query for the above inserted record in Redis (target) 

demo$ sudo docker exec -it re-node1 bash -c 'redis-cli -p 12000 ft.search idx_emp "@FNAME:allen"'
1) (integer) 1
2) "EMP:1001"
3)  1) "COMM"
    2) "123517.13"
    3) "LNAME"
    4) "Terleto"
    5) "HIREDATE"
    6) "1533442070000"
    7) "EMPNO"
    8) "1001"
    9) "MGR"
   10) "19"
   11) "DEPT"
   12) "96"
   13) "JOB"
   14) "FieldCTO"
   15) "FNAME"
   16) "Allen"
   17) "SAL"
   18) "167105.34"

Dashboard

Prerequisites

  1. Redis Connect cluster timeseries metrics should be enabled.
  2. Redis Connect job metrics should be enabled.
Standalone Grafana

Use your existing Grafana install and add redis-datasource plugin.

  1. Configure two Redis data sources, one with JobManager and another one with Target name using redis-datasource plugin.
  2. Import Sample Redis Connect Dashboard
  3. Validate and fix dashboard variables if they don't load properly
Containerized Grafana

If you used the demo script to set up Redis Enterprise in a docker container then, a containerized Grafana with redis-datasource plugin should already be installed and running.

  1. Open Grafana UI by going to http://localhost:13000 and login with redisconnect/Redis@123 credentials.
  2. Import Sample Redis Connect Dashboard
  3. Validate and fix dashboard variables if they don't load properly