Skip to content

Commit 18f571c

Browse files
jruauxjruaux
authored
Merge pull request #1 from redis-field-engineering/snowflake-demo
feat: Include demo scripts for snowflake
2 parents 0eab7fc + acf2718 commit 18f571c

File tree

5 files changed

+509
-0
lines changed

5 files changed

+509
-0
lines changed

‎.gitignore

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,8 @@
11
/.project
22
/.settings/
33
/.idea/
4+
5+
# Environment files - contain sensitive credentials
6+
.env
7+
8+
TASK-MEMORY.md

‎demos/snowflake/.env.example

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Snowflake Demo Environment Variables
2+
# Copy this file to .env and fill in your actual values
3+
4+
# Redis Configuration
5+
REDIS_HOST=your-redis-host.com
6+
REDIS_PORT=12001
7+
REDIS_PASS=your-redis-password
8+
9+
# Snowflake Configuration
10+
SNOWFLAKE_ACCOUNT=your-snowflake-account-id
11+
SNOWFLAKE_USER=your-snowflake-username
12+
SNOWFLAKE_PRIVATE_KEY_FILE=/path/to/your/snowflake_private_key.p8
13+
SNOWFLAKE_ROLE=accountadmin
14+
SNOWFLAKE_WAREHOUSE=compute_wh
15+
16+
# Snowflake Database/Schema
17+
SNOWFLAKE_DATABASE=tb_101
18+
SNOWFLAKE_SCHEMA=raw_pos
19+
SNOWFLAKE_TABLE=incremental_order_header
20+
SNOWFLAKE_CDC_SCHEMA=raw_pos_cdc

‎demos/snowflake/DEMO-GUIDE.md

Lines changed: 217 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,217 @@
1+
# Snowflake to Redis Data Integration Demo Guide
2+
3+
## Overview
4+
This demo showcases real-time data integration from Snowflake to Redis using RIOTX. The demo will demonstrate CDC (Change Data Capture) capabilities, showing how data changes in Snowflake are automatically synchronized to Redis.
5+
6+
## Setup Verification (Before Demo)
7+
8+
### 1. Environment Setup
9+
```bash
10+
# Copy environment template and configure your values
11+
cp .env.example .env
12+
# Edit .env with your actual Snowflake and Redis credentials
13+
```
14+
15+
### 2. Environment Check
16+
```bash
17+
# Verify tools are available
18+
which riotx
19+
which snowsql
20+
```
21+
22+
### 3. Snowflake Connection Test
23+
```bash
24+
# Load environment variables
25+
source .env
26+
snowsql -c $SNOWFLAKE_ACCOUNT -q "SELECT COUNT(*) FROM $SNOWFLAKE_DATABASE.$SNOWFLAKE_SCHEMA.$SNOWFLAKE_TABLE;"
27+
```
28+
29+
### 4. Redis Connection Test
30+
```bash
31+
# Test Redis connection (should be accessible via Redis Insights GUI)
32+
redis-cli -h $REDIS_HOST -p $REDIS_PORT -a '$REDIS_PASS' ping
33+
```
34+
35+
## Demo Flow (3-Window Setup)
36+
37+
### Window 1: Snowflake Terminal
38+
**Purpose**: Show source data and simulate changes
39+
**Command**: `snowsql -c $SNOWFLAKE_ACCOUNT`
40+
41+
### Window 2: RIOTX Data Sync
42+
**Purpose**: Run the data synchronization process
43+
**Command**: See main command below
44+
45+
### Window 3: Redis Insights GUI
46+
**Purpose**: Visualize data in Redis in real-time
47+
**URL**: Open Redis Insights and connect to Redis cluster
48+
49+
## Main Demo Script
50+
51+
### Phase 1: Initial Data Load (5 minutes)
52+
53+
#### 1.1 Show Current Snowflake Data
54+
```sql
55+
-- In Snowflake terminal (Window 1)
56+
USE SCHEMA $SNOWFLAKE_DATABASE.$SNOWFLAKE_SCHEMA;
57+
SELECT COUNT(*) FROM $SNOWFLAKE_TABLE;
58+
SELECT * FROM $SNOWFLAKE_TABLE LIMIT 10;
59+
```
60+
61+
#### 1.2 Start RIOTX Sync Process
62+
```bash
63+
# In RIOTX terminal (Window 2)
64+
riotx snowflake-import \
65+
-h $REDIS_HOST \
66+
-p $REDIS_PORT \
67+
-a $REDIS_PASS \
68+
$SNOWFLAKE_DATABASE.$SNOWFLAKE_SCHEMA.$SNOWFLAKE_TABLE \
69+
--cdc-schema $SNOWFLAKE_CDC_SCHEMA \
70+
--role $SNOWFLAKE_ROLE \
71+
--warehouse $SNOWFLAKE_WAREHOUSE \
72+
--jdbc-url "jdbc:snowflake://$SNOWFLAKE_ACCOUNT.snowflakecomputing.com?private_key_file=$SNOWFLAKE_PRIVATE_KEY_FILE" \
73+
--jdbc-user $SNOWFLAKE_USER \
74+
hset 'orders:#{ORDER_ID}'
75+
```
76+
77+
#### 1.3 Verify Initial Data in Redis
78+
- **In Redis Insights**: Browse to see the `orders:*` keys
79+
- **Expected**: Hash keys for each ORDER_ID with all order details
80+
81+
### Phase 2: CDC Demonstration (10 minutes)
82+
83+
#### 2.1 Insert New Orders
84+
```sql
85+
-- In Snowflake terminal (Window 1)
86+
INSERT INTO $SNOWFLAKE_DATABASE.$SNOWFLAKE_SCHEMA.$SNOWFLAKE_TABLE
87+
(ORDER_ID, TRUCK_ID, LOCATION_ID, ORDER_TS, ORDER_CURRENCY, ORDER_AMOUNT, ORDER_TOTAL)
88+
VALUES
89+
(9999001, 99, 99999, CURRENT_TIMESTAMP(), 'USD', 150.00, 150.00),
90+
(9999002, 99, 99999, CURRENT_TIMESTAMP(), 'USD', 75.50, 75.50);
91+
92+
-- Verify insertion
93+
SELECT * FROM $SNOWFLAKE_TABLE WHERE ORDER_ID >= 9999001;
94+
```
95+
96+
#### 2.2 Monitor RIOTX Output
97+
- **In RIOTX terminal**: Watch for new records being processed
98+
- **Expected**: Log messages showing CDC events being captured and sent to Redis
99+
100+
#### 2.3 Verify in Redis
101+
- **In Redis Insights**:
102+
- Refresh the key browser
103+
- Look for new keys: `orders:9999001` and `orders:9999002`
104+
- Click on keys to see the hash field values
105+
106+
#### 2.4 Update Existing Orders
107+
```sql
108+
-- In Snowflake terminal (Window 1)
109+
UPDATE $SNOWFLAKE_DATABASE.$SNOWFLAKE_SCHEMA.$SNOWFLAKE_TABLE
110+
SET ORDER_AMOUNT = 200.00, ORDER_TOTAL = 200.00
111+
WHERE ORDER_ID = 9999001;
112+
113+
-- Verify update
114+
SELECT * FROM $SNOWFLAKE_TABLE WHERE ORDER_ID = 9999001;
115+
```
116+
117+
#### 2.5 Verify Update in Redis
118+
- **In Redis Insights**: Check `orders:9999001` to see updated values
119+
- **Expected**: ORDER_AMOUNT and ORDER_TOTAL fields should reflect new values
120+
121+
### Phase 3: Scale and Performance (5 minutes)
122+
123+
#### 3.1 Bulk Insert - Load More Data from Main Table
124+
```sql
125+
-- In Snowflake terminal (Window 1)
126+
-- The main order_header table has 1,698,440 rows vs incremental table's 100 rows
127+
-- Let's add 1000 more records from the main table
128+
INSERT INTO $SNOWFLAKE_DATABASE.$SNOWFLAKE_SCHEMA.$SNOWFLAKE_TABLE
129+
SELECT * FROM $SNOWFLAKE_DATABASE.$SNOWFLAKE_SCHEMA.order_header
130+
WHERE ORDER_ID NOT IN (SELECT ORDER_ID FROM $SNOWFLAKE_DATABASE.$SNOWFLAKE_SCHEMA.$SNOWFLAKE_TABLE)
131+
LIMIT 1000;
132+
133+
-- Verify the insert
134+
SELECT COUNT(*) FROM $SNOWFLAKE_DATABASE.$SNOWFLAKE_SCHEMA.$SNOWFLAKE_TABLE;
135+
```
136+
137+
#### 3.2 Monitor Bulk Processing
138+
- **In RIOTX terminal**: Watch batch processing metrics
139+
- **In Redis Insights**: Watch key count increase in real-time
140+
141+
## Key Demo Points to Highlight
142+
143+
### 1. Real-Time CDC
144+
- Changes in Snowflake appear in Redis within seconds
145+
- No polling - event-driven architecture
146+
- Maintains data consistency
147+
148+
### 2. Scalability
149+
- Batch processing for bulk operations
150+
- Efficient handling of large datasets
151+
- Minimal impact on source system
152+
153+
### 3. Data Structure
154+
- Snowflake rows → Redis hashes
155+
- Flexible key naming patterns
156+
- Preserves all data types and relationships
157+
158+
### 4. Enterprise Features
159+
- Secure connections (TLS, authentication)
160+
- Role-based access control
161+
- Monitoring and observability
162+
163+
## Troubleshooting Commands
164+
165+
### Check Snowflake Stream Status
166+
```sql
167+
-- In Snowflake terminal
168+
SHOW STREAMS IN SCHEMA $SNOWFLAKE_CDC_SCHEMA;
169+
```
170+
171+
### Redis Connection Test
172+
```bash
173+
redis-cli -h $REDIS_HOST -p $REDIS_PORT -a '$REDIS_PASS' info replication
174+
```
175+
176+
### RIOTX Debug Mode
177+
```bash
178+
# Add --debug flag to the main command for verbose logging
179+
riotx snowflake-import --debug [... other parameters ...]
180+
```
181+
182+
## Q&A Preparation
183+
184+
### Common Questions:
185+
1. **"How does CDC work?"** - Snowflake Streams capture changes, RIOTX polls streams
186+
2. **"What's the latency?"** - Typically 1-5 seconds depending on configuration
187+
3. **"Can it handle schema changes?"** - Yes, but requires restart for new columns
188+
4. **"What about failover?"** - Redis Enterprise provides high availability
189+
5. **"Cost implications?"** - Snowflake compute costs, Redis memory usage
190+
191+
### Demo Recovery:
192+
- If RIOTX fails: Restart the command (it will resume from last position)
193+
- If Redis connection fails: Check Redis Insights connection settings
194+
- If Snowflake fails: Verify network connectivity and credentials
195+
196+
## Post-Demo Cleanup
197+
198+
```sql
199+
-- Optional: Clean up demo data
200+
DELETE FROM $SNOWFLAKE_DATABASE.$SNOWFLAKE_SCHEMA.$SNOWFLAKE_TABLE WHERE ORDER_ID >= 9999001;
201+
```
202+
203+
```bash
204+
# Optional: Clean up Redis keys
205+
redis-cli -h $REDIS_HOST -p $REDIS_PORT -a '$REDIS_PASS' --scan --pattern "orders:9999*" | xargs redis-cli -h $REDIS_HOST -p $REDIS_PORT -a '$REDIS_PASS' del
206+
```
207+
208+
## Success Metrics
209+
- [ ] Data appears in Redis within 5 seconds of Snowflake changes
210+
- [ ] All 3 windows show synchronized activity
211+
- [ ] Bulk operations process smoothly
212+
- [ ] Q&A handled confidently
213+
- [ ] Audience understands the value proposition
214+
215+
---
216+
217+
**Remember**: Take your time, breathe, and focus on the story: "Real-time data integration made simple with Snowflake and Redis."

‎demos/snowflake/run-demo-riotx.sh

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
#!/bin/bash
2+
3+
# Demo Script for RIOTX - Snowflake to Redis Integration
4+
# This script walks through the RIOTX portions of the demo
5+
6+
# Load environment variables
7+
if [ ! -f ".env" ]; then
8+
echo "Error: .env file not found!"
9+
echo "Please copy .env.example to .env and configure your values."
10+
exit 1
11+
fi
12+
13+
set -a # automatically export all variables
14+
source .env
15+
set +a
16+
17+
RED='\033[0;31m'
18+
GREEN='\033[0;32m'
19+
YELLOW='\033[1;33m'
20+
BLUE='\033[0;34m'
21+
DIM='\033[2m'
22+
NC='\033[0m' # No Color
23+
24+
# Redis host will be loaded from .env file
25+
26+
# Clear terminal and show realistic prompt
27+
clear
28+
29+
wait_for_enter() {
30+
# echo -e "${DIM}Press ENTER to continue...${NC}"
31+
read
32+
}
33+
34+
type_command() {
35+
local cmd="$1"
36+
echo -e "${DIM}\$ ${NC}${cmd}"
37+
}
38+
39+
echo -e "${GREEN}jeremy@macbook${NC}:${BLUE}~/work/snowflake-demo${NC}\$ ./run-demo-riotx.sh"
40+
echo
41+
42+
echo -e "${GREEN}SETUP PHASE - Redis Database Cleanup${NC}"
43+
echo -e "Would you like to flush the Redis database before starting? (y/n)"
44+
read -p "Enter choice: " flush_choice
45+
46+
if [[ $flush_choice =~ ^[Yy]$ ]]; then
47+
echo "Flushing Redis database..."
48+
type_command "redis-cli -h $REDIS_HOST -p $REDIS_PORT -a '$REDIS_PASS' flushdb"
49+
#redis-cli -h $REDIS_HOST -p $REDIS_PORT -a '$REDIS_PASS' flushdb
50+
redis-cli flushdb
51+
echo -e "${GREEN}Redis database flushed successfully${NC}"
52+
else
53+
echo "Skipping Redis flush"
54+
fi
55+
echo
56+
57+
clear
58+
59+
echo -e "${GREEN}SETUP PHASE - Environment Verification${NC}"
60+
echo "About to verify that riotx and Redis are accessible"
61+
wait_for_enter
62+
63+
#echo "Checking riotx installation..."
64+
#type_command "which riotx"
65+
#which riotx
66+
#echo
67+
68+
echo "Testing Redis connection..."
69+
#type_command "redis-cli -h $REDIS_HOST -p $REDIS_PORT -a '$REDIS_PASS' ping"
70+
type_command "redis-cli ping"
71+
#redis-cli -h $REDIS_HOST -p $REDIS_PORT -a '$REDIS_PASS' ping
72+
redis-cli ping
73+
echo
74+
75+
echo -e "${GREEN}PHASE 1 - Initial Data Sync${NC}"
76+
echo "About to start the main RIOTX sync process"
77+
echo "This will:"
78+
echo " - Connect to Snowflake table tb_101.raw_pos.incremental_order_header"
79+
echo " - Set up CDC stream in raw_pos_cdc schema"
80+
echo " - Start syncing data to Redis as 'orders:{ORDER_ID}' hashes"
81+
echo " - Monitor for real-time changes"
82+
echo
83+
echo -e "${RED}NOTE: This will run continuously until you stop it with Ctrl+C${NC}"
84+
85+
type_command "riotx snowflake-import \\"
86+
#echo -e "${DIM} -h $REDIS_HOST \\"
87+
#echo -e "${DIM} -p 12001 \\"
88+
#echo -e "${DIM} -a '$REDIS_PASS' \\"
89+
echo -e "${DIM} -h $REDIS_HOST \\"
90+
echo -e "${DIM} -p $REDIS_PORT \\"
91+
echo -e "${DIM} $SNOWFLAKE_DATABASE.$SNOWFLAKE_SCHEMA.$SNOWFLAKE_TABLE \\"
92+
echo -e "${DIM} --cdc-schema $SNOWFLAKE_CDC_SCHEMA \\"
93+
echo -e "${DIM} --role $SNOWFLAKE_ROLE \\"
94+
echo -e "${DIM} --warehouse $SNOWFLAKE_WAREHOUSE \\"
95+
echo -e "${DIM} --jdbc-url 'jdbc:snowflake://$SNOWFLAKE_ACCOUNT.snowflakecomputing.com?private_key_file=$SNOWFLAKE_PRIVATE_KEY_FILE' \\"
96+
echo -e "${DIM} --jdbc-user $SNOWFLAKE_USER \\"
97+
echo -e "${DIM} hset 'orders:#{ORDER_ID}'${NC}"
98+
echo
99+
100+
wait_for_enter
101+
102+
riotx snowflake-import \
103+
-h $REDIS_HOST \
104+
-p $REDIS_PORT \
105+
$SNOWFLAKE_DATABASE.$SNOWFLAKE_SCHEMA.$SNOWFLAKE_TABLE \
106+
--cdc-schema $SNOWFLAKE_CDC_SCHEMA \
107+
--role $SNOWFLAKE_ROLE \
108+
--warehouse $SNOWFLAKE_WAREHOUSE \
109+
--jdbc-url "jdbc:snowflake://$SNOWFLAKE_ACCOUNT.snowflakecomputing.com?private_key_file=$SNOWFLAKE_PRIVATE_KEY_FILE" \
110+
--jdbc-user $SNOWFLAKE_USER \
111+
hset 'orders:#{ORDER_ID}'
112+
113+
echo
114+
echo -e "${GREEN}Demo Complete!${NC}"
115+
echo "The RIOTX process has been stopped."
116+
echo "Check Redis Insights to see the synchronized data."
117+
echo
118+
echo -e "${GREEN}jeremy@macbook${NC}:${BLUE}~/work/snowflake-demo${NC}\$ "

0 commit comments

Comments
 (0)