11# Spark-Redis
22
3- Spark-Redis is a connector for reading/writing from Redis cluster directly via Spark. It supports all the
4- types of Redis structures: Plain Key/Value, Hash, ZSet, Set, List
3+ Spark-Redis is a connector for reading/writing from Redis cluster or non-cluster directly via Spark.
4+ It supports all the types of Redis structures: Plain Key/Value, Hash, ZSet, Set, List.
55In Spark, the data from Redis is represented as an RDD with the tolerance of reshard and down of nodes.
66
77Integrating Redis and Spark gives us a system that combines the best of both worlds.
@@ -79,50 +79,50 @@ In the example we can see how to read from Redis Server.
7979```
8080import com.redislabs.provider.redis._
8181val keysRDD = sc.fromRedisKeyPattern(("127.0.0.1", 7000), "keyPattern", 5)
82+ #keyPattern should be a plain string or a RedisRegex.
83+ #keysRDD is a RDD holds all the keys of keyPattern of the redis server.
84+ #keysRDD is divided into 5(default 3) partitions by hash slots.
8285```
83- keyPattern should be a plain string or a RedisRegex.
84- keysRDD is a RDD holds all the keys of keyPattern of the redis server.
85- keysRDD is divided into 5(default 3) partitions by hash slots.
8686
8787Using Redis' Key/Values
8888```
8989import com.redislabs.provider.redis._
9090val keysRDD = sc.fromRedisKeyPattern(("127.0.0.1", 7000), "keyPattern", 5)
9191val kvRDD = keysRDD.getKV
92+ #kvRDD is a RDD holds all the k/v pairs whose k's pattern is keyPattern, and k must be of 'string' type in redis-server.
9293```
93- kvRDD is a RDD holds all the k/v pairs whose k's pattern is keyPattern, and k must be of 'string' type in redis-server.
9494
9595Using Redis' Hash
9696```
9797import com.redislabs.provider.redis._
9898val keysRDD = sc.fromRedisKeyPattern(("127.0.0.1", 7000), "keyPattern", 5)
9999val hashRDD = keysRDD.getHash
100+ #hashRDD is a RDD holds all the dicts' contents, and the dicts' names must be of keyPattern and exists in the redis-server.
100101```
101- hashRDD is a RDD holds all the dicts' contents, and the dicts' names must be of keyPattern and exists in the redis-server.
102102
103103Using Redis' ZSet
104104```
105105import com.redislabs.provider.redis._
106106val keysRDD = sc.fromRedisKeyPattern(("127.0.0.1", 7000), "keyPattern", 5)
107107val zsetRDD = keysRDD.getZSet
108+ #zsetRDD is a RDD holds all the zsets' contents(key, score), and the zsets' names must be of keyPattern and exists in the redis-server.
108109```
109- zsetRDD is a RDD holds all the zsets' contents(key, score), and the zsets' names must be of keyPattern and exists in the redis-server.
110110
111111Using Redis' List
112112```
113113import com.redislabs.provider.redis._
114114val keysRDD = sc.fromRedisKeyPattern(("127.0.0.1", 7000), "keyPattern", 5)
115115val listRDD = keysRDD.getList
116+ #listRDD is a RDD holds all the lists' contents, and the lists' names must be of keyPattern and exists in the redis-server.
116117```
117- listRDD is a RDD holds all the lists' contents, and the lists' names must be of keyPattern and exists in the redis-server.
118118
119119Using Redis' Set
120120```
121121import com.redislabs.provider.redis._
122122val keysRDD = sc.fromRedisKeyPattern(("127.0.0.1", 7000), "keyPattern", 5)
123123val setRDD = keysRDD.getSet
124+ #setRDD is a RDD holds all the sets' contents, score), and the sets' names must be of keyPattern and exists in the redis-server.
124125```
125- setRDD is a RDD holds all the sets' contents, score), and the sets' names must be of keyPattern and exists in the redis-server.
126126
127127*****
128128
@@ -135,37 +135,37 @@ Saving as Redis' Key/Values
135135import com.redislabs.provider.redis._
136136val kvRDD = ...
137137sc.toRedisKV(kvRDD, ("127.0.0.1", 7000))
138+ #kvRDD is a RDD holds k/v pairs, we will store all the k/v pairs of kvRDD to the redis-server
138139```
139- kvRDD is a RDD holds k/v pairs, we will store all the k/v pairs of kvRDD to the redis-server
140140
141141Saving as Redis' Hash
142142```
143143import com.redislabs.provider.redis._
144144val hashRDD = ...
145145sc.toRedisHASH(hashRDD, hashName, ("127.0.0.1", 7000))
146+ #hashRDD is a RDD holds k/v pairs, we will store all the k/v pairs of hashRDD to a dict named hashName to the redis-server
146147```
147- hashRDD is a RDD holds k/v pairs, we will store all the k/v pairs of hashRDD to a dict named hashName to the redis-server
148148
149149Saving as Redis' ZSet
150150```
151151import com.redislabs.provider.redis._
152152val zsetRDD = ...
153153sc.toRedisZSET(zsetRDD, zsetName, ("127.0.0.1", 7000))
154+ #zsetRDD is a RDD holds k/v pairs, we will store all the k/v pairs of zsetRDD to a zset named zsetName to the redis-server
154155```
155- zsetRDD is a RDD holds k/v pairs, we will store all the k/v pairs of zsetRDD to a zset named zsetName to the redis-server
156156
157157Saving as Redis' List
158158```
159159import com.redislabs.provider.redis._
160160val listRDD = ...
161161sc.toRedisLIST(listRDD, listName, ("127.0.0.1", 7000))
162+ #listRDD is a RDD holds strings, we will store all the strings of listRDD to a list named listName to the redis-server
162163```
163- listRDD is a RDD holds strings, we will store all the strings of listRDD to a list named listName to the redis-server
164164
165165Saving as Redis' Set
166166```
167167import com.redislabs.provider.redis._
168168val setRDD = ...
169169sc.toRedisSET(setRDD, setName, ("127.0.0.1", 7000))
170+ #setRDD is a RDD holds strings, we will store all the unique strings of setRDD to a set named setName to the redis-server
170171```
171- setRDD is a RDD holds strings, we will store all the unique strings of setRDD to a set named setName to the redis-server
0 commit comments