1
1
# Spark-Redis
2
2
3
- Spark-Redis is a connector for reading/writing from Redis cluster directly via Spark. It supports all the
4
- types of Redis structures: Plain Key/Value, Hash, ZSet, Set, List
3
+ Spark-Redis is a connector for reading/writing from Redis cluster or non-cluster directly via Spark.
4
+ It supports all the types of Redis structures: Plain Key/Value, Hash, ZSet, Set, List.
5
5
In Spark, the data from Redis is represented as an RDD with the tolerance of reshard and down of nodes.
6
6
7
7
Integrating Redis and Spark gives us a system that combines the best of both worlds.
@@ -79,50 +79,50 @@ In the example we can see how to read from Redis Server.
79
79
```
80
80
import com.redislabs.provider.redis._
81
81
val keysRDD = sc.fromRedisKeyPattern(("127.0.0.1", 7000), "keyPattern", 5)
82
+ #keyPattern should be a plain string or a RedisRegex.
83
+ #keysRDD is a RDD holds all the keys of keyPattern of the redis server.
84
+ #keysRDD is divided into 5(default 3) partitions by hash slots.
82
85
```
83
- keyPattern should be a plain string or a RedisRegex.
84
- keysRDD is a RDD holds all the keys of keyPattern of the redis server.
85
- keysRDD is divided into 5(default 3) partitions by hash slots.
86
86
87
87
Using Redis' Key/Values
88
88
```
89
89
import com.redislabs.provider.redis._
90
90
val keysRDD = sc.fromRedisKeyPattern(("127.0.0.1", 7000), "keyPattern", 5)
91
91
val kvRDD = keysRDD.getKV
92
+ #kvRDD is a RDD holds all the k/v pairs whose k's pattern is keyPattern, and k must be of 'string' type in redis-server.
92
93
```
93
- kvRDD is a RDD holds all the k/v pairs whose k's pattern is keyPattern, and k must be of 'string' type in redis-server.
94
94
95
95
Using Redis' Hash
96
96
```
97
97
import com.redislabs.provider.redis._
98
98
val keysRDD = sc.fromRedisKeyPattern(("127.0.0.1", 7000), "keyPattern", 5)
99
99
val hashRDD = keysRDD.getHash
100
+ #hashRDD is a RDD holds all the dicts' contents, and the dicts' names must be of keyPattern and exists in the redis-server.
100
101
```
101
- hashRDD is a RDD holds all the dicts' contents, and the dicts' names must be of keyPattern and exists in the redis-server.
102
102
103
103
Using Redis' ZSet
104
104
```
105
105
import com.redislabs.provider.redis._
106
106
val keysRDD = sc.fromRedisKeyPattern(("127.0.0.1", 7000), "keyPattern", 5)
107
107
val zsetRDD = keysRDD.getZSet
108
+ #zsetRDD is a RDD holds all the zsets' contents(key, score), and the zsets' names must be of keyPattern and exists in the redis-server.
108
109
```
109
- zsetRDD is a RDD holds all the zsets' contents(key, score), and the zsets' names must be of keyPattern and exists in the redis-server.
110
110
111
111
Using Redis' List
112
112
```
113
113
import com.redislabs.provider.redis._
114
114
val keysRDD = sc.fromRedisKeyPattern(("127.0.0.1", 7000), "keyPattern", 5)
115
115
val listRDD = keysRDD.getList
116
+ #listRDD is a RDD holds all the lists' contents, and the lists' names must be of keyPattern and exists in the redis-server.
116
117
```
117
- listRDD is a RDD holds all the lists' contents, and the lists' names must be of keyPattern and exists in the redis-server.
118
118
119
119
Using Redis' Set
120
120
```
121
121
import com.redislabs.provider.redis._
122
122
val keysRDD = sc.fromRedisKeyPattern(("127.0.0.1", 7000), "keyPattern", 5)
123
123
val setRDD = keysRDD.getSet
124
+ #setRDD is a RDD holds all the sets' contents, score), and the sets' names must be of keyPattern and exists in the redis-server.
124
125
```
125
- setRDD is a RDD holds all the sets' contents, score), and the sets' names must be of keyPattern and exists in the redis-server.
126
126
127
127
*****
128
128
@@ -135,37 +135,37 @@ Saving as Redis' Key/Values
135
135
import com.redislabs.provider.redis._
136
136
val kvRDD = ...
137
137
sc.toRedisKV(kvRDD, ("127.0.0.1", 7000))
138
+ #kvRDD is a RDD holds k/v pairs, we will store all the k/v pairs of kvRDD to the redis-server
138
139
```
139
- kvRDD is a RDD holds k/v pairs, we will store all the k/v pairs of kvRDD to the redis-server
140
140
141
141
Saving as Redis' Hash
142
142
```
143
143
import com.redislabs.provider.redis._
144
144
val hashRDD = ...
145
145
sc.toRedisHASH(hashRDD, hashName, ("127.0.0.1", 7000))
146
+ #hashRDD is a RDD holds k/v pairs, we will store all the k/v pairs of hashRDD to a dict named hashName to the redis-server
146
147
```
147
- hashRDD is a RDD holds k/v pairs, we will store all the k/v pairs of hashRDD to a dict named hashName to the redis-server
148
148
149
149
Saving as Redis' ZSet
150
150
```
151
151
import com.redislabs.provider.redis._
152
152
val zsetRDD = ...
153
153
sc.toRedisZSET(zsetRDD, zsetName, ("127.0.0.1", 7000))
154
+ #zsetRDD is a RDD holds k/v pairs, we will store all the k/v pairs of zsetRDD to a zset named zsetName to the redis-server
154
155
```
155
- zsetRDD is a RDD holds k/v pairs, we will store all the k/v pairs of zsetRDD to a zset named zsetName to the redis-server
156
156
157
157
Saving as Redis' List
158
158
```
159
159
import com.redislabs.provider.redis._
160
160
val listRDD = ...
161
161
sc.toRedisLIST(listRDD, listName, ("127.0.0.1", 7000))
162
+ #listRDD is a RDD holds strings, we will store all the strings of listRDD to a list named listName to the redis-server
162
163
```
163
- listRDD is a RDD holds strings, we will store all the strings of listRDD to a list named listName to the redis-server
164
164
165
165
Saving as Redis' Set
166
166
```
167
167
import com.redislabs.provider.redis._
168
168
val setRDD = ...
169
169
sc.toRedisSET(setRDD, setName, ("127.0.0.1", 7000))
170
+ #setRDD is a RDD holds strings, we will store all the unique strings of setRDD to a set named setName to the redis-server
170
171
```
171
- setRDD is a RDD holds strings, we will store all the unique strings of setRDD to a set named setName to the redis-server
0 commit comments