@@ -34,17 +34,18 @@ fun main() {
34
34
}
35
35
```
36
36
37
- ## Building the application with Maven
38
-
37
+ ## Building the application
39
38
Because Kotlin Spark API is not part of the official Apache Spark distribution yet, it is not enough to add Spark
40
- as a dependency in your pom.xml file.
39
+ as a dependency to your build file.
41
40
You need to:
42
41
- Add Spark as a dependency
43
42
- Add Kotlin Spark API as a dependency
44
43
- Add Kotlin Standard Library as a dependency
45
44
46
45
When packaging your project into a jar file, you need to explicitly include Kotlin Spark API and Kotlin Standard Library
47
- dependencies, for example, using ` maven-shade-plugin ` .
46
+ dependencies. Here you can find an example of building your application with Maven, and with Gradle.
47
+
48
+ ### Building the application with Maven
48
49
49
50
Here's what the ` pom.xml ` looks like for this example:
50
51
``` xml
@@ -148,17 +149,55 @@ Here's what the project structure should look like:
148
149
149
150
Now you can package the application using Maven:
150
151
` mvn package `
151
-
152
- When done, you can execute the packaged application with ` ./bin/spark-submit ` :
153
152
154
- ` YOUR_SPARK_HOME/bin/spark-submit --class "SimpleApp" --master local YOUR_PROJECT/target/kotlin-spark-example-1.0-SNAPSHOT.jar `
153
+ ### Building the application with Gradle
155
154
156
- This example is also available as a [ GitHub repo ] ( https://github.com/MKhalusova/kotlin-spark-example ) , feel free to give it a try.
155
+ Here's what the ` build.gradle ` looks like for this example:
157
156
157
+ ```
158
+ plugins {
159
+ id 'org.jetbrains.kotlin.jvm' version '1.3.72'
160
+ id 'com.github.johnrengelman.shadow' version '5.2.0'
161
+ }
158
162
163
+ group = 'org.example'
164
+ version = '1.0-SNAPSHOT'
159
165
160
-
166
+ repositories {
167
+ mavenCentral()
168
+ maven { url = 'https://jitpack.io' }
169
+ }
161
170
171
+ dependencies {
172
+ // Kotlin stdlib
173
+ implementation 'org.jetbrains.kotlin:kotlin-stdlib:1.3.72'
174
+ // Kotlin Spark API
175
+ implementation 'com.github.JetBrains.kotlin-spark-api:kotlin-spark-api:0.1.0'
176
+ // Apache Spark
177
+ compileOnly 'org.apache.spark:spark-sql_2.12:3.0.0-preview2'
178
+ }
179
+
180
+ compileKotlin {
181
+ kotlinOptions.jvmTarget = '1.8'
182
+ }
183
+
184
+ shadowJar {
185
+ dependencies {
186
+ exclude(dependency {
187
+ it.moduleGroup == 'org.apache.spark' || it.moduleGroup == "org.scala-lang"
188
+ })
189
+ }
190
+ }
191
+ ```
192
+
193
+ Now you can package the application using Gradle:
194
+ ` gradle shadowJar `
162
195
163
196
164
-
197
+ ## Executing the application with spark-submit
198
+
199
+ Once you have your jar, you can execute the packaged application with ` ./bin/spark-submit ` :
200
+
201
+ ` YOUR_SPARK_HOME/bin/spark-submit --class "SimpleApp" --master local [path to your jar] `
202
+
203
+ This example is also available as a [ GitHub repo] ( https://github.com/MKhalusova/kotlin-spark-example ) , feel free to give it a try.
0 commit comments