Skip to content

Commit 47c4a53

Browse files
committed
feat: review Comments
Address the review comments Also adjusted the build so it was connected to the main top level build (resulted in removing quite a bit of the gradle build that really wasn't needed here) Converted to Kotlin format (hopefully corectly) Signed-off-by: MBWhite <[email protected]>
1 parent d7ea257 commit 47c4a53

27 files changed

+117
-591
lines changed

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,5 @@ out/**
99
*.iws
1010
.vscode
1111
.pmdCache
12+
13+
*/bin

examples/substrait-spark/.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,4 @@
1+
spark-warehouse
2+
derby.log
13
_apps
24
_data

examples/substrait-spark/README.md

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
# Introduction to the Substrait-Spark library
22

3-
The Substrait-Spark library was recently added to the [substrait-java](https://github.com/substrait-io/substrait-java) project; this library allows Substrait plans to convert to and from Spark Plans.
4-
3+
The Substrait-Spark library allows Substrait plans to convert to and from Spark Plans. This example will show how this can be used.
54

65
## How does this work in practice?
76

@@ -27,7 +26,7 @@ To run these you will need:
2726
- Java 17 or greater
2827
- Docker to start a test Spark Cluster
2928
- you could use your own cluster, but would need to adjust file locations defined in [SparkHelper](./app/src/main/java/io/substrait/examples/SparkHelper.java)
30-
- [just task runner](https://github.com/casey/just#installation) optional, but very helpful to run the bash commands
29+
- The [just task runner](https://github.com/casey/just#installation) is optional, but very helpful to run the bash commands
3130
- [Two datafiles](./app/src/main/resources/) are provided (CSV format)
3231

3332
For building using the `substrait-spark` library youself, using the [mvn repository](https://mvnrepository.com/artifact/io.substrait/spark)

examples/substrait-spark/app/.gitignore

Lines changed: 0 additions & 2 deletions
This file was deleted.

examples/substrait-spark/app/build.gradle

Lines changed: 0 additions & 62 deletions
This file was deleted.

examples/substrait-spark/build-logic/build.gradle

Lines changed: 0 additions & 16 deletions
This file was deleted.

examples/substrait-spark/build-logic/settings.gradle

Lines changed: 0 additions & 15 deletions
This file was deleted.

examples/substrait-spark/build-logic/src/main/groovy/buildlogic.java-application-conventions.gradle

Lines changed: 0 additions & 13 deletions
This file was deleted.

examples/substrait-spark/build-logic/src/main/groovy/buildlogic.java-common-conventions.gradle

Lines changed: 0 additions & 39 deletions
This file was deleted.

examples/substrait-spark/build-logic/src/main/groovy/buildlogic.java-library-conventions.gradle

Lines changed: 0 additions & 13 deletions
This file was deleted.
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
2+
plugins {
3+
// Apply the application plugin to add support for building a CLI application in Java.
4+
id("java")
5+
}
6+
7+
repositories {
8+
// Use Maven Central for resolving dependencies.
9+
mavenCentral()
10+
}
11+
12+
dependencies {
13+
implementation("org.apache.spark:spark-core_2.12:3.5.1")
14+
implementation("io.substrait:spark:0.36.0")
15+
implementation("io.substrait:core:0.36.0")
16+
implementation("org.apache.spark:spark-sql_2.12:3.5.1")
17+
18+
// For a real Spark application, these would not be required since they would be in the Spark server classpath
19+
runtimeOnly("org.apache.spark:spark-core_2.12:3.5.1")
20+
runtimeOnly("org.apache.spark:spark-hive_2.12:3.5.1")
21+
22+
}
23+
24+
tasks.jar {
25+
isZip64 = true
26+
exclude ("META-INF/*.RSA")
27+
exclude ("META-INF/*.SF")
28+
exclude ("META-INF/*.DSA")
29+
30+
duplicatesStrategy = DuplicatesStrategy.EXCLUDE
31+
manifest.attributes["Main-Class"] = "io.substrait.examples.App"
32+
from(configurations.runtimeClasspath.get().map({ if (it.isDirectory) it else zipTree(it) }))
33+
34+
}
35+
36+
tasks.named<Test>("test") {
37+
// Use JUnit Platform for unit tests.
38+
useJUnitPlatform()
39+
}
40+
41+
java {
42+
toolchain { languageVersion.set(JavaLanguageVersion.of(17)) }
43+
}
44+

examples/substrait-spark/gradle.properties

Lines changed: 0 additions & 6 deletions
This file was deleted.

examples/substrait-spark/gradle/libs.versions.toml

Lines changed: 0 additions & 14 deletions
This file was deleted.
Binary file not shown.

examples/substrait-spark/gradle/wrapper/gradle-wrapper.properties

Lines changed: 0 additions & 7 deletions
This file was deleted.

0 commit comments

Comments
 (0)