Skip to content

Commit 661a6d9

Browse files
angelcerveraangelcervera
committed
Update documentation before release.
1 parent db1c75b commit 661a6d9

File tree

3 files changed

+23
-23
lines changed

3 files changed

+23
-23
lines changed

‎examples/spark-utilities/README.md

+4-4
Original file line numberDiff line numberDiff line change
@@ -24,19 +24,19 @@ Tags extraction.
2424

2525
```shell script
2626
bin/spark-submit \
27-
--packages 'com.github.scopt:scopt_2.12:3.7.1,com.acervera.osm4scala:osm4scala-spark-shaded_2.12:1.0.10' \
27+
--packages 'com.github.scopt:scopt_2.12:3.7.1,com.acervera.osm4scala:osm4scala-spark-shaded_2.12:1.0.11' \
2828
--class com.acervera.osm4scala.examples.spark.Driver \
29-
"osm4scala-examples-spark-utilities_2.12-1.0.10-SNAPSHOT.jar" \
29+
"osm4scala-examples-spark-utilities_2.12-1.0.11.jar" \
3030
counter \
3131
-i <pbf files path> \
3232
-o <output> \
3333
-c 1 \
3434
-f csv
3535

3636
bin/spark-submit \
37-
--packages 'com.github.scopt:scopt_2.12:3.7.1,com.acervera.osm4scala:osm4scala-spark-shaded_2.12:1.0.10' \
37+
--packages 'com.github.scopt:scopt_2.12:3.7.1,com.acervera.osm4scala:osm4scala-spark-shaded_2.12:1.0.11' \
3838
--class com.acervera.osm4scala.examples.spark.Driver \
39-
"osm4scala-examples-spark-utilities_2.12-1.0.10-SNAPSHOT.jar" \
39+
"osm4scala-examples-spark-utilities_2.12-1.0.11.jar" \
4040
tag_keys \
4141
-i <pbf files path> \
4242
-o <output> \

‎website/docs/spark-connector.mdx

+18-18
Original file line numberDiff line numberDiff line change
@@ -90,13 +90,13 @@ necessary.
9090
The fat jar is near 5MB, so the size should be not a problem.
9191

9292
As you probably know, Spark is base in Scala. Different Spark distributions are using different Scala versions.
93-
This is the Spark/Scala version combination available for latest release v1.0.10:
93+
This is the Spark/Scala version combination available for latest release v1.0.11:
9494

9595
| Spark Branch | Scala | Packages |
9696
|:------------:|:------:|:---------|
97-
| 2.4 | 2.11 | [`com.acervera.osm4scala:osm4scala-spark2-shaded_2.11:1.0.10`](https://search.maven.org/artifact/com.acervera.osm4scala/osm4scala-spark2-shaded_2.11/1.0.10/jar)
98-
| 2.4 | 2.12 | [`com.acervera.osm4scala:osm4scala-spark2-shaded_2.12:1.0.10`](https://search.maven.org/artifact/com.acervera.osm4scala/osm4scala-spark2-shaded_2.12/1.0.10/jar)
99-
| 3.0 / 3.1 | 2.12 | [`com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.10`](https://search.maven.org/artifact/com.acervera.osm4scala/osm4scala-spark3-shaded_2.12/1.0.10/jar)
97+
| 2.4 | 2.11 | [`com.acervera.osm4scala:osm4scala-spark2-shaded_2.11:1.0.11`](https://search.maven.org/artifact/com.acervera.osm4scala/osm4scala-spark2-shaded_2.11/1.0.11/jar)
98+
| 2.4 | 2.12 | [`com.acervera.osm4scala:osm4scala-spark2-shaded_2.12:1.0.11`](https://search.maven.org/artifact/com.acervera.osm4scala/osm4scala-spark2-shaded_2.12/1.0.11/jar)
99+
| 3.0 / 3.1 | 2.12 | [`com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11`](https://search.maven.org/artifact/com.acervera.osm4scala/osm4scala-spark3-shaded_2.12/1.0.11/jar)
100100

101101
Although following sections are focus on Spark Shell and Notebooks, you can use the same technique in other situations where
102102
you want to use the shaded version.
@@ -115,13 +115,13 @@ To solve the conflict, I published the library in two fashion:
115115
1. Start the spark shell as usual, using the `--packages` option to add the right dependency. The dependency will depend to
116116
the Spark Version that you are using. Please, check the reference table in the previous section.
117117
```shell title="Scala"
118-
bin/spark-shell --packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.10'
118+
bin/spark-shell --packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11'
119119
```
120120
```scala title="PySpark"
121-
bin/pyspark --packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.10'
121+
bin/pyspark --packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11'
122122
```
123123
```scala title="SQL"
124-
bin/spark-sql --packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.10'
124+
bin/spark-sql --packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11'
125125
```
126126

127127
2. Create the Dataframe using the osm.pbf format, pointing to the pbf file or folder containing pbf files.
@@ -258,7 +258,7 @@ If you prefer an online option, you can try [MyBinder](https://mybinder.org/v2/g
258258
of the [All in one jar](#all-in-one-jar) section. In or case, the version used is `Spark v3.1.1` with `Scala 2.12`.
259259
```jypiter
260260
%%init_spark
261-
launcher.packages = ["com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.10"]
261+
launcher.packages = ["com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11"]
262262
```
263263
If you did not execute anything before, running the cell will start the Spark session. Sometime, depending to the
264264
Notebook used, **you will need to restart the Spark session (or Kernel session)**.
@@ -304,13 +304,13 @@ When we need to write more complex analysis, data extractions, ETLs, etc, it is
304304
can not *easily* to import and use facilities from the Scala library. So in this case, you can jump to the next step.
305305

306306
```sbt title="Sbt"
307-
libraryDependencies += "com.acervera.osm4scala" % "osm4scala-spark3-shaded_2.12" % "1.0.10"
307+
libraryDependencies += "com.acervera.osm4scala" % "osm4scala-spark3-shaded_2.12" % "1.0.11"
308308
```
309309
```xml title="Maven"
310310
<dependency>
311311
<groupId>com.acervera.osm4scala</groupId>
312312
<artifactId>osm4scala-spark3-shaded_2.12</artifactId>
313-
<version>1.0.10</version>
313+
<version>1.0.11</version>
314314
</dependency>
315315
```
316316
:::tip Reduce artifact size.
@@ -364,20 +364,20 @@ When we need to write more complex analysis, data extractions, ETLs, etc, it is
364364
3. Submit the application to your Spark cluster.
365365
```shell title="Scala"
366366
bin/spark-submit \
367-
--packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.10' \
368-
examples/spark-documentation/target/scala-2.12/osm4scala-examples-spark-documentation_2.12-1.0.10-SNAPSHOT.jar \
367+
--packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11' \
368+
examples/spark-documentation/target/scala-2.12/osm4scala-examples-spark-documentation_2.12-1.0.11.jar \
369369
/tmp/osm/monaco-anonymized.osm.pbf
370370
```
371371
```shell title="PySpark"
372372
bin/spark-submit \
373-
--packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.10' \
373+
--packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11' \
374374
examples/spark-documentation/src/main/scala/com/acervera/osm4scala/examples/spark/documentation/PrimiriveCounter.py \
375375
/tmp/osm/monaco-anonymized.osm.pbf
376376
```
377377

378378
:::note Optional --packages.
379379

380-
You will not need to add `--packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.10'` if it is part of the
380+
You will not need to add `--packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11'` if it is part of the
381381
deployed artifact.
382382

383383
:::
@@ -545,13 +545,13 @@ In that case, the best practice is to manage dependencies using `sbt` or `maven`
545545
OSM Pbf files are based on [Protocol Buffer](https://developers.google.com/protocol-buffers), so [Scalapb](https://scalapb.github.io) is
546546
used as deserializer so it's the unique transitive dependency.
547547

548-
This is the Spark/Scala version combination available for latest release v1.0.10:
548+
This is the Spark/Scala version combination available for latest release v1.0.11:
549549

550550
| Spark branch | Scalapb | Scala | Packages |
551551
|:------------:|:-------:|:------:|:---------|
552-
| 2.4 | 0.9.7 | 2.11 | [`com.acervera.osm4scala:osm4scala-spark2_2.11:1.0.10`](https://search.maven.org/artifact/com.acervera.osm4scala/osm4scala-spark2_2.11/1.0.10/jar)
553-
| 2.4 | 0.10.2 | 2.12 | [`com.acervera.osm4scala:osm4scala-spark2_2.12:1.0.10`](https://search.maven.org/artifact/com.acervera.osm4scala/osm4scala-spark2_2.12/1.0.10/jar)
554-
| 3.0 / 3.1 | 0.10.2 | 2.12 | [`com.acervera.osm4scala:osm4scala-spark3_2.12:1.0.10`](https://search.maven.org/artifact/com.acervera.osm4scala/osm4scala-spark3_2.12/1.0.10/jar)
552+
| 2.4 | 0.9.7 | 2.11 | [`com.acervera.osm4scala:osm4scala-spark2_2.11:1.0.11`](https://search.maven.org/artifact/com.acervera.osm4scala/osm4scala-spark2_2.11/1.0.11/jar)
553+
| 2.4 | 0.10.2 | 2.12 | [`com.acervera.osm4scala:osm4scala-spark2_2.12:1.0.11`](https://search.maven.org/artifact/com.acervera.osm4scala/osm4scala-spark2_2.12/1.0.11/jar)
554+
| 3.0 / 3.1 | 0.10.2 | 2.12 | [`com.acervera.osm4scala:osm4scala-spark3_2.12:1.0.11`](https://search.maven.org/artifact/com.acervera.osm4scala/osm4scala-spark3_2.12/1.0.11/jar)
555555

556556
After importing the connector, you can use it as we explained in the [All in one section](#all-in-one-jar). So lets see
557557
how to import the library in our project and few examples.

‎website/static/notebooks/spylon_notebook_example.ipynb

+1-1
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
"outputs": [],
99
"source": [
1010
"%%init_spark\n",
11-
"launcher.packages = [\"com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.10\"]"
11+
"launcher.packages = [\"com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11\"]"
1212
]
1313
},
1414
{

0 commit comments

Comments
 (0)