Merge pull request #33 from andre-marcos-perez/develop

Develop

Author: andre-marcos-perez

.github/ISSUE_TEMPLATE/bug_report.md

@@ -42,5 +42,5 @@ Please fill the template below.
 
 Please provide the following:
 
-- [] Docker Engine version: *Can be found using `docker version`, e.g.: 19.03.6*
-- [] Docker Compose version: *Can be found using `docker-compose version`, e.g.: 1.21.0*
+- [ ] Docker Engine version: *Can be found using `docker version`, e.g.: 19.03.6*
+- [ ] Docker Compose version: *Can be found using `docker-compose version`, e.g.: 1.21.0*

.github/pull_request_template.md

@@ -7,7 +7,7 @@ parallel computing in distributed environments through our projects. :sparkles:
 
 ### Issue
 
-- *Issue number with link, e.g.: [#22](https://github.com/andre-marcos-perez/spark-standalone-cluster-on-docker/issues/22)*
+- *Issue number with link, e.g.: #22
 
 ### Changes
 
@@ -23,5 +23,5 @@ parallel computing in distributed environments through our projects. :sparkles:
 
 Please make sure to check the following:
 
-- [] I have followed the steps in the [CONTRIBUTING.md](../CONTRIBUTING.md) file.
-- [] I am aware that pull requests that do not follow the rules will be automatically rejected.
+- [ ] I have followed the steps in the [CONTRIBUTING.md](../CONTRIBUTING.md) file.
+- [ ] I am aware that pull requests that do not follow the rules will be automatically rejected.

CHANGELOG.md

@@ -2,7 +2,14 @@
 
 All notable changes to this project will be documented in this file.
 
-## [v1.1.0](https://github.com/andre-marcos-perez/spark-standalone-cluster-on-docker/releases/tag/v1.1.0) (2020-08-09)
+## [1.2.0](https://github.com/andre-marcos-perez/spark-standalone-cluster-on-docker/releases/tag/v1.2.0) (2020-08-19)
+
+### Features
+
+ - R kernel for JupyterLab;
+ - Jupyter notebook with Spark R API (SparkR) example.
+
+## [1.1.0](https://github.com/andre-marcos-perez/spark-standalone-cluster-on-docker/releases/tag/v1.1.0) (2020-08-09)
 
 ### Features
 
@@ -14,7 +21,7 @@ All notable changes to this project will be documented in this file.
  - Docs general improvements;
  - Pull request template refactored.
 
-## [v1.0.0](https://github.com/andre-marcos-perez/spark-standalone-cluster-on-docker/releases/tag/v1.0.0) (2020-07-30)
+## [1.0.0](https://github.com/andre-marcos-perez/spark-standalone-cluster-on-docker/releases/tag/v1.0.0) (2020-07-30)
 
 ### Tech Stack
 

CONTRIBUTING.md

@@ -15,11 +15,12 @@ parallel computing in distributed environments through our projects. :sparkles:
 
 ### Contributions ideas
 
-- [] Microsoft Windows build script;
+- [ ] Microsoft Windows build script;
 - [x] Docker Hub CI/CD integration;
-- [] Spark submit support;
+- [ ] Spark submit support;
 - [x] JupyterLab Scala kernel;
 - [x] Jupyter notebook with Apache Spark Scala API examples;
-- [] JupyterLab R kernel;
-- [] Jupyter notebook with Apache Spark R API examples;
-- [] Test coverage.
+- [x] JupyterLab R kernel;
+- [x] Jupyter notebook with Apache Spark R API examples;
+- [ ] Test coverage;
+- [ ] Ever growing examples.

README.md

@@ -1,8 +1,9 @@
 # Apache Spark Standalone Cluster on Docker
+
 > The project just got its [own article](https://towardsdatascience.com/apache-spark-cluster-on-docker-ft-a-juyterlab-interface-418383c95445) at Towards Data Science Medium blog! :sparkles:
 
 This project gives you an **Apache Spark** cluster in standalone mode with a **JupyterLab** interface built on top of **Docker**.
-Learn Apache Spark through its Scala and Python API (PySpark) by running the Jupyter [notebooks](build/workspace/) with examples on how to read, process and write data.
+Learn Apache Spark through its **Scala**, **Python** (PySpark) and **R** (SparkR) API by running the Jupyter [notebooks](build/workspace/) with examples on how to read, process and write data.
 
 <p align="center"><img src="docs/image/cluster-architecture.png"></p>
 
@@ -13,6 +14,7 @@ Learn Apache Spark through its Scala and Python API (PySpark) by running the Jup
 ![docker-compose-file-version](https://img.shields.io/badge/docker--compose-v1.10.0%2B-blue)
 ![spark-scala-api](https://img.shields.io/badge/spark%20api-scala-red)
 ![spark-pyspark-api](https://img.shields.io/badge/spark%20api-pyspark-red)
+![spark-sparkr-api](https://img.shields.io/badge/spark%20api-sparkr-red)
 
 ## TL;DR
 
@@ -33,12 +35,12 @@ docker-compose up
 
 ### Cluster overview
 
-| Application            | URL                                      | Description                                                 |
-| ---------------------- | ---------------------------------------- | ----------------------------------------------------------- |
-| JupyterLab             | [localhost:8888](http://localhost:8888/) | Cluster interface with Scala and PySpark built-in notebooks |
-| Apache Spark Master    | [localhost:8080](http://localhost:8080/) | Spark Master node                                           |
-| Apache Spark Worker I  | [localhost:8081](http://localhost:8081/) | Spark Worker node with 1 core and 512m of memory (default)  |
-| Apache Spark Worker II | [localhost:8082](http://localhost:8082/) | Spark Worker node with 1 core and 512m of memory (default)  |
+| Application            | URL                                      | Description                                                |
+| ---------------------- | ---------------------------------------- | ---------------------------------------------------------- |
+| JupyterLab             | [localhost:8888](http://localhost:8888/) | Cluster interface with built-in Jupyter notebooks          |
+| Apache Spark Master    | [localhost:8080](http://localhost:8080/) | Spark Master node                                          |
+| Apache Spark Worker I  | [localhost:8081](http://localhost:8081/) | Spark Worker node with 1 core and 512m of memory (default) |
+| Apache Spark Worker II | [localhost:8082](http://localhost:8082/) | Spark Worker node with 1 core and 512m of memory (default) |
 
 ### Prerequisites
 
@@ -54,7 +56,7 @@ docker-compose up
 docker-compose up
 ```
 
-4. Run Apache Spark code using the provided Jupyter [notebooks](build/workspace/) with Scala and PySpark examples;
+4. Run Apache Spark code using the provided Jupyter [notebooks](build/workspace/) with Scala, PySpark and SparkR examples;
 5. Stop the cluster by typing `ctrl+c`.
 
 ### Build from your local machine
@@ -82,7 +84,7 @@ chmod +x build.sh ; ./build.sh
 docker-compose up
 ```
 
-7. Run Apache Spark code using the provided Jupyter [notebooks](build/workspace/) with Scala and PySpark examples;
+7. Run Apache Spark code using the provided Jupyter [notebooks](build/workspace/) with Scala, PySpark and SparkR examples;
 8. Stop the cluster by typing `ctrl+c`.
 
 ## <a name="tech-stack"></a>Tech Stack
@@ -93,15 +95,17 @@ docker-compose up
 | -------------- | ------- |
 | Docker Engine  | 1.13.0+ |
 | Docker Compose | 1.10.0+ |
-| Python         | 3.7     |
-| Scala          | 2.12    |
+| Python         | 3.7.3   |
+| Scala          | 2.12.11 |
+| R              | 3.5.2   |
 
 - Jupyter Kernels
 
-| Component      | Version | Provider                        |
-| -------------- | ------- | ------------------------------- |
-| Python         | 2.1.4   | [Jupyter](https://jupyter.org/) |
-| Scala          | 0.10.0  | [Almond](https://almond.sh/)    |
+| Component      | Version | Provider                                |
+| -------------- | ------- | --------------------------------------- |
+| Python         | 2.1.4   | [Jupyter](https://jupyter.org/)         |
+| Scala          | 0.10.0  | [Almond](https://almond.sh/)            |
+| R              | 1.1.1   | [IRkernel](https://irkernel.github.io/) |
 
 - Applications
 
@@ -110,18 +114,22 @@ docker-compose up
 | Apache Spark   | 2.4.0 \| 2.4.4 \| 3.0.0 | **\<spark-version>**-hadoop-2.7                      |
 | JupyterLab     | 2.1.4                   | **\<jupyterlab-version>**-spark-**\<spark-version>** |
 
+> Apache Spark R API (SparkR) is only supported on version **2.4.4**. Full list can be found [here](https://cran.r-project.org/src/contrib/Archive/SparkR/).
+
 ## <a name="docker-hub-metrics"></a>Docker Hub Metrics
 
-| Image                                                          | Latest Version Size                                                                                 | Downloads                                                                 |
-| -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
-| [JupyterLab](https://hub.docker.com/r/andreper/jupyterlab)     | ![docker-size](https://img.shields.io/docker/image-size/andreper/jupyterlab/latest)                 | ![docker-pull](https://img.shields.io/docker/pulls/andreper/jupyterlab)   |
-| [Spark Master](https://hub.docker.com/r/andreper/spark-master) | ![docker-size](https://img.shields.io/docker/image-size/andreper/spark-master/latest)               | ![docker-pull](https://img.shields.io/docker/pulls/andreper/spark-master) |
-| [Spark Worker](https://hub.docker.com/r/andreper/spark-worker) | ![docker-size](https://img.shields.io/docker/image-size/andreper/spark-worker/latest)               | ![docker-pull](https://img.shields.io/docker/pulls/andreper/spark-worker) |
+| Image                                                          | Size                                                                                           | Downloads                                                                 |
+| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
+| [JupyterLab](https://hub.docker.com/r/andreper/jupyterlab)     | ![docker-size-jupyterlab](https://img.shields.io/docker/image-size/andreper/jupyterlab/latest) | ![docker-pull](https://img.shields.io/docker/pulls/andreper/jupyterlab)   |
+| [Spark Master](https://hub.docker.com/r/andreper/spark-master) | ![docker-size-master](https://img.shields.io/docker/image-size/andreper/spark-master/latest)   | ![docker-pull](https://img.shields.io/docker/pulls/andreper/spark-master) |
+| [Spark Worker](https://hub.docker.com/r/andreper/spark-worker) | ![docker-size-worker](https://img.shields.io/docker/image-size/andreper/spark-worker/latest)   | ![docker-pull](https://img.shields.io/docker/pulls/andreper/spark-worker) |
 
 ## <a name="contributing"></a>Contributing
 
 We'd love some help. To contribute, please read [this file](CONTRIBUTING.md).
 
+> Staring us on GitHub is also an awesome way to show your support :star:
+
 ## <a name="contributors"></a>Contributors
 
- - **André Perez** - [dekoperez](https://twitter.com/dekoperez) - andre.marcos.perez@gmail.com
+ - **André Perez** - [dekoperez](https://twitter.com/dekoperez) - andre.marcos.perez@gmail.com

build/build.yml

@@ -1,6 +1,6 @@
 applications:
   scala: "2.12.11"
-  spark: "3.0.0"
+  spark: "2.4.4"
   hadoop: "2.7"
   jupyterlab: "2.1.4"
 build:

build/docker-compose.yml

@@ -9,22 +9,22 @@ volumes:
     driver: local
 services:
   jupyterlab:
-    image: jupyterlab:2.1.4-spark-3.0.0
+    image: jupyterlab:2.1.4-spark-2.4.4
     container_name: jupyterlab
     ports:
       - 8888:8888
     volumes:
       - shared-workspace:/opt/workspace
   spark-master:
-    image: spark-master:3.0.0-hadoop-2.7
+    image: spark-master:2.4.4-hadoop-2.7
     container_name: spark-master
     ports:
       - 8080:8080
       - 7077:7077
     volumes:
       - shared-workspace:/opt/workspace
   spark-worker-1:
-    image: spark-worker:3.0.0-hadoop-2.7
+    image: spark-worker:2.4.4-hadoop-2.7
     container_name: spark-worker-1
     environment:
       - SPARK_WORKER_CORES=1
@@ -36,7 +36,7 @@ services:
     depends_on:
       - spark-master
   spark-worker-2:
-    image: spark-worker:3.0.0-hadoop-2.7
+    image: spark-worker:2.4.4-hadoop-2.7
     container_name: spark-worker-2
     environment:
       - SPARK_WORKER_CORES=1

build/docker/base/Dockerfile

@@ -20,7 +20,7 @@ ARG shared_workspace=/opt/workspace
 RUN mkdir -p ${shared_workspace}/data && \
     mkdir -p /usr/share/man/man1 && \
     apt-get update -y && \
-    apt-get install -y curl python3 scala && \
+    apt-get install -y curl python3 r-base && \
     ln -s /usr/bin/python3 /usr/bin/python && \
     curl https://downloads.lightbend.com/scala/${scala_version}/scala-${scala_version}.deb -k -o scala.deb && \
     apt install -y ./scala.deb && \

build/docker/jupyterlab/Dockerfile

@@ -11,7 +11,7 @@ LABEL org.label-schema.description="JupyterLab image"
 LABEL org.label-schema.url="https://github.com/andre-marcos-perez/spark-cluster-on-docker"
 LABEL org.label-schema.schema-version="1.0"
 
-# -- Layer: JupyterLab + Python kernel
+# -- Layer: JupyterLab + Python kernel for PySpark
 
 ARG spark_version
 ARG jupyterlab_version
@@ -21,7 +21,7 @@ RUN apt-get update -y && \
     pip3 install --upgrade pip && \
     pip3 install pyspark==${spark_version} jupyterlab==${jupyterlab_version}
 
-# -- Layer: Scala kernel
+# -- Layer: Scala kernel for Spark
 
 ARG scala_version
 
@@ -31,6 +31,17 @@ RUN apt-get install -y ca-certificates-java --no-install-recommends && \
     ./coursier launch --fork almond:0.10.0 --scala ${scala_version} -- --display-name "Scala ${scala_version}" --install && \
     rm -f coursier
 
+# -- Layer: R kernel for SparkR
+
+COPY ./script/sparkr.sh ./sparkr.sh
+
+RUN apt-get install -y r-base-dev && \
+    R -e "install.packages('IRkernel')" && \
+    R -e "IRkernel::installspec(displayname = 'R 3.5', user = FALSE)" && \
+    chmod +x ./sparkr.sh && \
+    ./sparkr.sh ${spark_version} && \
+    rm -f sparkr.sh
+
 # -- Runtime
 
 EXPOSE 8888

build/script/sparkr.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+#
+# -- Download and install Apache Spark R API (SparkR)
+
+# ----------------------------------------------------------------------------------------------------------------------
+# -- Variables ---------------------------------------------------------------------------------------------------------
+# ----------------------------------------------------------------------------------------------------------------------
+
+SPARK_VERSION="${1}"
+
+# ----------------------------------------------------------------------------------------------------------------------
+# -- Main --------------------------------------------------------------------------------------------------------------
+# ----------------------------------------------------------------------------------------------------------------------
+
+if [[ "${SPARK_VERSION}" =~ ^(2.1.2|2.3.0|2.4.1|2.4.2|2.4.3|2.4.4|2.4.5|2.4.6)$ ]]
+then
+  curl https://cran.r-project.org/src/contrib/Archive/SparkR/SparkR_${SPARK_VERSION}.tar.gz -k -o sparkr.tar.gz
+  R CMD INSTALL sparkr.tar.gz
+  rm -f sparkr.tar.gz
+fi