- Drop support for all Spark version less than 2.y.z
- Add Spark 3.5.1
- Add Hadoop 3.3.6
- Add support for Java 17 for Spark 3.5.1
- Fix Ubuntu-based images to use
jre-focalvariant instead ofjrewhich was recently upgraded to Ubuntu Jammy to v22.y.z and causing system level python package installation to fail due to PEP 668 - Remove docker tags without self version
- (Temporarily drop support for R due to keyserver issues)
- Only supports for for 3.1.3, 3.2.2, 3.3.0, 3.4.1 (dropped 2.4.8).
- Supports both Java 8 and 11 for Spark 3 builds.
- Add Ubuntu-based image since the migration to eclipse-temurin for jre image source.
- Add support up to 3.1.1 for Spark 3.y.z.
- Drop 2.4.5 and 2.4.6 and only support 2.4.7 as the last supported 2.4.z version.
- Previously supported Spark 3.0.0 previews, but drop in favor to use the official Spark 3.0.0. Also support from 2.4.5 and 2.4.6 that comes with Debian-based image support.
- Debian-based image since the earliest preview release tag
v3.0.0-preview-rc1usesopenjdk:8-jdk-slim. See:
- Basic setup for Spark 2.3.4 and Spark 2.4.4.
- Alpine-based image since Spark pre-v3 uses
openjdk:8-alpine. See: https://github.com/apache/spark/blob/v2.4.4/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile#L18