Task 3.2 b.) -  Mutli-class classifier using PySpark .ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"JRkv266S972W"},"source":["#Task 3.2 b.) -  Mutli-class classifier using PySpark .ipynb\n","By Abdulwahab Adeshina Abdulliameed\n"]},{"cell_type":"markdown","metadata":{"id":"82nnS1p_J8vA"},"source":["**Setting Up PySpark in Colab**"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":4720,"status":"ok","timestamp":1648898851039,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"},"user_tz":-60},"id":"lNBpRPQXJPg9","outputId":"7905618c-65c3-446d-81bf-0c9d7825a9aa"},"outputs":[{"output_type":"stream","name":"stdout","text":["\r0% [Working]\r            \rHit:1 http://archive.ubuntu.com/ubuntu bionic InRelease\n","\r0% [Waiting for headers] [Connecting to security.ubuntu.com (185.125.190.36)] [\r0% [1 InRelease gpgv 242 kB] [Waiting for headers] [Connecting to security.ubun\r                                                                               \rGet:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]\n","\r0% [1 InRelease gpgv 242 kB] [2 InRelease 14.2 kB/88.7 kB 16%] [Waiting for hea\r                                                                               \rGet:3 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ InRelease [3,626 B]\n","\r0% [1 InRelease gpgv 242 kB] [2 InRelease 35.9 kB/88.7 kB 40%] [Waiting for hea\r                                                                               \rIgn:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease\n","Get:5 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]\n","Get:6 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]\n","Get:7 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic InRelease [15.9 kB]\n","Ign:8 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease\n","Get:9 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release [696 B]\n","Hit:10 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release\n","Get:11 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release.gpg [836 B]\n","Get:12 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ Packages [80.8 kB]\n","Hit:13 http://ppa.launchpad.net/cran/libgit2/ubuntu bionic InRelease\n","Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3,132 kB]\n","Get:15 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic InRelease [15.9 kB]\n","Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [918 kB]\n","Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2,264 kB]\n","Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [29.8 kB]\n","Get:19 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [12.2 kB]\n","Get:20 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [12.9 kB]\n","Get:22 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease [21.3 kB]\n","Get:23 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Packages [950 kB]\n","Get:24 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main Sources [1,830 kB]\n","Get:25 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,486 kB]\n","Get:26 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [884 kB]\n","Get:27 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2,693 kB]\n","Get:28 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main amd64 Packages [937 kB]\n","Get:29 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic/main amd64 Packages [45.3 kB]\n","Get:30 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic/main amd64 Packages [44.3 kB]\n","Fetched 15.6 MB in 3s (4,525 kB/s)\n","Reading package lists... Done\n"]}],"source":["#Update system packages\n","!sudo apt-get update"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":33923,"status":"ok","timestamp":1648900123282,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"},"user_tz":-60},"id":"u7saEvNpKXEy","outputId":"864ee899-3c2d-4588-e0b2-17c4e9d4d34d"},"outputs":[{"output_type":"stream","name":"stdout","text":["Reading package lists... Done\n","Building dependency tree       \n","Reading state information... Done\n","The following additional packages will be installed:\n","  default-jdk-headless openjdk-11-jdk openjdk-11-jdk-headless openjdk-11-jre\n","  openjdk-11-jre-headless\n","Suggested packages:\n","  openjdk-11-demo openjdk-11-source visualvm libnss-mdns fonts-dejavu-extra\n","  fonts-ipafont-gothic fonts-ipafont-mincho fonts-wqy-microhei\n","  | fonts-wqy-zenhei fonts-indic\n","Recommended packages:\n","  libatk-wrapper-java-jni fonts-dejavu-extra\n","The following NEW packages will be installed:\n","  default-jdk default-jdk-headless openjdk-11-jdk\n","The following packages will be upgraded:\n","  openjdk-11-jdk-headless openjdk-11-jre openjdk-11-jre-headless\n","3 upgraded, 3 newly installed, 0 to remove and 79 not upgraded.\n","Need to get 260 MB of archives.\n","After this operation, 1,610 kB of additional disk space will be used.\n","Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 openjdk-11-jre amd64 11.0.14.1+1-0ubuntu1~18.04 [174 kB]\n","Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 openjdk-11-jdk-headless amd64 11.0.14.1+1-0ubuntu1~18.04 [221 MB]\n","Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 openjdk-11-jre-headless amd64 11.0.14.1+1-0ubuntu1~18.04 [37.2 MB]\n","Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 default-jdk-headless amd64 2:1.11-68ubuntu1~18.04.1 [1,132 B]\n","Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 openjdk-11-jdk amd64 11.0.14.1+1-0ubuntu1~18.04 [1,544 kB]\n","Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 default-jdk amd64 2:1.11-68ubuntu1~18.04.1 [1,092 B]\n","Fetched 260 MB in 5s (51.9 MB/s)\n","debconf: unable to initialize frontend: Dialog\n","debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 6.)\n","debconf: falling back to frontend: Readline\n","debconf: unable to initialize frontend: Readline\n","debconf: (This frontend requires a controlling tty.)\n","debconf: falling back to frontend: Teletype\n","dpkg-preconfigure: unable to re-open stdin: \n","(Reading database ... 156210 files and directories currently installed.)\n","Preparing to unpack .../0-openjdk-11-jre_11.0.14.1+1-0ubuntu1~18.04_amd64.deb ...\n","Unpacking openjdk-11-jre:amd64 (11.0.14.1+1-0ubuntu1~18.04) over (11.0.14+9-0ubuntu2~18.04) ...\n","Preparing to unpack .../1-openjdk-11-jdk-headless_11.0.14.1+1-0ubuntu1~18.04_amd64.deb ...\n","Unpacking openjdk-11-jdk-headless:amd64 (11.0.14.1+1-0ubuntu1~18.04) over (11.0.14+9-0ubuntu2~18.04) ...\n","Preparing to unpack .../2-openjdk-11-jre-headless_11.0.14.1+1-0ubuntu1~18.04_amd64.deb ...\n","Unpacking openjdk-11-jre-headless:amd64 (11.0.14.1+1-0ubuntu1~18.04) over (11.0.14+9-0ubuntu2~18.04) ...\n","Selecting previously unselected package default-jdk-headless.\n","Preparing to unpack .../3-default-jdk-headless_2%3a1.11-68ubuntu1~18.04.1_amd64.deb ...\n","Unpacking default-jdk-headless (2:1.11-68ubuntu1~18.04.1) ...\n","Selecting previously unselected package openjdk-11-jdk:amd64.\n","Preparing to unpack .../4-openjdk-11-jdk_11.0.14.1+1-0ubuntu1~18.04_amd64.deb ...\n","Unpacking openjdk-11-jdk:amd64 (11.0.14.1+1-0ubuntu1~18.04) ...\n","Selecting previously unselected package default-jdk.\n","Preparing to unpack .../5-default-jdk_2%3a1.11-68ubuntu1~18.04.1_amd64.deb ...\n","Unpacking default-jdk (2:1.11-68ubuntu1~18.04.1) ...\n","Setting up openjdk-11-jre-headless:amd64 (11.0.14.1+1-0ubuntu1~18.04) ...\n","Setting up openjdk-11-jdk-headless:amd64 (11.0.14.1+1-0ubuntu1~18.04) ...\n","Setting up default-jdk-headless (2:1.11-68ubuntu1~18.04.1) ...\n","Setting up openjdk-11-jre:amd64 (11.0.14.1+1-0ubuntu1~18.04) ...\n","Setting up openjdk-11-jdk:amd64 (11.0.14.1+1-0ubuntu1~18.04) ...\n","update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jconsole to provide /usr/bin/jconsole (jconsole) in auto mode\n","Setting up default-jdk (2:1.11-68ubuntu1~18.04.1) ...\n","Processing triggers for hicolor-icon-theme (0.17-2) ...\n","Processing triggers for mime-support (3.60ubuntu1) ...\n"]}],"source":["#install Java\n","!sudo apt install default-jdk"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":14300,"status":"ok","timestamp":1648900169447,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"},"user_tz":-60},"id":"di3lVZJjKgSD","outputId":"26ebbd29-05d2-41fc-ebe6-e83756277d79"},"outputs":[{"output_type":"stream","name":"stdout","text":["--2022-04-02 11:49:14--  https://archive.apache.org/dist/spark/spark-3.0.3/spark-3.0.3-bin-hadoop2.7.tgz\n","Resolving archive.apache.org (archive.apache.org)... 138.201.131.134, 2a01:4f8:172:2ec5::2\n","Connecting to archive.apache.org (archive.apache.org)|138.201.131.134|:443... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 220400553 (210M) [application/x-gzip]\n","Saving to: ‘spark-3.0.3-bin-hadoop2.7.tgz’\n","\n","spark-3.0.3-bin-had 100%[===================>] 210.19M  26.8MB/s    in 10s     \n","\n","2022-04-02 11:49:25 (20.2 MB/s) - ‘spark-3.0.3-bin-hadoop2.7.tgz’ saved [220400553/220400553]\n","\n","spark-3.0.3-bin-hadoop2.7/\n","spark-3.0.3-bin-hadoop2.7/NOTICE\n","spark-3.0.3-bin-hadoop2.7/kubernetes/\n","spark-3.0.3-bin-hadoop2.7/kubernetes/tests/\n","spark-3.0.3-bin-hadoop2.7/kubernetes/tests/worker_memory_check.py\n","spark-3.0.3-bin-hadoop2.7/kubernetes/tests/py_container_checks.py\n","spark-3.0.3-bin-hadoop2.7/kubernetes/tests/pyfiles.py\n","spark-3.0.3-bin-hadoop2.7/kubernetes/dockerfiles/\n","spark-3.0.3-bin-hadoop2.7/kubernetes/dockerfiles/spark/\n","spark-3.0.3-bin-hadoop2.7/kubernetes/dockerfiles/spark/entrypoint.sh\n","spark-3.0.3-bin-hadoop2.7/kubernetes/dockerfiles/spark/bindings/\n","spark-3.0.3-bin-hadoop2.7/kubernetes/dockerfiles/spark/bindings/R/\n","spark-3.0.3-bin-hadoop2.7/kubernetes/dockerfiles/spark/bindings/R/Dockerfile\n","spark-3.0.3-bin-hadoop2.7/kubernetes/dockerfiles/spark/bindings/python/\n","spark-3.0.3-bin-hadoop2.7/kubernetes/dockerfiles/spark/bindings/python/Dockerfile\n","spark-3.0.3-bin-hadoop2.7/kubernetes/dockerfiles/spark/Dockerfile\n","spark-3.0.3-bin-hadoop2.7/jars/\n","spark-3.0.3-bin-hadoop2.7/jars/jackson-xc-1.9.13.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-digester-1.8.jar\n","spark-3.0.3-bin-hadoop2.7/jars/api-util-1.0.0-M20.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hive-vector-code-gen-2.3.7.jar\n","spark-3.0.3-bin-hadoop2.7/jars/derby-10.12.1.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-beanutils-1.9.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/httpcore-4.4.12.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hadoop-yarn-api-2.7.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/scala-library-2.12.10.jar\n","spark-3.0.3-bin-hadoop2.7/jars/parquet-format-2.4.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/kryo-shaded-4.0.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/xercesImpl-2.12.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-logging-1.1.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/okio-1.15.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-compiler-3.0.16.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jdo-api-3.0.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spire-macros_2.12-0.17.0-M1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/arrow-memory-0.15.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/JLargeArrays-1.5.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jsp-api-2.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/logging-interceptor-3.12.6.jar\n","spark-3.0.3-bin-hadoop2.7/jars/javax.servlet-api-3.1.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jcl-over-slf4j-1.7.30.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hive-cli-2.3.7.jar\n","spark-3.0.3-bin-hadoop2.7/jars/apacheds-i18n-2.0.0-M15.jar\n","spark-3.0.3-bin-hadoop2.7/jars/parquet-common-1.10.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/stax-api-1.0.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jackson-dataformat-yaml-2.10.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/algebra_2.12-2.0.0-M2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-graphx_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hadoop-annotations-2.7.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/guice-servlet-3.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/kubernetes-client-4.9.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spire-util_2.12-0.17.0-M1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jakarta.activation-api-1.2.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-network-shuffle_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jline-2.14.6.jar\n","spark-3.0.3-bin-hadoop2.7/jars/breeze_2.12-1.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/metrics-jvm-4.1.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-launcher_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/machinist_2.12-0.6.8.jar\n","spark-3.0.3-bin-hadoop2.7/jars/scala-compiler-2.12.10.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-pool-1.5.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/arrow-format-0.15.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-mllib_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jta-1.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-yarn_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/istack-commons-runtime-3.0.8.jar\n","spark-3.0.3-bin-hadoop2.7/jars/metrics-graphite-4.1.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hadoop-hdfs-2.7.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-collections-3.2.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/audience-annotations-0.5.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jpam-1.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jakarta.annotation-api-1.3.5.jar\n","spark-3.0.3-bin-hadoop2.7/jars/xmlenc-0.52.jar\n","spark-3.0.3-bin-hadoop2.7/jars/api-asn1-api-1.0.0-M20.jar\n","spark-3.0.3-bin-hadoop2.7/jars/py4j-0.10.9.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hadoop-mapreduce-client-common-2.7.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spire-platform_2.12-0.17.0-M1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/avro-1.8.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-core_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/curator-client-2.7.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/antlr4-runtime-4.7.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/javax.inject-1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-kubernetes_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jersey-media-jaxb-2.30.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-lang3-3.9.jar\n","spark-3.0.3-bin-hadoop2.7/jars/javax.jdo-3.2.0-m3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hadoop-auth-2.7.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-io-2.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/objenesis-2.5.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/paranamer-2.8.jar\n","spark-3.0.3-bin-hadoop2.7/jars/cats-kernel_2.12-2.0.0-M4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/flatbuffers-java-1.9.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jersey-server-2.30.jar\n","spark-3.0.3-bin-hadoop2.7/jars/stream-2.9.6.jar\n","spark-3.0.3-bin-hadoop2.7/jars/datanucleus-api-jdo-4.2.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/gson-2.2.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/xml-apis-1.4.01.jar\n","spark-3.0.3-bin-hadoop2.7/jars/apacheds-kerberos-codec-2.0.0-M15.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hadoop-mapreduce-client-shuffle-2.7.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-hive_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hive-jdbc-2.3.7.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hive-exec-2.3.7-core.jar\n","spark-3.0.3-bin-hadoop2.7/jars/opencsv-2.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-sketch_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/avro-mapred-1.8.2-hadoop2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jsr305-3.0.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/univocity-parsers-2.9.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/macro-compat_2.12-1.1.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-dbcp-1.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jersey-container-servlet-core-2.30.jar\n","spark-3.0.3-bin-hadoop2.7/jars/orc-core-1.5.10.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hk2-api-2.6.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/parquet-jackson-1.10.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/mesos-1.4.0-shaded-protobuf.jar\n","spark-3.0.3-bin-hadoop2.7/jars/chill-java-0.9.5.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-tags_2.12-3.0.3-tests.jar\n","spark-3.0.3-bin-hadoop2.7/jars/automaton-1.11-8.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-lang-2.6.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jackson-annotations-2.10.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hk2-utils-2.6.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-network-common_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-repl_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/velocity-1.5.jar\n","spark-3.0.3-bin-hadoop2.7/jars/htrace-core-3.1.0-incubating.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jul-to-slf4j-1.7.30.jar\n","spark-3.0.3-bin-hadoop2.7/jars/JTransforms-3.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/json4s-ast_2.12-3.6.6.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jersey-client-2.30.jar\n","spark-3.0.3-bin-hadoop2.7/jars/httpclient-4.5.6.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jetty-sslengine-6.1.26.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-kvstore_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spire_2.12-0.17.0-M1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jersey-hk2-2.30.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-math3-3.4.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/xbean-asm7-shaded-4.15.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jakarta.validation-api-2.0.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/parquet-encoding-1.10.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/snappy-java-1.1.8.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/slf4j-log4j12-1.7.30.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hadoop-mapreduce-client-app-2.7.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/zstd-jni-1.4.4-3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/slf4j-api-1.7.30.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hive-shims-0.23-2.3.7.jar\n","spark-3.0.3-bin-hadoop2.7/jars/guava-14.0.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/stax-api-1.0-2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/zjsonpatch-0.3.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hive-storage-api-2.7.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/shapeless_2.12-2.3.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/kubernetes-model-common-4.9.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/oro-2.0.8.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hadoop-yarn-client-2.7.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/libfb303-0.9.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/core-1.1.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jersey-container-servlet-2.30.jar\n","spark-3.0.3-bin-hadoop2.7/jars/datanucleus-rdbms-4.1.19.jar\n","spark-3.0.3-bin-hadoop2.7/jars/super-csv-2.2.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/compress-lzf-1.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jackson-module-paranamer-2.10.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/aopalliance-1.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/osgi-resource-locator-1.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jackson-module-scala_2.12-2.10.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hive-shims-2.3.7.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hadoop-yarn-server-common-2.7.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/json-1.8.jar\n","spark-3.0.3-bin-hadoop2.7/jars/antlr-runtime-3.5.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/threeten-extra-1.5.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jetty-6.1.26.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-tags_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jackson-module-jaxb-annotations-2.10.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jersey-common-2.30.jar\n","spark-3.0.3-bin-hadoop2.7/jars/aircompressor-0.10.jar\n","spark-3.0.3-bin-hadoop2.7/jars/lz4-java-1.7.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hadoop-client-2.7.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/activation-1.1.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/shims-0.7.45.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-crypto-1.1.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-mesos_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/libthrift-0.12.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/orc-mapreduce-1.5.10.jar\n","spark-3.0.3-bin-hadoop2.7/jars/HikariCP-2.5.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/generex-1.0.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/okhttp-3.12.6.jar\n","spark-3.0.3-bin-hadoop2.7/jars/breeze-macros_2.12-1.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/leveldbjni-all-1.8.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jaxb-runtime-2.3.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-httpclient-3.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jackson-core-asl-1.9.13.jar\n","spark-3.0.3-bin-hadoop2.7/jars/scala-xml_2.12-1.2.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hk2-locator-2.6.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hadoop-common-2.7.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/curator-recipes-2.7.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jackson-mapper-asl-1.9.13.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hive-common-2.3.7.jar\n","spark-3.0.3-bin-hadoop2.7/jars/xz-1.5.jar\n","spark-3.0.3-bin-hadoop2.7/jars/ST4-4.0.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hive-shims-scheduler-2.3.7.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-unsafe_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/parquet-hadoop-1.10.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/chill_2.12-0.9.5.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-configuration-1.6.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jackson-databind-2.10.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-hive-thriftserver_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/scala-reflect-2.12.10.jar\n","spark-3.0.3-bin-hadoop2.7/jars/joda-time-2.10.5.jar\n","spark-3.0.3-bin-hadoop2.7/jars/minlog-1.3.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jakarta.ws.rs-api-2.1.6.jar\n","spark-3.0.3-bin-hadoop2.7/jars/orc-shims-1.5.10.jar\n","spark-3.0.3-bin-hadoop2.7/jars/aopalliance-repackaged-2.6.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-compress-1.20.jar\n","spark-3.0.3-bin-hadoop2.7/jars/bonecp-0.8.0.RELEASE.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jodd-core-3.5.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hadoop-yarn-server-web-proxy-2.7.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jackson-core-2.10.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/protobuf-java-2.5.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-net-3.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hadoop-mapreduce-client-jobclient-2.7.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/datanucleus-core-4.1.17.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jackson-jaxrs-1.9.13.jar\n","spark-3.0.3-bin-hadoop2.7/jars/janino-3.0.16.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hive-shims-common-2.3.7.jar\n","spark-3.0.3-bin-hadoop2.7/jars/javolution-5.5.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/transaction-api-1.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-streaming_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jackson-datatype-jsr310-2.10.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-mllib-local_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jaxb-api-2.2.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-cli-1.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jakarta.inject-2.6.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/metrics-jmx-4.1.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/arrow-vector-0.15.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jakarta.xml.bind-api-2.3.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-codec-1.10.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-catalyst_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hadoop-yarn-common-2.7.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/curator-framework-2.7.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/arpack_combined_all-0.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hive-metastore-2.3.7.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hive-beeline-2.3.7.jar\n","spark-3.0.3-bin-hadoop2.7/jars/metrics-core-4.1.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hive-llap-common-2.3.7.jar\n","spark-3.0.3-bin-hadoop2.7/jars/netty-all-4.1.47.Final.jar\n","spark-3.0.3-bin-hadoop2.7/jars/javassist-3.25.0-GA.jar\n","spark-3.0.3-bin-hadoop2.7/jars/scala-collection-compat_2.12-2.1.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/spark-sql_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/jars/parquet-column-1.10.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/jetty-util-6.1.26.jar\n","spark-3.0.3-bin-hadoop2.7/jars/kubernetes-model-4.9.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/avro-ipc-1.8.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/guice-3.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/commons-text-1.6.jar\n","spark-3.0.3-bin-hadoop2.7/jars/json4s-scalap_2.12-3.6.6.jar\n","spark-3.0.3-bin-hadoop2.7/jars/metrics-json-4.1.1.jar\n","spark-3.0.3-bin-hadoop2.7/jars/snakeyaml-1.24.jar\n","spark-3.0.3-bin-hadoop2.7/jars/json4s-jackson_2.12-3.6.6.jar\n","spark-3.0.3-bin-hadoop2.7/jars/ivy-2.4.0.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hadoop-mapreduce-client-core-2.7.4.jar\n","spark-3.0.3-bin-hadoop2.7/jars/json4s-core_2.12-3.6.6.jar\n","spark-3.0.3-bin-hadoop2.7/jars/pyrolite-4.30.jar\n","spark-3.0.3-bin-hadoop2.7/jars/log4j-1.2.17.jar\n","spark-3.0.3-bin-hadoop2.7/jars/hive-serde-2.3.7.jar\n","spark-3.0.3-bin-hadoop2.7/jars/scala-parser-combinators_2.12-1.1.2.jar\n","spark-3.0.3-bin-hadoop2.7/jars/RoaringBitmap-0.7.45.jar\n","spark-3.0.3-bin-hadoop2.7/jars/zookeeper-3.4.14.jar\n","spark-3.0.3-bin-hadoop2.7/data/\n","spark-3.0.3-bin-hadoop2.7/data/mllib/\n","spark-3.0.3-bin-hadoop2.7/data/mllib/sample_lda_data.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/sample_libsvm_data.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/sample_svm_data.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/sample_multiclass_classification_data.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/sample_linear_regression_data.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/sample_lda_libsvm_data.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/sample_fpgrowth.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/sample_binary_classification_data.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/sample_isotonic_regression_libsvm_data.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/iris_libsvm.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/sample_movielens_data.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/als/\n","spark-3.0.3-bin-hadoop2.7/data/mllib/als/test.data\n","spark-3.0.3-bin-hadoop2.7/data/mllib/als/sample_movielens_ratings.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/sample_kmeans_data.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/pic_data.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/origin/\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/origin/kittens/\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/origin/kittens/54893.jpg\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/origin/kittens/not-image.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/origin/kittens/DP802813.jpg\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/origin/kittens/29.5.a_b_EGDP022204.jpg\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/origin/kittens/DP153539.jpg\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/origin/license.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/origin/multi-channel/\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/origin/multi-channel/BGRA.png\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/origin/multi-channel/BGRA_alpha_60.png\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/origin/multi-channel/grayscale.jpg\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/origin/multi-channel/chr30.4.184.jpg\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/partitioned/\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/partitioned/cls=multichannel/\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/partitioned/cls=multichannel/date=2018-01/\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/partitioned/cls=multichannel/date=2018-01/BGRA.png\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/partitioned/cls=multichannel/date=2018-01/BGRA_alpha_60.png\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/partitioned/cls=multichannel/date=2018-02/\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/partitioned/cls=multichannel/date=2018-02/grayscale.jpg\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/partitioned/cls=multichannel/date=2018-02/chr30.4.184.jpg\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/partitioned/cls=kittens/\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/partitioned/cls=kittens/date=2018-01/\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/partitioned/cls=kittens/date=2018-01/not-image.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/partitioned/cls=kittens/date=2018-01/29.5.a_b_EGDP022204.jpg\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/partitioned/cls=kittens/date=2018-02/\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/partitioned/cls=kittens/date=2018-02/54893.jpg\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/partitioned/cls=kittens/date=2018-02/DP802813.jpg\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/partitioned/cls=kittens/date=2018-02/DP153539.jpg\n","spark-3.0.3-bin-hadoop2.7/data/mllib/images/license.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/ridge-data/\n","spark-3.0.3-bin-hadoop2.7/data/mllib/ridge-data/lpsa.data\n","spark-3.0.3-bin-hadoop2.7/data/mllib/kmeans_data.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/streaming_kmeans_data_test.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/pagerank_data.txt\n","spark-3.0.3-bin-hadoop2.7/data/mllib/gmm_data.txt\n","spark-3.0.3-bin-hadoop2.7/data/graphx/\n","spark-3.0.3-bin-hadoop2.7/data/graphx/users.txt\n","spark-3.0.3-bin-hadoop2.7/data/graphx/followers.txt\n","spark-3.0.3-bin-hadoop2.7/data/streaming/\n","spark-3.0.3-bin-hadoop2.7/data/streaming/AFINN-111.txt\n","spark-3.0.3-bin-hadoop2.7/R/\n","spark-3.0.3-bin-hadoop2.7/R/lib/\n","spark-3.0.3-bin-hadoop2.7/R/lib/sparkr.zip\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/tests/\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/tests/testthat/\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/tests/testthat/test_basic.R\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/DESCRIPTION\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/profile/\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/profile/shell.R\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/profile/general.R\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/INDEX\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/Meta/\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/Meta/features.rds\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/Meta/links.rds\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/Meta/nsInfo.rds\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/Meta/package.rds\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/Meta/Rd.rds\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/Meta/hsearch.rds\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/help/\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/help/SparkR.rdb\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/help/aliases.rds\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/help/SparkR.rdx\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/help/AnIndex\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/help/paths.rds\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/R/\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/R/SparkR.rdb\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/R/SparkR.rdx\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/R/SparkR\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/NAMESPACE\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/html/\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/html/00Index.html\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/html/R.css\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/worker/\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/worker/worker.R\n","spark-3.0.3-bin-hadoop2.7/R/lib/SparkR/worker/daemon.R\n","spark-3.0.3-bin-hadoop2.7/README.md\n","spark-3.0.3-bin-hadoop2.7/RELEASE\n","spark-3.0.3-bin-hadoop2.7/yarn/\n","spark-3.0.3-bin-hadoop2.7/yarn/spark-3.0.3-yarn-shuffle.jar\n","spark-3.0.3-bin-hadoop2.7/LICENSE\n","spark-3.0.3-bin-hadoop2.7/sbin/\n","spark-3.0.3-bin-hadoop2.7/sbin/stop-mesos-shuffle-service.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/start-master.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/spark-config.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/start-history-server.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/start-slaves.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/spark-daemon.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/stop-mesos-dispatcher.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/start-mesos-shuffle-service.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/slaves.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/stop-history-server.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/start-thriftserver.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/stop-thriftserver.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/start-slave.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/start-all.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/stop-slave.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/spark-daemons.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/stop-slaves.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/stop-all.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/start-mesos-dispatcher.sh\n","spark-3.0.3-bin-hadoop2.7/sbin/stop-master.sh\n","spark-3.0.3-bin-hadoop2.7/examples/\n","spark-3.0.3-bin-hadoop2.7/examples/src/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/survreg.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/glm.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/prefixSpan.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/powerIterationClustering.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/lda.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/kstest.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/isoreg.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/ml.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/naiveBayes.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/mlp.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/als.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/kmeans.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/svmLinear.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/logit.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/randomForest.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/gbt.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/decisionTree.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/gaussianMixture.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/bisectingKmeans.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/ml/fpm.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/dataframe.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/RSparkSQLExample.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/data-manipulation.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/streaming/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/r/streaming/structured_network_wordcount.R\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SparkPageRank.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SkewedGroupByTest.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/AccumulatorMetricsTest.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/HdfsTest.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/Word2VecExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/DeveloperApiExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/MinHashLSHExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/IndexToStringExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/BisectingKMeansExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/CorrelationExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/MulticlassLogisticRegressionWithElasticNetExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/VectorSizeHintExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/NGramExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/BucketedRandomProjectionLSHExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/BinarizerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/StandardScalerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/FeatureHasherExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/OneVsRestExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/StopWordsRemoverExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/LinearSVCExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/GaussianMixtureExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/MaxAbsScalerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/IsotonicRegressionExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/ModelSelectionViaCrossValidationExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/EstimatorTransformerParamExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeRegressionExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/RandomForestRegressorExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/PrefixSpanExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/KMeansExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/ModelSelectionViaTrainValidationSplitExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionSummaryExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/MultilayerPerceptronClassifierExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/PCAExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/DCTExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/FPGrowthExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/ElementwiseProductExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/GeneralizedLinearRegressionExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/CountVectorizerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/MinMaxScalerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/ChiSquareTestExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionWithElasticNetExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/FMRegressorExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/VectorSlicerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/LinearRegressionExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/StringIndexerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/GradientBoostedTreeRegressorExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/ALSExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeClassificationExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/PolynomialExpansionExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/FMClassifierExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/UnaryTransformerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/ImputerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/TokenizerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/GradientBoostedTreeClassifierExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/PowerIterationClusteringExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/SQLTransformerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/SummarizerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/DataFrameExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/BucketizerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/TfIdfExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/AFTSurvivalRegressionExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/QuantileDiscretizerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/OneHotEncoderExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/PipelineExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/InteractionExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/LDAExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/RFormulaExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/VectorIndexerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/VectorAssemblerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/NormalizerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/ChiSqSelectorExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/NaiveBayesExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/RandomForestClassifierExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/LinearRegressionWithElasticNetExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/RobustScalerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/LogQuery.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SparkRemoteFileTest.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ExceptionHandlingTest.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/pythonconverters/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/pythonconverters/AvroConverters.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/BroadcastTest.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SparkHdfsLR.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/Word2VecExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostedTreesRunner.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/RandomRDDGeneration.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/PMMLModelExportExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/BisectingKMeansExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeRunner.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/MultiLabelMetricsExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/CorrelationsExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/LBFGSExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/TFIDFExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/StandardScalerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/LogisticRegressionWithLBFGSExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/StreamingTestExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassification.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/DenseKMeans.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostingClassificationExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/RankingMetricsExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/GaussianMixtureExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/MultivariateSummarizer.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/IsotonicRegressionExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/PCAOnSourceVectorExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/Correlations.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeRegressionExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/AssociationRulesExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/StreamingLogisticRegression.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/SVDExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/PrefixSpanExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/KMeansExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/SparseNaiveBayes.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/AbstractParams.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassificationMetricsExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/SampledRDDs.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/FPGrowthExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/ElementwiseProductExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/RecommendationExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/TallSkinnySVD.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingKolmogorovSmirnovTestExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostingRegressionExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/StreamingKMeansExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/CosineSimilarity.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/RandomForestRegressionExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/StreamingLinearRegressionExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/StratifiedSamplingExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeClassificationExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/TallSkinnyPCA.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/RandomForestClassificationExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/SimpleFPGrowth.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/LatentDirichletAllocationExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/KernelDensityEstimationExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/PowerIterationClusteringExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/SVMWithSGDExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/MovieLensALS.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/NormalizerExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/MulticlassMetricsExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/ChiSqSelectorExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/SummaryStatisticsExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/PCAOnRowMatrixExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/NaiveBayesExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SparkTC.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/DriverSubmissionTest.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/MultiBroadcastTest.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SparkLR.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SparkPi.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/ConnectedComponentsExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/LiveJournalPageRank.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/AggregateMessagesExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/SSSPExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/SynthBenchmark.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/TriangleCountingExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/PageRankExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/ComprehensiveExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/Analytics.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SparkKMeans.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/clickstream/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/clickstream/PageViewGenerator.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/clickstream/PageViewStream.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/DirectKafkaWordCount.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/HdfsWordCount.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/DirectKerberizedKafkaWordCount.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/RecoverableNetworkWordCount.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/QueueStream.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/SqlNetworkWordCount.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/CustomReceiver.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/RawNetworkGrep.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/StatefulNetworkWordCount.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/StreamingExamples.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/LocalLR.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/LocalKMeans.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/GroupByTest.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/LocalALS.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/SimpleTypedAggregator.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/UserDefinedUntypedAggregation.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/hive/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/hive/SparkHiveExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/RDDRelation.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/SparkSQLExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/SQLDataSourceExample.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/streaming/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredKerberizedKafkaWordCount.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredNetworkWordCount.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredNetworkWordCountWindowed.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredKafkaWordCount.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredSessionization.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/UserDefinedTypedAggregation.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/UserDefinedScalar.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SimpleSkewedGroupByTest.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/LocalFileLR.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/LocalPi.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/DFSReadWriteTest.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SparkALS.scala\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/resources/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/resources/people.json\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/resources/users.avro\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/resources/people.csv\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/resources/users.parquet\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/resources/users.orc\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/resources/dir1/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/resources/dir1/file1.parquet\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/resources/dir1/dir2/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/resources/dir1/dir2/file2.parquet\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/resources/dir1/file3.json\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/resources/user.avsc\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/resources/full_user.avsc\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/resources/kv1.txt\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/resources/people.txt\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/resources/employees.json\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaStringIndexerExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaChiSqSelectorExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaLabeledDocument.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaLogisticRegressionWithElasticNetExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaPolynomialExpansionExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaTfIdfExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaBucketizerExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaLogisticRegressionSummaryExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaSummarizerExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaQuantileDiscretizerExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaOneHotEncoderExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaStandardScalerExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaDCTExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeRegressionExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaRandomForestClassifierExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaOneVsRestExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaGeneralizedLinearRegressionExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaGaussianMixtureExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaWord2VecExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaNormalizerExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaLDAExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaCountVectorizerExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaMultilayerPerceptronClassifierExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaMinMaxScalerExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaCorrelationExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaNGramExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaImputerExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaMinHashLSHExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaVectorSizeHintExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaFMClassifierExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaFeatureHasherExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaModelSelectionViaTrainValidationSplitExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaSQLTransformerExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaKMeansExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaVectorAssemblerExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeRegressorExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaPrefixSpanExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaFMRegressorExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaNaiveBayesExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaMulticlassLogisticRegressionWithElasticNetExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaModelSelectionViaCrossValidationExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaDocument.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeClassifierExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaRandomForestRegressorExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaAFTSurvivalRegressionExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaIsotonicRegressionExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaElementwiseProductExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaALSExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaBucketedRandomProjectionLSHExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaVectorSlicerExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaTokenizerExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaRFormulaExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaEstimatorTransformerParamExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaPowerIterationClusteringExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaBisectingKMeansExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaMaxAbsScalerExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaPCAExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaPipelineExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaStopWordsRemoverExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaIndexToStringExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaFPGrowthExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaVectorIndexerExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaInteractionExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaChiSquareTestExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaRobustScalerExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaLinearRegressionWithElasticNetExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaLinearSVCExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/JavaWordCount.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/JavaHdfsLR.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/JavaStatusTrackerDemo.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaChiSqSelectorExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingKolmogorovSmirnovTestExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaRandomForestRegressionExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaSimpleFPGrowth.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaRandomForestClassificationExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaMulticlassClassificationMetricsExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaKernelDensityEstimationExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaSummaryStatisticsExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaDecisionTreeRegressionExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingClassificationExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaGaussianMixtureExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaStratifiedSamplingExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaSVDExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaAssociationRulesExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaALS.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaKMeansExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaSVMWithSGDExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaPrefixSpanExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaNaiveBayesExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaLogisticRegressionWithLBFGSExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaLBFGSExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingRegressionExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaRankingMetricsExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaIsotonicRegressionExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaElementwiseProductExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaCorrelationsExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaPowerIterationClusteringExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaBisectingKMeansExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaDecisionTreeClassificationExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaPCAExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaRecommendationExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaLatentDirichletAllocationExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaMultiLabelClassificationMetricsExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaBinaryClassificationMetricsExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaStreamingTestExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/JavaPageRank.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/JavaLogQuery.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/JavaStatefulNetworkWordCount.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/JavaCustomReceiver.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/JavaRecord.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/JavaNetworkWordCount.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKerberizedKafkaWordCount.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKafkaWordCount.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/JavaSqlNetworkWordCount.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/JavaRecoverableNetworkWordCount.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/JavaQueueStream.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/JavaSparkPi.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/JavaUserDefinedScalar.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/hive/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/hive/JavaSparkHiveExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/JavaUserDefinedTypedAggregation.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/JavaSQLDataSourceExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/JavaSparkSQLExample.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/JavaUserDefinedUntypedAggregation.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/streaming/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKafkaWordCount.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredNetworkWordCountWindowed.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredSessionization.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredNetworkWordCount.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKerberizedKafkaWordCount.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/JavaTC.java\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scripts/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/scripts/getGpusResources.sh\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/kmeans.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/dct_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/count_vectorizer_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/chisq_selector_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/gradient_boosted_tree_regressor_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/tf_idf_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/cross_validator.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/linear_regression_with_elastic_net.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/normalizer_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/fm_regressor_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/polynomial_expansion_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/pipeline_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/generalized_linear_regression_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/multilayer_perceptron_classification.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/bisecting_k_means_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/chi_square_test_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/train_validation_split.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/stopwords_remover_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/linearsvc.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/lda_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/random_forest_regressor_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/vector_assembler_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/word2vec_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/string_indexer_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/decision_tree_classification_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/index_to_string_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/bucketizer_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/vector_size_hint_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/vector_indexer_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/tokenizer_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/robust_scaler_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/standard_scaler_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/kmeans_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/power_iteration_clustering_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/naive_bayes_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/pca_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/aft_survival_regression.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/min_max_scaler_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/elementwise_product_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/dataframe_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/n_gram_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/rformula_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/one_vs_rest_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/als_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/onehot_encoder_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/vector_slicer_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/logistic_regression_with_elastic_net.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/sql_transformer.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/multiclass_logistic_regression_with_elastic_net.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/summarizer_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/quantile_discretizer_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/feature_hasher_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/imputer_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/bucketed_random_projection_lsh_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/estimator_transformer_param_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/gradient_boosted_tree_classifier_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/logistic_regression_summary_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/decision_tree_regression_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/fm_classifier_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/gaussian_mixture_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/fpgrowth_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/random_forest_classifier_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/correlation_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/min_hash_lsh_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/binarizer_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/prefixspan_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/max_abs_scaler_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/isotonic_regression_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/ml/interaction_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/logistic_regression.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/als.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/word2vec.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/correlations_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/gradient_boosting_classification_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/kernel_density_estimation_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/hypothesis_testing_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/svd_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/kmeans.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/latent_dirichlet_allocation_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/tf_idf_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/correlations.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/random_forest_regression_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/normalizer_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/logistic_regression.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/multi_label_metrics_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/gradient_boosting_regression_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/random_forest_classification_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/streaming_linear_regression_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/sampled_rdds.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/bisecting_k_means_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/logistic_regression_with_lbfgs_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/k_means_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/streaming_k_means_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/word2vec_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/multi_class_metrics_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/decision_tree_classification_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/summary_statistics_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/binary_classification_metrics_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/standard_scaler_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/power_iteration_clustering_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/gaussian_mixture_model.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/regression_metrics_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/naive_bayes_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/elementwise_product_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/linear_regression_with_sgd_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/pca_rowmatrix_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/hypothesis_testing_kolmogorov_smirnov_test_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/random_rdd_generation.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/decision_tree_regression_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/gaussian_mixture_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/recommendation_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/fpgrowth_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/ranking_metrics_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/svm_with_sgd_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/isotonic_regression_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/mllib/stratified_sampling_example.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/wordcount.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/status_api_demo.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/pagerank.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/sort.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/transitive_closure.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/avro_inputformat.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/pi.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/streaming/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/streaming/queue_stream.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/streaming/stateful_network_wordcount.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/streaming/network_wordjoinsentiments.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/streaming/sql_network_wordcount.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/streaming/network_wordcount.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/streaming/hdfs_wordcount.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/streaming/recoverable_network_wordcount.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/sql/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/sql/datasource.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/sql/hive.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/sql/arrow.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/sql/streaming/\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/sql/streaming/structured_network_wordcount_windowed.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/sql/streaming/structured_network_wordcount.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/sql/streaming/structured_kafka_wordcount.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/sql/basic.py\n","spark-3.0.3-bin-hadoop2.7/examples/src/main/python/parquet_inputformat.py\n","spark-3.0.3-bin-hadoop2.7/examples/jars/\n","spark-3.0.3-bin-hadoop2.7/examples/jars/spark-examples_2.12-3.0.3.jar\n","spark-3.0.3-bin-hadoop2.7/examples/jars/scopt_2.12-3.7.1.jar\n","spark-3.0.3-bin-hadoop2.7/conf/\n","spark-3.0.3-bin-hadoop2.7/conf/slaves.template\n","spark-3.0.3-bin-hadoop2.7/conf/metrics.properties.template\n","spark-3.0.3-bin-hadoop2.7/conf/fairscheduler.xml.template\n","spark-3.0.3-bin-hadoop2.7/conf/log4j.properties.template\n","spark-3.0.3-bin-hadoop2.7/conf/spark-defaults.conf.template\n","spark-3.0.3-bin-hadoop2.7/conf/spark-env.sh.template\n","spark-3.0.3-bin-hadoop2.7/bin/\n","spark-3.0.3-bin-hadoop2.7/bin/sparkR.cmd\n","spark-3.0.3-bin-hadoop2.7/bin/sparkR\n","spark-3.0.3-bin-hadoop2.7/bin/spark-submit\n","spark-3.0.3-bin-hadoop2.7/bin/pyspark2.cmd\n","spark-3.0.3-bin-hadoop2.7/bin/spark-class\n","spark-3.0.3-bin-hadoop2.7/bin/pyspark.cmd\n","spark-3.0.3-bin-hadoop2.7/bin/spark-submit2.cmd\n","spark-3.0.3-bin-hadoop2.7/bin/load-spark-env.cmd\n","spark-3.0.3-bin-hadoop2.7/bin/spark-sql\n","spark-3.0.3-bin-hadoop2.7/bin/docker-image-tool.sh\n","spark-3.0.3-bin-hadoop2.7/bin/find-spark-home.cmd\n","spark-3.0.3-bin-hadoop2.7/bin/load-spark-env.sh\n","spark-3.0.3-bin-hadoop2.7/bin/pyspark\n","spark-3.0.3-bin-hadoop2.7/bin/spark-shell.cmd\n","spark-3.0.3-bin-hadoop2.7/bin/spark-shell2.cmd\n","spark-3.0.3-bin-hadoop2.7/bin/spark-submit.cmd\n","spark-3.0.3-bin-hadoop2.7/bin/beeline.cmd\n","spark-3.0.3-bin-hadoop2.7/bin/find-spark-home\n","spark-3.0.3-bin-hadoop2.7/bin/spark-class.cmd\n","spark-3.0.3-bin-hadoop2.7/bin/sparkR2.cmd\n","spark-3.0.3-bin-hadoop2.7/bin/beeline\n","spark-3.0.3-bin-hadoop2.7/bin/spark-class2.cmd\n","spark-3.0.3-bin-hadoop2.7/bin/spark-sql.cmd\n","spark-3.0.3-bin-hadoop2.7/bin/run-example\n","spark-3.0.3-bin-hadoop2.7/bin/spark-shell\n","spark-3.0.3-bin-hadoop2.7/bin/run-example.cmd\n","spark-3.0.3-bin-hadoop2.7/bin/spark-sql2.cmd\n","spark-3.0.3-bin-hadoop2.7/python/\n","spark-3.0.3-bin-hadoop2.7/python/.gitignore\n","spark-3.0.3-bin-hadoop2.7/python/run-tests-with-coverage\n","spark-3.0.3-bin-hadoop2.7/python/pylintrc\n","spark-3.0.3-bin-hadoop2.7/python/MANIFEST.in\n","spark-3.0.3-bin-hadoop2.7/python/README.md\n","spark-3.0.3-bin-hadoop2.7/python/test_coverage/\n","spark-3.0.3-bin-hadoop2.7/python/test_coverage/coverage_daemon.py\n","spark-3.0.3-bin-hadoop2.7/python/test_coverage/conf/\n","spark-3.0.3-bin-hadoop2.7/python/test_coverage/conf/spark-defaults.conf\n","spark-3.0.3-bin-hadoop2.7/python/test_coverage/sitecustomize.py\n","spark-3.0.3-bin-hadoop2.7/python/run-tests.py\n","spark-3.0.3-bin-hadoop2.7/python/setup.py\n","spark-3.0.3-bin-hadoop2.7/python/test_support/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/userlibrary.py\n","spark-3.0.3-bin-hadoop2.7/python/test_support/hello/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/hello/sub_hello/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/hello/sub_hello/sub_hello.txt\n","spark-3.0.3-bin-hadoop2.7/python/test_support/hello/hello.txt\n","spark-3.0.3-bin-hadoop2.7/python/test_support/userlib-0.1.zip\n","spark-3.0.3-bin-hadoop2.7/python/test_support/SimpleHTTPServer.py\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/people.json\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/people_array.json\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/people_array_utf16le.json\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/text-test.txt\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/ages.csv\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/.part-r-00005.gz.parquet.crc\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-00005.gz.parquet\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/.part-r-00002.gz.parquet.crc\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/.part-r-00004.gz.parquet.crc\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00004.gz.parquet\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00002.gz.parquet\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=9/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/.part-r-00007.gz.parquet.crc\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/part-r-00007.gz.parquet\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/_metadata\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2014/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2014/month=9/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/.part-r-00008.gz.parquet.crc\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/part-r-00008.gz.parquet\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/_common_metadata\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/_SUCCESS\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/orc_partitioned/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/orc_partitioned/b=1/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/orc_partitioned/b=1/c=1/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/orc_partitioned/b=1/c=1/part-r-00000-829af031-b970-49d6-ad39-30460a0be2c8.orc\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/orc_partitioned/b=1/c=1/.part-r-00000-829af031-b970-49d6-ad39-30460a0be2c8.orc.crc\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/orc_partitioned/b=0/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/orc_partitioned/b=0/c=0/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/orc_partitioned/b=0/c=0/part-r-00000-829af031-b970-49d6-ad39-30460a0be2c8.orc\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/orc_partitioned/b=0/c=0/.part-r-00000-829af031-b970-49d6-ad39-30460a0be2c8.orc.crc\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/orc_partitioned/_SUCCESS\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/ages_newlines.csv\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/streaming/\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/streaming/text-test.txt\n","spark-3.0.3-bin-hadoop2.7/python/test_support/sql/people1.json\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/test_rddbarrier.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/test_worker.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/test_serializers.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/test_util.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/test_rdd.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/__init__.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/test_broadcast.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/test_appsubmit.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/test_profiler.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/test_pin_thread.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/test_shuffle.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/test_join.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/test_taskcontext.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/test_context.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/test_readwrite.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/test_conf.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/tests/test_daemon.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/testing/\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/testing/mlutils.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/testing/__init__.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/testing/mllibutils.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/testing/utils.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/testing/sqlutils.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/testing/streamingutils.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/accumulators.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/rddsampler.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tests/\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tests/test_algorithms.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tests/test_evaluation.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tests/test_util.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tests/test_feature.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tests/test_pipeline.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tests/test_wrapper.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tests/__init__.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tests/test_tuning.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tests/test_persistence.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tests/test_param.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tests/test_training_summary.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tests/test_linalg.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tests/test_image.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tests/test_stat.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tests/test_base.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/functions.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tuning.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/pipeline.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/base.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/feature.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/__init__.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/stat.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/image.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/classification.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/recommendation.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/regression.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/param/\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/param/_shared_params_code_gen.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/param/__init__.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/param/shared.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/tree.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/fpm.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/wrapper.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/clustering.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/common.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/linalg/\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/linalg/__init__.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/evaluation.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/ml/util.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/find_spark_home.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/heapq3.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/serializers.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/java_gateway.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/traceback_utils.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/conf.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/__init__.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/tests/\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/tests/test_algorithms.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/tests/test_streaming_algorithms.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/tests/test_util.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/tests/test_feature.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/tests/__init__.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/tests/test_linalg.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/tests/test_stat.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/feature.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/__init__.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/classification.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/recommendation.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/regression.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/tree.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/fpm.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/random.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/stat/\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/stat/distribution.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/stat/__init__.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/stat/KernelDensity.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/stat/test.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/stat/_statistics.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/clustering.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/common.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/linalg/\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/linalg/__init__.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/linalg/distributed.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/evaluation.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/mllib/util.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/resultiterable.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/profiler.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/statcounter.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/join.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/daemon.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/rdd.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/context.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/cloudpickle.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/version.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/resource.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/files.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/worker.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/shell.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/streaming/\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/streaming/tests/\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/streaming/tests/test_listener.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/streaming/tests/test_kinesis.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/streaming/tests/__init__.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/streaming/tests/test_dstream.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/streaming/tests/test_context.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/streaming/dstream.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/streaming/__init__.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/streaming/kinesis.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/streaming/listener.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/streaming/context.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/streaming/util.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/status.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_functions.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_readwriter.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_utils.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_pandas_grouped_map.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_dataframe.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_pandas_map.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_udf.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_streaming.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/__init__.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_serde.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_pandas_udf_window.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_group.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_pandas_udf.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_pandas_cogrouped_map.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_pandas_udf_grouped_agg.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_pandas_udf_scalar.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_catalog.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_datasources.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_pandas_udf_typehints.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_types.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_column.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_context.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_conf.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_arrow.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/tests/test_session.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/pandas/\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/pandas/functions.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/pandas/serializers.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/pandas/__init__.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/pandas/typehints.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/pandas/map_ops.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/pandas/types.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/pandas/group_ops.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/pandas/utils.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/pandas/conversion.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/functions.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/readwriter.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/catalog.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/window.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/udf.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/conf.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/__init__.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/session.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/column.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/group.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/context.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/types.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/dataframe.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/avro/\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/avro/functions.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/avro/__init__.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/utils.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/sql/streaming.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/shuffle.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/taskcontext.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/_globals.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/broadcast.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/util.py\n","spark-3.0.3-bin-hadoop2.7/python/pyspark/storagelevel.py\n","spark-3.0.3-bin-hadoop2.7/python/.coveragerc\n","spark-3.0.3-bin-hadoop2.7/python/docs/\n","spark-3.0.3-bin-hadoop2.7/python/docs/index.rst\n","spark-3.0.3-bin-hadoop2.7/python/docs/conf.py\n","spark-3.0.3-bin-hadoop2.7/python/docs/pyspark.ml.rst\n","spark-3.0.3-bin-hadoop2.7/python/docs/pyspark.rst\n","spark-3.0.3-bin-hadoop2.7/python/docs/_templates/\n","spark-3.0.3-bin-hadoop2.7/python/docs/_templates/layout.html\n","spark-3.0.3-bin-hadoop2.7/python/docs/_static/\n","spark-3.0.3-bin-hadoop2.7/python/docs/_static/pyspark.css\n","spark-3.0.3-bin-hadoop2.7/python/docs/_static/copybutton.js\n","spark-3.0.3-bin-hadoop2.7/python/docs/_static/pyspark.js\n","spark-3.0.3-bin-hadoop2.7/python/docs/pyspark.mllib.rst\n","spark-3.0.3-bin-hadoop2.7/python/docs/make2.bat\n","spark-3.0.3-bin-hadoop2.7/python/docs/pyspark.streaming.rst\n","spark-3.0.3-bin-hadoop2.7/python/docs/pyspark.sql.rst\n","spark-3.0.3-bin-hadoop2.7/python/docs/make.bat\n","spark-3.0.3-bin-hadoop2.7/python/docs/Makefile\n","spark-3.0.3-bin-hadoop2.7/python/docs/pyspark.resource.rst\n","spark-3.0.3-bin-hadoop2.7/python/lib/\n","spark-3.0.3-bin-hadoop2.7/python/lib/PY4J_LICENSE.txt\n","spark-3.0.3-bin-hadoop2.7/python/lib/py4j-0.10.9-src.zip\n","spark-3.0.3-bin-hadoop2.7/python/lib/pyspark.zip\n","spark-3.0.3-bin-hadoop2.7/python/run-tests\n","spark-3.0.3-bin-hadoop2.7/python/setup.cfg\n","spark-3.0.3-bin-hadoop2.7/licenses/\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-respond.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-sbt-launch-lib.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-antlr.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-dagre-d3.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-pyrolite.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-sorttable.js.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-janino.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-protobuf.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-jquery.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-scopt.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-netlib.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-d3.min.js.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-graphlib-dot.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-AnchorJS.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-datatables.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-pmml-model.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-paranamer.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-jakarta-ws-rs-api\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-dnsjava.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-jakarta.xml.bind-api.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-jakarta-annotation-api\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-CC0.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-jodd.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-f2j.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-heapq.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-machinist.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-javolution.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-modernizr.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-spire.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-leveldbjni.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-join.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-zstd-jni.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-slf4j.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-arpack.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-jsp-api.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-JTransforms.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-JLargeArrays.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-bootstrap.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-reflectasm.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-javassist.html\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-zstd.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-json-formatter.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-matchMedia-polyfill.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-scala.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-jakarta.activation-api.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-automaton.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-javax-transaction-transaction-api.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-jaxb-runtime.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-minlog.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-mustache.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-xmlenc.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-jline.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-istack-commons-runtime.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-py4j.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-vis-timeline.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-re2j.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-kryo.txt\n","spark-3.0.3-bin-hadoop2.7/licenses/LICENSE-cloudpickle.txt\n"]}],"source":["# Download Apache Spark and extract\n","!wget https://archive.apache.org/dist/spark/spark-3.0.3/spark-3.0.3-bin-hadoop2.7.tgz\n","!tar xvf spark-3.0.3-bin-hadoop2.7.tgz"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":60019,"status":"ok","timestamp":1648900234957,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"},"user_tz":-60},"id":"COCOvti5KySL","outputId":"0a0dd77b-ded8-465a-8b5b-e7960e547a5f"},"outputs":[{"output_type":"stream","name":"stdout","text":["Collecting pyspark\n","  Downloading pyspark-3.2.1.tar.gz (281.4 MB)\n","\u001b[K     |████████████████████████████████| 281.4 MB 24 kB/s \n","\u001b[?25hCollecting py4j==0.10.9.3\n","  Downloading py4j-0.10.9.3-py2.py3-none-any.whl (198 kB)\n","\u001b[K     |████████████████████████████████| 198 kB 42.2 MB/s \n","\u001b[?25hBuilding wheels for collected packages: pyspark\n","  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for pyspark: filename=pyspark-3.2.1-py2.py3-none-any.whl size=281853642 sha256=28376afa9552d9d13c2f69b2438a478de5e302d34b67dbfc23b38854ee9ac647\n","  Stored in directory: /root/.cache/pip/wheels/9f/f5/07/7cd8017084dce4e93e84e92efd1e1d5334db05f2e83bcef74f\n","Successfully built pyspark\n","Installing collected packages: py4j, pyspark\n","Successfully installed py4j-0.10.9.3 pyspark-3.2.1\n"]}],"source":["#Install pyspark\n","!pip install -q findspark\n","!pip install pyspark"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"AnE6ejahLDYa"},"outputs":[],"source":["#setting the environment path \n","import os\n","os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-11-openjdk-amd64\"\n","os.environ[\"SPARK_HOME\"] = \"/content/spark-3.0.3-bin-hadoop2.7\""]},{"cell_type":"code","execution_count":null,"metadata":{"id":"TqHTb6K-Dc29"},"outputs":[],"source":["#Running a local SparkSession\n","import findspark\n","findspark.init()\n","from pyspark.sql import SparkSession\n","spark = SparkSession.builder\\\n",".master(\"local\")\\\n",".appName(\"Colab\")\\\n",".config('spark.ui.port', '4050')\\\n",".getOrCreate()"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":440,"status":"ok","timestamp":1648900268531,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"},"user_tz":-60},"id":"V6UKIfaBwy1R","outputId":"5ef8e296-a632-4938-bd2c-8671467427af"},"outputs":[{"output_type":"stream","name":"stdout","text":["3.0.3\n"]}],"source":["# Check the pyspark version\n","import pyspark\n","print(pyspark.__version__)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"7wVS5hGWUmdN"},"outputs":[],"source":["from pyspark.sql import SparkSession\n","# Numerical analysis package\n","import numpy as np\n","# Package to work with data\n","import pandas as pd\n","# Package for Matplotlib Pyplot\n","# Pyplot is a collection of functions in the popular visualization package Matplotlib.\n","import matplotlib as mpl\n","import matplotlib.pyplot as plt\n","# Package for Seaborn\n","# Seaborn is a Python data visualization library built on top of Matplotlib.\n","import seaborn as sns\n","from pyspark.ml.linalg import DenseMatrix, Vectors\n","from pyspark.ml.stat import Correlation\n","import json\n","#PySpark SQL Package \n","import pyspark.sql.functions as F\n","import pyspark.sql\n","from pyspark.sql.functions import col, skewness, kurtosis\n","from pyspark.context import SparkContext\n","from pyspark.sql.functions import * \n","from pyspark.sql.functions import isnan, when, count, col\n","from pyspark.sql.functions import when\n","from pyspark.sql.functions import UserDefinedFunction\n","from pyspark.sql.functions import from_unixtime, unix_timestamp\n","from pyspark.sql.types import StringType\n","from datetime import date, timedelta, datetime\n","import time"]},{"cell_type":"markdown","metadata":{"id":"OVr_jzwa_j_q"},"source":["**Connect/Mount Google Drive to Colab**"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":21627,"status":"ok","timestamp":1648900305604,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"},"user_tz":-60},"id":"UqwaPjGCQWCS","outputId":"71eb7780-430a-41d8-a473-480fbbca6c60"},"outputs":[{"output_type":"stream","name":"stdout","text":["Mounted at /content/drive\n"]}],"source":["#Connect/Mount Google Drive to Colab\n","from google.colab import drive\n","drive.mount('/content/drive')"]},{"cell_type":"markdown","metadata":{"id":"dc46wz78ADHt"},"source":["**Define the UNSW-NB15 Dataframe**"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"koboL_Ic-Wya"},"outputs":[],"source":["#Define the UNSW-NB15 Dataframe\n","UNSWNB15 = spark.read.csv('/content/drive/MyDrive/UEL-UNICAF/Module 3: Big Data Analytics/Assessment/UNSW-NB15.csv', inferSchema=True )"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"iBI8fkBGLpLk"},"outputs":[],"source":["#Rename the Columns Header\n","UNSWNB15 = UNSWNB15.withColumnRenamed('_c0','srcip') \\\n",".withColumnRenamed('_c1','sport') \\\n",".withColumnRenamed('_c2','dstip') \\\n",".withColumnRenamed('_c3','dsport') \\\n",".withColumnRenamed('_c4','proto') \\\n",".withColumnRenamed('_c5','state') \\\n",".withColumnRenamed('_c6','dur') \\\n",".withColumnRenamed('_c7','sbytes') \\\n",".withColumnRenamed('_c8','dbytes') \\\n",".withColumnRenamed('_c9','sttl') \\\n",".withColumnRenamed('_c10','dttl') \\\n",".withColumnRenamed('_c11','sloss') \\\n",".withColumnRenamed('_c12','dloss') \\\n",".withColumnRenamed('_c13','service') \\\n",".withColumnRenamed('_c14','Sload') \\\n",".withColumnRenamed('_c15','Dload') \\\n",".withColumnRenamed('_c16','Spkts') \\\n",".withColumnRenamed('_c17','Dpkts') \\\n",".withColumnRenamed('_c18','swin') \\\n",".withColumnRenamed('_c19','dwin') \\\n",".withColumnRenamed('_c20','stcpb') \\\n",".withColumnRenamed('_c21','dtcpb') \\\n",".withColumnRenamed('_c22','smeansz') \\\n",".withColumnRenamed('_c23','dmeansz') \\\n",".withColumnRenamed('_c24','trans_depth') \\\n",".withColumnRenamed('_c25','res_bdy_len') \\\n",".withColumnRenamed('_c26','Sjit') \\\n",".withColumnRenamed('_c27','Djit') \\\n",".withColumnRenamed('_c28','Stime') \\\n",".withColumnRenamed('_c29','Ltime') \\\n",".withColumnRenamed('_c30','Spkt') \\\n",".withColumnRenamed('_c31','Dpkt') \\\n",".withColumnRenamed('_c32','tcprtt') \\\n",".withColumnRenamed('_c33','synack') \\\n",".withColumnRenamed('_c34','ackdat') \\\n",".withColumnRenamed('_c35','is_sm_ips_ports') \\\n",".withColumnRenamed('_c36','ct_state_ttl') \\\n",".withColumnRenamed('_c37','ct_flw_http_mthd') \\\n",".withColumnRenamed('_c38','is_ftp_login') \\\n",".withColumnRenamed('_c39','ct_ftp_cmd') \\\n",".withColumnRenamed('_c40','ct_srv_src') \\\n",".withColumnRenamed('_c41','ct_srv_dst') \\\n",".withColumnRenamed('_c42','ct_dst_ltm') \\\n",".withColumnRenamed('_c43','ct_src_ ltm') \\\n",".withColumnRenamed('_c44','ct_src_dport_ltm') \\\n",".withColumnRenamed('_c45','ct_dst_sport_ltm') \\\n",".withColumnRenamed('_c46','ct_dst_src_ltm') \\\n",".withColumnRenamed('_c47','attack_cat') \\\n",".withColumnRenamed('_c48','Label') \\"]},{"cell_type":"markdown","metadata":{"id":"PRHrtpRhAM1U"},"source":["**Working with PySpark**"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":356,"status":"ok","timestamp":1648900429042,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"},"user_tz":-60},"id":"hXqmEGunDz_q","outputId":"b18722ad-3331-4ad0-a511-05187c2d0673"},"outputs":[{"output_type":"stream","name":"stdout","text":["root\n"," |-- srcip: string (nullable = true)\n"," |-- sport: integer (nullable = true)\n"," |-- dstip: string (nullable = true)\n"," |-- dsport: integer (nullable = true)\n"," |-- proto: string (nullable = true)\n"," |-- state: string (nullable = true)\n"," |-- dur: double (nullable = true)\n"," |-- sbytes: integer (nullable = true)\n"," |-- dbytes: integer (nullable = true)\n"," |-- sttl: integer (nullable = true)\n"," |-- dttl: integer (nullable = true)\n"," |-- sloss: integer (nullable = true)\n"," |-- dloss: integer (nullable = true)\n"," |-- service: string (nullable = true)\n"," |-- Sload: double (nullable = true)\n"," |-- Dload: double (nullable = true)\n"," |-- Spkts: integer (nullable = true)\n"," |-- Dpkts: integer (nullable = true)\n"," |-- swin: integer (nullable = true)\n"," |-- dwin: integer (nullable = true)\n"," |-- stcpb: long (nullable = true)\n"," |-- dtcpb: long (nullable = true)\n"," |-- smeansz: integer (nullable = true)\n"," |-- dmeansz: integer (nullable = true)\n"," |-- trans_depth: integer (nullable = true)\n"," |-- res_bdy_len: integer (nullable = true)\n"," |-- Sjit: double (nullable = true)\n"," |-- Djit: double (nullable = true)\n"," |-- Stime: integer (nullable = true)\n"," |-- Ltime: integer (nullable = true)\n"," |-- Spkt: double (nullable = true)\n"," |-- Dpkt: double (nullable = true)\n"," |-- tcprtt: double (nullable = true)\n"," |-- synack: double (nullable = true)\n"," |-- ackdat: double (nullable = true)\n"," |-- is_sm_ips_ports: integer (nullable = true)\n"," |-- ct_state_ttl: integer (nullable = true)\n"," |-- ct_flw_http_mthd: integer (nullable = true)\n"," |-- is_ftp_login: integer (nullable = true)\n"," |-- ct_ftp_cmd: integer (nullable = true)\n"," |-- ct_srv_src: integer (nullable = true)\n"," |-- ct_srv_dst: integer (nullable = true)\n"," |-- ct_dst_ltm: integer (nullable = true)\n"," |-- ct_src_ ltm: integer (nullable = true)\n"," |-- ct_src_dport_ltm: integer (nullable = true)\n"," |-- ct_dst_sport_ltm: integer (nullable = true)\n"," |-- ct_dst_src_ltm: integer (nullable = true)\n"," |-- attack_cat: string (nullable = true)\n"," |-- Label: integer (nullable = true)\n","\n"]}],"source":["#Display Schema\n","UNSWNB15.printSchema()"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":2069,"status":"ok","timestamp":1648900420838,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"},"user_tz":-60},"id":"FzsO-MGXzOx4","outputId":"bea8a5e6-68d1-4bd0-dfb6-4dace10e9df0"},"outputs":[{"output_type":"stream","name":"stdout","text":["+----------+-----+-------------+------+-----+-----+------------+------+------+----+----+-----+-----+--------+---------+---------+-----+-----+----+----+----------+----------+-------+-------+-----------+-----------+----------+---------+----------+----------+------------+------------+------------+------------+------------+---------------+------------+----------------+------------+----------+----------+----------+----------+-----------+----------------+----------------+--------------+----------+-----+\n","|     srcip|sport|        dstip|dsport|proto|state|         dur|sbytes|dbytes|sttl|dttl|sloss|dloss| service|    Sload|    Dload|Spkts|Dpkts|swin|dwin|     stcpb|     dtcpb|smeansz|dmeansz|trans_depth|res_bdy_len|      Sjit|     Djit|     Stime|     Ltime|        Spkt|        Dpkt|      tcprtt|      synack|      ackdat|is_sm_ips_ports|ct_state_ttl|ct_flw_http_mthd|is_ftp_login|ct_ftp_cmd|ct_srv_src|ct_srv_dst|ct_dst_ltm|ct_src_ ltm|ct_src_dport_ltm|ct_dst_sport_ltm|ct_dst_src_ltm|attack_cat|Label|\n","+----------+-----+-------------+------+-----+-----+------------+------+------+----+----+-----+-----+--------+---------+---------+-----+-----+----+----+----------+----------+-------+-------+-----------+-----------+----------+---------+----------+----------+------------+------------+------------+------------+------------+---------------+------------+----------------+------------+----------+----------+----------+----------+-----------+----------------+----------------+--------------+----------+-----+\n","|59.166.0.3|56716|149.171.126.8|   143|  tcp|  FIN|  0.82546002|  7812| 16236|  31|  29|   30|   32|       -| 75090.25|156111.73|  122|  126| 255| 255|2751097753|2748686736|     64|    129|          0|          0| 445.25928| 474.9451|1421970774|1421970775|   6.8190908|    6.599896|5.9700001E-4|4.6899999E-4|     1.28E-4|              0|           0|               0|           0|         0|         2|         7|         1|          4|               1|               1|             1|      null|    0|\n","|59.166.0.0|43467|149.171.126.6| 49729|  tcp|  FIN|    0.101815|  4238| 65628|  31|  29|    7|   30|       -|328438.84|5087030.5|   72|   74| 255| 255| 961515433|3225510659|     59|    887|          0|          0|       0.0|91.579567|1421970775|1421970775|    1.429493|    1.387192|      6.8E-4|5.4600002E-4|     1.34E-4|              0|           0|               0|           0|         0|         7|         4|         1|          6|               1|               1|             1|      null|    0|\n","|59.166.0.5|41289|149.171.126.2|  9574|  tcp|  FIN| 0.044002999|  2750| 29104|  31|  29|    7|   17|       -|488693.97|5181101.5|   44|   48| 255| 255|3291096757|1191410228|     63|    606|          0|          0| 78.126968|62.206562|1421970775|1421970775|    1.014977|  0.92583001|     0.00125|     4.85E-4|     7.65E-4|              0|           0|               0|           0|         0|         3|         5|         3|          3|               1|               1|             1|      null|    0|\n","|59.166.0.9|43785|149.171.126.0|  6881|  tcp|  FIN|   2.7908299| 10476|395734|  31|  29|   16|  143|       -|29863.518|1130840.8|  180|  320| 255| 255|3934392726|3961690324|     58|   1237|          0|          0| 2707.4927| 2018.976|1421970772|1421970775|   15.589459|   8.7470121|6.8400003E-4|5.3199998E-4|1.5199999E-4|              0|           0|               0|           0|         0|        11|         4|         3|          2|               1|               1|             1|      null|    0|\n","|59.166.0.8|40691|149.171.126.9|  6881|  tcp|  FIN|   2.6335001| 13350|548216|  31|  29|   21|  197|       -|40381.238|1661560.6|  232|  438| 255| 255|   1518931|  18267719|     58|   1252|          0|          0| 718.33679|500.57288|1421970773|1421970775|   11.399026|   6.0251832|     6.19E-4|     4.89E-4|      1.3E-4|              0|           0|               0|           0|         0|        16|         7|         7|          1|               1|               1|             1|      null|    0|\n","|59.166.0.3|20393|149.171.126.3|  5190|  tcp|  FIN|    0.115048|  1958|  2308|  31|  29|    6|    6|       -|129963.15|153814.06|   22|   24| 255| 255|3646899201|3651364285|     89|     96|          0|          0| 435.26627|417.08563|1421970775|1421970775|    5.460381|    4.976913|7.0999999E-4|     5.73E-4|     1.37E-4|              0|           0|               0|           0|         0|         2|         6|         1|          4|               1|               1|             1|      null|    0|\n","|59.166.0.7|19792|149.171.126.0|    53|  udp|  CON|    0.003362|   146|   178|  31|  29|    0|    0|     dns|173706.13| 211778.7|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970775|1421970775|       0.011|0.0060000001|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         3|         2|         3|          3|               3|               1|             1|      null|    0|\n","|59.166.0.3|14382|149.171.126.9|  3354|  tcp|  FIN|  0.45305201|   424|  8824|  31|  29|    1|    4|ftp-data| 6551.124| 142835.7|    8|   12| 255| 255|2206905053|3307670308|     53|    735|          0|          0| 3906.7949|3074.6694|1421970775|1421970775|   64.671288|   41.134998|      6.8E-4|5.5900001E-4|     1.21E-4|              0|           0|               0|           0|         0|         4|         6|         7|          4|               1|               1|             2|      null|    0|\n","|59.166.0.9|37074|149.171.126.2|    53|  udp|  CON|    0.001088|   146|   178|  31|  29|    0|    0|     dns|536764.69|654411.75|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970775|1421970775|       0.001|0.0089999996|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         2|         5|         3|          2|               1|               1|             1|      null|    0|\n","|59.166.0.7|12569|149.171.126.5|    53|  udp|  CON|9.6899999E-4|   146|   178|  31|  29|    0|    0|     dns|602683.19|734778.13|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970775|1421970775|0.0099999998|       0.003|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         3|         1|         2|          3|               3|               1|             1|      null|    0|\n","|59.166.0.1|12792|149.171.126.7|    53|  udp|  CON|0.0010629999|   146|   178|  31|  29|    0|    0|     dns|549388.56| 669802.5|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970780|1421970780|0.0020000001|       0.003|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         5|         3|         2|          4|               3|               1|             1|      null|    0|\n","|59.166.0.0|63414|149.171.126.9| 10330|  tcp|  FIN|  0.26501101|  8928|   320|  31|  29|    4|    1|ftp-data|250283.94|8060.0425|   14|    6| 255| 255|2386904726|3699988372|    638|     53|          0|          0| 1725.2916|71.464249|1421970775|1421970776|   20.385462|   52.644802|7.7500002E-4|6.3299999E-4|     1.42E-4|              0|           0|               0|           0|         0|         3|         6|         6|          5|               1|               1|             2|      null|    0|\n","|59.166.0.1|33555|149.171.126.3|  6881|  tcp|  FIN|  0.51712799|  1540|  1644|  31|  29|    4|    4|       -|22338.764|  24025.0|   16|   18| 255| 255|1741520309|3943579644|     96|     91|          0|          0| 2036.1301| 51.91766|1421970776|1421970776|   34.433331|   30.388353|6.5399997E-4|5.2200002E-4|     1.32E-4|              0|           0|               0|           0|         0|         4|         6|         5|          7|               4|               1|             4|      null|    0|\n","|59.166.0.8|10867|149.171.126.8|   111|  udp|  CON|0.0053389999|   568|   312|  31|  29|    0|    0|       -|638321.81|350627.47|    4|    4|   0|   0|         0|         0|    142|     78|          0|          0| 1.7430201| 1.757632|1421970776|1421970776|        1.24|    1.252333|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|        16|         7|         5|          5|               1|               1|             4|      null|    0|\n","|59.166.0.8|12411|149.171.126.8|  1715|  udp|  CON|    0.001739|   512|   304|  31|  29|    0|    0|       -|1766532.5|1048878.6|    4|    4|   0|   0|         0|         0|    128|     76|          0|          0|0.63026798| 0.318434|1421970776|1421970776|  0.44966701|    0.227667|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|        16|         7|         5|          5|               1|               1|             4|      null|    0|\n","|59.166.0.8|46725|149.171.126.2|    53|  udp|  CON|    0.001018|   146|   178|  31|  29|    0|    0|     dns|573673.88|699410.63|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970776|1421970776|       0.011|       0.011|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         2|         5|         3|          5|               1|               1|             1|      null|    0|\n","|59.166.0.1|51562|149.171.126.4|    53|  udp|  CON|    0.001044|   146|   178|  31|  29|    0|    0|     dns|559386.94|681992.31|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970776|1421970776|       0.011|       0.003|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         2|         3|         5|          7|               2|               1|             3|      null|    0|\n","|59.166.0.3|48838|149.171.126.2|    53|  udp|  CON|9.8699995E-4|   146|   178|  31|  29|    0|    0|     dns| 591692.0|721377.94|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970776|1421970776|0.0080000004|       0.003|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         2|         5|         3|          3|               1|               1|             1|      null|    0|\n","|59.166.0.0|16907|149.171.126.9|    21|  tcp|  FIN|   2.2547121|  2934|  3740|  31|  29|   11|   15|     ftp|10211.503|13025.166|   52|   54| 255| 255| 241515551|2584135680|     56|     69|          0|          0| 3141.3708| 99.93412|1421970773|1421970776|   44.203648|   42.531734|6.6100003E-4|5.2499998E-4|     1.36E-4|              0|           0|               0|           0|         0|         1|         3|         6|          5|               1|               1|             2|      null|    0|\n","|59.166.0.0| 1915|149.171.126.4| 32945|  tcp|  FIN| 0.051220998|  2854| 29104|  31|  29|    7|   17|       -|436071.16|4450987.0|   46|   48| 255| 255|1921515932|1974066994|     62|    606|          0|          0| 78.226654|76.387978|1421970776|1421970776|   1.1294219|   1.0781699|      8.1E-4|     5.43E-4|     2.67E-4|              0|           0|               0|           0|         0|         7|         2|         5|          5|               1|               1|             2|      null|    0|\n","+----------+-----+-------------+------+-----+-----+------------+------+------+----+----+-----+-----+--------+---------+---------+-----+-----+----+----+----------+----------+-------+-------+-----------+-----------+----------+---------+----------+----------+------------+------------+------------+------------+------------+---------------+------------+----------------+------------+----------+----------+----------+----------+-----------+----------------+----------------+--------------+----------+-----+\n","only showing top 20 rows\n","\n"]}],"source":["#Show Dataframe UNSWNB15\n","UNSWNB15.show()"]},{"cell_type":"code","source":["#Define dataset by by replacing null values in attack_cat column with normal\n","dataset = UNSWNB15.na.fill(value='normal',subset=[\"attack_cat\"])\n","dataset.show()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"H_6Ei-0yjGw8","executionInfo":{"status":"ok","timestamp":1648900495866,"user_tz":-60,"elapsed":1156,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}},"outputId":"e333e5d7-8875-4ce5-f25c-414de3d77636"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["+----------+-----+-------------+------+-----+-----+------------+------+------+----+----+-----+-----+--------+---------+---------+-----+-----+----+----+----------+----------+-------+-------+-----------+-----------+----------+---------+----------+----------+------------+------------+------------+------------+------------+---------------+------------+----------------+------------+----------+----------+----------+----------+-----------+----------------+----------------+--------------+----------+-----+\n","|     srcip|sport|        dstip|dsport|proto|state|         dur|sbytes|dbytes|sttl|dttl|sloss|dloss| service|    Sload|    Dload|Spkts|Dpkts|swin|dwin|     stcpb|     dtcpb|smeansz|dmeansz|trans_depth|res_bdy_len|      Sjit|     Djit|     Stime|     Ltime|        Spkt|        Dpkt|      tcprtt|      synack|      ackdat|is_sm_ips_ports|ct_state_ttl|ct_flw_http_mthd|is_ftp_login|ct_ftp_cmd|ct_srv_src|ct_srv_dst|ct_dst_ltm|ct_src_ ltm|ct_src_dport_ltm|ct_dst_sport_ltm|ct_dst_src_ltm|attack_cat|Label|\n","+----------+-----+-------------+------+-----+-----+------------+------+------+----+----+-----+-----+--------+---------+---------+-----+-----+----+----+----------+----------+-------+-------+-----------+-----------+----------+---------+----------+----------+------------+------------+------------+------------+------------+---------------+------------+----------------+------------+----------+----------+----------+----------+-----------+----------------+----------------+--------------+----------+-----+\n","|59.166.0.3|56716|149.171.126.8|   143|  tcp|  FIN|  0.82546002|  7812| 16236|  31|  29|   30|   32|       -| 75090.25|156111.73|  122|  126| 255| 255|2751097753|2748686736|     64|    129|          0|          0| 445.25928| 474.9451|1421970774|1421970775|   6.8190908|    6.599896|5.9700001E-4|4.6899999E-4|     1.28E-4|              0|           0|               0|           0|         0|         2|         7|         1|          4|               1|               1|             1|    normal|    0|\n","|59.166.0.0|43467|149.171.126.6| 49729|  tcp|  FIN|    0.101815|  4238| 65628|  31|  29|    7|   30|       -|328438.84|5087030.5|   72|   74| 255| 255| 961515433|3225510659|     59|    887|          0|          0|       0.0|91.579567|1421970775|1421970775|    1.429493|    1.387192|      6.8E-4|5.4600002E-4|     1.34E-4|              0|           0|               0|           0|         0|         7|         4|         1|          6|               1|               1|             1|    normal|    0|\n","|59.166.0.5|41289|149.171.126.2|  9574|  tcp|  FIN| 0.044002999|  2750| 29104|  31|  29|    7|   17|       -|488693.97|5181101.5|   44|   48| 255| 255|3291096757|1191410228|     63|    606|          0|          0| 78.126968|62.206562|1421970775|1421970775|    1.014977|  0.92583001|     0.00125|     4.85E-4|     7.65E-4|              0|           0|               0|           0|         0|         3|         5|         3|          3|               1|               1|             1|    normal|    0|\n","|59.166.0.9|43785|149.171.126.0|  6881|  tcp|  FIN|   2.7908299| 10476|395734|  31|  29|   16|  143|       -|29863.518|1130840.8|  180|  320| 255| 255|3934392726|3961690324|     58|   1237|          0|          0| 2707.4927| 2018.976|1421970772|1421970775|   15.589459|   8.7470121|6.8400003E-4|5.3199998E-4|1.5199999E-4|              0|           0|               0|           0|         0|        11|         4|         3|          2|               1|               1|             1|    normal|    0|\n","|59.166.0.8|40691|149.171.126.9|  6881|  tcp|  FIN|   2.6335001| 13350|548216|  31|  29|   21|  197|       -|40381.238|1661560.6|  232|  438| 255| 255|   1518931|  18267719|     58|   1252|          0|          0| 718.33679|500.57288|1421970773|1421970775|   11.399026|   6.0251832|     6.19E-4|     4.89E-4|      1.3E-4|              0|           0|               0|           0|         0|        16|         7|         7|          1|               1|               1|             1|    normal|    0|\n","|59.166.0.3|20393|149.171.126.3|  5190|  tcp|  FIN|    0.115048|  1958|  2308|  31|  29|    6|    6|       -|129963.15|153814.06|   22|   24| 255| 255|3646899201|3651364285|     89|     96|          0|          0| 435.26627|417.08563|1421970775|1421970775|    5.460381|    4.976913|7.0999999E-4|     5.73E-4|     1.37E-4|              0|           0|               0|           0|         0|         2|         6|         1|          4|               1|               1|             1|    normal|    0|\n","|59.166.0.7|19792|149.171.126.0|    53|  udp|  CON|    0.003362|   146|   178|  31|  29|    0|    0|     dns|173706.13| 211778.7|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970775|1421970775|       0.011|0.0060000001|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         3|         2|         3|          3|               3|               1|             1|    normal|    0|\n","|59.166.0.3|14382|149.171.126.9|  3354|  tcp|  FIN|  0.45305201|   424|  8824|  31|  29|    1|    4|ftp-data| 6551.124| 142835.7|    8|   12| 255| 255|2206905053|3307670308|     53|    735|          0|          0| 3906.7949|3074.6694|1421970775|1421970775|   64.671288|   41.134998|      6.8E-4|5.5900001E-4|     1.21E-4|              0|           0|               0|           0|         0|         4|         6|         7|          4|               1|               1|             2|    normal|    0|\n","|59.166.0.9|37074|149.171.126.2|    53|  udp|  CON|    0.001088|   146|   178|  31|  29|    0|    0|     dns|536764.69|654411.75|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970775|1421970775|       0.001|0.0089999996|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         2|         5|         3|          2|               1|               1|             1|    normal|    0|\n","|59.166.0.7|12569|149.171.126.5|    53|  udp|  CON|9.6899999E-4|   146|   178|  31|  29|    0|    0|     dns|602683.19|734778.13|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970775|1421970775|0.0099999998|       0.003|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         3|         1|         2|          3|               3|               1|             1|    normal|    0|\n","|59.166.0.1|12792|149.171.126.7|    53|  udp|  CON|0.0010629999|   146|   178|  31|  29|    0|    0|     dns|549388.56| 669802.5|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970780|1421970780|0.0020000001|       0.003|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         5|         3|         2|          4|               3|               1|             1|    normal|    0|\n","|59.166.0.0|63414|149.171.126.9| 10330|  tcp|  FIN|  0.26501101|  8928|   320|  31|  29|    4|    1|ftp-data|250283.94|8060.0425|   14|    6| 255| 255|2386904726|3699988372|    638|     53|          0|          0| 1725.2916|71.464249|1421970775|1421970776|   20.385462|   52.644802|7.7500002E-4|6.3299999E-4|     1.42E-4|              0|           0|               0|           0|         0|         3|         6|         6|          5|               1|               1|             2|    normal|    0|\n","|59.166.0.1|33555|149.171.126.3|  6881|  tcp|  FIN|  0.51712799|  1540|  1644|  31|  29|    4|    4|       -|22338.764|  24025.0|   16|   18| 255| 255|1741520309|3943579644|     96|     91|          0|          0| 2036.1301| 51.91766|1421970776|1421970776|   34.433331|   30.388353|6.5399997E-4|5.2200002E-4|     1.32E-4|              0|           0|               0|           0|         0|         4|         6|         5|          7|               4|               1|             4|    normal|    0|\n","|59.166.0.8|10867|149.171.126.8|   111|  udp|  CON|0.0053389999|   568|   312|  31|  29|    0|    0|       -|638321.81|350627.47|    4|    4|   0|   0|         0|         0|    142|     78|          0|          0| 1.7430201| 1.757632|1421970776|1421970776|        1.24|    1.252333|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|        16|         7|         5|          5|               1|               1|             4|    normal|    0|\n","|59.166.0.8|12411|149.171.126.8|  1715|  udp|  CON|    0.001739|   512|   304|  31|  29|    0|    0|       -|1766532.5|1048878.6|    4|    4|   0|   0|         0|         0|    128|     76|          0|          0|0.63026798| 0.318434|1421970776|1421970776|  0.44966701|    0.227667|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|        16|         7|         5|          5|               1|               1|             4|    normal|    0|\n","|59.166.0.8|46725|149.171.126.2|    53|  udp|  CON|    0.001018|   146|   178|  31|  29|    0|    0|     dns|573673.88|699410.63|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970776|1421970776|       0.011|       0.011|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         2|         5|         3|          5|               1|               1|             1|    normal|    0|\n","|59.166.0.1|51562|149.171.126.4|    53|  udp|  CON|    0.001044|   146|   178|  31|  29|    0|    0|     dns|559386.94|681992.31|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970776|1421970776|       0.011|       0.003|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         2|         3|         5|          7|               2|               1|             3|    normal|    0|\n","|59.166.0.3|48838|149.171.126.2|    53|  udp|  CON|9.8699995E-4|   146|   178|  31|  29|    0|    0|     dns| 591692.0|721377.94|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970776|1421970776|0.0080000004|       0.003|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         2|         5|         3|          3|               1|               1|             1|    normal|    0|\n","|59.166.0.0|16907|149.171.126.9|    21|  tcp|  FIN|   2.2547121|  2934|  3740|  31|  29|   11|   15|     ftp|10211.503|13025.166|   52|   54| 255| 255| 241515551|2584135680|     56|     69|          0|          0| 3141.3708| 99.93412|1421970773|1421970776|   44.203648|   42.531734|6.6100003E-4|5.2499998E-4|     1.36E-4|              0|           0|               0|           0|         0|         1|         3|         6|          5|               1|               1|             2|    normal|    0|\n","|59.166.0.0| 1915|149.171.126.4| 32945|  tcp|  FIN| 0.051220998|  2854| 29104|  31|  29|    7|   17|       -|436071.16|4450987.0|   46|   48| 255| 255|1921515932|1974066994|     62|    606|          0|          0| 78.226654|76.387978|1421970776|1421970776|   1.1294219|   1.0781699|      8.1E-4|     5.43E-4|     2.67E-4|              0|           0|               0|           0|         0|         7|         2|         5|          5|               1|               1|             2|    normal|    0|\n","+----------+-----+-------------+------+-----+-----+------------+------+------+----+----+-----+-----+--------+---------+---------+-----+-----+----+----+----------+----------+-------+-------+-----------+-----------+----------+---------+----------+----------+------------+------------+------------+------------+------------+---------------+------------+----------------+------------+----------+----------+----------+----------+-----------+----------------+----------------+--------------+----------+-----+\n","only showing top 20 rows\n","\n"]}]},{"cell_type":"code","source":["#Update all values in attack_cat column\n","#from pyspark.sql.functions import col,when\n","from pyspark.sql.functions import col,when\n","updated_dataset = dataset.withColumn(\"attack_cat\", when( col(\"attack_cat\")==\"normal\",\"Normal\" )\\\n",".when( col(\"attack_cat\")==\"Generic\",\"Generic\" )\\\n",".when( col(\"attack_cat\")==\"Exploits\",\"Exploits\" )\\\n",".when( col(\"attack_cat\")==\"DoS\",\"DoS\" )\\\n",".when( col(\"attack_cat\")==\"Fuzzers\",\"Fuzzers\" )\\\n",".when( col(\"attack_cat\")==\"Analysis\",\"Analysis\" )\\\n",".when( col(\"attack_cat\")==\"Reconnaissance\",\"Reconnaissance\" )\\\n",".when( col(\"attack_cat\")==\"Backdoors\",\"Backdoors\" )\\\n",".when( col(\"attack_cat\")==\"Shellcode\",\"Shellcode\" )\\\n",".when( col(\"attack_cat\")==\"Worms\",\"Worms\" )\\\n",".when( col(\"attack_cat\")==\"Fuzzers \", \"Fuzzers\")\\\n",".when( col(\"attack_cat\")==\"Reconnaissance \",\"Reconnaissance\" )\\\n",".when( col(\"attack_cat\")==\"Backdoor\",\"Backdoors\" )\\\n",".when( col(\"attack_cat\")==\"Shellcode \",\"Shellcode\" )\\\n",")"],"metadata":{"id":"zRGa-OjgWDa8","executionInfo":{"status":"ok","timestamp":1648908537828,"user_tz":-60,"elapsed":322,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":90,"outputs":[]},{"cell_type":"code","source":["#Confirm the dataset and the number of records\n","updated_dataset.show()\n","updated_dataset.count()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"4oSb9QufnBBG","executionInfo":{"status":"ok","timestamp":1648908595648,"user_tz":-60,"elapsed":4196,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}},"outputId":"5b0bb376-6f7c-4e56-a8da-62409411e56b"},"execution_count":92,"outputs":[{"output_type":"stream","name":"stdout","text":["+----------+-----+-------------+------+-----+-----+------------+------+------+----+----+-----+-----+--------+---------+---------+-----+-----+----+----+----------+----------+-------+-------+-----------+-----------+----------+---------+----------+----------+------------+------------+------------+------------+------------+---------------+------------+----------------+------------+----------+----------+----------+----------+-----------+----------------+----------------+--------------+----------+-----+\n","|     srcip|sport|        dstip|dsport|proto|state|         dur|sbytes|dbytes|sttl|dttl|sloss|dloss| service|    Sload|    Dload|Spkts|Dpkts|swin|dwin|     stcpb|     dtcpb|smeansz|dmeansz|trans_depth|res_bdy_len|      Sjit|     Djit|     Stime|     Ltime|        Spkt|        Dpkt|      tcprtt|      synack|      ackdat|is_sm_ips_ports|ct_state_ttl|ct_flw_http_mthd|is_ftp_login|ct_ftp_cmd|ct_srv_src|ct_srv_dst|ct_dst_ltm|ct_src_ ltm|ct_src_dport_ltm|ct_dst_sport_ltm|ct_dst_src_ltm|attack_cat|Label|\n","+----------+-----+-------------+------+-----+-----+------------+------+------+----+----+-----+-----+--------+---------+---------+-----+-----+----+----+----------+----------+-------+-------+-----------+-----------+----------+---------+----------+----------+------------+------------+------------+------------+------------+---------------+------------+----------------+------------+----------+----------+----------+----------+-----------+----------------+----------------+--------------+----------+-----+\n","|59.166.0.3|56716|149.171.126.8|   143|  tcp|  FIN|  0.82546002|  7812| 16236|  31|  29|   30|   32|       -| 75090.25|156111.73|  122|  126| 255| 255|2751097753|2748686736|     64|    129|          0|          0| 445.25928| 474.9451|1421970774|1421970775|   6.8190908|    6.599896|5.9700001E-4|4.6899999E-4|     1.28E-4|              0|           0|               0|           0|         0|         2|         7|         1|          4|               1|               1|             1|    Normal|    0|\n","|59.166.0.0|43467|149.171.126.6| 49729|  tcp|  FIN|    0.101815|  4238| 65628|  31|  29|    7|   30|       -|328438.84|5087030.5|   72|   74| 255| 255| 961515433|3225510659|     59|    887|          0|          0|       0.0|91.579567|1421970775|1421970775|    1.429493|    1.387192|      6.8E-4|5.4600002E-4|     1.34E-4|              0|           0|               0|           0|         0|         7|         4|         1|          6|               1|               1|             1|    Normal|    0|\n","|59.166.0.5|41289|149.171.126.2|  9574|  tcp|  FIN| 0.044002999|  2750| 29104|  31|  29|    7|   17|       -|488693.97|5181101.5|   44|   48| 255| 255|3291096757|1191410228|     63|    606|          0|          0| 78.126968|62.206562|1421970775|1421970775|    1.014977|  0.92583001|     0.00125|     4.85E-4|     7.65E-4|              0|           0|               0|           0|         0|         3|         5|         3|          3|               1|               1|             1|    Normal|    0|\n","|59.166.0.9|43785|149.171.126.0|  6881|  tcp|  FIN|   2.7908299| 10476|395734|  31|  29|   16|  143|       -|29863.518|1130840.8|  180|  320| 255| 255|3934392726|3961690324|     58|   1237|          0|          0| 2707.4927| 2018.976|1421970772|1421970775|   15.589459|   8.7470121|6.8400003E-4|5.3199998E-4|1.5199999E-4|              0|           0|               0|           0|         0|        11|         4|         3|          2|               1|               1|             1|    Normal|    0|\n","|59.166.0.8|40691|149.171.126.9|  6881|  tcp|  FIN|   2.6335001| 13350|548216|  31|  29|   21|  197|       -|40381.238|1661560.6|  232|  438| 255| 255|   1518931|  18267719|     58|   1252|          0|          0| 718.33679|500.57288|1421970773|1421970775|   11.399026|   6.0251832|     6.19E-4|     4.89E-4|      1.3E-4|              0|           0|               0|           0|         0|        16|         7|         7|          1|               1|               1|             1|    Normal|    0|\n","|59.166.0.3|20393|149.171.126.3|  5190|  tcp|  FIN|    0.115048|  1958|  2308|  31|  29|    6|    6|       -|129963.15|153814.06|   22|   24| 255| 255|3646899201|3651364285|     89|     96|          0|          0| 435.26627|417.08563|1421970775|1421970775|    5.460381|    4.976913|7.0999999E-4|     5.73E-4|     1.37E-4|              0|           0|               0|           0|         0|         2|         6|         1|          4|               1|               1|             1|    Normal|    0|\n","|59.166.0.7|19792|149.171.126.0|    53|  udp|  CON|    0.003362|   146|   178|  31|  29|    0|    0|     dns|173706.13| 211778.7|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970775|1421970775|       0.011|0.0060000001|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         3|         2|         3|          3|               3|               1|             1|    Normal|    0|\n","|59.166.0.3|14382|149.171.126.9|  3354|  tcp|  FIN|  0.45305201|   424|  8824|  31|  29|    1|    4|ftp-data| 6551.124| 142835.7|    8|   12| 255| 255|2206905053|3307670308|     53|    735|          0|          0| 3906.7949|3074.6694|1421970775|1421970775|   64.671288|   41.134998|      6.8E-4|5.5900001E-4|     1.21E-4|              0|           0|               0|           0|         0|         4|         6|         7|          4|               1|               1|             2|    Normal|    0|\n","|59.166.0.9|37074|149.171.126.2|    53|  udp|  CON|    0.001088|   146|   178|  31|  29|    0|    0|     dns|536764.69|654411.75|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970775|1421970775|       0.001|0.0089999996|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         2|         5|         3|          2|               1|               1|             1|    Normal|    0|\n","|59.166.0.7|12569|149.171.126.5|    53|  udp|  CON|9.6899999E-4|   146|   178|  31|  29|    0|    0|     dns|602683.19|734778.13|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970775|1421970775|0.0099999998|       0.003|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         3|         1|         2|          3|               3|               1|             1|    Normal|    0|\n","|59.166.0.1|12792|149.171.126.7|    53|  udp|  CON|0.0010629999|   146|   178|  31|  29|    0|    0|     dns|549388.56| 669802.5|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970780|1421970780|0.0020000001|       0.003|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         5|         3|         2|          4|               3|               1|             1|    Normal|    0|\n","|59.166.0.0|63414|149.171.126.9| 10330|  tcp|  FIN|  0.26501101|  8928|   320|  31|  29|    4|    1|ftp-data|250283.94|8060.0425|   14|    6| 255| 255|2386904726|3699988372|    638|     53|          0|          0| 1725.2916|71.464249|1421970775|1421970776|   20.385462|   52.644802|7.7500002E-4|6.3299999E-4|     1.42E-4|              0|           0|               0|           0|         0|         3|         6|         6|          5|               1|               1|             2|    Normal|    0|\n","|59.166.0.1|33555|149.171.126.3|  6881|  tcp|  FIN|  0.51712799|  1540|  1644|  31|  29|    4|    4|       -|22338.764|  24025.0|   16|   18| 255| 255|1741520309|3943579644|     96|     91|          0|          0| 2036.1301| 51.91766|1421970776|1421970776|   34.433331|   30.388353|6.5399997E-4|5.2200002E-4|     1.32E-4|              0|           0|               0|           0|         0|         4|         6|         5|          7|               4|               1|             4|    Normal|    0|\n","|59.166.0.8|10867|149.171.126.8|   111|  udp|  CON|0.0053389999|   568|   312|  31|  29|    0|    0|       -|638321.81|350627.47|    4|    4|   0|   0|         0|         0|    142|     78|          0|          0| 1.7430201| 1.757632|1421970776|1421970776|        1.24|    1.252333|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|        16|         7|         5|          5|               1|               1|             4|    Normal|    0|\n","|59.166.0.8|12411|149.171.126.8|  1715|  udp|  CON|    0.001739|   512|   304|  31|  29|    0|    0|       -|1766532.5|1048878.6|    4|    4|   0|   0|         0|         0|    128|     76|          0|          0|0.63026798| 0.318434|1421970776|1421970776|  0.44966701|    0.227667|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|        16|         7|         5|          5|               1|               1|             4|    Normal|    0|\n","|59.166.0.8|46725|149.171.126.2|    53|  udp|  CON|    0.001018|   146|   178|  31|  29|    0|    0|     dns|573673.88|699410.63|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970776|1421970776|       0.011|       0.011|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         2|         5|         3|          5|               1|               1|             1|    Normal|    0|\n","|59.166.0.1|51562|149.171.126.4|    53|  udp|  CON|    0.001044|   146|   178|  31|  29|    0|    0|     dns|559386.94|681992.31|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970776|1421970776|       0.011|       0.003|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         2|         3|         5|          7|               2|               1|             3|    Normal|    0|\n","|59.166.0.3|48838|149.171.126.2|    53|  udp|  CON|9.8699995E-4|   146|   178|  31|  29|    0|    0|     dns| 591692.0|721377.94|    2|    2|   0|   0|         0|         0|     73|     89|          0|          0|       0.0|      0.0|1421970776|1421970776|0.0080000004|       0.003|         0.0|         0.0|         0.0|              0|           0|               0|           0|         0|         2|         5|         3|          3|               1|               1|             1|    Normal|    0|\n","|59.166.0.0|16907|149.171.126.9|    21|  tcp|  FIN|   2.2547121|  2934|  3740|  31|  29|   11|   15|     ftp|10211.503|13025.166|   52|   54| 255| 255| 241515551|2584135680|     56|     69|          0|          0| 3141.3708| 99.93412|1421970773|1421970776|   44.203648|   42.531734|6.6100003E-4|5.2499998E-4|     1.36E-4|              0|           0|               0|           0|         0|         1|         3|         6|          5|               1|               1|             2|    Normal|    0|\n","|59.166.0.0| 1915|149.171.126.4| 32945|  tcp|  FIN| 0.051220998|  2854| 29104|  31|  29|    7|   17|       -|436071.16|4450987.0|   46|   48| 255| 255|1921515932|1974066994|     62|    606|          0|          0| 78.226654|76.387978|1421970776|1421970776|   1.1294219|   1.0781699|      8.1E-4|     5.43E-4|     2.67E-4|              0|           0|               0|           0|         0|         7|         2|         5|          5|               1|               1|             2|    Normal|    0|\n","+----------+-----+-------------+------+-----+-----+------------+------+------+----+----+-----+-----+--------+---------+---------+-----+-----+----+----+----------+----------+-------+-------+-----------+-----------+----------+---------+----------+----------+------------+------------+------------+------------+------------+---------------+------------+----------------+------------+----------+----------+----------+----------+-----------+----------------+----------------+--------------+----------+-----+\n","only showing top 20 rows\n","\n"]},{"output_type":"execute_result","data":{"text/plain":["2539739"]},"metadata":{},"execution_count":92}]},{"cell_type":"code","source":["from pyspark.ml import Pipeline\n","from pyspark.ml.classification import LogisticRegression\n","from pyspark.ml.feature import HashingTF, Tokenizer\n","from pyspark.sql import Row\n","from pyspark.sql.functions import UserDefinedFunction\n","from pyspark.sql.types import *\n","import pyspark\n","from pyspark.context import SparkContext\n","from pyspark.sql.session import SparkSession\n","from py4j.protocol import Py4JJavaError"],"metadata":{"id":"YOk4ZtHl_AhY","executionInfo":{"status":"ok","timestamp":1648908559800,"user_tz":-60,"elapsed":407,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":91,"outputs":[]},{"cell_type":"code","source":["#Define Dataframe df\n","df = updated_dataset.select('srcip','attack_cat')"],"metadata":{"id":"_y44v9hAGIxX","executionInfo":{"status":"ok","timestamp":1648908637837,"user_tz":-60,"elapsed":602,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":93,"outputs":[]},{"cell_type":"code","source":["#Count and show Dataframe df\n","df.show()\n","df.count()"],"metadata":{"id":"i0YAnRHbHMyY","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648908643585,"user_tz":-60,"elapsed":2664,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}},"outputId":"2f8dcf5d-827f-467e-a2d8-2764d1679cdf"},"execution_count":94,"outputs":[{"output_type":"stream","name":"stdout","text":["+----------+----------+\n","|     srcip|attack_cat|\n","+----------+----------+\n","|59.166.0.3|    Normal|\n","|59.166.0.0|    Normal|\n","|59.166.0.5|    Normal|\n","|59.166.0.9|    Normal|\n","|59.166.0.8|    Normal|\n","|59.166.0.3|    Normal|\n","|59.166.0.7|    Normal|\n","|59.166.0.3|    Normal|\n","|59.166.0.9|    Normal|\n","|59.166.0.7|    Normal|\n","|59.166.0.1|    Normal|\n","|59.166.0.0|    Normal|\n","|59.166.0.1|    Normal|\n","|59.166.0.8|    Normal|\n","|59.166.0.8|    Normal|\n","|59.166.0.8|    Normal|\n","|59.166.0.1|    Normal|\n","|59.166.0.3|    Normal|\n","|59.166.0.0|    Normal|\n","|59.166.0.0|    Normal|\n","+----------+----------+\n","only showing top 20 rows\n","\n"]},{"output_type":"execute_result","data":{"text/plain":["2539739"]},"metadata":{},"execution_count":94}]},{"cell_type":"code","source":["#Drop null values in attack_cat\n","df = df.dropna(subset=('attack_cat'))"],"metadata":{"id":"5rlUU6hkK1xo"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["#Count and show Dataframe df\n","df.show()\n","df.count()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"bSo_COCVK53m","executionInfo":{"status":"ok","timestamp":1648908134045,"user_tz":-60,"elapsed":4101,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}},"outputId":"d32b542b-bfef-4c46-afd9-e520b5b12d54"},"execution_count":81,"outputs":[{"output_type":"stream","name":"stdout","text":["+----------+----------+\n","|     srcip|attack_cat|\n","+----------+----------+\n","|59.166.0.3|    Normal|\n","|59.166.0.0|    Normal|\n","|59.166.0.5|    Normal|\n","|59.166.0.9|    Normal|\n","|59.166.0.8|    Normal|\n","|59.166.0.3|    Normal|\n","|59.166.0.7|    Normal|\n","|59.166.0.3|    Normal|\n","|59.166.0.9|    Normal|\n","|59.166.0.7|    Normal|\n","|59.166.0.1|    Normal|\n","|59.166.0.0|    Normal|\n","|59.166.0.1|    Normal|\n","|59.166.0.8|    Normal|\n","|59.166.0.8|    Normal|\n","|59.166.0.8|    Normal|\n","|59.166.0.1|    Normal|\n","|59.166.0.3|    Normal|\n","|59.166.0.0|    Normal|\n","|59.166.0.0|    Normal|\n","+----------+----------+\n","only showing top 20 rows\n","\n"]},{"output_type":"execute_result","data":{"text/plain":["2539739"]},"metadata":{},"execution_count":81}]},{"cell_type":"code","source":["#Give a name to the app.\n","spark = SparkSession.builder.appName(\"UNSWNB15ClassifierApp\").getOrCreate()\n","df = updated_dataset.select('srcip','attack_cat')"],"metadata":{"id":"fg5a9jfrmAnV","executionInfo":{"status":"ok","timestamp":1648908701700,"user_tz":-60,"elapsed":387,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":100,"outputs":[]},{"cell_type":"code","source":["#Import all the packages required for feature engineering\n","import pyspark.ml.feature\n","#Import the packages required to initialize the pipeline stages.\n","from pyspark.ml.feature import Tokenizer,StopWordsRemover,CountVectorizer,IDF\n","#Import package to add labels to our dataset\n","from pyspark.ml.feature import StringIndexer"],"metadata":{"id":"XhrttYhgSp-W","executionInfo":{"status":"ok","timestamp":1648908704156,"user_tz":-60,"elapsed":629,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":101,"outputs":[]},{"cell_type":"code","source":["#Sequential process starting from the tokenizer stage to the idf stage to create the pipeline stages.\n","tokenizer = Tokenizer(inputCol='srcip',outputCol='mytokens')\n","stopwords_remover = StopWordsRemover(inputCol='mytokens',outputCol='filtered_tokens')\n","vectorizer = CountVectorizer(inputCol='filtered_tokens',outputCol='rawFeatures')\n","idf = IDF(inputCol='rawFeatures',outputCol='vectorizedFeatures')"],"metadata":{"id":"1TLn4OgWeqcy","executionInfo":{"status":"ok","timestamp":1648908706564,"user_tz":-60,"elapsed":522,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":102,"outputs":[]},{"cell_type":"code","source":["#Add labels into the attact_cat column to be used when predicting the type of Attack.\n","labelEncoder = StringIndexer(inputCol='attack_cat',outputCol='label').fit(df)"],"metadata":{"id":"tIRgyoYtKa4y","executionInfo":{"status":"ok","timestamp":1648908719960,"user_tz":-60,"elapsed":11273,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":103,"outputs":[]},{"cell_type":"code","source":["dftest = labelEncoder.transform(df)"],"metadata":{"id":"8kgPu5sVL1lA","executionInfo":{"status":"ok","timestamp":1648908751710,"user_tz":-60,"elapsed":348,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":105,"outputs":[]},{"cell_type":"code","source":["#Register the DataFrame as a temporary table in the SQLContext.\n","dftest.registerTempTable(\"dftest\")\n","spark.sql(\"SELECT DISTINCT attack_cat, label FROM dftest ORDER BY label ASC\").show()"],"metadata":{"id":"BJA3CrlNL8nN","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648908768906,"user_tz":-60,"elapsed":12914,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}},"outputId":"4be683fd-04e2-473c-c351-629835f8f78c"},"execution_count":106,"outputs":[{"output_type":"stream","name":"stdout","text":["+--------------+-----+\n","|    attack_cat|label|\n","+--------------+-----+\n","|        Normal|  0.0|\n","|       Generic|  1.0|\n","|      Exploits|  2.0|\n","|       Fuzzers|  3.0|\n","|           DoS|  4.0|\n","|Reconnaissance|  5.0|\n","|      Analysis|  6.0|\n","|     Backdoors|  7.0|\n","|     Shellcode|  8.0|\n","|         Worms|  9.0|\n","+--------------+-----+\n","\n"]}]},{"cell_type":"code","source":["#See how the different Attacts are labeled\n","labelEncoder.transform(df).show()"],"metadata":{"id":"BxjsNyhrUAQ5","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648901445705,"user_tz":-60,"elapsed":775,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}},"outputId":"b52bfe25-dcbb-4c1f-cd82-9e217fb2ad8e"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["+----------+----------+-----+\n","|     srcip|attack_cat|label|\n","+----------+----------+-----+\n","|59.166.0.3|    normal|  0.0|\n","|59.166.0.0|    normal|  0.0|\n","|59.166.0.5|    normal|  0.0|\n","|59.166.0.9|    normal|  0.0|\n","|59.166.0.8|    normal|  0.0|\n","|59.166.0.3|    normal|  0.0|\n","|59.166.0.7|    normal|  0.0|\n","|59.166.0.3|    normal|  0.0|\n","|59.166.0.9|    normal|  0.0|\n","|59.166.0.7|    normal|  0.0|\n","|59.166.0.1|    normal|  0.0|\n","|59.166.0.0|    normal|  0.0|\n","|59.166.0.1|    normal|  0.0|\n","|59.166.0.8|    normal|  0.0|\n","|59.166.0.8|    normal|  0.0|\n","|59.166.0.8|    normal|  0.0|\n","|59.166.0.1|    normal|  0.0|\n","|59.166.0.3|    normal|  0.0|\n","|59.166.0.0|    normal|  0.0|\n","|59.166.0.0|    normal|  0.0|\n","+----------+----------+-----+\n","only showing top 20 rows\n","\n"]}]},{"cell_type":"code","source":["#This was the result before I updated the attact_cat in the dataset to define updated_dataset\n","#Register the DataFrame as a temporary table in the SQLContext.\n","updateddf.registerTempTable(\"updateddf\")\n","spark.sql(\"SELECT DISTINCT attack_cat, label FROM updateddf ORDER BY label ASC\").show()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"rPkdta53b5cU","executionInfo":{"status":"ok","timestamp":1648907789616,"user_tz":-60,"elapsed":14782,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}},"outputId":"2dbc923a-8ace-4adf-9b1c-81f0101efaba"},"execution_count":75,"outputs":[{"output_type":"stream","name":"stdout","text":["+--------------+-----+\n","|    attack_cat|label|\n","+--------------+-----+\n","|        Normal|  0.0|\n","|       Generic|  1.0|\n","|      Exploits|  2.0|\n","|       Fuzzers|  3.0|\n","|           DoS|  4.0|\n","|Reconnaissance|  5.0|\n","|       Fuzzers|  6.0|\n","|      Analysis|  7.0|\n","|     Backdoors|  8.0|\n","|Reconnaissance|  9.0|\n","|     Shellcode| 10.0|\n","|     Backdoors| 11.0|\n","|     Shellcode| 12.0|\n","|         Worms| 13.0|\n","+--------------+-----+\n","\n"]}]},{"cell_type":"code","source":["#Assign numeric values to the Attact categories available in the dataset for easy predictions.\n","label_dict = {'normal':0.0,\n","    'Generic':1.0,\n","    'Exploits':2.0,\n","    'Fuzzers':3.0,\n","    'DoS':4.0,\n","    'Reconnaissance':5.0,\n","    'Analysis':6.0,\n","    'Backdoor':7.0,\n","    'Shellcode':8.0,\n","    'Worms':9.0}"],"metadata":{"id":"gMZbwmr715yX","executionInfo":{"status":"ok","timestamp":1648908951997,"user_tz":-60,"elapsed":360,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":108,"outputs":[]},{"cell_type":"code","source":["#Add these labels into the dataset by using transform() method to add the labels to the respective subject categories.\n","df = labelEncoder.transform(df)"],"metadata":{"id":"UtGTh-T4NAQt","executionInfo":{"status":"ok","timestamp":1648908973178,"user_tz":-60,"elapsed":511,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":109,"outputs":[]},{"cell_type":"code","source":["#The last stage involves building the model using the LogisticRegression algorithm.\n","(mainDataFrame,testDataFrame) = df.randomSplit((0.7,0.3),seed=42)\n","#70% of our dataset will be used for training and 30% for testing"],"metadata":{"id":"7olmvQ86dZNr","executionInfo":{"status":"ok","timestamp":1648908977718,"user_tz":-60,"elapsed":545,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":110,"outputs":[]},{"cell_type":"code","source":["#Import the LogisticRegression algorithm which we will use in building the model to perform classification.\n","from pyspark.ml.classification import LogisticRegression"],"metadata":{"id":"bebUZMSx2N-E","executionInfo":{"status":"ok","timestamp":1648908980355,"user_tz":-60,"elapsed":318,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":111,"outputs":[]},{"cell_type":"code","source":["#Creating estimator \n","#An estimator is a function that takes data as input, \n","#fits the data, \n","#and creates a model used to make predictions.\n","lr = LogisticRegression(featuresCol='vectorizedFeatures',labelCol='label')"],"metadata":{"id":"a9uuUXgqOP9g","executionInfo":{"status":"ok","timestamp":1648908987819,"user_tz":-60,"elapsed":396,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":112,"outputs":[]},{"cell_type":"code","source":["#Import the Pipeline() method that we’ll use to build the model.\n","from pyspark.ml import Pipeline"],"metadata":{"id":"goLn4mrO2-YN","executionInfo":{"status":"ok","timestamp":1648908990626,"user_tz":-60,"elapsed":340,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":113,"outputs":[]},{"cell_type":"code","source":["#Fitting the five stages by add the initialized 5 stages into the Pipeline() method.\n","pipeline = Pipeline(stages=[tokenizer,stopwords_remover,vectorizer,idf,lr])"],"metadata":{"id":"a183ihss3E1I","executionInfo":{"status":"ok","timestamp":1648908992127,"user_tz":-60,"elapsed":4,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":114,"outputs":[]},{"cell_type":"code","source":["#Building model\n","#The model is built by fitting the model into the main dataset using the fit() method and passing the mainDataFrame as our parameter.\n","#Let’s initialize the model pipeline as lr_model.\n","lr_model = pipeline.fit(mainDataFrame)"],"metadata":{"id":"wZ5qlaBz2341","executionInfo":{"status":"ok","timestamp":1648909157595,"user_tz":-60,"elapsed":162811,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":115,"outputs":[]},{"cell_type":"code","source":["#Testing model\n","#The model is tested using the test dataset to see if it can classify the course title and assign the right subject.\n","predictions = lr_model.transform(testDataFrame)"],"metadata":{"id":"unvv2BEj3K0H","executionInfo":{"status":"ok","timestamp":1648909238858,"user_tz":-60,"elapsed":525,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":116,"outputs":[]},{"cell_type":"code","source":["#To see if the model was able to do the right classification\n","predictions.show()"],"metadata":{"id":"wfVDnZ3A3l6J","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648909280145,"user_tz":-60,"elapsed":4847,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}},"outputId":"47ae6c35-603a-437c-f472-63ff292a29b0"},"execution_count":119,"outputs":[{"output_type":"stream","name":"stdout","text":["+-----------+----------+-----+-------------+---------------+---------------+--------------------+--------------------+--------------------+----------+\n","|      srcip|attack_cat|label|     mytokens|filtered_tokens|    rawFeatures|  vectorizedFeatures|       rawPrediction|         probability|prediction|\n","+-----------+----------+-----+-------------+---------------+---------------+--------------------+--------------------+--------------------+----------+\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","|10.40.170.2|    Normal|  0.0|[10.40.170.2]|  [10.40.170.2]|(43,[24],[1.0])|(43,[24],[7.08854...|[14.6446021663145...|[0.99999887785934...|       0.0|\n","+-----------+----------+-----+-------------+---------------+---------------+--------------------+--------------------+--------------------+----------+\n","only showing top 20 rows\n","\n"]}]},{"cell_type":"code","source":["#To get all the available columns\n","predictions.columns"],"metadata":{"id":"h-llsqyXQhDr","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648909439194,"user_tz":-60,"elapsed":7,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}},"outputId":"4cb1245e-beff-4d71-d91e-064c7f7d6ded"},"execution_count":120,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['srcip',\n"," 'attack_cat',\n"," 'label',\n"," 'mytokens',\n"," 'filtered_tokens',\n"," 'rawFeatures',\n"," 'vectorizedFeatures',\n"," 'rawPrediction',\n"," 'probability',\n"," 'prediction']"]},"metadata":{},"execution_count":120}]},{"cell_type":"code","source":["#From the above columns, select the necessary columns used for predictions and view the first 10 rows.\n","predictions.select('rawPrediction','probability','attack_cat','label','prediction').show(10)"],"metadata":{"id":"XZZ9-Y5jQrrK","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648909447783,"user_tz":-60,"elapsed":5702,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}},"outputId":"eccb5f29-8c7c-4393-968d-ea20c6de144c"},"execution_count":121,"outputs":[{"output_type":"stream","name":"stdout","text":["+--------------------+--------------------+----------+-----+----------+\n","|       rawPrediction|         probability|attack_cat|label|prediction|\n","+--------------------+--------------------+----------+-----+----------+\n","|[14.6446021663145...|[0.99999887785934...|    Normal|  0.0|       0.0|\n","|[14.6446021663145...|[0.99999887785934...|    Normal|  0.0|       0.0|\n","|[14.6446021663145...|[0.99999887785934...|    Normal|  0.0|       0.0|\n","|[14.6446021663145...|[0.99999887785934...|    Normal|  0.0|       0.0|\n","|[14.6446021663145...|[0.99999887785934...|    Normal|  0.0|       0.0|\n","|[14.6446021663145...|[0.99999887785934...|    Normal|  0.0|       0.0|\n","|[14.6446021663145...|[0.99999887785934...|    Normal|  0.0|       0.0|\n","|[14.6446021663145...|[0.99999887785934...|    Normal|  0.0|       0.0|\n","|[14.6446021663145...|[0.99999887785934...|    Normal|  0.0|       0.0|\n","|[14.6446021663145...|[0.99999887785934...|    Normal|  0.0|       0.0|\n","+--------------------+--------------------+----------+-----+----------+\n","only showing top 10 rows\n","\n"]}]},{"cell_type":"code","source":["#Model evaluation\n","#This is checking the model accuracy to know how well we trained our model.\n","#Import the MulticlassClassificationEvaluator which will be used to evaluate the model and calculate the accuracy score.\n","from pyspark.ml.evaluation import MulticlassClassificationEvaluator"],"metadata":{"id":"jrzEAZx1Q9fT","executionInfo":{"status":"ok","timestamp":1648909465189,"user_tz":-60,"elapsed":508,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":122,"outputs":[]},{"cell_type":"code","source":["#The MulticlassClassificationEvaluator uses the label, column and prediction columns to calculate the accuracy. \n","#If the two-column matches, it increases the accuracy score of the model.\n","evaluator = MulticlassClassificationEvaluator(labelCol='label',predictionCol='prediction',metricName='accuracy')"],"metadata":{"id":"-VR6xP72RBka","executionInfo":{"status":"ok","timestamp":1648909468443,"user_tz":-60,"elapsed":442,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":123,"outputs":[]},{"cell_type":"code","source":["accuracy = evaluator.evaluate(predictions)"],"metadata":{"id":"6mHzKI2kREfu","executionInfo":{"status":"ok","timestamp":1648909497969,"user_tz":-60,"elapsed":26888,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":124,"outputs":[]},{"cell_type":"code","source":["#To get the accuracy\n","accuracy"],"metadata":{"id":"NCPJM2jiRI8W","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648909505333,"user_tz":-60,"elapsed":556,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}},"outputId":"dfb8a16f-8a38-4a6b-dcef-d5da0f6d9e5e"},"execution_count":125,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.9451756602933821"]},"metadata":{},"execution_count":125}]},{"cell_type":"code","source":["#To perform a single prediction, prepare the sample input as a string using  the StringType() function.\n","#Import StringType package\n","from pyspark.sql.types import StringType"],"metadata":{"id":"3L0jnz0_RMDq","executionInfo":{"status":"ok","timestamp":1648909533388,"user_tz":-60,"elapsed":411,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":126,"outputs":[]},{"cell_type":"code","source":["#Create a sample data frame made up of the srcip column.\n","example1 = spark.createDataFrame([\n","    (\"Applying a multi-class classifier to classify data into ten classes (categories)\",StringType())\n","],\n","[\"srcip\"]\n","\n",")"],"metadata":{"id":"X-NPeoguRR9K","executionInfo":{"status":"ok","timestamp":1648909537072,"user_tz":-60,"elapsed":321,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}}},"execution_count":127,"outputs":[]},{"cell_type":"code","source":["#Output the data frame without truncating\n","example1.show(truncate=False)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sZy1YCvARwv9","executionInfo":{"status":"ok","timestamp":1648909541457,"user_tz":-60,"elapsed":1506,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}},"outputId":"107b0dea-b730-431d-a37c-6c09eb07fb3a"},"execution_count":128,"outputs":[{"output_type":"stream","name":"stdout","text":["+--------------------------------------------------------------------------------+---+\n","|srcip                                                                           |_2 |\n","+--------------------------------------------------------------------------------+---+\n","|Applying a multi-class classifier to classify data into ten classes (categories)|[] |\n","+--------------------------------------------------------------------------------+---+\n","\n"]}]},{"cell_type":"code","source":["#Make a prediction\n","pred_example1 = lr_model.transform(example1)\n","#Show the output\n","pred_example1.show()"],"metadata":{"id":"YsTvlavWSImU","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648909546903,"user_tz":-60,"elapsed":868,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}},"outputId":"e9d9d834-504d-4b05-839b-a5515c551c18"},"execution_count":129,"outputs":[{"output_type":"stream","name":"stdout","text":["+--------------------+---+--------------------+--------------------+-----------+------------------+--------------------+--------------------+----------+\n","|               srcip| _2|            mytokens|     filtered_tokens|rawFeatures|vectorizedFeatures|       rawPrediction|         probability|prediction|\n","+--------------------+---+--------------------+--------------------+-----------+------------------+--------------------+--------------------+----------+\n","|Applying a multi-...| []|[applying, a, mul...|[applying, multi-...| (43,[],[])|        (43,[],[])|[8.82155526802020...|[0.99890082821522...|       0.0|\n","+--------------------+---+--------------------+--------------------+-----------+------------------+--------------------+--------------------+----------+\n","\n"]}]},{"cell_type":"code","source":["#Show all the available columns\n","pred_example1.columns"],"metadata":{"id":"uFSDnGwBSatZ","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648909555097,"user_tz":-60,"elapsed":499,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}},"outputId":"9aeb7a67-6167-49f9-9531-54103f449dfd"},"execution_count":130,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['srcip',\n"," '_2',\n"," 'mytokens',\n"," 'filtered_tokens',\n"," 'rawFeatures',\n"," 'vectorizedFeatures',\n"," 'rawPrediction',\n"," 'probability',\n"," 'prediction']"]},"metadata":{},"execution_count":130}]},{"cell_type":"code","source":["#From the above columns, select the necessary columns that give the prediction results.\n","pred_example1.select('srcip','rawPrediction','probability','prediction').show()"],"metadata":{"id":"wZtq7BorSi6e","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648909562265,"user_tz":-60,"elapsed":474,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}},"outputId":"29b1b839-81d4-420b-d383-0332e46441c9"},"execution_count":131,"outputs":[{"output_type":"stream","name":"stdout","text":["+--------------------+--------------------+--------------------+----------+\n","|               srcip|       rawPrediction|         probability|prediction|\n","+--------------------+--------------------+--------------------+----------+\n","|Applying a multi-...|[8.82155526802020...|[0.99890082821522...|       0.0|\n","+--------------------+--------------------+--------------------+----------+\n","\n"]}]},{"cell_type":"code","source":["#Show label dictionary\n","label_dict"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"jeCIid6sSrKT","executionInfo":{"status":"ok","timestamp":1648909588885,"user_tz":-60,"elapsed":599,"user":{"displayName":"Abdulwahab Abdulliameed","userId":"10732377996348789858"}},"outputId":"3a4e0f48-a27e-42a3-b296-fd62e5bc6066"},"execution_count":132,"outputs":[{"output_type":"execute_result","data":{"text/plain":["{'Analysis': 6.0,\n"," 'Backdoor': 7.0,\n"," 'DoS': 4.0,\n"," 'Exploits': 2.0,\n"," 'Fuzzers': 3.0,\n"," 'Generic': 1.0,\n"," 'Reconnaissance': 5.0,\n"," 'Shellcode': 8.0,\n"," 'Worms': 9.0,\n"," 'normal': 0.0}"]},"metadata":{},"execution_count":132}]}],"metadata":{"colab":{"collapsed_sections":[],"name":"Task 3.2 b.) -  Mutli-class classifier using PySpark .ipynb","provenance":[{"file_id":"19jb2Q7oSOC3zFSm0chXYs4C5LCNpGWT8","timestamp":1648730152751}],"authorship_tag":"ABX9TyOWQcc7s4xfSiBOFdWh5Vvk"},"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"name":"python"}},"nbformat":4,"nbformat_minor":0}