welcome_distributed.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Welcome to BlazingSQL Notebooks!\n",
    "\n",
    "BlazingSQL Notebooks is a fully managed, high-performance JupyterLab environment. \n",
    "\n",
    "**No setup required.** You just login and start writing code, immediately.\n",
    "\n",
    "Every Notebooks environment has:   \n",
    "- An attached CUDA GPU\n",
    "- Pre-Installed GPU Data Science Packages ([BlazingSQL](https://github.com/BlazingDB/blazingsql), [RAPIDS](https://github.com/rapidsai), [Dask](https://github.com/dask), and many more)\n",
    "\n",
    "Start running GPU-accelerated code below!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The GPU DataFrame\n",
    "The RAPIDS ecosystem is built on the concept of a shared GPU DataFrame, built on [Apache Arrow](http://arrow.apache.org/), between all of the different libraries and packages. This was achieved with the `cudf.DataFrame`.\n",
    "\n",
    "There are two libraries specific to data manipulation:\n",
    "- **BlazingSQL**:  SQL commands on a `cudf.DataFrame`\n",
    "- **cuDF**: pandas-like commands on a `cudf.DataFrame`\n",
    "\n",
    "### BlazingSQL (BSQL) \n",
    "[GitHub](https://github.com/BlazingDB/blazingsql) | [Intro Notebook](intro_notebooks/the_dataframe.ipynb)\n",
    "\n",
    "BlazingSQL is a distributed SQL engine built on top of cuDF. Easily run SQL on files and DataFrames.\n",
    "\n",
    "We start with a BlazingContext, which acts like a session of the SQL engine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dask\n",
    "from dask.distributed import Client\n",
    "dask_scheduler_ip_port = '172.31.12.105:8786'\n",
    "client = Client(dask_scheduler_ip_port)\n",
    "client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from blazingsql import BlazingContext\n",
    "network_interface = 'ens5'\n",
    "bc = BlazingContext(dask_client=client, network_interface=network_interface)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With `.create_table('table_name', 'file_path')` you can create tables from many formats. Here we infer the schema from a CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bc.create_table('taxi', 'data/sample_taxi.csv', header=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can run a SQL query directly on that CSV file with `.sql()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bc.sql('SELECT * FROM taxi').compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Learn more about [creating](https://docs.blazingdb.com/docs/creating-tables) and [querying](https://docs.blazingdb.com/docs/single-gpu) BlazingSQL tables, or the [BlazingContext API](https://docs.blazingdb.com/docs/methods-arguments).\n",
    "\n",
    "BlazingSQL returns each query's results as a cuDF DataFrame, making for easy handoff to GPU or non-GPU solutions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(bc.sql('select * from taxi limit 10'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### cuDF\n",
    "[GitHub](https://github.com/rapidsai/cudf) | [Intro Notebook](intro_notebooks/the_dataframe.ipynb)\n",
    "\n",
    "cuDF is a GPU DataFrame Library similar to pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cudf\n",
    "s = cudf.Series([3, 5, 0.01, None, 4])\n",
    "s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can make a `cudf.DataFrame` from a SQL statement, each column being a `cudf.Series`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = bc.sql('select * from taxi where trip_distance < 10')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Utilize DataFrame methods like `.head()`, `.tail()`, or `.describe()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.tail(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe().compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also filter cuDF DataFrames just like pandas DataFrames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[(df['passenger_count'] != 1) & (df['trip_distance'] < 10)].compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To ensure interoperability, you can also easily convert from cuDF to pandas with `.to_pandas()`. This grants you access to all pandas methods, in this example, `.sample()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.compute().to_pandas().sample(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Learn more about [BlazingSQL + cuDF](intro_notebooks/the_dataframe.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Visualization\n",
    "cuDF DataFrames easily plug into current and GPU-accelerated visualization.\n",
    "\n",
    "\n",
    "### Matplotlib\n",
    "\n",
    "[GitHub](https://github.com/matplotlib/matplotlib) | [Intro Notebook](intro_notebooks/data_visualization.ipynb#Matplotlib)\n",
    "\n",
    "Calling the `.to_pandas()` method, we can convert a `cudf.DataFrame` into a `pandas.DataFrame` and hand off to Matplotlib or other CPU visualization packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bc.sql('SELECT passenger_count, tip_amount FROM taxi').compute().to_pandas().plot(kind='scatter', x='passenger_count', y='tip_amount')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Datashader\n",
    "\n",
    "[GitHub](https://github.com/holoviz/datashader) | [Intro Notebook](intro_notebooks/data_visualization.ipynb#Datashader)\n",
    "\n",
    "Datashader is a data rasterization pipeline for automating the process of creating meaningful representations of large amounts of data.\n",
    "\n",
    "Datashader is one of the first visualization tools to support GPU DataFrames, so we can directly pass in `cudf.DataFrame` query results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datashader import Canvas, transfer_functions\n",
    "from colorcet import fire"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We execute and pass a query as a GPU DataFrame to datashader to render taxi dropoff locations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nyc = Canvas().points(bc.sql('SELECT dropoff_x, dropoff_y FROM taxi').compute(), 'dropoff_x', 'dropoff_y')\n",
    "\n",
    "transfer_functions.set_background(transfer_functions.shade(nyc, cmap=fire), \"black\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine Learning\n",
    "### cuML \n",
    "[GitHub](https://github.com/rapidsai/cuml) | [Intro Notebook](intro_notebooks/machine_learning.ipynb)\n",
    "\n",
    "cuML is a GPU-accelerated machine learning library similar to scikit-learn but made to run on GPU.\n",
    "\n",
    "Let's predict fare amount of the `taxi` table we've been querying with a linear regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from cuml.linear_model import LinearRegression\n",
    "from cuml.preprocessing.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pull feature (X) and target (y) values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = bc.sql('SELECT trip_distance, tolls_amount, pickup_x, pickup_y, dropoff_x, dropoff_y FROM taxi')\n",
    "y = bc.sql('SELECT fare_amount FROM taxi')['fare_amount']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split data into train and test sets (80:20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X.compute(), y.compute(), train_size=0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run a Linear Regression Model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# call Linear Regression model\n",
    "lr = LinearRegression(fit_intercept=True, normalize=True)\n",
    "\n",
    "# train the model\n",
    "lr.fit(X_train, y_train)\n",
    "\n",
    "# make predictions for test X values\n",
    "y_pred = lr.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test the model's predicted values with sklearn's r2_score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cuml.metrics.regression as cuml_reg\n",
    "print(f'R-squared: {cuml_reg.r2_score(y_test,y_pred):.3}')\n",
    "print(f'MAE: {float(cuml_reg.mean_absolute_error(y_test,y_pred)):.3}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## That is the Quick Tour!\n",
    "There are in fact many more packages that are integrating the GPU DataFrame, and therefore providing interoperability with the rest of the stack.\n",
    "\n",
    "Some of those not mentioned here are:\n",
    "- **cuGraph**: a graph analytics library similar to NetworkX \n",
    "- **cuSignal**: a signal analytics library similar to SciPy Signal \n",
    "- **CLX**: a collection of cyber security use cases with the RAPIDS stack \n",
    "\n",
    "[Continue to The DataFrame introductory Notebook](intro_notebooks/the_dataframe.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Rapids Stable",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}