nb3_keras-nn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook 3: Neural networks with Keras\n",
    "\n",
    "On to the most exciting machine learning technique: Neural networks. as you will see they are just as easy to use as the sklearn methods.\n",
    "\n",
    "There are several deep learning libraries for Python. The three most commonly used are:\n",
    "\n",
    "- Keras: High level library based on Tensorflow (or others) that is easy to use and flexible enough for most standard users. It has a great documentation and online support.\n",
    "- Tensorflow: Google's neural network library. Most widely used in ML research. Flexible and powerful but also (unnecessarily?) complicated.\n",
    "- Pytorch: The newcomer developed by Facebook. Flexible like Tensorflow but with a nicer, more Pythonic API.\n",
    "\n",
    "Here we will use Keras which is a great start for most tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "DATADIR = './dataset1/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Copy from previous notebook\n",
    "def create_sub(preds, fn=None):\n",
    "    df =  pd.DataFrame({'Id': range(len(preds)), 'Expected': preds})\n",
    "    if fn is not None: df.to_csv(DATADIR + fn, index=False)\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Load the preprocessed data\n",
    "with open('./tmp/preproc_data.pkl', 'rb') as f:\n",
    "    X_train, y_train, X_valid, y_valid, X_test = pickle.load(f)\n",
    "with open('./tmp/dfs.pkl', 'rb') as f:\n",
    "    df_train, df_test = pickle.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear regression: The Keras way\n",
    "\n",
    "First, let's build our own linear regression algorithm with stochastic gradient descent. This should give us the same solution as the sklearn linear regression we did in the last notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import *\n",
    "from keras.optimizers import SGD, Adam"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(728008, 22)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two ways to build a model in Keras. We will start with the easier, a Sequential model. This means that it is a succession of layers. For linear regression we only have one linear layer that maps the inputs to the outputs. Layers where all inputs are connected to all outputs are called fully-connected or Dense layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ln = Sequential([Dense(1, input_shape=(22,), activation='linear')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 1)                 23        \n",
      "=================================================================\n",
      "Total params: 23\n",
      "Trainable params: 23\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "ln.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the model has 23 parameters, 22 coefficients plus one bias term.\n",
    "\n",
    "Next we need to compile the model, which basically means telling Keras which optimizer to use and which loss function to minimize. In the background it also randomly initializes the weights and biases at this stage.\n",
    "\n",
    "We will use the Adam optimizer, which is a fancy version of SGD: https://arxiv.org/abs/1412.6980"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ln.compile(Adam(1e-1), 'mse')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can train/fit the model. For this we specify the training data, the batch size, the number of epochs. Optionally, we can pass on the validation data, so that we get a validation score every epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 728008 samples, validate on 180849 samples\n",
      "Epoch 1/12\n",
      "728008/728008 [==============================] - 4s 6us/step - loss: 26.7081 - val_loss: 6.7833\n",
      "Epoch 2/12\n",
      "728008/728008 [==============================] - 2s 2us/step - loss: 3.8377 - val_loss: 3.5299\n",
      "Epoch 3/12\n",
      "728008/728008 [==============================] - 2s 2us/step - loss: 3.1384 - val_loss: 3.4176\n",
      "Epoch 4/12\n",
      "728008/728008 [==============================] - 2s 2us/step - loss: 3.0784 - val_loss: 3.3439\n",
      "Epoch 5/12\n",
      "728008/728008 [==============================] - 2s 2us/step - loss: 3.0438 - val_loss: 3.2908\n",
      "Epoch 6/12\n",
      "728008/728008 [==============================] - 2s 2us/step - loss: 3.0253 - val_loss: 3.2790\n",
      "Epoch 7/12\n",
      "728008/728008 [==============================] - 1s 2us/step - loss: 3.0143 - val_loss: 3.2588\n",
      "Epoch 8/12\n",
      "728008/728008 [==============================] - 2s 2us/step - loss: 3.0098 - val_loss: 3.2515\n",
      "Epoch 9/12\n",
      "728008/728008 [==============================] - 1s 2us/step - loss: 3.0063 - val_loss: 3.2536\n",
      "Epoch 10/12\n",
      "728008/728008 [==============================] - 2s 2us/step - loss: 3.0062 - val_loss: 3.2421\n",
      "Epoch 11/12\n",
      "728008/728008 [==============================] - 2s 2us/step - loss: 3.0059 - val_loss: 3.2538\n",
      "Epoch 12/12\n",
      "728008/728008 [==============================] - 2s 2us/step - loss: 3.0067 - val_loss: 3.2487\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x21517677080>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ln.fit(X_train, y_train, 10_000, epochs=12, validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import r2_score\n",
    "def print_scores(m, X_train=X_train, X_valid=X_valid):\n",
    "    preds = m.predict(X_valid, 10000)\n",
    "    print('Train R2 = ', r2_score(y_train, m.predict(X_train, 10000)), \n",
    "          ', Valid R2 = ', r2_score(y_valid, preds), ', Valid MSE = ', \n",
    "          m.evaluate(X_valid, y_valid, 10000, 0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train R2 =  0.931859979389 , Valid R2 =  0.918576221325 , Valid MSE =  3.24872250179\n"
     ]
    }
   ],
   "source": [
    "print_scores(ln)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that with the sklearn linear regression algorithm we got a score of around 3.24. This indicates that we are doing pretty much the same here. But we can do better!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural network with a hidden layer\n",
    "\n",
    "Now let's actually build a neural network with a hidden layer. For this we simply add another Dense layer but this time with a non-linear activation function. In our case this is a Rectified Linear Unit or relu. There is no set rule for how many hidden layers or nodes to use. For this we just need to employ trial-and-error.\n",
    "\n",
    "BTW: You will often see people using powers of two for the batch size or the number of nodes. This is for optimization purposes on the GPU, which are not crucial in our case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nn = Sequential([\n",
    "    Dense(256, input_shape=(22,), activation='relu'),\n",
    "    Dense(1, activation='linear')\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_2 (Dense)              (None, 256)               5888      \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 1)                 257       \n",
      "=================================================================\n",
      "Total params: 6,145\n",
      "Trainable params: 6,145\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "nn.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nn.compile(Adam(1e-3), 'mse')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 728008 samples, validate on 180849 samples\n",
      "Epoch 1/12\n",
      "728008/728008 [==============================] - 9s 13us/step - loss: 8.5953 - val_loss: 3.4008\n",
      "Epoch 2/12\n",
      "728008/728008 [==============================] - 7s 9us/step - loss: 2.8585 - val_loss: 3.0352\n",
      "Epoch 3/12\n",
      "728008/728008 [==============================] - 7s 10us/step - loss: 2.6590 - val_loss: 2.9863\n",
      "Epoch 4/12\n",
      "728008/728008 [==============================] - 7s 9us/step - loss: 2.5823 - val_loss: 2.9497\n",
      "Epoch 5/12\n",
      "728008/728008 [==============================] - 7s 9us/step - loss: 2.5296 - val_loss: 2.9034\n",
      "Epoch 6/12\n",
      "728008/728008 [==============================] - 6s 9us/step - loss: 2.4887 - val_loss: 2.9127\n",
      "Epoch 7/12\n",
      "728008/728008 [==============================] - 6s 9us/step - loss: 2.4530 - val_loss: 2.9263\n",
      "Epoch 8/12\n",
      "728008/728008 [==============================] - 6s 9us/step - loss: 2.4237 - val_loss: 2.9107\n",
      "Epoch 9/12\n",
      "728008/728008 [==============================] - 6s 9us/step - loss: 2.3969 - val_loss: 2.9081\n",
      "Epoch 10/12\n",
      "728008/728008 [==============================] - 6s 9us/step - loss: 2.3768 - val_loss: 2.9076\n",
      "Epoch 11/12\n",
      "728008/728008 [==============================] - 6s 9us/step - loss: 2.3544 - val_loss: 2.9907\n",
      "Epoch 12/12\n",
      "728008/728008 [==============================] - 6s 9us/step - loss: 2.3376 - val_loss: 2.9084\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x21517799b70>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nn.fit(X_train, y_train, 1024, epochs=12, validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train R2 =  0.947471880236 , Valid R2 =  0.927105055996 , Valid MSE =  2.90843115181\n"
     ]
    }
   ],
   "source": [
    "print_scores(nn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Some notes on the results*\n",
    "\n",
    "- We see that we are quite heavily overfitting. This makes sense in a model with 6k parameters. There are techniques to prevent overfitting in neural networks, most notably dropout and weight decay (L2 regularization). We will cover those later. For some reason, I found those techniques not to work for this model and dataset. Can you figure out why?\n",
    "- The score is quite significantly better than our best single random forest model. This suggests that the nonlinear computing power of the neural network is useful.\n",
    "- This is not yet a deep neural network. DNNs have several hidden layers. Again, I found that for this dataset a DNN didn't perform better. This might be because the nonlinearities are not very strong, but feel free to prove me wrong."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ADVANCED TECHNIQUE ALERT: Embeddings\n",
    "\n",
    "We saw in the previous notebook that the station information might be really important. We could train a separate neural network for each station but that would reduce the amount of training data for each individual model and probably lead to stronger overfitting. \n",
    "\n",
    "Here is a different method of using categorical variables in neural networks. Don't worry if you don't understand embeddings right away. The concept is easy but it takes a while to wrap your head around it (it did for me anyways). \n",
    "\n",
    "First we need to get continuous station IDs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "split_date = '2015-01-01'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stations = df_train.station\n",
    "stations_train = df_train.station[df_train.time < split_date]\n",
    "stations_valid = df_train.station[df_train.time >= split_date]\n",
    "stations_test = df_test.station"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "unique_stations = pd.concat([df_train.station, df_test.station]).unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stat2id = {s: i for i, s in enumerate(unique_stations)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ids = stations.apply(lambda x: stat2id[x])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ids_train = ids[df_train.time < split_date]\n",
    "ids_valid = ids[df_train.time >= split_date]\n",
    "ids_test = stations_test.apply(lambda x: stat2id[x])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will have to use the Keras's Functional API. Check out the official documentation.\n",
    "\n",
    "We now have two separate inputs. Our regular features and the station ID.\n",
    "\n",
    "An embedding is a mapping from an integer to a vector of real numbers. In our case the vector has length two. The elements are also called latent features. These are then concatenates with the regular features and passed through one hidden layer as before.\n",
    "\n",
    "The latent features are updated along with the weights and biases during training and can now represent station-specific information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.models import Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "features_in = Input(shape=(22,))\n",
    "id_in = Input(shape=(1,))\n",
    "emb = Embedding(len(unique_stations), 2)(id_in)\n",
    "emb = Flatten()(emb)\n",
    "x = Concatenate()([features_in, emb])\n",
    "x = Dense(100, activation='relu')(x)\n",
    "out = Dense(1, activation='linear')(x)\n",
    "model = Model(inputs=[features_in, id_in], outputs=out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_2 (InputLayer)            (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 1, 2)         1044        input_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "input_1 (InputLayer)            (None, 22)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "flatten_1 (Flatten)             (None, 2)            0           embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 24)           0           input_1[0][0]                    \n",
      "                                                                 flatten_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_4 (Dense)                 (None, 100)          2500        concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_5 (Dense)                 (None, 1)            101         dense_4[0][0]                    \n",
      "==================================================================================================\n",
      "Total params: 3,645\n",
      "Trainable params: 3,645\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.compile('adam', 'mse')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 728008 samples, validate on 180849 samples\n",
      "Epoch 1/10\n",
      "728008/728008 [==============================] - 9s 12us/step - loss: 10.6545 - val_loss: 3.2888\n",
      "Epoch 2/10\n",
      "728008/728008 [==============================] - 5s 7us/step - loss: 2.7366 - val_loss: 2.8709\n",
      "Epoch 3/10\n",
      "728008/728008 [==============================] - 5s 7us/step - loss: 2.4955 - val_loss: 2.7256\n",
      "Epoch 4/10\n",
      "728008/728008 [==============================] - 4s 5us/step - loss: 2.3679 - val_loss: 2.6462\n",
      "Epoch 5/10\n",
      "728008/728008 [==============================] - 4s 5us/step - loss: 2.2623 - val_loss: 2.5664\n",
      "Epoch 6/10\n",
      "728008/728008 [==============================] - 5s 6us/step - loss: 2.1857 - val_loss: 2.5130\n",
      "Epoch 7/10\n",
      "728008/728008 [==============================] - 5s 7us/step - loss: 2.1422 - val_loss: 2.5165\n",
      "Epoch 8/10\n",
      "728008/728008 [==============================] - 4s 6us/step - loss: 2.1102 - val_loss: 2.5112\n",
      "Epoch 9/10\n",
      "728008/728008 [==============================] - 4s 5us/step - loss: 2.0844 - val_loss: 2.5081\n",
      "Epoch 10/10\n",
      "728008/728008 [==============================] - 4s 5us/step - loss: 2.0605 - val_loss: 2.4832\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x2151b033240>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit([X_train, ids_train], y_train, 1024, 10, \n",
    "          validation_data=([X_valid, ids_valid], y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 728008 samples, validate on 180849 samples\n",
      "Epoch 1/10\n",
      "728008/728008 [==============================] - 3s 4us/step - loss: 2.0414 - val_loss: 2.4992\n",
      "Epoch 2/10\n",
      "728008/728008 [==============================] - 3s 4us/step - loss: 2.0259 - val_loss: 2.4634\n",
      "Epoch 3/10\n",
      "728008/728008 [==============================] - 3s 4us/step - loss: 2.0115 - val_loss: 2.4528\n",
      "Epoch 4/10\n",
      "728008/728008 [==============================] - 3s 4us/step - loss: 2.0003 - val_loss: 2.4630\n",
      "Epoch 5/10\n",
      "728008/728008 [==============================] - 3s 4us/step - loss: 1.9911 - val_loss: 2.4446\n",
      "Epoch 6/10\n",
      "728008/728008 [==============================] - 3s 4us/step - loss: 1.9813 - val_loss: 2.4740\n",
      "Epoch 7/10\n",
      "728008/728008 [==============================] - 3s 4us/step - loss: 1.9734 - val_loss: 2.5073\n",
      "Epoch 8/10\n",
      "728008/728008 [==============================] - 3s 4us/step - loss: 1.9675 - val_loss: 2.4546\n",
      "Epoch 9/10\n",
      "728008/728008 [==============================] - 3s 4us/step - loss: 1.9604 - val_loss: 2.4480\n",
      "Epoch 10/10\n",
      "728008/728008 [==============================] - 3s 4us/step - loss: 1.9532 - val_loss: 2.4633\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x21518331198>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit([X_train, ids_train], y_train, 1024, 10, \n",
    "          validation_data=([X_valid, ids_valid], y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train R2 =  0.955964364947 , Valid R2 =  0.938262241398 , Valid MSE =  2.46327159392\n"
     ]
    }
   ],
   "source": [
    "print_scores(model, X_train=[X_train, ids_train], X_valid=[X_valid, ids_valid])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Expected</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>4.093746</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1.777899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>0.641055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>3.240536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>1.727655</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  Expected\n",
       "0   0  4.093746\n",
       "1   1  1.777899\n",
       "2   2  0.641055\n",
       "3   3  3.240536\n",
       "4   4  1.727655"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Submit to Kaggle\n",
    "df_sub = create_sub(model.predict([X_test, ids_test], 10000).squeeze(), 'nn_emb.csv'); df_sub.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This technique allows us to build a single model that incorporates station information and gives us the best score. Yay!\n",
    "\n",
    "But can you do better?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Your turn\n",
    "\n",
    "1. As we have seen now there are a lot of hyperparameters. Try playing around with them and get the best score. How do the parameters influence the skill?\n",
    "2. Try an ensemble of techniques. This means training a few models (can be several NNs or some NNs with some RFs) and averaging the predictions. This is also a way to prevent overfitting and might just increase your score ;)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": false,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}