Workflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HpBandSter + CAVE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will present you a example procedure of how to efficiently optimize a algorithm using our frameworks\n",
    "<a href=\"https://github.com/automl/CAVE\" target=\"_blank\">CAVE</a> and <a href=\"https://github.com/automl/HpBandSter\" target=\"_blank\">HpBandSter</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Short introduction\n",
    "\n",
    "### HpBandSter\n",
    "\n",
    "HpBandSter (HyperBand on STERoids) implements recently published methods for optimizing hyperparameters of machine learning algorithms. We designed HpBandSter such that it scales from running sequentially on a local machine to running on a distributed system in parallel\n",
    "\n",
    "One of the implemented algorithms is BOHB, which combines Bayesian Optimization and HyperBand to efficiently search for well performing configurations. Learn more about this method by reading out paper, published at <a href=\"http://proceedings.mlr.press/v80/falkner18a.html\" target=\"_blank\">ICML 2018</a>\n",
    "\n",
    "### CAVE\n",
    "\n",
    "CAVE stands for Configuration Assessment, Visualization and Evaluation. It is designed to create comprehensive reports about an optimization process. The resulting figures and interactive plots can be used to gain insights in the parameter importance, feature importance, search behaviour and quality. For a detailed report on the various analysis methods that CAVE offers, please see the documentation.\n",
    "For HpBandSter/BOHB CAVE offers a report that can be used to compare the optimization process over the individual budgets. Each analysis method is performed for each individual budget.\n",
    "\n",
    "**NOTE:** This notebook is just a simple example of how to combine those tools. For a more detailed insight into the frameworks, please take a look at the corresponding documentation.\n",
    "- <a href=\"https://automl.github.io/CAVE/stable/\" target=\"_blank\">CAVE-Documentation</a>\n",
    "- <a href=\"https://automl.github.io/HpBandSter/build/html/index.html\" target=\"_blank\">HpBandSter-Documentation</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The workflow\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next section, we guide you through the following tutorial:\n",
    "#### 1) Run BOHB on a given algorithm to optimize and a *configuration space*. \n",
    "This step contains: \n",
    "1. Setting up a worker, which is responsible for evaluating a given model with a single configuration on a single budget at a time. Here, the model is a simple scipy implementation of a SVM classifing the MNIST-dataset.\n",
    "2. Defining a configuration space for the classifier using the <a href=\"https://github.com/automl/ConfigSpace\" target=\"_blank\">ConfigSpace module</a>.\n",
    "3. Setting up a nameserver, which organizes multiple workers and starting the optimizer, here BOHB.\n",
    "\n",
    "#### 2) We will pass the BOHB results into the CAVE-tool.\n",
    "It will give insights into the \n",
    "1. Parameter importance, \n",
    "2. performance analysis,\n",
    "3. feature analysis and \n",
    "4. configuration behaviour."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1) Setting up a worker class\n",
    "The Worker is responsible to evaluate a hyperparameter setting and returning the associated loss that is minimized. By deriving from the base class, encoding a new problem consists of implementing two methods: **__init__** and **compute**. The first allows to perform inital computations, e.g. loading the dataset, when the worker is started, while the latter is called repeatedly called during the optimization and evaluates a given configuration yielding the associated loss.\n",
    "\n",
    "In the implementation below the following steps are perfomed: \n",
    "+ inherit from the hpbandster.core.worker class\n",
    "+ In the *__init__*-method, the mnist data is loaded\n",
    "+ overwrite the *compute* methode: the training of the model happens here\n",
    "+ **NOTE**: make sure that the returned dictionary contains the fields *loss* and *info*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from sklearn import datasets, neural_network, metrics\n",
    "from hpbandster.core.worker import Worker\n",
    "\n",
    "class MyWorker(Worker):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super(MyWorker, self).__init__(*args, **kwargs)\n",
    "        # load the MNIST dataset\n",
    "        digits = datasets.load_digits()\n",
    "        n_samples = len(digits.images)\n",
    "        data = digits.images.reshape((n_samples, -1))\n",
    "        \n",
    "        # split it into training and test set.\n",
    "        self.train_x = data[:n_samples // 2]\n",
    "        self.train_y = digits.target[:n_samples // 2]\n",
    "        self.test_x = data[n_samples // 2:]\n",
    "        self.test_y = digits.target[n_samples // 2:]\n",
    "\n",
    "    def compute(self, config, budget, *args, **kwargs):\n",
    "        \"\"\"\n",
    "        Simple example for a compute function. It'll be repeatedly called by the optimizer. \n",
    "        \n",
    "        Args:\n",
    "            config: dictionary containing the sampled configurations by the optimizer\n",
    "            budget: (float) amount of time/epochs/etc. the model can use to train\n",
    "\n",
    "        Returns:\n",
    "            dictionary with mandatory fields:\n",
    "                'loss' (scalar)\n",
    "                'info' (dict)\n",
    "        \"\"\"\n",
    "        beta_1 = 0  if 'beta_1' not in config else config['beta_1']\n",
    "        beta_2 = 0  if 'beta_2' not in config else config['beta_2']\n",
    "        \n",
    "        clf = neural_network.MLPClassifier(max_iter=int(budget),\n",
    "                                           learning_rate='constant',\n",
    "                                           learning_rate_init=config['learning_rate_init'],\n",
    "                                           activation=config['activation'],\n",
    "                                           solver=config['solver'],\n",
    "                                           beta_1=beta_1, \n",
    "                                           beta_2=beta_2\n",
    "                                          )\n",
    "        clf.fit(self.train_x, self.train_y)\n",
    "        \n",
    "        predicted = clf.predict(self.test_x)\n",
    "        loss_train = metrics.log_loss(self.train_y, clf.predict_proba(self.train_x))\n",
    "        loss_test = metrics.log_loss(self.test_y, clf.predict_proba(self.test_x))\n",
    "\n",
    "        accuracy_train = clf.score(self.train_x, self.train_y)\n",
    "        accuracy_test = clf.score(self.test_x, self.test_y)\n",
    "        \n",
    "        return ({\n",
    "            'loss': loss_train,  # this is the a mandatory field to run hyperband\n",
    "            'info': {'loss_train': loss_train,\n",
    "                     'loss_test': loss_test,\n",
    "                     'accuracy_train': accuracy_train,\n",
    "                     'accuracy_test': accuracy_test,\n",
    "                    }  # can be used for any user-defined information - also mandatory\n",
    "        })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2) ConfigSpace:\n",
    "Every problem needs a description of the search space to be complete. In HpBandSter, a ConfigurationSpace-object defining all hyperparameters, their ranges, and potential dependencies between them.\n",
    "\n",
    "In our example here, the search space consists of the hyperparameters:\n",
    "\n",
    "|         Name        |     Type    |      Values      |     Condition    |\n",
    "|:-------------------:|:-----------:|:----------------:|:----------------:|\n",
    "| activation-function | categorical | {'relu', 'tanh'} |                  |\n",
    "|    learning-rate    |    float    |   [1e-6 - 1e-2]  |                  |\n",
    "|        solver       | categorical |  {'sgd', 'adam'} |                  |\n",
    "|        beta_1       |    float    |      [0, 1]      | solver == 'adam' |\n",
    "|        beta_2       |    float    |      [0, 1]      | solver == 'adam' |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import ConfigSpace as CS\n",
    "import ConfigSpace.hyperparameters as CSH\n",
    "\n",
    "def sample_configspace():\n",
    "    config_space = CS.ConfigurationSpace()\n",
    "    config_space.add_hyperparameter(CSH.CategoricalHyperparameter('activation', ['tanh', 'relu']))\n",
    "    config_space.add_hyperparameter(CS.UniformFloatHyperparameter('learning_rate_init', lower=1e-6, upper=1e-2, log=True))\n",
    "    \n",
    "    solver = CSH.CategoricalHyperparameter('solver', ['sgd', 'adam'])\n",
    "    config_space.add_hyperparameter(solver)\n",
    "    \n",
    "    beta_1 = CS.UniformFloatHyperparameter('beta_1', lower=0, upper=1)\n",
    "    config_space.add_hyperparameter(beta_1)\n",
    "    \n",
    "    condition = CS.EqualsCondition(beta_1, solver, 'adam')\n",
    "    config_space.add_condition(condition)\n",
    "    \n",
    "    beta_2 = CS.UniformFloatHyperparameter('beta_2', lower=0, upper=1)\n",
    "    config_space.add_hyperparameter(beta_2)\n",
    "    \n",
    "    condition = CS.EqualsCondition(beta_2, solver, 'adam')\n",
    "    config_space.add_condition(condition)\n",
    "    \n",
    "    return config_space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3) Setting up the HpBandSter Nameserver and starting the optimization run\n",
    "\n",
    "**NOTE:** Unfortunately, the configuration space is *not saved automatically* to file but this step is mandatory for the analysis with CAVE.  \n",
    "We recommend to save the configuration space every time you use BOHB.\n",
    "We do this by using the ConfigSpace-to-json-writer.\n",
    "\n",
    "#### Step 1:\n",
    "To initiate the communication between the worker(s) and the optimizer, HpBandSter requires a nameserver to be present. This is a small service that keeps track of all running processes and their IP addresses and ports.\n",
    "\n",
    "It could be a 'static' server with a permanent address, but here it will be started for the local machine with a random port. \n",
    "\n",
    "The run_id is used to identify individual runs and needs to be given to all other components as well (see below).\n",
    "\n",
    "#### Step 2: \n",
    "The worker implements the actual problem that is optimized. Its 'compute'-method will be called later by the BOHB-optimizer repeatedly with the sampled configurations and return the computed loss (and additional information).\n",
    "\n",
    "#### Step 3:  \n",
    "In the last of the 3 Steps, we create an optimizer object. The optimizer decides which configurations are evaluated, and how the budgets are distributed. It samples configurations from the ConfigurationSpace, using succesive halving. \n",
    "To take advantage of lower fidelity approximation, i.e. budgets lower than *max_budget*, those lower accuracy evaluations have to be meaningful. As these budgets can mean very different things (epochs of training a neural network, number of data points to train the model, or number of cross-validation folds to name a few), these have to be user specified. This is done by two parameters, called *min_budget* and *max_budget* for all optimizers. For better speed ups, the lower budget should be as small as possible while still being informative. By informative, we mean that the performance is a ok indicator for the loss on higher budgets. \n",
    "\n",
    "**NOTE:** BOHB does not build a new model at the beginning of every SuccessiveHalving run. Instead it collects all evaluations on all budgets and uses the largest budget with enough evaluations as a base for future evaluations.  \n",
    "\n",
    "#### Step 4:\n",
    "After the run is finished, the services started above need to be shutdown. This ensures that the worker, the nameserver and the master all properly exit and no (daemon) threads keep running afterwards. In particular we shutdown the optimizer (which shuts down all workers) and the nameserver.\n",
    "After a run is finished, one might be interested in all kinds of information. HpBandSter offers full access to all evaluated configurations including timing information and potential error messages for failed runs.\n",
    "In this example, we simply look up the best configuration (called incumbent), count the number of configurations and evaluations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:hpbandster:DISPATCHER: started the 'discover_worker' thread\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start listening for jobs\n",
      "INFO:hpbandster:DISPATCHER: started the 'job_runner' thread\n",
      "INFO:hpbandster:DISPATCHER: Pyro daemon running on localhost:59676\n",
      "INFO:hpbandster:DISPATCHER: discovered new worker, hpbandster.run_0.worker.DESKTOP-KV1H67P.1608415988\n",
      "INFO:hpbandster:HBMASTER: adjusted queue size to (0, 1)\n",
      "INFO:hpbandster:DISPATCHER: A new worker triggered discover_worker\n",
      "INFO:hpbandster:HBMASTER: starting run at 1535468705.7045996\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 0)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (6) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 0) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 1)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (6) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 1) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 2)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (6) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 2) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 3)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (6) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 3) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 4)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (6) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 4) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 5)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (6) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 5) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 6)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (6) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 6) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 7)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (6) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 7) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 8)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (6) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 8) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 9)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (6) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 9) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 10)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (6) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 10) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 11)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (6) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 11) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 12)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (6) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 12) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 13)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (6) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 13) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 14)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (6) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 14) with dispatcher\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\statsmodels\\nonparametric\\kernels.py:64: RuntimeWarning: divide by zero encountered in true_divide\n",
      "  kernel_value = np.ones(Xi.size) * h / (num_levels - 1)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 15)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (6) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 15) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 3)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (12) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 3) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 5)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (12) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 5) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 7)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (12) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 7) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 9)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (12) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 9) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 11)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (12) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 11) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 13)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (12) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 13) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 14)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (12) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 14) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 15)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (12) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 15) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 3)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (25) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 3) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 7)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (25) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 7) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 13)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (25) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 13) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 15)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (25) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 15) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 3)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 3) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 15)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 15) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (0, 0, 3)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (0, 0, 3) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (1, 0, 0)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (12) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (1, 0, 0) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (1, 0, 1)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (12) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (1, 0, 1) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (1, 0, 2)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (12) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (1, 0, 2) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (1, 0, 3)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (12) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (1, 0, 3) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (1, 0, 4)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (12) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (1, 0, 4) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (1, 0, 5)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (12) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (1, 0, 5) with dispatcher\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:hpbandster:sampled vector: [1, 0.6645311879776868, 1, 0.33303804599742226, 0.5509833576746997] has EI value inf\n",
      "WARNING:hpbandster:data in the KDEs:\n",
      "[[0.         0.99350633 1.         0.86538002 0.61996657]\n",
      " [0.         0.87372611 1.         0.44715457 0.47975806]\n",
      " [0.         0.88044106 1.         0.47676305 0.71592325]\n",
      " [1.         0.76355038 1.         0.11566473 0.03196209]\n",
      " [0.         0.82833534 1.         0.53123841 0.67118551]\n",
      " [0.         0.82032556 1.         0.58478843 0.64664503]]\n",
      "[[0.         0.81738103 1.         0.71362197 0.67817373]\n",
      " [0.         0.81596286 1.         0.61733232 0.42751703]\n",
      " [0.         0.82578798 1.         0.42932803 0.58552529]\n",
      " [0.         0.80866376 1.         0.67103611 0.87021057]\n",
      " [0.         0.77144743 1.         0.44132936 0.95769447]\n",
      " [0.         0.83808672 0.         0.65448923 0.35634805]\n",
      " [0.         0.30555762 1.         0.88732838 0.85928481]\n",
      " [0.         0.22773261 1.         0.65448923 0.35634805]]\n",
      "WARNING:hpbandster:bandwidth of the KDEs:\n",
      "[0.32372659 0.06173875 0.001      0.19183861 0.20270503]\n",
      "[0.001      0.20023808 0.27824128 0.11597421 0.19101884]\n",
      "WARNING:hpbandster:l(x) = 0.06473420078312318\n",
      "WARNING:hpbandster:g(x) = inf\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (1, 0, 6)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (12) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (1, 0, 6) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (1, 0, 7)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (12) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (1, 0, 7) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (1, 0, 0)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (25) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (1, 0, 0) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (1, 0, 1)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (25) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (1, 0, 1) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (1, 0, 3)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (1, 0, 3) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (1, 0, 7)\n",
      "C:\\Users\\Philipp\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (25) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (1, 0, 7) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (1, 0, 0)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (1, 0, 0) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (1, 0, 3)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (1, 0, 3) with dispatcher\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: start processing job (1, 0, 0)\n",
      "INFO:hpbandster.run_0.worker.DESKTOP-KV1H67P.16084:WORKER: registered result for job (1, 0, 0) with dispatcher\n",
      "INFO:hpbandster:DISPATCHER: Dispatcher shutting down\n",
      "INFO:hpbandster:DISPATCHER: shut down complete\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import logging\n",
    "logging.basicConfig(level=logging.INFO)\n",
    "\n",
    "from ConfigSpace.read_and_write import pcs_new\n",
    "import hpbandster.core.nameserver as hpns\n",
    "import hpbandster.core.result as hpres\n",
    "\n",
    "\n",
    "from hpbandster.optimizers import BOHB as BOHB\n",
    "\n",
    "# Create a configuration space\n",
    "config_space = sample_configspace()\n",
    "\n",
    "# Write the ConfigSpace for later use to file\n",
    "out_dir = 'workflow_result'\n",
    "os.makedirs(out_dir, exist_ok=True)\n",
    "with open(os.path.join(out_dir, 'configspace.pcs'), 'w') as fh:\n",
    "    fh.write(pcs_new.write(config_space))\n",
    "\n",
    "# Log the optimization results for later analysis\n",
    "result_logger = hpres.json_result_logger(directory=out_dir, overwrite=True)\n",
    "    \n",
    "# Every run has to have a unique (at runtime) id.\n",
    "run_id = '0'\n",
    "\n",
    "# Step 1: Set up the nameserver\n",
    "NS = hpns.NameServer(run_id=run_id, host='localhost', port=0)\n",
    "ns_host, ns_port = NS.start()\n",
    "\n",
    "# Step 2: Connect the worker with the nameserver\n",
    "w = MyWorker(nameserver=ns_host,\n",
    "             nameserver_port=ns_port,\n",
    "             run_id=run_id,  # unique Hyperband run id (same as nameserver's)\n",
    "            )\n",
    "w.run(background=True)\n",
    "\n",
    "# Step 3: Create the optimizer object and run it\n",
    "bohb = BOHB(  configspace=config_space,\n",
    "              run_id=run_id,  # same as nameserver's\n",
    "              eta=2, min_budget=5, max_budget=100,  # Hyperband parameters\n",
    "              nameserver=ns_host,\n",
    "              nameserver_port=ns_port,\n",
    "              result_logger=result_logger,\n",
    "              ping_interval=3600,  # how often master pings for workers (in seconds)\n",
    "           )\n",
    "\n",
    "# Then start the optimizer. The n_iterations parameter specifies\n",
    "# the number of iterations to be performed in this run\n",
    "res = bohb.run(n_iterations=2)\n",
    "\n",
    "# After the optimizer run, we shutdown the master.\n",
    "bohb.shutdown(shutdown_workers=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A total of 24 unique configurations where sampled.\n",
      "A total of 46 runs where executed.\n",
      "Best configuration found: {'activation': 'tanh', 'learning_rate_init': 0.003324790023143092, 'solver': 'adam', 'beta_1': 0.47676304566040895, 'beta_2': 0.7159232476419948}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEKCAYAAAARnO4WAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGTZJREFUeJzt3X20XXV95/H35zzc3CTkJlKuiCQY\nsFNaYAjg9QFRq9B2rFKk1Non7JJZqyy7OgVtq9W2o3XN2GU7M9RiO7YMPtCKOi3IouM4VqVK0FJq\nAnkgBKsFbAJSgkqeSHLvufc7f+x9kpub87DvSfbd9+z9ea2VlfOwz9nfyyKf87vf89u/nyICMzMr\nv1rRBZiZ2cJw4JuZVYQD38ysIhz4ZmYV4cA3M6sIB76ZWUU48M3MKsKBb2ZWEQ58M7OKaBRdwGyn\nnHJKrF27tugyzMyGxsaNG5+OiPEsxy6qwF+7di0bNmwougwzs6Eh6dtZj3VLx8ysIhz4ZmYV4cA3\nM6sIB76ZWUU48M3MKsKBb2ZWEQ58M7OKKEXg33jXN7n7n3cVXYaZ2aJWisD/87v/hXsc+GZmPZUi\n8Jv1Gq0Zb8ZuZtZLaQJ/cnqm6DLMzBa1UgT+SF1MtRz4Zma9lCLwG/UaUx7hm5n1lGvgS3q7pG2S\nHpT0KUmjeZynWRdT0+7hm5n1klvgSzoduA6YiIjzgDrw83mcq+kRvplZX3m3dBrAUkkNYBnwRB4n\nGWk48M3M+skt8CPiceC/A/8KfAfYHRFfmHucpGslbZC0YdeuwebSJyN8t3TMzHrJs6XzHOANwJnA\n84Hlkq6ee1xE3BQRExExMT6eaZeuYzRq8rRMM7M+8mzp/BjwaETsiogp4DPAy/M40UijRsuBb2bW\nU56B/6/AyyQtkyTgMmB7HidyS8fMrL88e/j3AbcB9wNb03PdlMe5kmmZHuGbmfXSyPPNI+K9wHvz\nPAd4aQUzsyxKcaWt5+GbmfVXksAXLffwzcx6Kknge4RvZtZPaQJ/0qtlmpn1VIrAT5ZWcEvHzKyX\nUgR+o+ZpmWZm/ZQi8NtbHEZ4lG9m1k0pAn+kkfwYbuuYmXVXisBv1gXgto6ZWQ8lCfz2CN+Bb2bW\nTSkCv5EGvpdXMDPrrhSBP5K2dHy1rZlZd6UIfLd0zMz6c+CbmVVEqQJ/suWWjplZNyUJfE/LNDPr\npySBn/wYrRkHvplZN6UKfLd0zMy6K0XgjzTc0jEz66cUge9ZOmZm/ZUi8Bs1B76ZWT+lCPwjLR33\n8M3MuilF4LulY2bWnwPfzKwiShX4k27pmJl1VZLAT3v4LY/wzcy6KUng+0pbM7N+ShX4nqVjZtZd\nSQI/aelMuqVjZtZVKQJfEs26PEvHzKyHUgQ+JFfbOvDNzLorTeAnI3z38M3MuilN4I80PMI3M+ul\nNIHfrDvwzcx6KVngu6VjZtZNaQK/UReTHuGbmXVVmsAfqddoOfDNzLoqTeC7pWNm1luugS9plaTb\nJD0sabuki/M6ly+8MjPrrZHz+/8J8PmIeKOkEWBZXidq1mteWsHMrIfcAl/SGPAq4C0AETEJTOZ1\nvma9xrOTrbze3sxs6OXZ0jkL2AV8TNIDkm6WtDyvkzXrojXjHr6ZWTd5Bn4DuAj4cERcCOwH3jX3\nIEnXStogacOuXbsGPplbOmZmveUZ+DuBnRFxX3r/NpIPgKNExE0RMRERE+Pj4wOfrOmlFczMesot\n8CPiSWCHpLPThy4DHsrrfCOelmlm1lPes3R+Hbg1naHzCHBNXidq1Dwt08ysl1wDPyI2ARN5nqMt\nael4hG9m1k1prrQd8WqZZmY99Q18Scsl1dLbPyTpCknN/EubH19pa2bWW5YR/npgVNLpwF0kffiP\n51nUILwevplZb1kCXxHxLHAV8KGI+GngnHzLmr9GOksnwn18M7NOMgV+uujZLwH/N30s79k98zZS\nF4CvtjUz6yJL4L8NeDdwR0Rsk3QW8OV8y5q/Zj35UdzWMTPrrO9IPSLuBu4GSL+8fToirsu7sPk6\nHPitgJGCizEzW4SyzNL5pKSxdOGzh4BvSHpH/qXNT7OR/Cje5tDMrLMsLZ1zImIPcCXwOeAM4M25\nVjWAZq3dw3fgm5l1kiXwm+m8+yuBOyNiClh034we1dIxM7NjZAn8vwAeA5YD6yW9ANiTZ1GDcEvH\nzKy3LF/a3gjcOOuhb0t6TX4lDaY9LdOzdMzMOsvype1KSTe0NymR9D9IRvuLiqdlmpn1lqWl81Fg\nL/Cm9M8e4GN5FjWIxuHAdw/fzKyTLFfMvjAifmbW/fdJ2pRXQYNquqVjZtZTlhH+AUmvaN+RdAlw\nIL+SBjPilo6ZWU9ZRvi/CtwiaSUg4HvAW/IsahDu4ZuZ9ZZlls4mYJ2ksfT+opuSCUcCf9Lz8M3M\nOuoa+JJ+o8vjAETEDTnVNJBm3Vfampn10muEv2LBqjgB3NIxM+uta+BHxPsWspDj1b7S1ksrmJl1\nVppNzNstHS+tYGbWWWkC39Myzcx6y7K0wplZHita+0rblq+0NTPrKMsI//YOj912ogs5Xm7pmJn1\n1mta5g8D5wIrJV0166kxYDTvwuarWXNLx8ysl17TMs8GLgdWAT816/G9wK/kWdQgajXRqMmBb2bW\nRa9pmXcCd0q6OCLuXcCaBtas17xapplZF1nW0vmWpN8B1s4+PiL+Y15FDapR9wjfzKybLIF/J3AP\n8CVgOt9yjs9IvebANzPrIkvgL4uI3869khOgWa/5Slszsy6yTMv8rKTX5V7JCdBsuKVjZtZNlsC/\nniT0D0raI2mvpEW7RLLn4ZuZdZZlPfyhWTWzWav5Slszsy6yLK0gSVdL+s/p/TWSXpJ/afPnlo6Z\nWXdZWjr/E7gY+MX0/j7gz3Kr6Di4pWNm1l2WWTovjYiLJD0AEBHflzSSc10DaXpapplZV1lG+FOS\n6kAASBoHFmWqjvhKWzOzrrIE/o3AHcCpkt4PfBX4g6wnkFSX9ICkzw5YY2aNumh5hG9m1lGWWTq3\nStoIXJY+dGVEbJ/HOa4HtpOsspmrpIfvEb6ZWSdZd7xaBtTT45dmfXNJq4HXAzfPv7T589IKZmbd\nZZmW+R7gFuBk4BTgY5J+L+P7fxB4JwvU82968TQzs66yzNL5BeDCiDgIIOkDwP3Af+31IkmXA09F\nxEZJr+5x3LXAtQBnnHFGxrI7S9bSceCbmXWSpaXzGEfvcLUE+JcMr7sEuELSY8CngUslfWLuQRFx\nU0RMRMTE+Ph4hrftrlGvMTXjHr6ZWSe9tjj8EMlUzEPANklfTO//OMlMnZ4i4t3Au9P3ejXwWxFx\n9QmouasRt3TMzLrq1dLZkP69kWRaZttXcqvmOLmlY2bWXa8tDm85USeJiK+wAB8UzYYvvDIz6ybL\nLJ3L0wunvjcsyyNHOPTNzObKMkvng8BVwNZY5EnarAmA6ZmgUVfB1ZiZLS5ZZunsAB5c7GEPSUsH\ncFvHzKyDLCP8dwKfk3Q3yYwdACLihtyqGlCzngT+5PQMS6kXXI2Z2eKSJfDfT7IG/iiwKJdFbhtJ\n2ziemmlmdqwsgX9yRPxE7pWcAO0Rvrc5NDM7VpYe/pckDUXgN+rtHr5H+GZmc2UJ/F8DPi/pwOKf\nlpm0dLzNoZnZsbKsh79iIQo5EUY8wjcz66pv4Et6VafHI2L9iS/n+LR7+FMt9/DNzObK8qXtO2bd\nHgVeQrK+zqW5VHQcDs/Dn/EI38xsriwtnZ+afV/SGuCPcqvoOLSvtPUCamZmx8q6xeFsO4HzTnQh\nJ4KvtDUz6y5LD7+9Lj4kHxAXAJvzLGpQTX9pa2bWVZYe/oZZt1vApyLiaznVc1w8LdPMrLssgX8b\ncDAipgEk1SUti4hn8y1t/kZ8pa2ZWVdZevh3AUtn3V8KfCmfco6Pr7Q1M+suS+CPRsS+9p309rL8\nShqcWzpmZt1lCfz9ki5q35H0IuBAfiUNzlfampl1l6WH/zbgbyQ9kd4/Dfi5/Eoa3JErbR34ZmZz\nZbnw6uuSfhg4GxDwcERM5V7ZANrz8Fsz/tLWzGyuLCN8gBcDa9PjL5RERPxlblUNqFFzD9/MrJss\nF179FfBCYBMwnT4cwKILfC+eZmbWXZYR/gRwzjBsYl6viXpN/tLWzKyDLLN0HgSel3chJ0qz7sA3\nM+skywj/FOAhSf8EHGo/GBFX5FbVcWjWa148zcysgyyB//t5F3EiJYHvEb6Z2VxZpmXevRCFnChu\n6ZiZddY18CV9NSJeIWkvR5ZHhmQufkTEWO7VDaBZr3lapplZB10DPyJekf49NJuYQ7K8gnv4ZmbH\nGmTHq0WtWa/R8gjfzOwYpQv8hnv4ZmYdlS7wkx6+WzpmZnOVLvBH6jWvlmlm1kHpAr/ZcEvHzKyT\n8gV+vcaUl0c2MztG6QK/UXNLx8ysk9IF/ohbOmZmHZUu8L2WjplZZ7kFvqQ1kr4sabukbZKuz+tc\ns3m1TDOzzrJucTiIFvCbEXG/pBXARklfjIiHcjynR/hmZl3kNsKPiO9ExP3p7b3AduD0vM7X5tUy\nzcw6W5AevqS1wIXAfR2eu1bSBkkbdu3addznckvHzKyz3ANf0knA7cDbImLP3Ocj4qaImIiIifHx\n8eM+n5dHNjPrLNfAl9QkCftbI+IzeZ6rbcQtHTOzjvKcpSPgI8D2iLghr/PM1azXiIBpX21rZnaU\nPEf4lwBvBi6VtCn987oczwfASaPJxKPv7j/U50gzs2rJbVpmRHyVZDvEBXXu81cCsHXnbi77kdGF\nPr2Z2aJVuittzzt9jJpg887dRZdiZraolC7wl400+KFTV7B5xzNFl2JmtqiULvAB1q1exZadzxDh\nL27NzNpKGfjnr1nJ95+dYsf3DhRdipnZolHKwF+3ehUAm3e6rWNm1lbKwD/7eStY0qi5j29mNksp\nA79Zr3Hu88fY4pk6ZmaHlTLwAdatWcXWx3fT8jILZmZAmQN/9SoOTE3zzaf2FV2KmdmiUN7AX5N8\ncbvFX9yamQElDvy1P7CMsdEGm3a4j29mBiUOfEmsW7PKI3wzs1RpAx/g/NUrefjJvRycmi66FDOz\nwpU68NetXsX0TLDtiWM22jIzq5xSB/4F6Re3vgDLzKzkgf/csVGeNzbqJRbMzCh54AOsW7PSV9ya\nmVGBwD9/9SoefXo/u5+dKroUM7NClT7w2338LY+7rWNm1Vb6wD/v9GSPW39xa2ZVV/rAX7m0yVnj\ny73HrZlVXukDH5L5+Jt2eMtDM6u2igT+SnbtPcSTew4WXYqZWWEqEfjnH74Ay20dM6uuSgT+OaeN\n0ajJF2CZWaVVIvBHm3V+5LQxr5xpZpVWicCHZOXMLTt2MzPjL27NrJoqE/jr1qxi76EWj353f9Gl\nmJkVojqBv9orZ5pZtVUm8H/wuSexbKTuhdTMrLIqE/j1mjjv9JVs8gjfzCqqMoEPyUJqDz2xh8nW\nTNGlmJktuEoF/vmrVzI5PcM3ntxbdClmZguuUoHf/uJ2k+fjm1kFVSrwVz9nKScvH2GL+/hmVkGV\nCnxJrFu90kssmFklVSrwIbkA65tP7WPfoVbRpZiZLahG0QUstHWrVxEBP/cX9/LcFUsYW9pkbLTJ\nitHG4dtjSxusGG0yNuuxFaMNRpv1oss3MxtYroEv6bXAnwB14OaI+ECe58viZWf9AFdddDpP7j7I\nrn2HeOTp/ew5MMWegy2m+6yzM9KodflAaKSPJ7dXpMe0H1uRPr9spI6kBfpJzcyOllvgS6oDfwb8\nOLAT+Lqkv42Ih/I6ZxZLR+rc8KYLjnk8Inh2cpq9B1vsOTiVfghMJffTD4TDf6fP7z3Y4vFnDrDn\nQPJYv/n99ZqO/UDI+NvF2NImK5Y0qNX8gWFmg8lzhP8S4FsR8QiApE8DbwAKDfxuJLF8SYPlSxo8\nb+XoQO9xcOroD4wjt1vph8eR2+3nH3l6H3sOtNh7cIr9k9N9aoSTljSO/ZCY9ZvGSaMN6rUadUG9\nXqMuUa9BTaJem/VHopb+PfvxI8eRvo+o1TjqNY2jjlOX9+bwe/u3GrPFIc/APx3YMev+TuClOZ6v\ncKPNOqPNOuMrlgz0+tb0zOEPiSO/Wcz6kJj1WPv5x585wMPtD5BDLRbjtr01cfQHg0S93vlDpyb6\nfkD0fLbPZ0u/jx5/OFkRTl42wl+/9eLcz5Nn4Hf6l3NMHEm6FrgW4IwzzsixnMWvUa/xnOUjPGf5\nyECvn5kJnp2aZno6mI6gNTPDzAxMRzAzE0zPBK2ZYCaS29Nzbk/PRHrskde05hw3E0FrOo68Z8f3\npsv7Hrl97PmPvKaXXs/226S+72dhnwOCQH0/Mszmb8XowsyfyfMsO4E1s+6vBp6Ye1BE3ATcBDAx\nMbEIx6fDo1YTJy2p3MQrM8soz3n4Xwf+naQzJY0APw/8bY7nMzOzHnIbDkZES9J/Av6OZFrmRyNi\nW17nMzOz3nL9/T8iPgd8Ls9zmJlZNpVbWsHMrKoc+GZmFeHANzOrCAe+mVlFOPDNzCpC/a5OXEiS\ndgHfnsdLTgGezqmcvLn24gxz/a69OIu1/hdExHiWAxdV4M+XpA0RMVF0HYNw7cUZ5vpde3GGvX5w\nS8fMrDIc+GZmFTHsgX9T0QUcB9denGGu37UXZ9jrH+4evpmZZTfsI3wzM8toKANf0mslfUPStyS9\nq+h65kPSRyU9JenBomuZL0lrJH1Z0nZJ2yRdX3RNWUkalfRPkjantb+v6JrmS1Jd0gOSPlt0LfMl\n6TFJWyVtkrSh6HrmQ9IqSbdJejj9fz//ralyMnQtnXRz9H9m1ubowC8UvTl6VpJeBewD/jIiziu6\nnvmQdBpwWkTcL2kFsBG4chj+2yvZu3B5ROyT1AS+ClwfEf9YcGmZSfoNYAIYi4jLi65nPiQ9BkxE\nxGKcx96TpFuAeyLi5nRvj2UR8UzRdQ1iGEf4hzdHj4hJoL05+lCIiPXA94quYxAR8Z2IuD+9vRfY\nTrJ38aIXiX3p3Wb6Z2hGO5JWA68Hbi66liqRNAa8CvgIQERMDmvYw3AGfqfN0YcidMpE0lrgQuC+\nYivJLm2JbAKeAr4YEUNTO/BB4J3ATNGFDCiAL0jamO5jPSzOAnYBH0vbaTdLWl50UYMaxsDPtDm6\n5UfSScDtwNsiYk/R9WQVEdMRcQHJ/sovkTQULTVJlwNPRcTGoms5DpdExEXATwK/lrY2h0EDuAj4\ncERcCOwHhup7w9mGMfAzbY5u+Uj737cDt0bEZ4quZxDpr+RfAV5bcClZXQJckfbBPw1cKukTxZY0\nPxHxRPr3U8AdJK3ZYbAT2Dnrt8HbSD4AhtIwBr43Ry9I+sXnR4DtEXFD0fXMh6RxSavS20uBHwMe\nLraqbCLi3RGxOiLWkvz//vcRcXXBZWUmaXn6JT9pO+QngKGYpRYRTwI7JJ2dPnQZsOgnKXST6562\neRj2zdElfQp4NXCKpJ3AeyPiI8VWldklwJuBrWkvHOB30r2LF7vTgFvSWV414K8jYuimNw6pU4E7\nkvECDeCTEfH5Ykual18Hbk0HmI8A1xRcz8CGblqmmZkNZhhbOmZmNgAHvplZRTjwzcwqwoFvZlYR\nDnwzs4pw4JuZVYQD34aKpK9ImkhvPybplIyve4ukPx3gfGv7LWWdHvOLs+5PSLpxvufKUMvHJT0q\n6a09jnmlpIeGcflty58D3+z4rQUOB35EbIiI63I61zsi4s+7PRkR9wCvy+ncNuQc+LbgJL1T0nXp\n7T+W9Pfp7cvaa8RI+rCkDYNsVpJukHN/utnJXR2ef4GkuyRtSf8+I338VEl3pK/bLOnlc153Vrpi\n4ovnvOUHgFemm3u8XdKr25uUSPp9SbdI+kL6G8lVkv4o3Qzk8+naREh6kaS709Uk/y7de6Dfz/mz\nkh5Ma10/n/9GVk0OfCvCeuCV6e0J4KQ0+F4B3JM+/rsRMQGcD/yopPOzvLGkceB/AT8TEeuAn+1w\n2J+SbEBzPnAr0G6/3Ajcnb7uIuDwkh3pWiq3A9dExNfnvN+7SDbIuCAi/rjD+V5Ispb9G4BPAF+O\niH8PHABen/7sHwLeGBEvAj4KvD/Dj/se4D+k9V6R4XirOAe+FWEj8KJ0Qa1DwL0kwf9KjgT+myTd\nDzwAnAuck/G9Xwasj4hHASKi02YzFwOfTG//FckHDcClwIfT101HxO708XHgTuDqiNjE/P2/iJgC\ntpKs/9ReR2YrSTvobOA84IvpGkW/R7IKbD9fAz4u6VfS9zXraegWT7PhFxFT6VK/1wD/AGwBXkMy\nEt4u6Uzgt4AXR8T3JX0cGM349mL++yP0O343yaY7lzBr1D8PhwAiYkbSVBxZwGqG5N+ggG0RMa+9\nUiPirZJeSvLbwyZJF0TEdweozyrCI3wrynqSUF9PMqp/K7ApDcMxko0mdks6lWTTjKzuJWkBnQkg\n6eQOx/wDyTLDAL9Esr8twF3Ar6avq6fb2wFMAlcCvzx7Ns4se4EV86hxrm8A40o3x5bUlHRuvxdJ\nemFE3BcR7wGe5uh9IsyO4cC3otxDsmTxvRHxb8DB9DEiYjNJK2cbST/7a1nfNCJ2AdcCn5G0Gfjf\nHQ67DrhG0haS5Z6vTx+/HniNpK0kbafDoRsR+4HLgbdLmruH8haglX55+vastc5670ngjcAfpjVv\nAl7e+1UA/Lf0y98HST44N8/33FYtXh7ZbEikra3PRsRtfY5bmx43FFs42sLxCN9seOwG/ku/C6+A\n/0PS4jE7ikf4ZmYV4RG+mVlFOPDNzCrCgW9mVhEOfDOzinDgm5lVxP8HpPVhUpzTalMAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Step 4 Plotting the results\n",
    "\n",
    "# The returned result object holds informations about the optimization run\n",
    "# like the incumbent (=best) configuration.\n",
    "id2config = res.get_id2config_mapping()\n",
    "print('A total of %i unique configurations where sampled.' % len(id2config.keys()))\n",
    "print('A total of %i runs where executed.' % len(res.get_all_runs()))\n",
    "print('Best configuration found: {}'.format(id2config[res.get_incumbent_id()]['config']))\n",
    "\n",
    "# The incumbent trajectory is a dictionary with all the configuration IDs, the times the runs\n",
    "# finished, their respective budgets, and corresponding losses.\n",
    "# It's used to do meaningful plots of the optimization process.\n",
    "incumbent_trajectory = res.get_incumbent_trajectory()\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "plt.plot(incumbent_trajectory['times_finished'], incumbent_trajectory['losses'])\n",
    "plt.xlabel('wall clock time [s]')\n",
    "plt.ylabel('incumbent loss')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the results in CAVE\n",
    "\n",
    "### 2.1) Creating a HTML-report with CAVE\n",
    "\n",
    "Creating the report with CAVE is very straight-forward. Simply provide the output-directory of the BOHB-analysis in CAVE's `--folders` argument and specify `--file_format` as `BOHB`. You can do this by commandline ('!' simply executes the command as if it was executed on the command line):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "! cave --folders workflow_result --file_format BOHB --output CAVE-workflow-result --verbose_level INFO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After CAVE finished the report, you can have a look at it with your favorite browser."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "! firefox CAVE-workflow-result/report.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2) Using CAVE from within Python\n",
    "\n",
    "Of course you can use CAVE on a module-level. Import and instantiate it (very similarily to the commandline). By default, CAVE even outputs all analysis results in a jupyter-cell-compatible way. Of course, the HTML-report is built meanwhile, so you don't have to run time-consuming analyzing-methods repeatedly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from cave.cavefacade import CAVE\n",
    "\n",
    "cave = CAVE(folders=[\"workflow_result\"],\n",
    "            output_dir=\"test_jupyter\",\n",
    "            ta_exec_dir=[\".\"],\n",
    "            file_format='BOHB',\n",
    "            #verbose_level='DEV_DEBUG'\n",
    "           )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most interesting plot for BOHB might be a visualization of the learning curves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "cave.bohb_learning_curves()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access the individual budgets via the 'run'-keyword-argument of each analysis-method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "cave.print_budgets()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "cave.overview_table(run='budget_12.5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each budget, we can compare the default against the incumbent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "cave.compare_default_incumbent(run='budget_25.0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "cave.performance_table(run='budget_25.0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For parameter-importance analysis, CAVE uses <a href=\"https://github.com/automl/ParameterImportance\" target=\"_blank\">PIMP</a> , a package that provides multiple approaches to parameter-importance analysis. We can easily invoke them via CAVE, of course. To estimate the importance, random forests are used to predict performances of configurations that were not executed. This is difficult for big budgets with few configurations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "cave.cave_fanova(run='budget_12.5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "cave.cave_ablation(run='budget_12.5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "cave.local_parameter_importance(run='budget_12.5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each budget, we can compare the different parameter-importance-methods that have already been run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "cave.pimp_comparison_table(run='budget_12.5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To analyze BOHB's behaviour, we can check out the configurator footprint, cost-over-time and parallel coordinated parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "cave.configurator_footprint(run='budget_12.5', time_slider=True, num_quantiles=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "cave.cost_over_time(run=\"budget_25.0\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "cave.parallel_coordinates(run='budget_12.5')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}