Python Training - Part 1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Make your Python Code Run Faster\n",
    "\n",
    "Dr. David Perry\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Me\n",
    "* Compute Integration Specialist at Research Platform Services (ResPlat)\n",
    "* I ❤️ Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What's a ResPlat?\n",
    "* Cloud & High Performance Computing\n",
    "* Data Storage & Management\n",
    "* Training & Community\n",
    "* Part of University Services — here for all academic departments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "So you probably already have basics down for intro course if you are here.\n",
    "\n",
    "I touch on HPC and cloud here, but the details are covered in seperate courses (or you can work it out on your own from docs).\n",
    "\n",
    "I'll mention the Pandas library, but that's a rich package in of itself, enough to fill a whole other course."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src=\"resbaz.png\" width=\"75%\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Related Courses\n",
    "\n",
    "* Intro to Python\n",
    "* HPC\n",
    "* Cloud Computing\n",
    "* Intro to Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Today\n",
    "* Part 1: Profiling (why is my code slow)\n",
    "* Part 2: Optimization (fix my slow code)\n",
    "* Part 3: Parallellization (run multiple bits of code at once)\n",
    "* Part 4: Cloud & High Performance Computing (f%^$ it, gimme a bigger computer please)\n",
    "\n",
    "**Goal**: Less waiting, more research."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The Buffet Disclaimer\n",
    "\n",
    "<center><img src=\"daisies.jpg\" width=\"75%\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src=\"too_much.gif\" width=\"75%\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "I'm going to unleash upon you a veritable feast of tools, technologies, jargon and acronyms. \n",
    "\n",
    "But just because it's on the table doesn't mean you have to eat it! \n",
    "\n",
    "And you'll probably forget most of the details when you wake up tomorrow.\n",
    "\n",
    "That's okay! \n",
    "\n",
    "These slides are here for you later, Google is your friend, and we're here to help. If nothing else, remember that you don't have to tolerate slow code if you don't want to, you do have options."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### After the Hangover\n",
    "* Link to slides/code: https://github.com/resbaz/high-performance-python-course\n",
    "* I'm here to help: perry.d@unimelb.edu.au\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Part 1: Profiling\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Before we do anything else, we should figure out where our code is slow. It's no use throwing a bunch of fancy tricks at the problem if you don't know where it lies. Profiling will help you figure out what's taking the most time in your code, and focus your efforts there."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### iPython Timing Magic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time a = \"abc\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%timeit -n 100 a = \"abc\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "The easiest thing is to just time your code -- these are built into Jupyter/IPython. Timeit will run your code multiple times, in case it's variable.\n",
    "\n",
    "You can try different things, and see how it impacts execution time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Python Profiler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "def batman():\n",
    "    out = \"\"\n",
    "    for i in range(15):\n",
    "        out += \"na\"\n",
    "    out += \"\\n batman!\"\n",
    "    time.sleep(1)\n",
    "    print(out)\n",
    "\n",
    "%prun batman()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Next level up is the built-in Python profiler. This will tell you what functions you're calling, how many times, and how long they take.\n",
    "\n",
    "That can help, but is sometimes cryptic when it describes underlying Python functions.\n",
    "\n",
    "BTW: If you have an IDE like PyCharm, it might give you other handy tools like call graphs an inline heatmaps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext line_profiler \n",
    "# ^ Not installed by default, you'll need to DIY - `pip install line_profiler`\n",
    "\n",
    "import time\n",
    "import math\n",
    "\n",
    "def batman():\n",
    "    out = \"\"\n",
    "    for i in range(15):\n",
    "        out += \"na\"\n",
    "        a = i**i * math.log(i+1) * math.factorial(i*10)  # Oh no, math!\n",
    "    out += \"\\n batman!\"\n",
    "    print(out)\n",
    "\n",
    "%lprun -f batman batman()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "More helpful I think is the line profiler, this will describe how long you spend on each line in a program. You can even get a heatmap!\n",
    "\n",
    "Line profiling is not built into Python, you'll have to install it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext heat\n",
    "# ^ Also not installed by default, 'pip install py-heat-magic'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "%%heat\n",
    "import time\n",
    "import math\n",
    "\n",
    "def batman():\n",
    "    out = \"\"\n",
    "    for i in range(15):\n",
    "        out += \"na\"\n",
    "        a = i**i * math.log(i+1) * math.factorial(i*10)  # Oh no, math!\n",
    "    out += \"\\n batman!\"\n",
    "    print(out)\n",
    "    \n",
    "batman()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Memory Profiling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%writefile mem_profile_example.py\n",
    "\n",
    "import random\n",
    "def make_big_array():\n",
    "    out = []\n",
    "    for i in range(100000):\n",
    "        out.append(random.random())\n",
    "    return out\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext memory_profiler\n",
    "from mem_profile_example import make_big_array\n",
    "\n",
    "%mprun -f make_big_array make_big_array()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Speed is one thing, but you also have a finite amount of memory (RAM) on your computer. If you're dealing with big datasets, your bottleneck might be memory, which limits how much data you can analyse at once, or maybe prevents you from loading your data altogether! Different Python libraries and programming techniques can have drastic impacts on memory usage, and if you run into problems you can use memory_profiler to investigate (there are other libraries for this as well).\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Learn More\n",
    "\n",
    "* %time and %timeit (built-in): http://ipython.readthedocs.io/en/stable/interactive/magics.html\n",
    "* Line Profiler: https://github.com/rkern/line_profiler\n",
    "* ...with heat map: https://github.com/csurfer/pyheat\n",
    "* Memory Profiler: https://github.com/pythonprofilers/memory_profiler\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Challenges!\n",
    "\n",
    "Go here: http://go.unimelb.edu.au/dgh6\n",
    "\n",
    "Select challenge (1 to 3), and click 'Clone' to start your own instance.\n",
    "\n",
    "You can login using your University email, or create a (free) Microsoft account."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Challenge 1**\n",
    "* Second approach a little slower because of function call overhead (tradeoff with readability?)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Challenge 2**\n",
    "* Second approach a little faster (maybe) because we avoid a call to 'append' and list comprehensions can be performed more efficiently by Python interpreter.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Challenge 3**\n",
    "* Reading CSV in chunks uses much less memory, but a little slower."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}