languages/python/Python_Environments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Environments\n",
    "\n",
    "![xkcd comic on Python environments](https://imgs.xkcd.com/comics/python_environment.png)\n",
    "\n",
    "## **Note: _This notebook will not work as designed unless ran from our [JupyterHub](https://europa.hpc.nrel.gov) instance_**\n",
    "\n",
    "## Table of Contents\n",
    "* [System Python](#system-python)\n",
    "* [User Environments](#user-environments)\n",
    "    * [`.local` User Environment](#Local-/home-install,-a.k.a-~/.local)\n",
    "    * [Python Virtual Environments](#Python-Virtual-Environments)\n",
    "    * [Anaconda Environments](#Anaconda-Environments)\n",
    "* [The Right Tool for the Job](#The-Right-Tool-for-the-Job)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## System Python\n",
    "This python environment is used by Operating System root user for package managers and system utilities. These are built on long-term stable versions so they don't have bleeding-edge versions of packages, let alone many packages outside of the standard library. Some systems are still using Python 2 as system python as of this workshop.\n",
    "\n",
    "Without loading any modules, the python you are referencing is the system-installation which does not offer much freedom to non-admin users. **You are likely not an admin user and do not have write-privileges to the system python directories, as that may compromise the integrity of the system.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Default Python paths on Eagle login/compute nodes:\n",
      "/usr/bin/python\n",
      "/usr/bin/python2\n",
      "/usr/bin/python3\n",
      "\n",
      "Default Python paths on Europa:\n",
      "/opt/jupyter/anaconda3/bin/python\n",
      "/bin/python2\n",
      "/opt/jupyter/anaconda3/bin/python3\n"
     ]
    }
   ],
   "source": [
    "%%bash -l\n",
    "module purge\n",
    "\n",
    "# Default Python on normal Eagle nodes is system python\n",
    "echo \"Default Python paths on Eagle login/compute nodes:\"\n",
    "ssh -T el1 <<EOF\n",
    "which \\\n",
    "    python  \\\n",
    "    python2 \\\n",
    "    python3\n",
    "EOF\n",
    "echo\n",
    "\n",
    "# Default Python on JupyterHub is a Jupyter installation\n",
    "echo \"Default Python paths on Europa:\"\n",
    "which \\\n",
    "    python  \\\n",
    "    python2 \\\n",
    "    python3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the sake of demonstration, let's try to install some libraries and notice we will get denied permission:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting colorz\n",
      "  Using cached https://files.pythonhosted.org/packages/6b/38/1dcc0641bfbf8edf1d3310a879cde418ff8e86c13f121c8d018940a992aa/colorz-1.0.3-py2.py3-none-any.whl\n",
      "Requirement already satisfied: Pillow in /opt/jupyter/anaconda3/lib/python3.7/site-packages (from colorz) (5.4.1)\n",
      "Requirement already satisfied: scipy in /opt/jupyter/anaconda3/lib/python3.7/site-packages (from colorz) (1.2.1)\n",
      "Installing collected packages: colorz\n",
      "\u001b[31mCould not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/opt/jupyter/anaconda3/lib/python3.7/site-packages/colorz.py'\n",
      "Consider using the `--user` option or check the permissions.\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# Timeout prevents pip from hanging forever\n",
    "!timeout 5 \\\n",
    "    pip install colorz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## User Environments\n",
    "\n",
    "Obviously, a python environment where we can't manage dependencies is not of much use. Since the HPC cluster is a shared system by design, it is important to provide tools for each user to manage their own Python environments, otherwise there would be rampant compability issues and version mismatches from many users trying to install the dependencies they need.\n",
    "\n",
    "*Enter Python virtual environments*\n",
    "\n",
    "Luckily, there is a plethora of options for users to manage their own virtual installations of Python in directories that they (*and usually **only** they*) have write privileges in. Let's demonstrate these methods:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Local `/home` install, a.k.a `~/.local`\n",
    "\n",
    "Perhaps you noticed in the error response we got from trying to install a module to the operating system python above that it suggests to use the `--user` flag. This will create a Python installation hierarchy in `$HOME/.local` which is a hidden directory in your `/home` where you certainly (or at least *should*) have write privileges while logged in. However, this is a singular Python environment&mdash;if you find yourself needing to use libraries which require different versions of a dependency, that is where this method starts to show its flaws.\n",
    "\n",
    "Let's see it in action:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No executables found named \"colorz\"\n",
      "\n",
      "Successfully installed colorz-1.0.3\n",
      "\n",
      "\"colorz\" found in /home/mbartlet/.local/bin/colorz\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "PATH=\"$PATH:$HOME/.local/bin\" # this is so our shell can find .local executables\n",
    "\n",
    "check_if_executable_exists() {\n",
    "    local path=\"$(command -v $1)\"\n",
    "    if [ -z \"$path\" ]\n",
    "        then\n",
    "            echo \"No executables found named \\\"$1\\\"\"\n",
    "        else\n",
    "            echo \"\\\"$1\\\" found in $path\"\n",
    "    fi\n",
    "    echo # Newline for readability in output\n",
    "}\n",
    "\n",
    "# There probably isn't any executable named this before installing\n",
    "check_if_executable_exists colorz\n",
    "\n",
    "# Timeout prevents pip from hanging forever\n",
    "timeout 5 \\\n",
    "    pip install --user \\\n",
    "        --no-warn-script-location \\\n",
    "        colorz \\\n",
    "    | tail -1 # We don't need the whole `pip` output\n",
    "        \n",
    "echo # Newline for readability in output\n",
    "    \n",
    "# Now this should print the path of the executable in $HOME/.local/bin\n",
    "check_if_executable_exists colorz\n",
    "\n",
    "timeout 5 \\\n",
    "    pip uninstall --yes --quiet colorz # Don't keep `colorz` unless you want to"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python Virtual Environments\n",
    "\n",
    "Virtual environments allow you to create a virtual python installation in any directory. Usually, they are created alongside any project source code they may accompany. Virtual environments give you the liberty of cleanly managing collections of dependencies with specific versions.\n",
    "\n",
    "*Note that earlier versions of python used the \"virtualenv\" package to install and manage virtual environments, however as of writing that module is largely deprecated relative to \"venv\"*\n",
    "\n",
    "We can create a virtual environment to accomplish the same as above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Path before activating virtual environment:\n",
      "\t/opt/jupyter/anaconda3/bin/python3\n",
      "\n",
      "Path after activating virtual environment:\n",
      "\t/tmp/tmp.SEwHHmJKIb/demo_virtual_environment/bin/python3\n",
      "\n",
      "`colorz` installed in virtual environment:\n",
      "\t/tmp/tmp.SEwHHmJKIb/demo_virtual_environment/bin/colorz\n",
      "\n",
      "Path after deactivating virtual environment:\n",
      "\t/opt/jupyter/anaconda3/bin/python3\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash -l\n",
    "\n",
    "TMP_DIR=$(mktemp -d) # Create a temporary directory\n",
    "VENV_NAME=\"demo_virtual_environment\"\n",
    "\n",
    "cd $TMP_DIR\n",
    "python3 -m venv  $VENV_NAME # Create a virtual environment\n",
    "\n",
    "printf \"Path before activating virtual environment:\\n\\t$(which python3)\\n\\n\"\n",
    "\n",
    "source  $VENV_NAME/bin/activate # Activate our virtual environment\n",
    "\n",
    "printf \"Path after activating virtual environment:\\n\\t$(which python3)\\n\\n\"\n",
    "\n",
    "timeout 10 \\\n",
    "    pip install --quiet colorz # Install a package in the virtual environment\n",
    "\n",
    "printf \"\\`colorz\\` installed in virtual environment:\\n\\t$(which colorz)\\n\\n\"\n",
    "\n",
    "deactivate # Deactivate the virtual environment\n",
    "\n",
    "printf \"Path after deactivating virtual environment:\\n\\t$(which python3)\\n\\n\"\n",
    "\n",
    "rm -rf $TMP_DIR # Destroy the demo virtual environment for cleanliness"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Anaconda Environments\n",
    "\n",
    "Virtual environment are well and good, however you are limited to use the version of Python that the `venv` module is installed with. Bleeding-edge versions of packages may start to require the latest python language features, or similarly known-stable versions of modules may not be compatible with later version of Python. This is part of Anaconda's niche, as it allows you to manage entire installations of many common data science softwares (not just limited to Python).\n",
    "\n",
    "On NREL HPC, Anaconda is accessible as an optional software module using LMOD. Here is how to source the Anaconda software module so your shell is able to utilize the installation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python path before loading Anaconda:\n",
      "\t/opt/jupyter/anaconda3/bin/python\n",
      "\n",
      "Python path after loading Anaconda:\n",
      "\t/nopt/nrel/apps/anaconda/5.3/bin/python\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash -l\n",
    "\n",
    "# Simply loading the Anaconda module will change your Python environment\n",
    "\n",
    "module purge # Make sure no other modules conflict \n",
    "\n",
    "printf \"Python path before loading Anaconda:\\n\\t$(which python)\\n\\n\"\n",
    "\n",
    "module load conda # Load Anaconda, named \"conda\" after its executable name.\n",
    "\n",
    "printf \"Python path after loading Anaconda:\\n\\t$(which python)\\n\\n\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's repeat what we were able to accomplish using the `venv` module, but using an Anaconda environment to install a specific version of Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting package metadata: done\n",
      "Solving environment: done\n",
      "\n",
      "\n",
      "==> WARNING: A newer version of conda exists. <==\n",
      "  current version: 4.6.14\n",
      "  latest version: 4.7.12\n",
      "\n",
      "Please update conda by running\n",
      "\n",
      "    $ conda update -n base -c defaults conda\n",
      "\n",
      "\n",
      "\n",
      "## Package Plan ##\n",
      "\n",
      "  environment location: /home/mbartlet/.conda/envs/conda_demo_env\n",
      "\n",
      "  added / updated specs:\n",
      "    - python=3.5\n",
      "\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-main\n",
      "  bzip2              conda-forge/linux-64::bzip2-1.0.8-h516909a_1\n",
      "  ca-certificates    conda-forge/linux-64::ca-certificates-2019.11.28-hecc5488_0\n",
      "  certifi            conda-forge/linux-64::certifi-2018.8.24-py35_1001\n",
      "  libffi             conda-forge/linux-64::libffi-3.2.1-he1b5a44_1006\n",
      "  libgcc-ng          conda-forge/linux-64::libgcc-ng-9.2.0-hdf63c60_0\n",
      "  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-9.2.0-hdf63c60_0\n",
      "  ncurses            conda-forge/linux-64::ncurses-6.1-hf484d3e_1002\n",
      "  openssl            conda-forge/linux-64::openssl-1.0.2t-h14c3975_0\n",
      "  pip                conda-forge/linux-64::pip-18.0-py35_1001\n",
      "  python             conda-forge/linux-64::python-3.5.5-h5001a0f_2\n",
      "  readline           conda-forge/linux-64::readline-7.0-hf8c457e_1001\n",
      "  setuptools         conda-forge/linux-64::setuptools-40.4.3-py35_0\n",
      "  sqlite             conda-forge/linux-64::sqlite-3.28.0-h8b20d00_0\n",
      "  tk                 conda-forge/linux-64::tk-8.6.10-hed695b0_0\n",
      "  wheel              conda-forge/linux-64::wheel-0.32.0-py35_1000\n",
      "  xz                 conda-forge/linux-64::xz-5.2.4-h14c3975_1001\n",
      "  zlib               conda-forge/linux-64::zlib-1.2.11-h516909a_1006\n",
      "\n",
      "\n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n",
      "#\n",
      "# To activate this environment, use:\n",
      "# > conda activate conda_demo_env\n",
      "#\n",
      "# To deactivate an active environment, use:\n",
      "# > conda deactivate\n",
      "#\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# This block doesn't use the %%bash magic so output from conda can be observed in real-time\n",
    "\n",
    "ENV_NAME=\"conda_demo_env\"\n",
    "\n",
    "import os\n",
    "os.environ['DEMO_ENV_NAME']=ENV_NAME # Export variable to cells further down\n",
    "\n",
    "# Make sure the environment doesn't exist already\n",
    "!conda-env remove --name $ENV_NAME\n",
    "\n",
    "!conda create --use-local --yes --name $ENV_NAME python=3.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before activating conda environment:\n",
      "\tPython version: Python 2.7.15 :: Anaconda, Inc.\n",
      "\tPython path: /nopt/nrel/apps/anaconda/5.3/bin/python\n",
      "\tpip path: /nopt/nrel/apps/anaconda/5.3/bin/pip\n",
      "\n",
      "After activating conda environment:\n",
      "\tPython version: Python 3.5.5\n",
      "\tPython path: /home/mbartlet/.conda/envs/conda_demo_env/bin/python\n",
      "\tpip path: /home/mbartlet/.conda/envs/conda_demo_env/bin/pip\n",
      "\tcolorz path: /home/mbartlet/.conda/envs/conda_demo_env/bin/colorz\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash -l\n",
    "\n",
    "module purge ; module load conda\n",
    "ENV_NAME=$DEMO_ENV_NAME       # Exported from previous cell\n",
    "\n",
    "printf \"Before activating conda environment:\\n\"\n",
    "printf \"\\tPython version: $(python --version 2>&1)\\n\"\n",
    "printf \"\\tPython path: $(which python)\\n\"\n",
    "printf \"\\tpip path: $(which pip)\\n\\n\"\n",
    "\n",
    "source activate $ENV_NAME\n",
    "\n",
    "pip install --quiet colorz\n",
    "\n",
    "printf \"After activating conda environment:\\n\"\n",
    "printf \"\\tPython version: $(python --version)\\n\"\n",
    "printf \"\\tPython path: $(which python)\\n\"\n",
    "printf \"\\tpip path: $(which pip)\\n\"\n",
    "printf \"\\tcolorz path: $(which colorz)\\n\\n\"\n",
    "\n",
    "conda deactivate\n",
    "\n",
    "conda-env remove --yes --quiet --name $ENV_NAME &>/dev/null"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see that after activating our conda environment, we now have a Python 3.5 interpretter as requested in the `conda create` command.\n",
    "\n",
    "There are many other features that Anaconda provides to create/update/clone/export environments you create, such as creating environments from an easily shareable file, duplicating existing environments, or even completely modifying the underlying python installation of an existing environment. For intimate detail one should consult the [Anaconda docs on environments](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## The Right Tool for the Job\n",
    "\n",
    "Here are our recommendations for managing user environments on NREL HPC systems:\n",
    "\n",
    "* Avoid using the `--user` installation, even for dependencies you are likely to use very frequently such as numpy and pandas. Many users are not aware of how the `--user` flag operates and can create ambiguities about which environment is hosting certain packages and modules that are found by the python interpretter.\n",
    "\n",
    "* If your workflow uses Python exclusively and modestly you should prefer virtual environments created with the `venv` module over fully-loaded Anaconda environments. A frequent issue observed by support staff is HPC users will often exhaust their limited `/home` storage quotas by simply having a handful of Anaconda environments. There are techniques to minimize the footprint of your conda environments, but simplicity over versatility is one of the tenets of [The Zen of Python](https://www.python.org/dev/peps/pep-0020/#id3).\n",
    "\n",
    "* Where Anaconda shines is managing data-science software stacks. If you need to switch between various versions of Python, R, Julia, etc. then that is where the convenience of conda becomes obvious. Conda may also handle other external dependencies at the operating system level, such as automatically installing CUDA and cuDNN libraries when asked to install Tensorflow. If your needs end up transcending a single Python installation, then that is when Anaconda is most likely to be a convience.\n",
    "\n",
    "Below is a comparison between various management commands mentioned above as it is presented in the [Anaconda docs](https://conda.io/projects/conda/en/latest/commands.html#conda-vs-pip-vs-virtualenv-commands).\n",
    "\n",
    "\n",
    "\n",
    "| Task                                 | Conda package and environment manager command       | Pip package manager command                                       | Virtualenv environment manager command              |\n",
    "|--------------------------------------|-----------------------------------------------------|-------------------------------------------------------------------|-----------------------------------------------------|\n",
    "| Install a package                    | conda install <span>\\$</span>PACKAGE_NAME                         | pip install <span>\\$</span>PACKAGE_NAME                                         | X                                                   |\n",
    "| Update a package                     | conda update --name <span>\\$</span>ENVIRONMENT_NAME <span>\\$</span>PACKAGE_NAME | pip install --upgrade <span>\\$</span>PACKAGE_NAME                               | X                                                   |\n",
    "| Update package manager               | conda update conda                                  | Linux/macOS: pip install -U pip Win: python -m pip install -U pip | X                                                   |\n",
    "| Uninstall a package                  | conda remove --name <span>\\$</span>ENVIRONMENT_NAME <span>\\$</span>PACKAGE_NAME | pip uninstall <span>\\$</span>PACKAGE_NAME                                       | X                                                   |\n",
    "| Create an environment                | conda create --name <span>\\$</span>ENVIRONMENT_NAME python        | X                                                                 | cd <span>\\$</span>ENV_BASE_DIR; virtualenv <span>\\$</span>ENVIRONMENT_NAME      |\n",
    "| Activate an environment              | conda activate <span>\\$</span>ENVIRONMENT_NAME*                   | X                                                                 | source <span>\\$</span>ENV_BASE_DIR/<span>\\$</span>ENVIRONMENT_NAME/bin/activate |\n",
    "| Deactivate an environment            | conda deactivate                                    | X                                                                 | deactivate                                          |\n",
    "| Search available packages            | conda search <span>\\$</span>SEARCH_TERM                           | pip search <span>\\$</span>SEARCH_TERM                                           | X                                                   |\n",
    "| Install package from specific source | conda install --channel <span>\\$</span>URL <span>\\$</span>PACKAGE_NAME          | pip install --index-url <span>\\$</span>URL <span>\\$</span>PACKAGE_NAME                        | X                                                   |\n",
    "| List installed packages              | conda list --name <span>\\$</span>ENVIRONMENT_NAME                 | pip list                                                          | X                                                   |\n",
    "| Create requirements file             | conda list --export                                 | pip freeze                                                        | X                                                   |\n",
    "| List all environments                | conda info --envs                                   | X                                                                 | Install virtualenv wrapper, then lsvirtualenv       |\n",
    "| Install other package manager        | conda install pip                                   | pip install conda                                                 | X                                                   |\n",
    "| Install Python                       | conda install python=x.x                            | X                                                                 | X                                                   |\n",
    "| Update Python                        | conda update python*                                | X                                                                 | X                                                   |"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}