tu019-PythonModules.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "57eedfd7",
   "metadata": {},
   "source": [
    "# Writing your own python modules"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d61788bd",
   "metadata": {},
   "source": [
    "You will soon realize that for every project there are always a few lines of code that end up being extremely helpful and handy to keeparound. These lines end up being the ones applied multiple times and for multiple purpuses. \n",
    "\n",
    "Repeating operations and reusing lines of code is key to programming. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8dd2c83",
   "metadata": {},
   "source": [
    "In this tutorial we will learn how to write python [modules](https://docs.python.org/3/tutorial/modules.html). A module is nothing more than a file (ending in `.py`) containing collection of functions. A module can be as simple as containing a few functions, or as complicated as `numpy` or `seaborn`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e00451c8",
   "metadata": {},
   "source": [
    "We have learned so far how to write functions. Functions are a handy way to reuse the same lines of code.\n",
    "\n",
    "As the data science projects become more complex, or you become more expert at data science projects, the number of functions that end up needing to be carried around can grow fast. \n",
    "\n",
    "For any sizable project, the number of functions needed to be kept around is larger than the number of functions we are willing to copy and paste in every new script or jupyter notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fe3dee3",
   "metadata": {},
   "source": [
    "To avoid copying an pasting dozens of functions we can use python modules. Modules are collections of functions (and other python assertions, such as variables definitions) in a file saved on the current path accessible to `python`. \n",
    "\n",
    "Just like functions facilitate reusing dozens of lines of code, modules facilitates reusing dozens of functions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2e6d2d5",
   "metadata": {},
   "source": [
    "### Learning goals:\n",
    "\n",
    "* Understanding Python Modules\n",
    "* Practice building Python Modules\n",
    "* grouped data: aggregation and pivot tables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d9e3cf5",
   "metadata": {},
   "source": [
    "--- "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "733f93c4",
   "metadata": {},
   "source": [
    "## Our first module"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daa9c929",
   "metadata": {},
   "source": [
    "Python offers a convenient way to keep useful code and functions around by writing and importing modules. \n",
    "\n",
    "*What is a module?* Python modules are libraries of functions. We have encountered modules all along our tutorials. Indeed everytime we were invoking an `import` statement we were effectively loading a module.\n",
    "\n",
    "*How is a module defined?* A module is a python file (ending with exstension `.py`) with a series of functions definitions (i.e., statement starting with `def`) they live in the current path where your python code isrunning and because of that it can be imported.\n",
    "\n",
    "*How does a module work?* Python allows importing any `.py` file containing `def` statements. Importing a module file makes the functions in the file callable and usable (for example in Jupyter notebook). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63d5c74f",
   "metadata": {},
   "source": [
    "### How to write a python module \n",
    "\n",
    "Let's learn how to write and use Python modules! (we will start simple.)\n",
    "\n",
    "In a nutshell, the general process to write and use a python modules can be summarized as follows:\n",
    "\n",
    "**To write a module we need to:**\n",
    "A) Create a file with exstension `.py`.\n",
    "B) Write functions inside the file.\n",
    "C) Save the file on the path accessible to python (for simplicity say the current working directory).\n",
    "\n",
    "**To use python modules we need to:**\n",
    "A) Make sure the module is in the current working directory.\n",
    "B) import the module by typing `import` and `<moduleName>`\n",
    "C) Call the functions in the module withthe syntax `moduleName.functionName`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6334c07",
   "metadata": {},
   "source": [
    "### MyModule\n",
    "\n",
    "Hereafter, we will practice with the process described above. Write a module and then import and use the module. \n",
    "\n",
    "This means that we will write a file outside this jupyter notebook. This is something we have not done before and might feel a bit awkward (are we really leaving our safe `Jupyter Notebooks` heaven? Yes).\n",
    "\n",
    "Just as a heads start, our module will be called `mymodule`. The module will contain a function that will print the first few words of [Billie Eilish's song \"Ocean Eyes\"](https://www.youtube.com/watch?v=viimfQi_pUw&ab_channel=BillieEilish).\n",
    "\n",
    "So, to learn how to create a module, we will perform the following exercise.\n",
    "\n",
    "  - Open a new Jupyter Notebook (from the File menu, select \"New Notebook\"\n",
    "  \n",
    "<img src=\"./assets/jnb019/open_new_notebook.jpg\" alt=\"Drawing\" style=\"width: 300px;\"/>\n",
    "  \n",
    "  - Edit the name of the new notebook and rename it from \"untitled\"  `mymodule`\n",
    "  \n",
    "<img src=\"./assets/jnb019/rename_jupyter_notebook.jpg\" alt=\"Drawing\" style=\"width: 400px;\"/>\n",
    "  \n",
    "  - Copy and paste the code for the function provided below (`OceanEyes`) into the `mymodule` Jupyter Notebook. Note. Only create a single cell in the new notebook. Make sure no other cell is there.\n",
    "  \n",
    "<img src=\"./assets/jnb019/paste_function_code.jpg\" alt=\"Drawing\" style=\"width: 400px;\"/>\n",
    "\n",
    "  - Download the `mymodule` notebook into the current directory with the `.py` file exstension. To do so, from the File menu navigate to \"Downloads as\" and select the file type \"Python *.py*.\"\n",
    "  \n",
    "<img src=\"./assets/jnb019/download_module.jpg\" alt=\"Drawing\" style=\"width: 400px;\"/>\n",
    " \n",
    "  - Save the file in the same directory of the current tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0da08070",
   "metadata": {},
   "outputs": [],
   "source": [
    "def OceanEyes():\n",
    "    print('Can''t stop starin'' at those ocean eyes')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6c36888",
   "metadata": {},
   "source": [
    "Alright, after following the instructions above, and if all went well, we should be ready to load the module and use its function.\n",
    "\n",
    "To load the module we will tell python to import it. This is a simple as running the following statements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e03b2f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import mymodule"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76e2b8b3",
   "metadata": {},
   "source": [
    "If the previos cell executed withouterrors the module is loaded! \n",
    "(If error were returned, please read the errors and try to repeatthe previous steps.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3503463",
   "metadata": {},
   "source": [
    "Next, let's use the module! The module we created will \"only\" print the first few words of a song. But let's it.\n",
    "\n",
    "Our module is called is just like any other modules we have used before. For example, we have used `Pandas`, and `Numpy`, those are also modules. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e5c38bc",
   "metadata": {},
   "source": [
    "So, let's  take a look at our syntax! Our module is called `mymodule` and the function it contains `OceanEyes` so the call goes as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7db0c72a",
   "metadata": {},
   "outputs": [],
   "source": [
    "mymodule.OceanEyes()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3f937fe",
   "metadata": {},
   "source": [
    "Did you get it? Did you get the words from the song? If you did, congratulations you just wrote a python module.\n",
    "\n",
    "More complex modules just contains more functions, more complex functions etc. Butthe process (given what we have covered so far) can be summarized as above."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24f91f37",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - Make your module:\n",
    "       - Pick a song you like\n",
    "       - Make a new python module that is called with the first two words in the title of the song\n",
    "       - When invoked, the module should print the first phrase of the song you picked\n",
    "       - Import the module and show it works\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "009b4da4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "deb23f27",
   "metadata": {},
   "source": [
    "### More about modules"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2463b6e4",
   "metadata": {},
   "source": [
    "Note now that, the file name is also the name of the module (`mymodule`). The file name has the suffix `.py` appended, that suffix is not used in the code, when calling the module (in other words we do not `import mymodule.py` but we `import mymodule`). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9d56ecd",
   "metadata": {},
   "source": [
    "The name for modules imported in the current workspace is alwasy available as the value of the global variable __name__ (a string). \n",
    "\n",
    "We can extract themodulename into a string as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acaceac1",
   "metadata": {},
   "outputs": [],
   "source": [
    "mymodule.__name__ "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed77b928",
   "metadata": {},
   "source": [
    "Just like we have done in the past with Pandas and Numpy also our module can be imported with a different (shorter) name:|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de2fbbae",
   "metadata": {},
   "outputs": [],
   "source": [
    "import mymodule as mm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e7c8678",
   "metadata": {},
   "source": [
    "Now the function in mymodule should be called using `mm`, give it a try:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3a70ef2",
   "metadata": {},
   "outputs": [],
   "source": [
    "mm.OceanEyes()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fc60239",
   "metadata": {},
   "source": [
    "Functions inside a module can be imported directly and assigned a callable name.We have seen this before ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7e6fd92",
   "metadata": {},
   "outputs": [],
   "source": [
    "from mymodule import OceanEyes as oe "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38f925fb",
   "metadata": {},
   "source": [
    "Now we can call the function directly, avoiding the sintax `mymodule.<functionName>`. Try the following, it should work:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f91eada5",
   "metadata": {},
   "outputs": [],
   "source": [
    "oe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49e6b886",
   "metadata": {},
   "source": [
    "#### Let's break this\n",
    "\n",
    "OK now let's try something that should breakthings for us, but perhaps also help usunderstand. Move the file `mymodule.py` out of the current directly, for example, move it to your desktop instead.\n",
    "\n",
    "After doing that try importing the module again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e1c92ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "import mymodule"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d13106fb",
   "metadata": {},
   "source": [
    "Did that work? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e68b95d",
   "metadata": {},
   "source": [
    "#### Modules can import other modules. \n",
    "\n",
    "It is possible to add import operations inside a module. Say for example you want to load `Numpy` every time you load your module. You could add `import numpy as np` at the beginning of your module and the module will automatically add numpy to your current workspace as soon as you call your module. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27ba29f1",
   "metadata": {},
   "source": [
    "#### The standard modules in Python\n",
    "\n",
    "Python comes with a library of modules called standard: The [python standard modules lbrary](https://docs.python.org/3/library/). These modules are shipped with the Python3 distribution. This means that you can simply import them without saving, or moving files. The files are pythonmagically there for you.\n",
    "\n",
    "A list of standard modules can be found [here](https://docs.python.org/3/py-modindex.html). The lis "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c98200b",
   "metadata": {},
   "source": [
    "### In sum\n",
    "\n",
    "Write python modules is as easy as writing a file ending withthe `.py` exstension. The file should contain function definitions. The file could also contain variables definitions or other code statements, an aspect of modules that we have not experimented with in this tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4aa62bb0",
   "metadata": {},
   "source": [
    "## Make the best rat lab module\n",
    "\n",
    "To practice with modules we will make an exercise and make a module out of the code from a previous tutorial. \n",
    "\n",
    "Your goal will be to take these functions sve a the module and demonstrate that it runs from within this jupyter notebook "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01e8f2cd",
   "metadata": {},
   "source": [
    "First of all we will break down the code into the basic steps and make one function per step. After that, we will make a module, save it to disk and call it to use the function.\n",
    "\n",
    "Let's get started."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dabab21d",
   "metadata": {},
   "source": [
    "In a previous tutorial, we loaded data from files given to us from a lab and performed a series of operations to reorganize the data into a Tidy Data Format. In that tutorial (Tutorial 17 using 'datasets/017DataFile.csv') we performed four independent operations to reorganzie the data.\n",
    "\n",
    "  - We loaded reaction time data into a specific format.\n",
    "  \n",
    "  - We organized the labels for the strains of rats into the appropriate format for the data.\n",
    "  \n",
    "  - We organized the labels for the sexes of rats into the appropriate format for the data.\n",
    "  \n",
    "  - We combined the data and labels into a tidy format (one colum per variable/label)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ca2cdb9",
   "metadata": {},
   "source": [
    "Below we have four functions written to implement the operations described above and used in the previous tutorial. These functions can now be conveniently called multiple times within this Jupyter notebook. Yet, to call the functions in a new notebooks, or in future (many) notebooks, they must be copied and pasted into each new Jupyter notebook. Boring...\n",
    "\n",
    "Wouldn't it be easier if we could call them directly from a module? Let's do this."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c36ddf2",
   "metadata": {},
   "source": [
    "Below we first describe how we functionalized the code from the previous tutorial. We describe each function and what it does and then use them after loading the data.\n",
    "\n",
    "After that, we will open a new notebook and save it as a module. We will then repeat the data processing performed with the functions by loading the module we just created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac91ad63",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_data(filename) :\n",
    "    '''\n",
    "    get_data() \n",
    "    Loads the data from a filename.\n",
    "    Organizes the data and retunrs key data values\n",
    "    '''\n",
    "    import numpy as np\n",
    "    import pandas as pd\n",
    "    \n",
    "    my_input_data = pd.read_csv(filename)  # read the data\n",
    "\n",
    "    raw_data   = my_input_data.to_numpy()                    # convert to numpy array\n",
    "    obs, grps  = raw_data.shape                              # get the number of rows and columns\n",
    "    new_length = obs*grps                                    # compute total number of observations\n",
    "    values_col = np.reshape(raw_data, (new_length, 1), \n",
    "                            order = 'F')                     # reshape the array\n",
    "    values_col = np.squeeze(values_col)                      # squeeze to make 1D\n",
    "\n",
    "    return values_col, obs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d153bfd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_strains(obs=10, names=['wildtype', 'mutant']) :\n",
    "    '''\n",
    "    get_strains() \n",
    "    Takes names of rat types (e.g., names=['wildtype', 'mutant']) and \n",
    "    the number of observation per group (obs_per_grp=10).\n",
    "    Returns the variable `strain` containing.\n",
    "    User specifies a filename string.\n",
    "    '''\n",
    "    import pandas as pd\n",
    " \n",
    "    strain = pd.Series(names)                   # make the short series\n",
    "    strain = strain.repeat([2*obs])     # repeat each over two cell's worth of data\n",
    "    strain = strain.reset_index(drop=True)      # reset the series's index value\n",
    "    \n",
    "    return strain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38eb26d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_sexes(obs, sexLabels=['male', 'female']) :\n",
    "    '''\n",
    "    tidyMyData() Takes one-column-per-cell rat reaction time data as input.\n",
    "    Returns tidy one-column-per-variable data.\n",
    "    User specifies a filename string.\n",
    "    '''\n",
    "    import pandas as pd\n",
    " \n",
    "    sexes = pd.Series(sexLabels)                      # make the short series\n",
    "    sexes = sexes.repeat(obs)                 # repeat each over one cell's worth of data\n",
    "    sexes = pd.concat([sexes]*2, ignore_index=True)   # stack or \"concatonate\" two copies\n",
    "    \n",
    "    return sexes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fecd832",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tidy_data(values_col,strain,sexes) :\n",
    "    '''\n",
    "    tidyMyData() Takes \n",
    "    1. A one-column-per-cell rat reaction time data (values_col).\n",
    "    2. A sexes variables labelling each entry in values_col by rat-sex\n",
    "    3. A strain variable labelling entries in values_col by rat strain\n",
    "    \n",
    "    Returns one-column-per-variable data adhering to the tidy format.\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    import pandas as pd\n",
    "\n",
    "    # construct the data frame\n",
    "    my_new_tidy_data = pd.DataFrame(\n",
    "        {\n",
    "            \"RTs\": values_col,                               # make a column named RTs and put the values in\n",
    "            \"sex\": sexes,                                    # ditto for sex\n",
    "            \"strain\": strain                                 # and for genetic strain\n",
    "        }    \n",
    "    )\n",
    "    \n",
    "    return my_new_tidy_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56fc939a",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - Your goal is to make a module called `bestratlab.py` out of the above functions and to demonstrate that it can run from this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9696e9ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "import bestlabrat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e283349",
   "metadata": {},
   "source": [
    "## A note on recycling code\n",
    "\n",
    "We have learned early in our journey towards Data Science that it is convenient to keep helpful code around and recycle it. So far, we have learned of at least three ways to recycle code:\n",
    "  \n",
    "  - *Loops.* Loops facilitate reusing hundreds of operations. Loops allow repeating the same operations over and over avoiding actually copying and pasting the same lines of code.\n",
    "  \n",
    "  - *Functions.* Functions facilitate reusing hundreds of lines of code. Functions allow reusing the same lines of code for different instances ofthe same situation. \n",
    "  \n",
    "  - *Modules.* Modules allow facilitates reusing hundreds of functions. Modules provide a convenient way to save good work, functions, in an accessible file. Module files can be loaded the, or better imported in the current working python stack and that allow accessing and using the functions saved in the module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70b0ed22",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}