module-1-programming/Day1_2_Python_Packages.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Python Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why are packages useful? We can write less code and we don't need to reinvent the wheel!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "from IPython.core.display import HTML \n",
    "Image(url= \"https://i.imgflip.com/42bnn2.jpg\",width=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at this example for calculating the mean of a list of numbers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nums = [5,3,2] \n",
    "calc_mean = sum(nums)/len(nums)\n",
    "print(calc_mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```statistics``` package has a function called ```mean```. Let's import it and use the ```mean``` function to calculate the mean of the numbers in the list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from statistics import mean\n",
    "\n",
    "nums =[5,3,2] \n",
    "print(mean(nums))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Math-based packages: Numpy and random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random: Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's start by exploring the ```random``` package. The ```random```package is good for generating pseudo random float or int values and has many useful functions and applications. Let's start by importing the package:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also get a random number using the ```random()``` function. However, this will only return a float between 0.0 and 1.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.random()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like regular floats, we can perform mathematical operations on the value returned by ```random()```. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.random() * 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "beatles = ['George', 'Paul', 'Ringo', 'John']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.choice(beatles)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we'd like to get the same item each time, we can set a seed. A random seed is a starting point in generating random numbers. A random seed specifies the start point when a computer generates a random number sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.seed(22)\n",
    "random.choice(beatles)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and if we run it again...we should get the same value returned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.seed(22)\n",
    "random.choice(beatles)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Random to create pseudodata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```random``` function is very useful for generating pseudodata. Here we are generating a dataset of 100 float values randomly selelcted using the ```random``` function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#initialize a list which we will add random values to\n",
    "pseudodata = []\n",
    "\n",
    "#populating the pseudodata list with 100 float values\n",
    "for i in range(100):\n",
    "    pseudodata.append(random.random())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's print out the first 10 floats in the psuedodata list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(pseudodata[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```Numpy``` is a python package meant to support large arrays and matrices and allows for mathematical functions to be used with them. Let's start by importing the package. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a simple example of using the function ```sum()``` to find the sum of all data points within pseudodata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(pseudodata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How about the ```mean()```?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.mean(pseudodata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the variance, or ```var()```?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.var(pseudodata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we come accross a function and we're not sure what it does or how it works, we can use ```help()```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(np.sum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are times when we will want to transform our data. Some reasons may be that we want to improve the interpretability of the data or gain insight into the relationship between variables. Log10 transformation is a common tranformation method used. \n",
    "\n",
    "Let's use ```numpy```'s ```log10()``` function in order to complete a log10 transformation of the data and see the first 10 items that have been log10 transformed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.log10(pseudodata)[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Graphing packages: Seaborn and Matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Seaborn and Matplotlib are commonly used python packages for plotting graphs in python. Why do we need two packages for plotting? Matplotlib is generally used for more basic plotting such as plotting barplots, scatterplots, pie charts, etc. While both Matplotlib and Seaborn can work with lists and dataframes, Seaborn is better integrated for use with dataframes, provides a wider range of visualization patterns, and specializes in statistics visualizations, such as distribution plots.\n",
    "\n",
    "Let's start by importing seaborn as sns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by using seaborn's ```distplot()``` function to plot the distribution of our non-transformed pseudodata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.distplot(pseudodata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the moment, the plot does not have any x or y axis labels, so let's add some labels and a title using the ```set()``` function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = sns.distplot(pseudodata)\n",
    "a.set(xlabel='pseudodata', ylabel='density', title='Distribution of Pseudodata')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's start using our new superpowers of python and plotting to start doing some data visualization. Let's start by seeing what happens to the distribution if our pseudodata consists of very larger numbers (remember, right now they are all floats between 0.0 and 1.0). To do this, we will start by multiplying all of our floats in *pseudodata* by 1000000 to create a new dataset we will call *pseudodata_large*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#initialize your new list\n",
    "pseudodata_large = []\n",
    "\n",
    "#iterate through all values in pseudodata, multiplying each one by 1000000 before appending it to your new list\n",
    "for i in pseudodata:\n",
    "    pseudodata_large.append(i*1000000)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what the first 10 values of our new dataset are"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pseudodata_large[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Exercise 1:*** Now try plotting the distribution of your data. Make sure to set axes labels and a title. Has the distribution shifted and if so, how?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Exercise 2:*** When we are dealing with data with very large values, sometimes we will want to transform the data. For this exercise, log transform the pseudodata_large dataset and plot the distribution of the transformed data. Make sure to add axes labels and a title. After transforming the data, has the distribution shifted? And if so, how?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can make many different types of plots using seaborn. For example, we can make a violin plot using the ```violinplot()``` function as shown below. To see what other plots can be made with seaborn, you can read the documentation at https://seaborn.pydata.org/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = sns.violinplot(pseudodata)\n",
    "d.set(ylabel='pseudodata', title='Sample Violin Plot')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading in our own data and plotting with Matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we will be reading in our dataset which contains all of the sizes of the different chromosomes that correspond to the hg19 reference. The data file should already be available to you, but for your own reference, the table was taken from http://hgdownload.soe.ucsc.edu/goldenPath/hg19/bigZips/hg19.chrom.sizes.\n",
    "\n",
    "\n",
    "We will start by initializing two lists, x and y, to which we will append data to as we open and read through the file line-by-line using ```with open```. There are other methods you can use to open, read, and close a file in python. However, we encourage you to start with ```with open``` as it automatically takes care of more of the logistics of opening and closing file compared to other methods. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#list x will contain chromosome names\n",
    "#list y will contain the size of the chromosomes\n",
    "#Important: the index of the chromosome name and its corresponding size in lists x and y have to be the same\n",
    "x=[]\n",
    "y=[]\n",
    "\n",
    "#opens the file\n",
    "with open ('data/hg19.chrom.sizes.txt') as chromSize:\n",
    "    for line in chromSize:# The for loop allows you to read the file line-by-line\n",
    "        print('line:', line) #what does the line contain\n",
    "        x_value = line.split('\\t')[0].rstrip()\n",
    "        y_value = int(line.split('\\t')[1].strip('\\n').rstrip())\n",
    "        print('x value:', x_value, 'y value:', y_value) #the x and y values we are appending to our list\n",
    "        x.append(x_value)\n",
    "        y.append(y_value)\n",
    "  \n",
    "#just printing the first five values of lists x and y\n",
    "print(x[:5])\n",
    "print(y[:5])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our data stored as lists x and y. Let's get ready to plot our data using matplotlib! Start by importing matplotlib's pyplot as plt. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make a simple barplot using the ```bar()``` function and our lists, x and y, that contain our data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.bar(x,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! We have created a plot...but it's a bit of a mess. Let's see if we can clean it up by making it larger, rotating the x-axis tick labels, adding axes labels, and a title. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20,5))\n",
    "plt.bar(x, y)\n",
    "plt.xticks(x, rotation='vertical')\n",
    "plt.ylabel('Chromosome Length (bp)')\n",
    "plt.xlabel('Chromosome Name')\n",
    "plt.title('Length of Hg19 Chromosomes')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Brief Introduction to Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import the pandas package. But nickname it 'pd' for short."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read a CSV (or TSV or anything!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we are reading in a .tsv where the first column is \"geneSymbol\" and the second column is \"chromosome\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to save it to a variable so that we can manipulate it in other ways and CHECK IT OUT!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gene_chrom_table = pd.read_csv('data/gene_chrom.tsv', sep='\\t')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check out your data! What are we working with here?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can sneak a peek at just the first few rows..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gene_chrom_table.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gene_chrom_table.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or the last few rows..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gene_chrom_table.tail(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get a sense of the \"shape\" of the data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gene_chrom_table.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function shape:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(gene_chrom_table.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(gene_chrom_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How about indexing into a row? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gene_chrom_table['geneSymbol']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[gene_chrom_table.geneSymbol == 'TP53']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gene_chrom_table[gene_chrom_table.geneSymbol == 'TP53']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gene_chrom_table.loc[11939]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gene_chrom_table[gene_chrom_table.geneSymbol.isin(['TP53', 'BRCA1'])]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}