module-1-programming/FULL - Terminal_Commands_and_Bash_Scripting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 style = \"font-size: 35px\">Terminal Commands and Bash Scripting v2</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is an extended version of the Unix module lecture with extra content that we didn't have time to teach. You will **not** be evaluated on your knowledge of this material, but it contains lots of tidbits that we guarantee you will find useful in your research or day-to-day."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Anatomy of a Command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/command_anatomy.jpg\" width='750px'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# File System Navigation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `pwd`: where am I?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the path to my current location (aka working directory) in the file system?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/amassara/module-1-programming\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first slash in the path refers to the root directory, the folder that contains all of the other folders on your computer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `whoami`: who am I logged in as?\n",
    "To continue the existential questions...\n",
    "\n",
    "<img src=\"../module-1-programming/img/whoami.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Who am I logged in as? What is my username?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "amassara\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "whoami"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `tree`: show the file system structure in a tree format\n",
    "```bash\n",
    "tree ~/module-1-programming\n",
    "```\n",
    "```\n",
    "~/module-1-programming\n",
    "├── Day1_1_Intro_to_Programming.ipynb\n",
    "├── Day1_2_Python_Packages.ipynb\n",
    "├── Day1_3_Intro_to_Pandas.ipynb\n",
    "├── Day2_1_R_Basics.ipynb\n",
    "├── Day2_2_Terminal_Commands_and_Bash_Scripting.ipynb\n",
    "├── README.md\n",
    "├── bash_playground\n",
    "│   ├── scripts\n",
    "│   │   ├── hello.sh\n",
    "│   │   ├── hello_to_all_of_you.sh\n",
    "│   │   └── hello_you.sh\n",
    "│   └── sequences.txt\n",
    "├── data\n",
    "│   ├── chrom_lengths.tsv\n",
    "│   ├── gene_chrom.tsv\n",
    "│   └── hg19.chrom.sizes.txt\n",
    "└── img\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `cd`: **c**hange my working **d**irectory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `cd` command to navigate to a directory, given the path to that directory:\n",
    "```bash\n",
    "cd [path_to_dir]\n",
    "```\n",
    "Without any arguments, `cd` will go to your home directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now where are we?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/amassara\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually, you'll want to give `cd` the path to a directory, though. For example, let's go to the `bash_playground/` folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cd module-1-programming/bash_playground"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can go back *up* a level with the \"..\" notation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/amassara/module-1-programming\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cd ..\n",
    "pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And you can go back to your last directory by using `-` as the directory path. This will also print the new working directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/amassara/module-1-programming/bash_playground\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cd -"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `ls`: **l**i**s**t contents of a directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `ls` command to print the contents of a directory, given the path to that directory:\n",
    "```bash\n",
    "ls [path_to_dir_or_file]...\n",
    "```\n",
    "Without any arguments, `ls` will show you the contents of your current working directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scripts  sequences.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But we can also give it the path to a file or directory (or multiple files or directories)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scripts/hello.sh  scripts/hello_you.sh\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls scripts/hello.sh scripts/hello_you.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding flags get us more information. Let's use the short-format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r----- 1 amassara root 82 Jan  5 05:23 scripts/hello.sh\n",
      "-rw-r----- 1 amassara root 38 Jan  5 05:23 scripts/hello_you.sh\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls -lh scripts/hello.sh scripts/hello_you.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `-a` flag to show _all_ files, including hidden ones, prefixed with a single dot `.`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".  ..  scripts\tsequences.txt  .you_found_me.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls -a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like with `cd`, we can also specify the path to the directory above our current one using the \"..\" notation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " bash_playground   Day1_4_Terminal_Commands_and_Bash_Scripting.ipynb    img\n",
      " data\t\t  'FULL - Terminal_Commands_and_Bash_Scripting.ipynb'\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls .."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In any path, the tilde `~` represents our home directory, so we can view the files in that directory like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cmm262-2023  module-1-programming  private  public\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls ~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tilde actually gets replaced with the path to our home directory prior to us executing the command. You can see this by putting an `echo` in front of the command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ls /home/amassara\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "echo ls ~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Paths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Relative vs absolute paths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two types of paths:\n",
    "1. relative - interpreted relative to your current working directory\n",
    "2. absolute - independent of your current working directory (ie relative to the root directory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So these paths refer to the same file:\n",
    "- `scripts/hello.sh` (relative)\n",
    "- `~/module-1-programming/bash_playground/scripts/hello.sh` (absolute)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### When should I use a relative or absolute path?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whether it's best to use a relative or absolute path will depend on the context.\n",
    "\n",
    "If we changed our working directory from `bash_playground/`, then the first path would break but the second one wouldn't.\n",
    "\n",
    "But if we changed the location of the `module-1-programming/` directory, then the second path would be the one to break, instead. The first path wouldn't break, assuming our working directory is still `bash_playground/`!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Different ways of writing a path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are all equivalent paths!\n",
    "- `scripts/hello.sh`\n",
    "- `../bash_playground/scripts/hello.sh`\n",
    "- `scripts/../scripts/hello.sh`\n",
    "- `scripts/./hello.sh`\n",
    "\n",
    "The double-dot `..` refers to the folder above and the single-dot `.` refers to the current folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Understanding help and manual pages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most commands have usage documentation available via `--help` (long format) **and/or** `-h` (short format)\n",
    "```bash\n",
    "cmd --help\n",
    "```\n",
    "```bash\n",
    "cmd -h\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, some commands will also register their help via the built-in manual pages in the `man` command\n",
    "```bash\n",
    "man cmd\n",
    "```\n",
    "(press **q** to quit or exit-out after opening the manual)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An example manual page: `man ls`\n",
    "<img src=\"img/man_ls.png\" width='800px'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wildcards and globbing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Question marks `?` and asterisks `*` are considered _wildcards_.\n",
    "\n",
    "You can use a question-mark `?` in place of any character. Bash will find all of the files that match. This is called _globbing_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sequences.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls ?equences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use an asterisk `*` when you want to match zero or more characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sequences.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls *es.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is useful when you want to refer to a bunch of similarly named files simultaneously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scripts/hello.sh  scripts/hello_to_all_of_you.sh  scripts/hello_you.sh\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls scripts/*.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wildcards get expanded to a space-separated list prior to the command executing. A great way to see this is to prepend the command with an `echo`.\n",
    "\n",
    "We'll discuss the `echo` command in more depth later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ls scripts/hello.sh scripts/hello_to_all_of_you.sh scripts/hello_you.sh\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "echo ls scripts/*.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tab completion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While typing a command, you can press `<tab>` to use the terminal's auto-completions. This helps save you time and reduces the chance you'll make typos while you type. So you should make an effort to use tab completion as often as possible.\n",
    "\n",
    "For example, let's tab-complete the `scripts/` directory.\n",
    "```bash\n",
    "ls sc<tab>\n",
    "```\n",
    "## double tap when there are multiple options\n",
    "If there are multiple possible ways to complete your command, nothing will appear at first.\n",
    "```bash\n",
    "ls scripts/hello<tab>\n",
    "```\n",
    "Press `<tab>` twice in quick succession to display the possible options.\n",
    "```bash\n",
    "ls scripts/hello<tab><tab>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making use of your bash history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the up and down arrow keys to cycle through your past commands.\n",
    "\n",
    "To view all of your past commands, use the `history` command.\n",
    "\n",
    "**Caution**: this might spit out a lot of text!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    1  exit\n",
      "    2  PS1='[PEXP\\[\\]ECT_PROMPT>' PS2='[PEXP\\[\\]ECT_PROMPT+' PROMPT_COMMAND=''\n",
      "    3  export PAGER=cat\n",
      "    4  display () {     TMPFILE=$(mktemp ${TMPDIR-/tmp}/bash_kernel.XXXXXXXXXX);     cat > $TMPFILE;     echo \"bash_kernel: saved image data to: $TMPFILE\" >&2; }\n",
      "    5  PS1='[PEXP\\[\\]ECT_PROMPT>' PS2='[PEXP\\[\\]ECT_PROMPT+' PROMPT_COMMAND=''\n",
      "    6  export PAGER=cat\n",
      "    7  display () {     TMPFILE=$(mktemp ${TMPDIR-/tmp}/bash_kernel.XXXXXXXXXX);     cat > $TMPFILE;     echo \"bash_kernel: saved image data to: $TMPFILE\" >&2; }\n",
      "    8  pwd\n",
      "    9  echo $?\n",
      "   10  pwd\n",
      "   11  echo $?\n",
      "   12  whoami\n",
      "   13  echo $?\n",
      "   14  cd\n",
      "   15  echo $?\n",
      "   16  pwd\n",
      "   17  echo $?\n",
      "   18  cd module-1-programming/bash_playground\n",
      "   19  echo $?\n",
      "   20  cd ..\n",
      "   21  pwd\n",
      "   22  echo $?\n",
      "   23  cd -\n",
      "   24  echo $?\n",
      "   25  ls\n",
      "   26  echo $?\n",
      "   27  ls scripts/hello.sh scripts/hello_you.sh\n",
      "   28  echo $?\n",
      "   29  ls -lh scripts/hello.sh scripts/hello_you.sh\n",
      "   30  echo $?\n",
      "   31  ls -a\n",
      "   32  echo $?\n",
      "   33  ls ..\n",
      "   34  echo $?\n",
      "   35  ls ~\n",
      "   36  echo $?\n",
      "   37  echo ls ~\n",
      "   38  echo $?\n",
      "   39  ls ?equences.txt\n",
      "   40  echo $?\n",
      "   41  ls *es.txt\n",
      "   42  echo $?\n",
      "   43  ls scripts/*.sh\n",
      "   44  echo $?\n",
      "   45  echo ls scripts/*.sh\n",
      "   46  echo $?\n",
      "   47  history\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To search for a past command within the history:\n",
    "1. press _ctrl+r_\n",
    "2. start typing part of the command\n",
    "    The top ?hit will appear as you type\n",
    "3. To move onto the next best hit, press _ctrl+r_ again\n",
    "4. Once you've found the command, use _esc_ or the arrow keys (or press _enter_ if you want to execute the command immediately)\n",
    "\n",
    "**You try**: Search for the last `echo` command we executed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Altering the file system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `mkdir`: **m**a**k**e a new **dir**ectory\n",
    "```bash\n",
    "mkdir path_to_dir...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mkdir shiny_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look, it's there!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scripts  sequences.txt\tshiny_new\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `cp`: **c**o**p**y a file or directory\n",
    "```bash\n",
    "cp path_to_source... path_to_dest\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sequences_copy.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cp sequences.txt shiny_new/sequences_copy.txt\n",
    "ls shiny_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `cp -r` to copy a directory and all of its contents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "same_old  scripts  sequences.txt  shiny_new\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cp -r shiny_new same_old\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should now have two new directories in our bash playground: `same_old/` and `shiny_new/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sequences_copy.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls same_old"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And `shiny_new/` should contain the same contents as `same_old/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sequences_copy.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls shiny_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `touch`: create a new, empty file\n",
    "```bash\n",
    "touch path_to_file...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "touch empty_inside.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Don't read into that name too much. Just trust me, it's empty. Look at the size. Or go check it yourself later!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 13K\n",
      "-rw-rw---- 1 amassara root   0 Jan 10 05:44 empty_inside.txt\n",
      "drwxrwx--- 2 amassara root   3 Jan 10 05:44 same_old\n",
      "drwxr-x--- 2 amassara root   5 Jan  5 05:23 scripts\n",
      "-rw-r----- 1 amassara root 840 Jan  5 05:23 sequences.txt\n",
      "drwxrwx--- 2 amassara root   3 Jan 10 05:44 shiny_new\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls -lh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `mv`: **m**o**v**e a file or directory\n",
    "```bash\n",
    "mv path_to_source... path_to_dest\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's move `empty_inside.txt` to the `shiny_new/` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "same_old  scripts  sequences.txt  shiny_new\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mv empty_inside.txt shiny_new/\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check: is it there?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "empty_inside.txt  sequences_copy.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls shiny_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### renaming a file\n",
    "Just use `mv` where the destination is in the same folder!\n",
    "\n",
    "Let's convert `empty_inside.txt` to `fulfilled.txt`. Hopefully, you'll be the same way after this lesson ;)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "same_old  scripts  sequences.txt  shiny_new\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mv shiny_new/empty_inside.txt shiny_new/fulfilled.txt\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `rm`: **r**e**m**ove a file or directory\n",
    "```bash\n",
    "rm path_to_file_or_dir...\n",
    "```\n",
    "With great power comes great responsibility... use this carefully! There is no \"undo\" button or trash can in the terminal!\n",
    "\n",
    "Let's `echo` the command first to make sure that the wildcards are expanding correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rm shiny_new/fulfilled.txt shiny_new/sequences_copy.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "echo rm shiny_new/*.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004h\u001b[?2004l"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "rm shiny_new/*.txt\n",
    "ls shiny_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `rm -r`: delete a directory and all of its contents, **r**ecursively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scripts  sequences.txt\tshiny_new\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "rm -r same_old\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `rmdir`: **r**e**m**ove an empty **dir**ectory\n",
    "```bash\n",
    "rmdir path_to_empty_dir...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scripts  sequences.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "rmdir shiny_new\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `ln`: create a symlink to a file or directory\n",
    "```bash\n",
    "ln -s path_to_target... path_to_link\n",
    "```\n",
    "A symlink is a shortcut to a file or directory. This can be helpful when you want to have a file in multiple locations but don't want to copy it.\n",
    "\n",
    "Recall the `data/` directory in the folder above `bash_playground/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chrom_lengths.tsv  gene_chrom.tsv  hg19.chrom.sizes.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls ../data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a symlink to the `data/` directory from within the `bash_playground/` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data-sym  scripts  sequences.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ln -s ../data data-sym\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A symlink is a special file that just references another file (or directory). You can view the path of a symlink using `ls -l`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lrwxrwxrwx 1 amassara root 7 Jan 10 05:44 data-sym -> ../data\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls -l data-sym"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Symlinks will act exactly the same as their original file or directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chrom_lengths.tsv  gene_chrom.tsv  hg19.chrom.sizes.txt\n",
      "\u001b[?2004h\u001b[?2004l"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls data-sym\n",
    "cd data-sym"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chrom_lengths.tsv  gene_chrom.tsv  hg19.chrom.sizes.txt\n",
      "\u001b[?2004h\u001b[?2004l"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls\n",
    "cd .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scripts  sequences.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "rm data-sym\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deleting a symlink doesn't delete the target it points to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " bash_playground   Day1_4_Terminal_Commands_and_Bash_Scripting.ipynb    img\n",
      " data\t\t  'FULL - Terminal_Commands_and_Bash_Scripting.ipynb'\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls .."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try this on your own:\n",
    "1. Create an empty directory called `grad_school`\n",
    "2. Create an empty file in that directory and name it with your name\n",
    "3. Create a symlink to that file in the current directory called `linked_me`\n",
    "4. Delete the directory you created\n",
    "5. Delete the symlink"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004h\u001b[?2004l\u001b[?2004l\u001b[?2004l\u001b[?2004l"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mkdir grad_school\n",
    "touch grad_school/myname\n",
    "ln -s grad_school/myname linked_me\n",
    "rm -r grad_school\n",
    "rm linked_me"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Printing things"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `cat`: print the contents of files\n",
    "```bash\n",
    "cat path_to_file...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seqA\tCCCTAACCCCCTAACCCCCTAACCCTCAGTCGGGGAGGCGACAATAGCTG\n",
      "seqB\tGTCATATGTTCTGTACGTTATTGGCCAACTGATCATACCTGAATCGAGCC\n",
      "seqC\tGAACCGGGATTATCAAAGACGAACATGGTCGGGTCCTTGAACCAAACGAA\n",
      "seqE\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqD\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqF\tTTAGTTGCAAATAGGCTTACCTAGTAGAGGTCCGACAACCCCTACACGTC\n",
      "seqG\tCACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "seqH\tCAGAACGCACTTTCCGGCAGTGAATGTCGATGCCAACATTGCCTAAAACA\n",
      "seqI\tCACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "seqJ\tTGCGGGGCGGCGATATGCGGCAACGATGGCCGCGAGTTAAATGAGCCATA\n",
      "seqK\tCTTTGTGGGGTAATCTAGAAAGCAGAGTACGTATTTGGCGACCCGACATT\n",
      "seqL\tCTTATTTGCTCTCCCGGCTGTAGTGTGCACAGACTTCCAGATGTAATAAG\n",
      "seqM\tCTTTCTCTATTTACCGAGCAATGAAAACCACGGGGAACAATCCACCCTTT\n",
      "seqN\tGTCAAATGGCCGGCCCACTTCTCTCTTCGACACGGCGTCGGGCAGTCTGG\n",
      "seqO\tATGGTAAGACTAGCCAGCAGATCCTTAATCGGCTTTCCCGAAGGACAGTA\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cat sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you provide more than one file, `cat` will _concatenate_ them (aka append them to each other). This is where it gets its name."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember the hidden secret file? Let's take a look at it now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1[?2004l\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n",
      "6\n",
      "7\n",
      "8\n",
      "9\n",
      "10\n",
      "11\n",
      "12\n",
      "13\n",
      "14\n",
      "15\n",
      "I am secret text from a hidden file\n",
      "Here's chr21:\n",
      "https://hgdownload.soe.ucsc.edu/goldenPath/hg38/chromosomes/chr21.fa.gz\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cat .you_found_me.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now, let's concatenate the two files together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1[?2004l\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n",
      "6\n",
      "7\n",
      "8\n",
      "9\n",
      "10\n",
      "11\n",
      "12\n",
      "13\n",
      "14\n",
      "15\n",
      "I am secret text from a hidden file\n",
      "Here's chr21:\n",
      "https://hgdownload.soe.ucsc.edu/goldenPath/hg38/chromosomes/chr21.fa.gz\n",
      "seqA\tCCCTAACCCCCTAACCCCCTAACCCTCAGTCGGGGAGGCGACAATAGCTG\n",
      "seqB\tGTCATATGTTCTGTACGTTATTGGCCAACTGATCATACCTGAATCGAGCC\n",
      "seqC\tGAACCGGGATTATCAAAGACGAACATGGTCGGGTCCTTGAACCAAACGAA\n",
      "seqE\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqD\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqF\tTTAGTTGCAAATAGGCTTACCTAGTAGAGGTCCGACAACCCCTACACGTC\n",
      "seqG\tCACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "seqH\tCAGAACGCACTTTCCGGCAGTGAATGTCGATGCCAACATTGCCTAAAACA\n",
      "seqI\tCACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "seqJ\tTGCGGGGCGGCGATATGCGGCAACGATGGCCGCGAGTTAAATGAGCCATA\n",
      "seqK\tCTTTGTGGGGTAATCTAGAAAGCAGAGTACGTATTTGGCGACCCGACATT\n",
      "seqL\tCTTATTTGCTCTCCCGGCTGTAGTGTGCACAGACTTCCAGATGTAATAAG\n",
      "seqM\tCTTTCTCTATTTACCGAGCAATGAAAACCACGGGGAACAATCCACCCTTT\n",
      "seqN\tGTCAAATGGCCGGCCCACTTCTCTCTTCGACACGGCGTCGGGCAGTCTGG\n",
      "seqO\tATGGTAAGACTAGCCAGCAGATCCTTAATCGGCTTTCCCGAAGGACAGTA\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cat .you_found_me.txt sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `wc`: count the number of lines or words in a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "wc [path_to_file]...\n",
    "```\n",
    "You can use the `wc` command to obtain counts of the lines, words, or characters (respectively) in a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 15  30 840 sequences.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "wc sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use flags to get just the line count (`-l`), just the word count (`-w`), or just the character count (`-c`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15 sequences.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "wc -l sequences.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30 sequences.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "wc -w sequences.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "840 sequences.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "wc -c sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `head`: print the top of a file\n",
    "```bash\n",
    "head [path_to_file]...\n",
    "```\n",
    "By default, `head` will print the first 10 lines of a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seqA\tCCCTAACCCCCTAACCCCCTAACCCTCAGTCGGGGAGGCGACAATAGCTG\n",
      "seqB\tGTCATATGTTCTGTACGTTATTGGCCAACTGATCATACCTGAATCGAGCC\n",
      "seqC\tGAACCGGGATTATCAAAGACGAACATGGTCGGGTCCTTGAACCAAACGAA\n",
      "seqE\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqD\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqF\tTTAGTTGCAAATAGGCTTACCTAGTAGAGGTCCGACAACCCCTACACGTC\n",
      "seqG\tCACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "seqH\tCAGAACGCACTTTCCGGCAGTGAATGTCGATGCCAACATTGCCTAAAACA\n",
      "seqI\tCACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "seqJ\tTGCGGGGCGGCGATATGCGGCAACGATGGCCGCGAGTTAAATGAGCCATA\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "head sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But you can also specify the number of lines to the `-n` flag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seqA\tCCCTAACCCCCTAACCCCCTAACCCTCAGTCGGGGAGGCGACAATAGCTG\n",
      "seqB\tGTCATATGTTCTGTACGTTATTGGCCAACTGATCATACCTGAATCGAGCC\n",
      "seqC\tGAACCGGGATTATCAAAGACGAACATGGTCGGGTCCTTGAACCAAACGAA\n",
      "seqE\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqD\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "head -n 5 sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `tail`: print the end of a file\n",
    "```bash\n",
    "tail [path_to_file]...\n",
    "```\n",
    "You can use `tail` in the exact same way as `head`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seqF\tTTAGTTGCAAATAGGCTTACCTAGTAGAGGTCCGACAACCCCTACACGTC\n",
      "seqG\tCACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "seqH\tCAGAACGCACTTTCCGGCAGTGAATGTCGATGCCAACATTGCCTAAAACA\n",
      "seqI\tCACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "seqJ\tTGCGGGGCGGCGATATGCGGCAACGATGGCCGCGAGTTAAATGAGCCATA\n",
      "seqK\tCTTTGTGGGGTAATCTAGAAAGCAGAGTACGTATTTGGCGACCCGACATT\n",
      "seqL\tCTTATTTGCTCTCCCGGCTGTAGTGTGCACAGACTTCCAGATGTAATAAG\n",
      "seqM\tCTTTCTCTATTTACCGAGCAATGAAAACCACGGGGAACAATCCACCCTTT\n",
      "seqN\tGTCAAATGGCCGGCCCACTTCTCTCTTCGACACGGCGTCGGGCAGTCTGG\n",
      "seqO\tATGGTAAGACTAGCCAGCAGATCCTTAATCGGCTTTCCCGAAGGACAGTA\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "tail sequences.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seqK\tCTTTGTGGGGTAATCTAGAAAGCAGAGTACGTATTTGGCGACCCGACATT\n",
      "seqL\tCTTATTTGCTCTCCCGGCTGTAGTGTGCACAGACTTCCAGATGTAATAAG\n",
      "seqM\tCTTTCTCTATTTACCGAGCAATGAAAACCACGGGGAACAATCCACCCTTT\n",
      "seqN\tGTCAAATGGCCGGCCCACTTCTCTCTTCGACACGGCGTCGGGCAGTCTGG\n",
      "seqO\tATGGTAAGACTAGCCAGCAGATCCTTAATCGGCTTTCCCGAAGGACAGTA\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "tail -n 5 sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `echo`: print a string\n",
    "```bash\n",
    "echo [string]...\n",
    "```\n",
    "You can use the `echo` command to print a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hey2004l\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "echo hey"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `echo` command accepts multiple string arguments, so we can also do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "is anybody out there?\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "echo is anybody out there?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Under the hood: standard input, output, and error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Every command has three standard files: standard input (stdin), standard output (stdout), and standard error (stderr).\n",
    "\n",
    "When you run a command, its output will be written to stdout and any error messages will be written to stderr.\n",
    "\n",
    "<img src=\"img/stdin_out_err.png\" width='450px'>\n",
    "\n",
    "By default, stdout and stderr are connected to your terminal, which is why you can see them on your screen.\n",
    "\n",
    "Most commands will not read from stdin unless they detect that they should, or you explicitly request them to. More on that later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Redirecting stdout and stderr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can redirect the stdout and stderr from the terminal to custom files. Standard output is represented by the number 1 and standard error by the number 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ls: cannot access 'nonexistent_file': No such file or directory\n",
      "scripts/:\n",
      "hello.sh  hello_to_all_of_you.sh  hello_you.sh\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls scripts/ nonexistent_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls scripts/ nonexistent_file 1>file_out 2>file_err"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scripts/:\n",
      "hello.sh\n",
      "hello_to_all_of_you.sh\n",
      "hello_you.sh\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cat file_out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ls: cannot access 'nonexistent_file': No such file or directory\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cat file_err"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you omit the number prior to the redirection operator `>`, it will default to stdout (the number 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls scripts/ nonexistent_file >file_out_new 2>file_err_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scripts/:\n",
      "hello.sh\n",
      "hello_to_all_of_you.sh\n",
      "hello_you.sh\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cat file_out_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ls: cannot access 'nonexistent_file': No such file or directory\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cat file_err_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can redirect both stdout and stderr to the same file using the ampersand `&`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ls: cannot access 'nonexistent_file': No such file or directory\n",
      "scripts/:\n",
      "hello.sh\n",
      "hello_to_all_of_you.sh\n",
      "hello_you.sh\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls scripts/ nonexistent_file &>file_both\n",
    "cat file_both"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whichever redirection you leave off will default to the terminal, again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ls: cannot access 'nonexistent_file': No such file or directory\n",
      "scripts/:[?2004l\n",
      "hello.sh\n",
      "hello_to_all_of_you.sh\n",
      "hello_you.sh\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls scripts/ nonexistent_file >file_out\n",
    "cat file_out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Appending instead of overwriting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using a single redirection operator `>` will overwrite the contents of the target file. Let's overwrite the contents of `file_out`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  147  echo $?4l\n",
      "  148  cat file_err_new\n",
      "  149  echo $?\n",
      "  150  ls scripts/ nonexistent_file &>file_both\n",
      "  151  cat file_both\n",
      "  152  echo $?\n",
      "  153  ls scripts/ nonexistent_file >file_out\n",
      "  154  cat file_out\n",
      "  155  echo $?\n",
      "  156  history >file_out\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "history >file_out\n",
    "tail file_out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can append to the file (ie add to the end of it) instead of overwriting it by using two redirection operators in sequence `>>`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  154  cat file_out\n",
      "  155  echo $?\n",
      "  156  history >file_out\n",
      "file_both\n",
      "file_err\n",
      "file_err_new\n",
      "file_out\n",
      "file_out_new\n",
      "scripts\n",
      "sequences.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls >>file_out\n",
    "tail file_out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we move onto the next section, let's just clean up after ourselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rm file_both file_err file_err_new file_out file_out_new\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "echo rm file_*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "rm file_*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Piping from stdout to stdin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/mariopipe.png\">\n",
    "\n",
    "You can use the pipe operator **|** to connect the stdout of one command to the stdin of another.\n",
    "\n",
    "<img src=\"img/piping_stdout_stdin.png\" width='675px'>\n",
    "\n",
    "Without any arguments, most commands (including `tail`) will assume that the file should be read from stdin, instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  157  tail file_out\n",
      "  158  echo $?\n",
      "  159  ls >>file_out\n",
      "  160  tail file_out\n",
      "  161  echo $?\n",
      "  162  echo rm file_*\n",
      "  163  echo $?\n",
      "  164  rm file_*\n",
      "  165  echo $?\n",
      "  166  history | tail\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "history | tail"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can chain a series of pipes to create a _pipeline_!\n",
    "\n",
    "Let's try to get the 10 most recent commands from our history after excluding the 5 most recent ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  159  ls >>file_out\n",
      "  160  tail file_out\n",
      "  161  echo $?\n",
      "  162  echo rm file_*\n",
      "  163  echo $?\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "history | tail | head -n5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Piped commands run in parallel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A common misconception is that commands in a pipeline run sequentially. In fact, they actually run in parallel (at the same time)!\n",
    "\n",
    "Each command waits until it receives a line of text from its stdin. It processes this line and then writes to its stdout as soon as it can. In this way, the original text is said to be _streamed_ through the pipeline. Instead of waiting for each step to finish before moving onto the next step, we are running the commands at the same time!\n",
    "\n",
    "You can think of a chain of piped commands as a bunch of trains, each with their own engine, traversing on the same railroad track. Each train can go as fast or as slow as it wants, but none can overtake the other.\n",
    "\n",
    "The ability to _stream_ text makes command pipelines one of the fastest methods to process big data, and thus, a favorite among bioinformaticians."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to storing text within files, you can also store text within variables.\n",
    "\n",
    "Let's try storing the string \"hello\" inside a variable named `my_variable`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "my_variable=hello"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use the variable within another command, we must prefix the variable name with a dollar-sign `$`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello04l\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "echo $my_variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concatenating (joining) variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can join variables together by just putting them next to each other. Let's create two new variables and join them for the third."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "helloworld?2004l\u001b[?2004l\u001b[?2004l\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "hello_var=hello\n",
    "world_var=world\n",
    "my_variable=$hello_var$world_var\n",
    "echo $my_variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What would it take for us to get a space between \"hello\" and \"world\"? First, let's try this and see what happens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bash: world: command not found\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "my_variable=hello world"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It doesn't work! But why not?\n",
    "\n",
    "Spaces are important in bash. They're used to separate arguments from each other and delineate strings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use quotes when there are spaces involved"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we just need a way to denote that both words are part of a single string. Quotes to the rescue!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "my_variable=\"hello world\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Returning to our original example, it should be clear why we can just do something like this, now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello world2004l\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "my_variable=\"$hello_var $world_var\"\n",
    "echo $my_variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about if we want to put an underscore between the two variables, instead?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "world04h\u001b[?2004l\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "my_variable=$hello_var_$world_var\n",
    "echo $my_variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It doesn't work! But why not?\n",
    "\n",
    "We're telling bash to look for a variable called `hello_var_`. But since this variable doesn't exist, bash will replace it with an empty string `\"\"`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use braces to avoid ambiguity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we just need a way to denote where the variable is. The solution is to surround the variable name in braces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello_world2004l\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "my_variable=${hello_var}_$world_var\n",
    "echo $my_variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use single quotes when you want to be interpreted literally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004h\u001b[?2004l\r",
      "$hello_var $world_var\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "my_variable='$hello_var $world_var'\n",
    "echo $my_variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Within double quotes, expressions with $ are evaluated prior to the command executing. But within single quotes, everything is kept as it is.\n",
    "\n",
    "Once again, we can prefix the command with an `echo` to see the resolved command before it gets executed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "my_variable=hello world\n",
      "my_variable=$hello_var $world_var\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "echo my_variable=\"$hello_var $world_var\"\n",
    "echo my_variable='$hello_var $world_var'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storing the stdout of a command in a variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can store the stdout of a command in a variable by wrapping the command in parantheses.\n",
    "```bash\n",
    "my_variable=$(cmd)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seqA CCCTAACCCCCTAACCCCCTAACCCTCAGTCGGGGAGGCGACAATAGCTG\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "my_variable=$(head -n1 sequences.txt)\n",
    "echo $my_variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get fancy and actually try to manipulate some of these files.\n",
    "\n",
    "Note: all of these commands will also read from stdin if they aren't given a file argument."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `cut`: extract specific columns from a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "cut [path_to_file]...\n",
    "```\n",
    "To get a column of the file, provide the column number (ex: 2) to the `-f` flag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seqA\tCCCTAACCCCCTAACCCCCTAACCCTCAGTCGGGGAGGCGACAATAGCTG\n",
      "seqB\tGTCATATGTTCTGTACGTTATTGGCCAACTGATCATACCTGAATCGAGCC\n",
      "seqC\tGAACCGGGATTATCAAAGACGAACATGGTCGGGTCCTTGAACCAAACGAA\n",
      "seqE\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqD\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqF\tTTAGTTGCAAATAGGCTTACCTAGTAGAGGTCCGACAACCCCTACACGTC\n",
      "seqG\tCACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "seqH\tCAGAACGCACTTTCCGGCAGTGAATGTCGATGCCAACATTGCCTAAAACA\n",
      "seqI\tCACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "seqJ\tTGCGGGGCGGCGATATGCGGCAACGATGGCCGCGAGTTAAATGAGCCATA\n",
      "seqK\tCTTTGTGGGGTAATCTAGAAAGCAGAGTACGTATTTGGCGACCCGACATT\n",
      "seqL\tCTTATTTGCTCTCCCGGCTGTAGTGTGCACAGACTTCCAGATGTAATAAG\n",
      "seqM\tCTTTCTCTATTTACCGAGCAATGAAAACCACGGGGAACAATCCACCCTTT\n",
      "seqN\tGTCAAATGGCCGGCCCACTTCTCTCTTCGACACGGCGTCGGGCAGTCTGG\n",
      "seqO\tATGGTAAGACTAGCCAGCAGATCCTTAATCGGCTTTCCCGAAGGACAGTA\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cat sequences.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CCCTAACCCCCTAACCCCCTAACCCTCAGTCGGGGAGGCGACAATAGCTG\n",
      "GTCATATGTTCTGTACGTTATTGGCCAACTGATCATACCTGAATCGAGCC\n",
      "GAACCGGGATTATCAAAGACGAACATGGTCGGGTCCTTGAACCAAACGAA\n",
      "TCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "TCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "TTAGTTGCAAATAGGCTTACCTAGTAGAGGTCCGACAACCCCTACACGTC\n",
      "CACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "CAGAACGCACTTTCCGGCAGTGAATGTCGATGCCAACATTGCCTAAAACA\n",
      "CACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "TGCGGGGCGGCGATATGCGGCAACGATGGCCGCGAGTTAAATGAGCCATA\n",
      "CTTTGTGGGGTAATCTAGAAAGCAGAGTACGTATTTGGCGACCCGACATT\n",
      "CTTATTTGCTCTCCCGGCTGTAGTGTGCACAGACTTCCAGATGTAATAAG\n",
      "CTTTCTCTATTTACCGAGCAATGAAAACCACGGGGAACAATCCACCCTTT\n",
      "GTCAAATGGCCGGCCCACTTCTCTCTTCGACACGGCGTCGGGCAGTCTGG\n",
      "ATGGTAAGACTAGCCAGCAGATCCTTAATCGGCTTTCCCGAAGGACAGTA\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cut -f 2 sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `tr`: substitution of characters in a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "tr character_set1 character_set2\n",
    "```\n",
    "You can use the `tr` command to replace all instances of a T with a U. Let's turn our DNA sequences into RNA!\n",
    "\n",
    "Note that the `tr` command doesn't have a file argument. It only accepts input from stdin. We can attach a file to stdin using the `<` operator (rather than `>` for stdout)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seqA\tCCCUAACCCCCUAACCCCCUAACCCUCAGUCGGGGAGGCGACAAUAGCUG\n",
      "seqB\tGUCAUAUGUUCUGUACGUUAUUGGCCAACUGAUCAUACCUGAAUCGAGCC\n",
      "seqC\tGAACCGGGAUUAUCAAAGACGAACAUGGUCGGGUCCUUGAACCAAACGAA\n",
      "seqE\tUCUCCGUCCGCUGGCGUGUUUUUCUUUUCUCAAGUGGGCAAGUUACCCGG\n",
      "seqD\tUCUCCGUCCGCUGGCGUGUUUUUCUUUUCUCAAGUGGGCAAGUUACCCGG\n",
      "seqF\tUUAGUUGCAAAUAGGCUUACCUAGUAGAGGUCCGACAACCCCUACACGUC\n",
      "seqG\tCACACGCGGACGAGCAUGAAGCACUCGACUCACCUCGAAUAGUAGGGGGA\n",
      "seqH\tCAGAACGCACUUUCCGGCAGUGAAUGUCGAUGCCAACAUUGCCUAAAACA\n",
      "seqI\tCACACGCGGACGAGCAUGAAGCACUCGACUCACCUCGAAUAGUAGGGGGA\n",
      "seqJ\tUGCGGGGCGGCGAUAUGCGGCAACGAUGGCCGCGAGUUAAAUGAGCCAUA\n",
      "seqK\tCUUUGUGGGGUAAUCUAGAAAGCAGAGUACGUAUUUGGCGACCCGACAUU\n",
      "seqL\tCUUAUUUGCUCUCCCGGCUGUAGUGUGCACAGACUUCCAGAUGUAAUAAG\n",
      "seqM\tCUUUCUCUAUUUACCGAGCAAUGAAAACCACGGGGAACAAUCCACCCUUU\n",
      "seqN\tGUCAAAUGGCCGGCCCACUUCUCUCUUCGACACGGCGUCGGGCAGUCUGG\n",
      "seqO\tAUGGUAAGACUAGCCAGCAGAUCCUUAAUCGGCUUUCCCGAAGGACAGUA\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "tr T U <sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `tr` command can also replace more than one character at a time. Let's replace each base with its complement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seqT\tGGGATTGGGGGATTGGGGGATTGGGAGTCAGCCCCTCCGCTGTTATCGAC\n",
      "seqB\tCAGTATACAAGACATGCAATAACCGGTTGACTAGTATGGACTTAGCTCGG\n",
      "seqG\tCTTGGCCCTAATAGTTTCTGCTTGTACCAGCCCAGGAACTTGGTTTGCTT\n",
      "seqE\tAGAGGCAGGCGACCGCACAAAAAGAAAAGAGTTCACCCGTTCAATGGGCC\n",
      "seqD\tAGAGGCAGGCGACCGCACAAAAAGAAAAGAGTTCACCCGTTCAATGGGCC\n",
      "seqF\tAATCAACGTTTATCCGAATGGATCATCTCCAGGCTGTTGGGGATGTGCAG\n",
      "seqC\tGTGTGCGCCTGCTCGTACTTCGTGAGCTGAGTGGAGCTTATCATCCCCCT\n",
      "seqH\tGTCTTGCGTGAAAGGCCGTCACTTACAGCTACGGTTGTAACGGATTTTGT\n",
      "seqI\tGTGTGCGCCTGCTCGTACTTCGTGAGCTGAGTGGAGCTTATCATCCCCCT\n",
      "seqJ\tACGCCCCGCCGCTATACGCCGTTGCTACCGGCGCTCAATTTACTCGGTAT\n",
      "seqK\tGAAACACCCCATTAGATCTTTCGTCTCATGCATAAACCGCTGGGCTGTAA\n",
      "seqL\tGAATAAACGAGAGGGCCGACATCACACGTGTCTGAAGGTCTACATTATTC\n",
      "seqM\tGAAAGAGATAAATGGCTCGTTACTTTTGGTGCCCCTTGTTAGGTGGGAAA\n",
      "seqN\tCAGTTTACCGGCCGGGTGAAGAGAGAAGCTGTGCCGCAGCCCGTCAGACC\n",
      "seqO\tTACCATTCTGATCGGTCGTCTAGGAATTAGCCGAAAGGGCTTCCTGTCAT\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "tr ACGT TGCA <sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `rev`: reverse text in a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "rev [path_to_file]...\n",
    "```\n",
    "The `rev` command reverses the characters on each line of a file. Let's use it to obtain the reverse complement of our sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CAGCTATTGTCGCCTCCCCGACTGAGGGTTAGGGGGTTAGGGGGTTAGGG\n",
      "GGCTCGATTCAGGTATGATCAGTTGGCCAATAACGTACAGAACATATGAC\n",
      "TTCGTTTGGTTCAAGGACCCGACCATGTTCGTCTTTGATAATCCCGGTTC\n",
      "CCGGGTAACTTGCCCACTTGAGAAAAGAAAAACACGCCAGCGGACGGAGA\n",
      "CCGGGTAACTTGCCCACTTGAGAAAAGAAAAACACGCCAGCGGACGGAGA\n",
      "GACGTGTAGGGGTTGTCGGACCTCTACTAGGTAAGCCTATTTGCAACTAA\n",
      "TCCCCCTACTATTCGAGGTGAGTCGAGTGCTTCATGCTCGTCCGCGTGTG\n",
      "TGTTTTAGGCAATGTTGGCATCGACATTCACTGCCGGAAAGTGCGTTCTG\n",
      "TCCCCCTACTATTCGAGGTGAGTCGAGTGCTTCATGCTCGTCCGCGTGTG\n",
      "TATGGCTCATTTAACTCGCGGCCATCGTTGCCGCATATCGCCGCCCCGCA\n",
      "AATGTCGGGTCGCCAAATACGTACTCTGCTTTCTAGATTACCCCACAAAG\n",
      "CTTATTACATCTGGAAGTCTGTGCACACTACAGCCGGGAGAGCAAATAAG\n",
      "AAAGGGTGGATTGTTCCCCGTGGTTTTCATTGCTCGGTAAATAGAGAAAG\n",
      "CCAGACTGCCCGACGCCGTGTCGAAGAGAGAAGTGGGCCGGCCATTTGAC\n",
      "TACTGTCCTTCGGGAAAGCCGATTAAGGATCTGCTGGCTAGTCTTACCAT\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cut -f2 sequences.txt | tr ACGT TGCA | rev"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `paste`: merge files horizontally"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "paste [path_to_file]...\n",
    "```\n",
    "The `paste` command is useful when you want to merge multiple files, each containing a table with **n** lines (rows)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\tseqA\tCCCTAACCCCCTAACCCCCTAACCCTCAGTCGGGGAGGCGACAATAGCTG\n",
      "2\tseqB\tGTCATATGTTCTGTACGTTATTGGCCAACTGATCATACCTGAATCGAGCC\n",
      "3\tseqC\tGAACCGGGATTATCAAAGACGAACATGGTCGGGTCCTTGAACCAAACGAA\n",
      "4\tseqE\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "5\tseqD\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "6\tseqF\tTTAGTTGCAAATAGGCTTACCTAGTAGAGGTCCGACAACCCCTACACGTC\n",
      "7\tseqG\tCACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "8\tseqH\tCAGAACGCACTTTCCGGCAGTGAATGTCGATGCCAACATTGCCTAAAACA\n",
      "9\tseqI\tCACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "10\tseqJ\tTGCGGGGCGGCGATATGCGGCAACGATGGCCGCGAGTTAAATGAGCCATA\n",
      "11\tseqK\tCTTTGTGGGGTAATCTAGAAAGCAGAGTACGTATTTGGCGACCCGACATT\n",
      "12\tseqL\tCTTATTTGCTCTCCCGGCTGTAGTGTGCACAGACTTCCAGATGTAATAAG\n",
      "13\tseqM\tCTTTCTCTATTTACCGAGCAATGAAAACCACGGGGAACAATCCACCCTTT\n",
      "14\tseqN\tGTCAAATGGCCGGCCCACTTCTCTCTTCGACACGGCGTCGGGCAGTCTGG\n",
      "15\tseqO\tATGGTAAGACTAGCCAGCAGATCCTTAATCGGCTTTCCCGAAGGACAGTA\n",
      "I am secret text from a hidden file\t\n",
      "Here's chr21:\t\n",
      "https://hgdownload.soe.ucsc.edu/goldenPath/hg38/chromosomes/chr21.fa.gz\t\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "paste .you_found_me.txt sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `sort`: sort the lines in a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "sort [path_to_file]...\n",
    "```\n",
    "The `sort` command will sort the lines of your file. By default, this is done alphabetically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seqA\tCCCTAACCCCCTAACCCCCTAACCCTCAGTCGGGGAGGCGACAATAGCTG\n",
      "seqB\tGTCATATGTTCTGTACGTTATTGGCCAACTGATCATACCTGAATCGAGCC\n",
      "seqC\tGAACCGGGATTATCAAAGACGAACATGGTCGGGTCCTTGAACCAAACGAA\n",
      "seqE\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqD\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "head -n 5 sequences.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seqA\tCCCTAACCCCCTAACCCCCTAACCCTCAGTCGGGGAGGCGACAATAGCTG\n",
      "seqB\tGTCATATGTTCTGTACGTTATTGGCCAACTGATCATACCTGAATCGAGCC\n",
      "seqC\tGAACCGGGATTATCAAAGACGAACATGGTCGGGTCCTTGAACCAAACGAA\n",
      "seqD\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqE\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "sort sequences.txt | head -n 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also sort the file by specific columns using the `-k` flag.\n",
    "\n",
    "Let's sort by the second column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seqO\tATGGTAAGACTAGCCAGCAGATCCTTAATCGGCTTTCCCGAAGGACAGTA\n",
      "seqG\tCACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "seqI\tCACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "seqH\tCAGAACGCACTTTCCGGCAGTGAATGTCGATGCCAACATTGCCTAAAACA\n",
      "seqA\tCCCTAACCCCCTAACCCCCTAACCCTCAGTCGGGGAGGCGACAATAGCTG\n",
      "seqL\tCTTATTTGCTCTCCCGGCTGTAGTGTGCACAGACTTCCAGATGTAATAAG\n",
      "seqM\tCTTTCTCTATTTACCGAGCAATGAAAACCACGGGGAACAATCCACCCTTT\n",
      "seqK\tCTTTGTGGGGTAATCTAGAAAGCAGAGTACGTATTTGGCGACCCGACATT\n",
      "seqC\tGAACCGGGATTATCAAAGACGAACATGGTCGGGTCCTTGAACCAAACGAA\n",
      "seqN\tGTCAAATGGCCGGCCCACTTCTCTCTTCGACACGGCGTCGGGCAGTCTGG\n",
      "seqB\tGTCATATGTTCTGTACGTTATTGGCCAACTGATCATACCTGAATCGAGCC\n",
      "seqD\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqE\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqJ\tTGCGGGGCGGCGATATGCGGCAACGATGGCCGCGAGTTAAATGAGCCATA\n",
      "seqF\tTTAGTTGCAAATAGGCTTACCTAGTAGAGGTCCGACAACCCCTACACGTC\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "sort -k 2 sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `uniq`: filter out duplicate lines in a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "uniq [path_to_file_input [path_to_file_output]]\n",
    "```\n",
    "You can use the `uniq` command to filter out duplicated lines or count the number of unique lines.\n",
    "\n",
    "Note that the text provided to `uniq` must be sorted first! Otherwise, it won't work properly.\n",
    "\n",
    "Let's find unique sequences in the `sequences.txt` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CCCTAACCCCCTAACCCCCTAACCCTCAGTCGGGGAGGCGACAATAGCTG\n",
      "GAACCGGGATTATCAAAGACGAACATGGTCGGGTCCTTGAACCAAACGAA\n",
      "GTCATATGTTCTGTACGTTATTGGCCAACTGATCATACCTGAATCGAGCC\n",
      "TCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "TCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "head -n 5 sequences.txt | cut -f2 | sort"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CCCTAACCCCCTAACCCCCTAACCCTCAGTCGGGGAGGCGACAATAGCTG\n",
      "GAACCGGGATTATCAAAGACGAACATGGTCGGGTCCTTGAACCAAACGAA\n",
      "GTCATATGTTCTGTACGTTATTGGCCAACTGATCATACCTGAATCGAGCC\n",
      "TCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "head -n 5 sequences.txt | cut -f2 | sort | uniq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I bet you didn't even notice the last two sequences were duplicates until now ;P"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `grep`: search for lines containing a keyword"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "grep pattern [path_to_file]...\n",
    "```\n",
    "To search for lines in the file containing a specific pattern, you can use `grep`.\n",
    "\n",
    "Let's look for all sequences with \"TAAC\" in them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seqA\tCCCTAACCCCCTAACCCCCTAACCCTCAGTCGGGGAGGCGACAATAGCTG\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "grep \"TAAC\" sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of these commands read from stdin and `grep` is no exception. For example, we can search through our bash history for the `echo` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    4  display () {     TMPFILE=$(mktemp ${TMPDIR-/tmp}/bash_kernel.XXXXXXXXXX);     cat > $TMPFILE;     echo \"bash_kernel: saved image data to: $TMPFILE\" >&2; }\n",
      "    7  display () {     TMPFILE=$(mktemp ${TMPDIR-/tmp}/bash_kernel.XXXXXXXXXX);     cat > $TMPFILE;     echo \"bash_kernel: saved image data to: $TMPFILE\" >&2; }\n",
      "    9  echo $?\n",
      "   11  echo $?\n",
      "   13  echo $?\n",
      "   15  echo $?\n",
      "   17  echo $?\n",
      "   19  echo $?\n",
      "   22  echo $?\n",
      "   24  echo $?\n",
      "   26  echo $?\n",
      "   28  echo $?\n",
      "   30  echo $?\n",
      "   32  echo $?\n",
      "   34  echo $?\n",
      "   36  echo $?\n",
      "   37  echo ls ~\n",
      "   38  echo $?\n",
      "   40  echo $?\n",
      "   42  echo $?\n",
      "   44  echo $?\n",
      "   45  echo ls scripts/*.sh\n",
      "   46  echo $?\n",
      "   48  echo $?\n",
      "   50  echo $?\n",
      "   52  echo $?\n",
      "   55  echo $?\n",
      "   58  echo $?\n",
      "   60  echo $?\n",
      "   62  echo $?\n",
      "   64  echo $?\n",
      "   66  echo $?\n",
      "   69  echo $?\n",
      "   71  echo $?\n",
      "   74  echo $?\n",
      "   75  echo rm shiny_new/*.txt\n",
      "   76  echo $?\n",
      "   79  echo $?\n",
      "   82  echo $?\n",
      "   85  echo $?\n",
      "   87  echo $?\n",
      "   90  echo $?\n",
      "   92  echo $?\n",
      "   95  echo $?\n",
      "   98  echo $?\n",
      "  101  echo $?\n",
      "  103  echo $?\n",
      "  109  echo $?\n",
      "  111  echo $?\n",
      "  113  echo $?\n",
      "  115  echo $?\n",
      "  117  echo $?\n",
      "  119  echo $?\n",
      "  121  echo $?\n",
      "  123  echo $?\n",
      "  125  echo $?\n",
      "  127  echo $?\n",
      "  129  echo $?\n",
      "  131  echo $?\n",
      "  132  echo hey\n",
      "  133  echo $?\n",
      "  134  echo is anybody out there?\n",
      "  135  echo $?\n",
      "  137  echo $?\n",
      "  139  echo $?\n",
      "  141  echo $?\n",
      "  143  echo $?\n",
      "  145  echo $?\n",
      "  147  echo $?\n",
      "  149  echo $?\n",
      "  152  echo $?\n",
      "  155  echo $?\n",
      "  158  echo $?\n",
      "  161  echo $?\n",
      "  162  echo rm file_*\n",
      "  163  echo $?\n",
      "  165  echo $?\n",
      "  167  echo $?\n",
      "  169  echo $?\n",
      "  171  echo $?\n",
      "  172  echo $my_variable\n",
      "  173  echo $?\n",
      "  177  echo $my_variable\n",
      "  178  echo $?\n",
      "  180  echo $?\n",
      "  182  echo $?\n",
      "  184  echo $my_variable\n",
      "  185  echo $?\n",
      "  187  echo $my_variable\n",
      "  188  echo $?\n",
      "  190  echo $my_variable\n",
      "  191  echo $?\n",
      "  193  echo $my_variable\n",
      "  194  echo $?\n",
      "  195  echo my_variable=\"$hello_var $world_var\"\n",
      "  196  echo my_variable='$hello_var $world_var'\n",
      "  197  echo $?\n",
      "  199  echo $my_variable\n",
      "  200  echo $?\n",
      "  202  echo $?\n",
      "  204  echo $?\n",
      "  206  echo $?\n",
      "  208  echo $?\n",
      "  210  echo $?\n",
      "  212  echo $?\n",
      "  214  echo $?\n",
      "  216  echo $?\n",
      "  218  echo $?\n",
      "  220  echo $?\n",
      "  222  echo $?\n",
      "  224  echo $?\n",
      "  225  history | grep echo\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "history | grep echo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you use the `-E` flag, `grep` will interpret the pattern `grep` as a _regex (aka regular expression)_. Regexes can get quite complicated, so we don't have the time to dive into them completely.\n",
    "\n",
    "But to give you an idea of how powerful this can be, let's try to grab sequences C through F or just sequence B and D."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seqC\tGAACCGGGATTATCAAAGACGAACATGGTCGGGTCCTTGAACCAAACGAA\n",
      "seqE\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqD\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqF\tTTAGTTGCAAATAGGCTTACCTAGTAGAGGTCCGACAACCCCTACACGTC\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "grep -E 'seq[C-F]' sequences.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seqB\tGTCATATGTTCTGTACGTTATTGGCCAACTGATCATACCTGAATCGAGCC\n",
      "seqD\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "grep -E 'seq(B|D)' sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `sed`: scan through a file and make edits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "sed script_pattern [path_to_file]...\n",
    "```\n",
    "The `tr` command made single-character substitutions, but what if you want to replace entire words or sentences? That's one of the most common uses for the `sed` command.\n",
    "\n",
    "Let's replace \"seqC\" with the label \"sample1\". The first argument to `sed` is a special pattern composed of the letter **s**, the search keyword, then the replacement keyword, where each is separated by some character -- in this case, we use a slash `/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seqA\tCCCTAACCCCCTAACCCCCTAACCCTCAGTCGGGGAGGCGACAATAGCTG\n",
      "seqB\tGTCATATGTTCTGTACGTTATTGGCCAACTGATCATACCTGAATCGAGCC\n",
      "sample1\tGAACCGGGATTATCAAAGACGAACATGGTCGGGTCCTTGAACCAAACGAA\n",
      "seqE\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqD\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "head -n 5 sequences.txt | sed 's/seqC/sample1/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `awk`: scan through a file and extract text\n",
    "```bash\n",
    "awk program-text [path_to_file]...\n",
    "```\n",
    "`awk` is a much more powerful command than many of the others. It implements its own programming language which can be used to do any number of tasks, including most of those of the other commands we just learned.\n",
    "\n",
    "We don't have time to discuss `awk`, but there are tons of great tutorials online you can peruse."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `vim`: terminal-based text editor\n",
    "```bash\n",
    "vim [path_to_file]...\n",
    "```\n",
    "`vim` is a fully-featured [text editor](https://learntocodewith.me/resources/text-editors/) that runs from within the terminal. You can learn how to use it at [openvim.com](https://openvim.com)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `less`: view large files and text one page at a time\n",
    "```bash\n",
    "less [path_to_file]...\n",
    "```\n",
    "You can use `less` to view large files one page at a time. This proves to be incredibly useful in bioinformatics, where most files are very, very large."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the `sequences.txt` file with `less`.\n",
    "```bash\n",
    "less sequences.txt\n",
    "```\n",
    "\n",
    "This will open a separate viewing window. You can quit out of it and return to the terminal prompt by pressing **q**.\n",
    "\n",
    "Now, let's take a look at our bash history. We can pipe into `less` for this.\n",
    "```bash\n",
    "history | less\n",
    "```\n",
    "\n",
    "You can use the arrow keys to scroll in `less`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### searching through the text\n",
    "The `less` command has a vast array of incredible helpful keyboard shortcuts. We don't have time to go into all of them, but one of the most useful is the ability to search within the text.\n",
    "\n",
    "To search down through the text...\n",
    "1. Press the forward-slash `/`\n",
    "2. Type your search keyword\n",
    "3. Press _enter_\n",
    "\n",
    "To search up from your current position, use a question-mark `?` instead of a forward-slash.\n",
    "\n",
    "**You try**: Search for `ls` commands within `history`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Try this:**\n",
    "Retrieve just the second column of the third line (seq C) from the sequences.txt file using a combination of tail, head, and cut. Use pipes!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GAACCGGGATTATCAAAGACGAACATGGTCGGGTCCTTGAACCAAACGAA\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "head -n 3 sequences.txt | tail -n 1 | cut -f 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Got long-running commands? Use job control."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Scenario:** You're about to execute a commmand that will take a long time.\n",
    "\n",
    "**Solution:** Send the command to the background by appending an ampersand `&`. Keep working while it runs.\n",
    "\n",
    "We'll use the `sleep` command to illustrate how this works, but you could use any command in theory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `sleep`: do nothing for n seconds\n",
    "```bash\n",
    "sleep num_seconds\n",
    "```\n",
    "Running `sleep 3` will cause `sleep` to execute for 3 seconds, doing absolutely nothing in the meantime."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "sleep 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try sending `sleep` to the background while it runs.\n",
    "```bash\n",
    "sleep 7 &\n",
    "```\n",
    "Now, we can continue working while it executes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## backgrounding a job while it is running"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'm generally not put-together enough to anticipate when I need to background a job. So is there a way to background a job after you've already started executing it?\n",
    "\n",
    "Yup!\n",
    "1. Press ctrl+z to temporarily pause the command\n",
    "2. Send it to the background using the `bg` command\n",
    "\n",
    "For example,\n",
    "1. `sleep 120`\n",
    "2. _ctrl+z_\n",
    "3. `bg`\n",
    "\n",
    "To bring it back to the foreground, use the `fg` command after backgrounding it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ending a command prematurely"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also permanently stop a command, if you need to. Usually, if your command isn't backgrounded, you can do this by pressing _ctrl+c_.\n",
    "\n",
    "For example,\n",
    "1. `sleep 120`\n",
    "2. _ctrl+c_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Otherwise, if your command is backgrounded or `ctrl+c` isn't working for whatever reason, you can use the `kill` command.\n",
    "\n",
    "There are two ways of doing this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `kill <PID>`\n",
    "Every running command has a process identifier (PID). To list all processes and their PIDs, use the `ps` command. We use the `-u` option to retrieve only those processes that our **u**ser owns.\n",
    "```bash\n",
    "ps -u $(whoami)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `kill <job ID>`\n",
    "Every backgrounded command has a job identifier. To list all jobs and their IDs, use the `jobs` command.\n",
    "```bash\n",
    "jobs\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## how to step away from your terminal for a while"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Backgrounded jobs will be killed when you try to close your terminal window or shut down your computer. You can prepend `nohup` to your command to ensure that it continues running.\n",
    "```bash\n",
    "nohup sleep 7 &\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### run your commands within `screen` sessions!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, `nohup` can have some unintended side effects (like redirecting stdout/stderr) and must be prepended before every command. A popular alternative is to use `screen` sessions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. First, create and enter a `screen` session. Let's call it `test`\n",
    "```bash\n",
    "screen -S test\n",
    "```\n",
    "2. Done! Now, you can just run your commands normally!\n",
    "3. If you'd like to step away from your computer, you can just detach from the `screen` session while commands within it are running.\n",
    "\n",
    "    Just press **ctrl + a** and then the letter **d**.\n",
    "    \n",
    "    **Note:** You will be detached automatically if your computer turns off or your `ssh` session dies because of a choppy internet connection.\n",
    "4. When you return to your computer or restart your `ssh` session, just reattach to the `screen` session and pick up where you left off.\n",
    "```bash\n",
    "screen -r test\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At any point, you can view your existing `screen` sessions.\n",
    "```bash\n",
    "screen -ls\n",
    "```\n",
    "To terminate a `screen` session, attach to it and then just exit normally.\n",
    "```bash\n",
    "exit\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with larger files\n",
    "Bioinformaticians frequently keep commands compressed so they don't take up too much space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `gzip`: compressing a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "gzip [path_to_file]...\n",
    "```\n",
    "You can compress a file using the `gzip` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "gzip sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how the file has a `.gz` extension now?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scripts  sequences.txt.gz\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And if we try to take a look inside, it'll just look like gibberish."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "�^�c\u0000\u0003sequences.txt\u0000��ArB1\f",
      "C��V�ۺPJ7�\u0002\u0007`��\f",
      "z\n",
      "�@��s\u001d",
      "q�?��\u0004��M����\u0011\u000eh��t�Ѓz(��*�O�*;�Q�\u0016��W�\u0000_e�])��1���9To�'\u0011U���(���\u0006w|�ﴦ3��l\u0019d(\u0015�F��\"\\��9\u0005���z���<@!�Ƞ�\u000b",
      ":Z�����rTd؆qp[\\@ۂ�®\t�/|`\u0000�\u001c",
      "��d\u001c",
      "��\u0019o��:�ğz��7��{���x\f",
      "\u0017�Ϟ���\u000f��I�i-���{�\u0007I���\u000f\u000f|61NH\u0003\u0000\u0000\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "head sequences.txt.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### decompress a file\n",
    "Let's revert the file back to the way it was. We can use the `-d` switch to activate decompression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scripts  sequences.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "gzip -d sequences.txt.gz\n",
    "ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seqA\tCCCTAACCCCCTAACCCCCTAACCCTCAGTCGGGGAGGCGACAATAGCTG\n",
      "seqB\tGTCATATGTTCTGTACGTTATTGGCCAACTGATCATACCTGAATCGAGCC\n",
      "seqC\tGAACCGGGATTATCAAAGACGAACATGGTCGGGTCCTTGAACCAAACGAA\n",
      "seqE\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqD\tTCTCCGTCCGCTGGCGTGTTTTTCTTTTCTCAAGTGGGCAAGTTACCCGG\n",
      "seqF\tTTAGTTGCAAATAGGCTTACCTAGTAGAGGTCCGACAACCCCTACACGTC\n",
      "seqG\tCACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "seqH\tCAGAACGCACTTTCCGGCAGTGAATGTCGATGCCAACATTGCCTAAAACA\n",
      "seqI\tCACACGCGGACGAGCATGAAGCACTCGACTCACCTCGAATAGTAGGGGGA\n",
      "seqJ\tTGCGGGGCGGCGATATGCGGCAACGATGGCCGCGAGTTAAATGAGCCATA\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "head sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, now that we know how to deal with small files, let's level-up to \"BIG DATA\"!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `wget`: downloading a file from a URL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "wget url...\n",
    "```\n",
    "For example, let's download chr21 from the hg38 reference genome:\n",
    "\n",
    "https://hgdownload.soe.ucsc.edu/goldenPath/hg38/chromosomes/chr21.fa.gz (also at https://bit.ly/3zjmX7L)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that the URL for chr21 is stored at the bottom of the hidden file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://hgdownload.soe.ucsc.edu/goldenPath/hg38/chromosomes/chr21.fa.gz\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "echo $(tail -n1 .you_found_me.txt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-01-10 05:46:22--  https://hgdownload.soe.ucsc.edu/goldenPath/hg38/chromosomes/chr21.fa.gz\n",
      "Resolving hgdownload.soe.ucsc.edu (hgdownload.soe.ucsc.edu)... 128.114.119.163\n",
      "Connecting to hgdownload.soe.ucsc.edu (hgdownload.soe.ucsc.edu)|128.114.119.163|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 12709705 (12M) [application/x-gzip]\n",
      "Saving to: ‘chr21.fa.gz’\n",
      "\n",
      "chr21.fa.gz         100%[===================>]  12.12M  24.2MB/s    in 0.5s    \n",
      "\n",
      "2023-01-10 05:46:23 (24.2 MB/s) - ‘chr21.fa.gz’ saved [12709705/12709705]\n",
      "\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "wget $(tail -n1 .you_found_me.txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check: do we have it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chr21.fa.gz  scripts  sequences.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `du`: print a file's **d**isk **u**sage\n",
    "```bash\n",
    "du path_to_file...\n",
    "```\n",
    "In addition to `ls`, we can use the `du` to check how big a file is."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How big is the file we downloaded compared to the sequences file? We can use the `-h` flag to print the size in human-readable format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9.0K\tsequences.txt\n",
      "1.0K\tchr21.fa.gz\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "du -h sequences.txt chr21.fa.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's several 1000 times larger!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `zcat`: output decompressed text from a gzipped file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "zcat [path_to_file]...\n",
    "```\n",
    "We want to take a look at the first ten lines of this file, but it's compressed. Rather than uncompress it all and waste storage space, let's see if we can obtain what we want using the `zcat` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ">chr214l\n",
      "NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN\n",
      "NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN\n",
      "NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN\n",
      "NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN\n",
      "NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN\n",
      "NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN\n",
      "NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN\n",
      "NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN\n",
      "NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN\n",
      "\n",
      "gzip: stdout: Broken pipe\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "zcat chr21.fa.gz | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first set of lines of chr21 represent unsequenced heterochromatin. Let's see if we can find real sequences by opening the file inside the `zless` command...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `zless`: `less` but for gzipped files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "zcat [path_to_file]...\n",
    "```\n",
    "Let's open this file in `less` and then look around for a nucleotide like a **g**.\n",
    "\n",
    "```bash\n",
    "zless chr21.fa.gz\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we finish this section, let's just clean up after ourselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scripts  sequences.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "rm chr21.fa.gz\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logic control\n",
    "Bash has for loops and if statements, just like python and R!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## for loops"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A for loop is written like this:\n",
    "```bash\n",
    "for my_var in <space_separated_list>; do\n",
    "    cmd \"$my_var\"\n",
    "done\n",
    "```\n",
    "A variable, in this case called `my_var`, is set to each item in the loop. The items are specified as a space separated list. Inside the body of the loop, we can run a `cmd` using the value of each item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello Dolly!004l\u001b[?2004l\n",
      "Hello Kitty!\n",
      "Hello Poppet!\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "for name in Dolly Kitty Poppet; do\n",
    "    echo Hello \"$name!\";\n",
    "done;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an example, let's iterate over all of the scripts in the `scripts/` directory and print out their paths. Recall that expressions with wildcards like the asterisk `*` get expanded to a space-separated list prior to command execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "check out this file called scripts/hello.sh\n",
      "check out this file called scripts/hello_to_all_of_you.sh\n",
      "check out this file called scripts/hello_you.sh\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "for file in scripts/*.sh; do\n",
    "    echo \"check out this file called $file\"\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## if statements\n",
    "An if-statement is written like this:\n",
    "```bash\n",
    "if [ <logical_statement> ]; then\n",
    "    cmd\n",
    "fi\n",
    "```\n",
    "The logical statement is something that we can test for.\n",
    "\n",
    "Here's an example. First, we make a variable called `$you` and then we test whether `$you` equals \"bored\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "okay onto the next adventure!04l\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "you=excited\n",
    "if [ $you != bored ]; then\n",
    "    echo \"okay onto the next adventure!\"\n",
    "fi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bash: also a programming language"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is _bash_ anyways?\n",
    "1. A shell programming language used within terminals\n",
    "2. An application that reads and interpets all of the commands (ie code) that you type into the terminal\n",
    "\n",
    "**Takeaway**: You can write scripts composed from all of the commands we learned!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've written a couple of scripts for you, already. Why don't you take a look?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#!/usr/bin/env bash\n",
      "\n",
      "# this is a comment; it will be ignored\n",
      "\n",
      "echo \"Hello world!\"\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cat scripts/hello.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is a shebang?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first line of `hello.sh` is called a _shebang_. It denotes the command that should be used to execute your script.\n",
    "\n",
    "In this case, we're using the `bash` command, since we're writing our script in the `bash` programming language. But we could have used any command -- like `python`, for example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Executing a script"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To execute the script, just type the path to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello world!\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "scripts/hello.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Passing arguments to a script"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scripts are, in fact, commands themselves! Within a script, `$1` refers to the first argument to the script, `$2` refers to the second argument, etc...\n",
    "\n",
    "We demonstrate this in `hello_you.sh`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#!/usr/bin/env bash\n",
      "\n",
      "echo \"Hello $1\"!\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cat scripts/hello_you.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello me!\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "scripts/hello_you.sh me"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calling scripts from within other scripts "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#!/usr/bin/env bash\n",
      "\n",
      "for name in \"Dolly\" \"Kitty\" \"Poppet\"; do\n",
      "    scripts/hello_you.sh $name\n",
      "done;\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cat scripts/hello_to_all_of_you.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello Dolly!\n",
      "Hello Kitty!\n",
      "Hello Poppet!\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "scripts/hello_to_all_of_you.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create your own script!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try to create your own script.\n",
    "1. Navigate to the `module-1-programming/bash_playground/scripts/` directory in DataHub, and create a new file called `my_hello.sh`.\n",
    "2. Copy the shebang from our other scripts and add your own command.\n",
    "3. Every time you want to convert a regular file into a script, you'll first need to designate the file as _executable_. You should do this now using the `chmod` command:\n",
    "\n",
    "    ```bash\n",
    "    chmod u+x scripts/my_hello.sh\n",
    "    ```\n",
    "4. Execute your script.\n",
    "\n",
    "    ```bash\n",
    "    scripts/my_hello.sh\n",
    "    ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making a script globally availale using the special `$PATH` environment variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I don’t want to have to type out the full path to `hello_you.sh` every time.\n",
    "\n",
    "How can I make it so that I can execute `hello_you.sh` from anywhere by just typing `hello_you.sh`?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`$PATH` is a special _environment_ variable that is created by _bash_ on startup. It contains a colon-separated list of directories.\n",
    "\n",
    "Whenever you type a command, _bash_ will look through these directories to find an executable file that matches it.\n",
    "\n",
    "What does `$PATH` look like right now?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/conda/condabin:/opt/k8s-support/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/rstudio-server/bin\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "echo $PATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how `/usr/bin` is listed in `$PATH`? Try this:\n",
    "```bash\n",
    "ls /usr/bin | less\n",
    "```\n",
    "Do you recognize any of those?! They're the commands we've been learning about today!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_All of the commands we've been using are just executable files in a folder on the filesystem somewhere!_\n",
    "\n",
    "Can we make our own scripts into commands?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add the `scripts/` directory to the end of `$PATH`. Note that we can obtain the absolute path to the `scripts/` directory using `pwd`:\n",
    "```\n",
    "echo \"$(pwd)/scripts\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/conda/condabin:/opt/k8s-support/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/rstudio-server/bin:/home/amassara/module-1-programming/bash_playground/scripts\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "PATH=\"$PATH:$(pwd)/scripts\"\n",
    "echo \"$PATH\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now, we should be able to execute our scripts as commands from anywhere :D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello world!\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "scripts/hello.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# We just scratched the surface! How can I learn more?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. [The missing semester of your CS education: an open MIT course](https://missing.csail.mit.edu/)\n",
    "2. [A curated list of resources for terminal-using bioinformaticians](https://docs.google.com/document/d/1cOjKC4OI4rsQUl_68PPHCn8vO11lwhpSWyJrGDZT80Q)\n",
    "3. [Software carpentry](https://swcarpentry.github.io/shell-novice/)\n",
    "4. [explainshell.com](http://explainshell.com) and help/manual pages\n",
    "5. It’s dry reading, but [the GNU bash manual](https://www.gnu.org/software/bash/manual/bash.html) is really quite good\n",
    "6. Other useful topics for bioinformaticians that we weren't able to cover today\n",
    "\t- git\n",
    "\t- ssh\n",
    "\t- sshfs\n",
    "\t- conda and conda environments\n",
    "\t- Snakemake\n",
    "\t- the screen command\n",
    "\t- process substitution\n",
    "\t- exit codes\n",
    "\t- functions\n",
    "\t- aliases\n",
    "\t- bash startup\n",
    "\t- using a computing cluster\n",
    "\t- regexes\n",
    "\t- manipulating strings\n",
    "\t- shell expansions\n",
    "\t- the diff and comm commands\n",
    "\t- file and directory permissions\n",
    "\t- the find and xargs commands\n",
    "\t- arrays\n",
    "\t- shell arithmetic\n",
    "\t- while loops\n",
    "\t- signals\n",
    "\t- the top command\n",
    "\t- the datamash tool\n",
    "\t- the /dev special files"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  },
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}