nb-1-ipyrad-assembly.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Canarium GBS Assembly\n",
    "### *Federman et al.*\n",
    "\n",
    "This notebook provides all code necessary to reproduce the assembled GBS data sets used in Federman et al. (xxxx). Starting from demultiplexed fastq data files we assemble the data into four complete data sets that were used in downstream analyses. All code in this notebook is written in Python and uses the *ipyrad* package for assembly. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Required software"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## conda install ipyrad -c ipyrad\n",
    "## conda install sra-tools -c bioconda\n",
    "## conda install entrez-direct -c bioconda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipyrad as ip\n",
    "import ipyrad.analysis as ipa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ipyrad v.0.7.20\n"
     ]
    }
   ],
   "source": [
    "print \"ipyrad v.{}\".format(ip.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect to cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "host compute node: [40 cores] on sacra\n"
     ]
    }
   ],
   "source": [
    "import ipyparallel as ipp\n",
    "ipyclient = ipp.Client()\n",
    "ip.cluster_info(ipyclient)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Links to Sections\n",
    "\n",
    "+ [Download Sequence Data](#Download-sequence-data-from-SRA)\n",
    "+ [ipyrad Assembly](#ipyrad-Assembly)\n",
    "+ [Assembly stats](#Assembly-stats)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download sequence data from SRA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Fetching project data..."
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Run</th>\n",
       "      <th>spots</th>\n",
       "      <th>spots_with_mates</th>\n",
       "      <th>ScientificName</th>\n",
       "      <th>SampleName</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>SRR5534921</td>\n",
       "      <td>12818693</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium lamianum</td>\n",
       "      <td>SF327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>SRR5534922</td>\n",
       "      <td>5675773</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium longistipulatum</td>\n",
       "      <td>D12950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>SRR5534923</td>\n",
       "      <td>34404746</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium ovatum</td>\n",
       "      <td>D14269</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>SRR5534924</td>\n",
       "      <td>3382649</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium pilicarpum</td>\n",
       "      <td>5573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SRR5534925</td>\n",
       "      <td>16632442</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium obtusifolium</td>\n",
       "      <td>SF228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>SRR5534926</td>\n",
       "      <td>10769159</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium odontophyllum</td>\n",
       "      <td>SFC1988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>SRR5534927</td>\n",
       "      <td>47625</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium ntidifolium</td>\n",
       "      <td>4304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>SRR5534928</td>\n",
       "      <td>17034881</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium obtusifolium</td>\n",
       "      <td>SF224</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>SRR5534929</td>\n",
       "      <td>211932</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium ferrugineum</td>\n",
       "      <td>SF343</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>SRR5534930</td>\n",
       "      <td>402493</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium galokense</td>\n",
       "      <td>D13101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>SRR5534931</td>\n",
       "      <td>5796976</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium galokense</td>\n",
       "      <td>SF155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>SRR5534932</td>\n",
       "      <td>3219083</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium globosum</td>\n",
       "      <td>SF200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>SRR5534933</td>\n",
       "      <td>1902860</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium globosum</td>\n",
       "      <td>SF209</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>SRR5534934</td>\n",
       "      <td>9709559</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium indicum</td>\n",
       "      <td>D13374</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>SRR5534935</td>\n",
       "      <td>2192159</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium lamianum</td>\n",
       "      <td>D13063</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>SRR5534936</td>\n",
       "      <td>450626</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium lamianum</td>\n",
       "      <td>SF160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>SRR5534937</td>\n",
       "      <td>12874061</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium pulchrebracteatum</td>\n",
       "      <td>SF286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>SRR5534938</td>\n",
       "      <td>2499902</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium compressum</td>\n",
       "      <td>D13090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>SRR5534939</td>\n",
       "      <td>7260851</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium ferrugineum</td>\n",
       "      <td>SF172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>SRR5534940</td>\n",
       "      <td>15736904</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium pulchrebracteatum</td>\n",
       "      <td>SF276</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>SRR5534941</td>\n",
       "      <td>12539878</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium pilicarpum</td>\n",
       "      <td>D13052</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>SRR5534942</td>\n",
       "      <td>23355083</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium compressum</td>\n",
       "      <td>D13097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>SRR5534943</td>\n",
       "      <td>2097537</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium egregium</td>\n",
       "      <td>D13103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>SRR5534944</td>\n",
       "      <td>1033763</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium elegans</td>\n",
       "      <td>D12963</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>SRR5534945</td>\n",
       "      <td>2725698</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium bengalense</td>\n",
       "      <td>D13852</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>SRR5534946</td>\n",
       "      <td>12132085</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium betamponae</td>\n",
       "      <td>SF175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>SRR5534947</td>\n",
       "      <td>18446248</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium betamponae</td>\n",
       "      <td>SF328</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>SRR5534948</td>\n",
       "      <td>400266</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium boivinii</td>\n",
       "      <td>D12962</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>SRR5534949</td>\n",
       "      <td>11647948</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium velutinifolium</td>\n",
       "      <td>D14505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>SRR5534950</td>\n",
       "      <td>12852942</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium velutinifolium</td>\n",
       "      <td>D14504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>SRR5534951</td>\n",
       "      <td>16100807</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium scholasticum</td>\n",
       "      <td>SF197</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>SRR5534952</td>\n",
       "      <td>186682</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium scholasticum</td>\n",
       "      <td>SF301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>SRR5534953</td>\n",
       "      <td>1152381</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium planifolium</td>\n",
       "      <td>SF153</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>SRR5534954</td>\n",
       "      <td>3803237</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium multiflorum</td>\n",
       "      <td>D14501</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>SRR5534955</td>\n",
       "      <td>4734247</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium multiflorum</td>\n",
       "      <td>D14485</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>SRR5534956</td>\n",
       "      <td>10744745</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium multinerve</td>\n",
       "      <td>D14482</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>SRR5534957</td>\n",
       "      <td>2757099</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium multiflorum</td>\n",
       "      <td>D14513</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>SRR5534958</td>\n",
       "      <td>6645549</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium multiflorum</td>\n",
       "      <td>D14477</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>SRR5534959</td>\n",
       "      <td>225668</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium madagascariense</td>\n",
       "      <td>D13091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>SRR5534960</td>\n",
       "      <td>7962974</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium multiflorum</td>\n",
       "      <td>D14480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>SRR5534961</td>\n",
       "      <td>19015238</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium multiflorum</td>\n",
       "      <td>D14478</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>SRR5534962</td>\n",
       "      <td>668679</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium scholasticum</td>\n",
       "      <td>D13075</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>SRR5534963</td>\n",
       "      <td>294419</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium multinerve</td>\n",
       "      <td>D14492</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>SRR5534964</td>\n",
       "      <td>11466007</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium multinerve</td>\n",
       "      <td>D14483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>SRR5534965</td>\n",
       "      <td>9829153</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium planifolium</td>\n",
       "      <td>SF164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>SRR5534966</td>\n",
       "      <td>1788748</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium pulchrebracteatum</td>\n",
       "      <td>D14528</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>SRR5534967</td>\n",
       "      <td>5687824</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium velutinifolium</td>\n",
       "      <td>D14506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>SRR5534968</td>\n",
       "      <td>1694555</td>\n",
       "      <td>0</td>\n",
       "      <td>Canarium ferrugineum</td>\n",
       "      <td>D13053</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Run     spots spots_with_mates              ScientificName  \\\n",
       "0   SRR5534921  12818693                0           Canarium lamianum   \n",
       "1   SRR5534922   5675773                0    Canarium longistipulatum   \n",
       "2   SRR5534923  34404746                0             Canarium ovatum   \n",
       "3   SRR5534924   3382649                0         Canarium pilicarpum   \n",
       "4   SRR5534925  16632442                0       Canarium obtusifolium   \n",
       "5   SRR5534926  10769159                0      Canarium odontophyllum   \n",
       "6   SRR5534927     47625                0        Canarium ntidifolium   \n",
       "7   SRR5534928  17034881                0       Canarium obtusifolium   \n",
       "8   SRR5534929    211932                0        Canarium ferrugineum   \n",
       "9   SRR5534930    402493                0          Canarium galokense   \n",
       "10  SRR5534931   5796976                0          Canarium galokense   \n",
       "11  SRR5534932   3219083                0           Canarium globosum   \n",
       "12  SRR5534933   1902860                0           Canarium globosum   \n",
       "13  SRR5534934   9709559                0            Canarium indicum   \n",
       "14  SRR5534935   2192159                0           Canarium lamianum   \n",
       "15  SRR5534936    450626                0           Canarium lamianum   \n",
       "16  SRR5534937  12874061                0  Canarium pulchrebracteatum   \n",
       "17  SRR5534938   2499902                0         Canarium compressum   \n",
       "18  SRR5534939   7260851                0        Canarium ferrugineum   \n",
       "19  SRR5534940  15736904                0  Canarium pulchrebracteatum   \n",
       "20  SRR5534941  12539878                0         Canarium pilicarpum   \n",
       "21  SRR5534942  23355083                0         Canarium compressum   \n",
       "22  SRR5534943   2097537                0           Canarium egregium   \n",
       "23  SRR5534944   1033763                0            Canarium elegans   \n",
       "24  SRR5534945   2725698                0         Canarium bengalense   \n",
       "25  SRR5534946  12132085                0         Canarium betamponae   \n",
       "26  SRR5534947  18446248                0         Canarium betamponae   \n",
       "27  SRR5534948    400266                0           Canarium boivinii   \n",
       "28  SRR5534949  11647948                0     Canarium velutinifolium   \n",
       "29  SRR5534950  12852942                0     Canarium velutinifolium   \n",
       "30  SRR5534951  16100807                0       Canarium scholasticum   \n",
       "31  SRR5534952    186682                0       Canarium scholasticum   \n",
       "32  SRR5534953   1152381                0        Canarium planifolium   \n",
       "33  SRR5534954   3803237                0        Canarium multiflorum   \n",
       "34  SRR5534955   4734247                0        Canarium multiflorum   \n",
       "35  SRR5534956  10744745                0         Canarium multinerve   \n",
       "36  SRR5534957   2757099                0        Canarium multiflorum   \n",
       "37  SRR5534958   6645549                0        Canarium multiflorum   \n",
       "38  SRR5534959    225668                0    Canarium madagascariense   \n",
       "39  SRR5534960   7962974                0        Canarium multiflorum   \n",
       "40  SRR5534961  19015238                0        Canarium multiflorum   \n",
       "41  SRR5534962    668679                0       Canarium scholasticum   \n",
       "42  SRR5534963    294419                0         Canarium multinerve   \n",
       "43  SRR5534964  11466007                0         Canarium multinerve   \n",
       "44  SRR5534965   9829153                0        Canarium planifolium   \n",
       "45  SRR5534966   1788748                0  Canarium pulchrebracteatum   \n",
       "46  SRR5534967   5687824                0     Canarium velutinifolium   \n",
       "47  SRR5534968   1694555                0        Canarium ferrugineum   \n",
       "\n",
       "   SampleName  \n",
       "0       SF327  \n",
       "1      D12950  \n",
       "2      D14269  \n",
       "3        5573  \n",
       "4       SF228  \n",
       "5     SFC1988  \n",
       "6        4304  \n",
       "7       SF224  \n",
       "8       SF343  \n",
       "9      D13101  \n",
       "10      SF155  \n",
       "11      SF200  \n",
       "12      SF209  \n",
       "13     D13374  \n",
       "14     D13063  \n",
       "15      SF160  \n",
       "16      SF286  \n",
       "17     D13090  \n",
       "18      SF172  \n",
       "19      SF276  \n",
       "20     D13052  \n",
       "21     D13097  \n",
       "22     D13103  \n",
       "23     D12963  \n",
       "24     D13852  \n",
       "25      SF175  \n",
       "26      SF328  \n",
       "27     D12962  \n",
       "28     D14505  \n",
       "29     D14504  \n",
       "30      SF197  \n",
       "31      SF301  \n",
       "32      SF153  \n",
       "33     D14501  \n",
       "34     D14485  \n",
       "35     D14482  \n",
       "36     D14513  \n",
       "37     D14477  \n",
       "38     D13091  \n",
       "39     D14480  \n",
       "40     D14478  \n",
       "41     D13075  \n",
       "42     D14492  \n",
       "43     D14483  \n",
       "44      SF164  \n",
       "45     D14528  \n",
       "46     D14506  \n",
       "47     D13053  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## get accession data from sra\n",
    "sra = ipa.sratools(accession=\"SRP106882\", workdir=\"./fastq-files\")\n",
    "\n",
    "## print run info for posterity\n",
    "run_info = sra.fetch_runinfo((1,4,6,29,30))\n",
    "run_info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[####################] 100%  Downloading fastq files | 0:24:21 |  \n",
      "48 fastq files downloaded to /home/deren/Documents/Canarium/fastq-files\n"
     ]
    }
   ],
   "source": [
    "## run parallel download\n",
    "sra.run(ipyclient=ipyclient)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ipyrad Assembly\n",
    "\n",
    "Enter parameter values for the ipyrad assembly . "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New Assembly: Canarium\n",
      "0   assembly_name               Canarium                                     \n",
      "1   project_dir                 ./analysis-ipyrad                            \n",
      "2   raw_fastq_path                                                           \n",
      "3   barcodes_path                                                            \n",
      "4   sorted_fastq_path           ./fastq-files/*.gz                           \n",
      "5   assembly_method             denovo                                       \n",
      "6   reference_sequence                                                       \n",
      "7   datatype                    gbs                                          \n",
      "8   restriction_overhang        ('CWGC', 'CWGC')                             \n",
      "9   max_low_qual_bases          5                                            \n",
      "10  phred_Qscore_offset         33                                           \n",
      "11  mindepth_statistical        6                                            \n",
      "12  mindepth_majrule            6                                            \n",
      "13  maxdepth                    10000                                        \n",
      "14  clust_threshold             0.9                                          \n",
      "15  max_barcode_mismatch        0                                            \n",
      "16  filter_adapters             2                                            \n",
      "17  filter_min_trim_len         35                                           \n",
      "18  max_alleles_consens         2                                            \n",
      "19  max_Ns_consens              (5, 5)                                       \n",
      "20  max_Hs_consens              (8, 8)                                       \n",
      "21  min_samples_locus           4                                            \n",
      "22  max_SNPs_locus              (10, 10)                                     \n",
      "23  max_Indels_locus            (8, 8)                                       \n",
      "24  max_shared_Hs_locus         4                                            \n",
      "25  trim_reads                  (0, 0, 0, 0)                                 \n",
      "26  trim_loci                   (0, 5)                                       \n",
      "27  output_formats              ('l', 'p', 's', 'v', 'k', 'a')               \n",
      "28  pop_assign_file                                                          \n"
     ]
    }
   ],
   "source": [
    "## create an Assembly\n",
    "data = ip.Assembly(\"Canarium\")\n",
    "\n",
    "## set params\n",
    "data.set_params(\"project_dir\", \"analysis-ipyrad\")\n",
    "data.set_params(\"sorted_fastq_path\", \"./fastq-files/*.gz\")\n",
    "data.set_params(\"restriction_overhang\", (\"CWGC\", \"CWGC\"))\n",
    "data.set_params(\"datatype\", \"gbs\")\n",
    "data.set_params(\"clust_threshold\", 0.90)\n",
    "data.set_params(\"filter_adapters\", 2)\n",
    "data.set_params(\"max_SNPs_locus\", (10, 10))\n",
    "data.set_params(\"max_shared_Hs_locus\", 4)\n",
    "data.set_params(\"trim_reads\", (0, 0))\n",
    "data.set_params(\"trim_loci\", (0, 5))\n",
    "data.set_params(\"output_formats\", list(\"lpsvka\"))\n",
    "\n",
    "## print params for posterity\n",
    "data.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assemble reads within each Sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data.run(\"12\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Assembly: Canarium\n",
      "[####################] 100%  dereplicating         | 0:01:55 | s3 | | \n",
      "[####################] 100%  clustering            | 15:30:31 | s3 | \n",
      "[####################] 100%  building clusters     | 0:04:02 | s3 | \n",
      "[####################] 100%  chunking              | 0:00:42 | s3 | \n",
      "[####################] 100%  aligning              | 1:00:07 | s3 | \n",
      "[####################] 100%  concatenating         | 0:03:13 | s3 | \n"
     ]
    }
   ],
   "source": [
    "data.run(\"3\", force=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Assembly: Canarium\n",
      "[####################] 100%  inferring [H, E]      | 0:12:06 | s4 | \n",
      "[####################] 100%  calculating depths    | 0:00:54 | s5 | \n",
      "[####################] 100%  chunking clusters     | 0:01:22 | s5 | \n",
      "[####################] 100%  consens calling       | 0:21:34 | s5 | \n"
     ]
    }
   ],
   "source": [
    "data.run(\"45\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Assembly: Canarium\n",
      "[####################] 100%  concat/shuffle input  | 0:01:06 | s6 | \n",
      "[####################] 100%  clustering across     | 5:43:06 | s6 | \n",
      "[####################] 100%  building clusters     | 0:00:53 | s6 | \n",
      "[####################] 100%  aligning clusters     | 0:04:13 | s6 | \n",
      "[####################] 100%  database indels       | 0:02:13 | s6 | \n",
      "[####################] 100%  indexing clusters     | 0:02:19 | s6 | \n",
      "[####################] 100%  building database     | 0:24:48 | s6 | \n"
     ]
    }
   ],
   "source": [
    "data.run(\"6\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Assembly: Canarium\n",
      "[####################] 100%  filtering loci        | 0:00:39 | s7 | \n",
      "[####################] 100%  building loci/stats   | 0:00:29 | s7 | \n",
      "[####################] 100%  building alleles      | 0:00:36 | s7 | \n",
      "[####################] 100%  building vcf file     | 0:01:06 | s7 | \n",
      "[####################] 100%  writing vcf file      | 0:00:00 | s7 | \n",
      "[####################] 100%  building arrays       | 0:00:31 | s7 | \n",
      "[####################] 100%  writing outfiles      | 0:01:00 | s7 | \n",
      "Outfiles written to: ~/Documents/Canarium/analysis-ipyrad/Canarium_outfiles\n",
      "\n"
     ]
    }
   ],
   "source": [
    "data.run(\"7\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Full assembly stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading Assembly: Canarium\n",
      "from saved path: ~/Documents/Canarium/analysis-ipyrad/Canarium.json\n"
     ]
    }
   ],
   "source": [
    "## re-load assembly in case coming back to this notebook later\n",
    "data = ip.load_json(\"analysis-ipyrad/Canarium.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total N reads: 365012834\n",
      "mean N reads/sample: 7604434.04167\n",
      "S.D. N reads/sample: 7407655.17926\n"
     ]
    }
   ],
   "source": [
    "## print some stats\n",
    "print \"total N reads:\", data.stats.reads_raw.sum()\n",
    "print \"mean N reads/sample:\", data.stats.reads_raw.mean()\n",
    "print \"S.D. N reads/sample:\", data.stats.reads_raw.std()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Excluding low-data samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sample_coverage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4304</th>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5573</th>\n",
       "      <td>37672</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D12950</th>\n",
       "      <td>44861</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D12962</th>\n",
       "      <td>1981</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D12963</th>\n",
       "      <td>10141</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D13052</th>\n",
       "      <td>75316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D13053</th>\n",
       "      <td>21812</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D13063</th>\n",
       "      <td>28227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D13075</th>\n",
       "      <td>4096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D13090</th>\n",
       "      <td>27541</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D13091</th>\n",
       "      <td>1059</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D13097</th>\n",
       "      <td>81399</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D13101</th>\n",
       "      <td>2860</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D13103</th>\n",
       "      <td>26359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D13374</th>\n",
       "      <td>56701</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D13852</th>\n",
       "      <td>13687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D14269</th>\n",
       "      <td>79114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D14477</th>\n",
       "      <td>61515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D14478</th>\n",
       "      <td>97836</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D14480</th>\n",
       "      <td>70588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D14482</th>\n",
       "      <td>69750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D14483</th>\n",
       "      <td>71352</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D14485</th>\n",
       "      <td>51841</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D14492</th>\n",
       "      <td>1272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D14501</th>\n",
       "      <td>49981</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D14504</th>\n",
       "      <td>77524</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D14505</th>\n",
       "      <td>77445</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D14506</th>\n",
       "      <td>60470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D14513</th>\n",
       "      <td>39505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D14528</th>\n",
       "      <td>26587</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SF153</th>\n",
       "      <td>9269</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SF155</th>\n",
       "      <td>53298</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SF160</th>\n",
       "      <td>2856</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SF164</th>\n",
       "      <td>69764</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SF172</th>\n",
       "      <td>64881</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SF175</th>\n",
       "      <td>69099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SF197</th>\n",
       "      <td>82914</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SF200</th>\n",
       "      <td>42029</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SF209</th>\n",
       "      <td>28325</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SF224</th>\n",
       "      <td>81039</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SF228</th>\n",
       "      <td>82338</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SF276</th>\n",
       "      <td>82961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SF286</th>\n",
       "      <td>78681</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SF301</th>\n",
       "      <td>785</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SF327</th>\n",
       "      <td>72815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SF328</th>\n",
       "      <td>80219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SF343</th>\n",
       "      <td>894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SFC1988</th>\n",
       "      <td>51364</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         sample_coverage\n",
       "4304                   7\n",
       "5573               37672\n",
       "D12950             44861\n",
       "D12962              1981\n",
       "D12963             10141\n",
       "D13052             75316\n",
       "D13053             21812\n",
       "D13063             28227\n",
       "D13075              4096\n",
       "D13090             27541\n",
       "D13091              1059\n",
       "D13097             81399\n",
       "D13101              2860\n",
       "D13103             26359\n",
       "D13374             56701\n",
       "D13852             13687\n",
       "D14269             79114\n",
       "D14477             61515\n",
       "D14478             97836\n",
       "D14480             70588\n",
       "D14482             69750\n",
       "D14483             71352\n",
       "D14485             51841\n",
       "D14492              1272\n",
       "D14501             49981\n",
       "D14504             77524\n",
       "D14505             77445\n",
       "D14506             60470\n",
       "D14513             39505\n",
       "D14528             26587\n",
       "SF153               9269\n",
       "SF155              53298\n",
       "SF160               2856\n",
       "SF164              69764\n",
       "SF172              64881\n",
       "SF175              69099\n",
       "SF197              82914\n",
       "SF200              42029\n",
       "SF209              28325\n",
       "SF224              81039\n",
       "SF228              82338\n",
       "SF276              82961\n",
       "SF286              78681\n",
       "SF301                785\n",
       "SF327              72815\n",
       "SF328              80219\n",
       "SF343                894\n",
       "SFC1988            51364"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## number of consens reads per sample. \n",
    "data.stats_dfs.s7_samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## make subset lists to exclude taxa with little data\n",
    "subs = [i.name for i in data.samples.values() if i.stats.reads_consens > 12000]\n",
    "subsnout = list(set(subs) - set([\"D14269\", \"D13374\", \"SFC1988\", \"D13852\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Final assemblies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## make new branches\n",
    "Can_min4 = data.branch(\"Canarium-min4\", subsamples=subs)\n",
    "Can_min10 = data.branch(\"Canarium-min10\", subsamples=subs)\n",
    "Can_min20 = data.branch(\"Canarium-min20\", subsamples=subs)\n",
    "Can_min30nout = data.branch(\"Canarium-min30-nout\", subsamples=subsnout)\n",
    "\n",
    "## set params on new assemblies\n",
    "Can_min10.set_params(\"min_samples_locus\", 10)\n",
    "Can_min20.set_params(\"min_samples_locus\", 20)\n",
    "Can_min30nout.set_params(\"min_samples_locus\", 30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Assembly: Canarium-min4\n",
      "[####################] 100%  filtering loci        | 0:00:50 | s7 | \n",
      "[####################] 100%  building loci/stats   | 0:00:29 | s7 | \n",
      "[####################] 100%  building alleles      | 0:00:36 | s7 | \n",
      "[####################] 100%  building vcf file     | 0:01:00 | s7 | \n",
      "[####################] 100%  writing vcf file      | 0:00:00 | s7 | \n",
      "[####################] 100%  building arrays       | 0:00:40 | s7 | \n",
      "[####################] 100%  writing outfiles      | 0:00:50 | s7 | \n",
      "Outfiles written to: ~/Documents/Canarium/analysis-ipyrad/Canarium-min4_outfiles\n",
      "\n",
      "Assembly: Canarium-min10\n",
      "[####################] 100%  filtering loci        | 0:00:41 | s7 | \n",
      "[####################] 100%  building loci/stats   | 0:00:29 | s7 | \n",
      "[####################] 100%  building alleles      | 0:00:35 | s7 | \n",
      "[####################] 100%  building vcf file     | 0:00:49 | s7 | \n",
      "[####################] 100%  writing vcf file      | 0:00:00 | s7 | \n",
      "[####################] 100%  building arrays       | 0:00:39 | s7 | \n",
      "[####################] 100%  writing outfiles      | 0:00:27 | s7 | \n",
      "Outfiles written to: ~/Documents/Canarium/analysis-ipyrad/Canarium-min10_outfiles\n",
      "\n",
      "Assembly: Canarium-min20\n",
      "[####################] 100%  filtering loci        | 0:00:41 | s7 | \n",
      "[####################] 100%  building loci/stats   | 0:00:29 | s7 | \n",
      "[####################] 100%  building alleles      | 0:00:35 | s7 | \n",
      "[####################] 100%  building vcf file     | 0:00:42 | s7 | \n",
      "[####################] 100%  writing vcf file      | 0:00:00 | s7 | \n",
      "[####################] 100%  building arrays       | 0:00:38 | s7 | \n",
      "[####################] 100%  writing outfiles      | 0:00:15 | s7 | \n",
      "Outfiles written to: ~/Documents/Canarium/analysis-ipyrad/Canarium-min20_outfiles\n",
      "\n",
      "Assembly: Canarium-min30-nout\n",
      "[####################] 100%  filtering loci        | 0:00:37 | s7 | \n",
      "[####################] 100%  building loci/stats   | 0:00:28 | s7 | \n",
      "[####################] 100%  building alleles      | 0:00:33 | s7 | \n",
      "[####################] 100%  building vcf file     | 0:00:35 | s7 | \n",
      "[####################] 100%  writing vcf file      | 0:00:00 | s7 | \n",
      "[####################] 100%  building arrays       | 0:00:35 | s7 | \n",
      "[####################] 100%  writing outfiles      | 0:00:06 | s7 | \n",
      "Outfiles written to: ~/Documents/Canarium/analysis-ipyrad/Canarium-min30-nout_outfiles\n",
      "\n"
     ]
    }
   ],
   "source": [
    "## final assemblies\n",
    "Can_min4.run(\"7\", force=True)\n",
    "Can_min10.run(\"7\", force=True)\n",
    "Can_min20.run(\"7\", force=True)\n",
    "Can_min30nout.run(\"7\", force=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assembly stats\n",
    "See the github page for stats of each assembly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading Assembly: Canarium-min4\n",
      "from saved path: ~/Documents/Canarium/analysis-ipyrad/Canarium-min4.json\n",
      "loading Assembly: Canarium-min10\n",
      "from saved path: ~/Documents/Canarium/analysis-ipyrad/Canarium-min10.json\n",
      "loading Assembly: Canarium-min20\n",
      "from saved path: ~/Documents/Canarium/analysis-ipyrad/Canarium-min20.json\n",
      "loading Assembly: Canarium-min30-nout\n",
      "from saved path: ~/Documents/Canarium/analysis-ipyrad/Canarium-min30-nout.json\n"
     ]
    }
   ],
   "source": [
    "## reoload assemblies from their JSON files\n",
    "Can_min4 = ip.load_json(\"analysis-ipyrad/Canarium-min4.json\")\n",
    "Can_min10 = ip.load_json(\"analysis-ipyrad/Canarium-min10.json\")\n",
    "Can_min20 = ip.load_json(\"analysis-ipyrad/Canarium-min20.json\")\n",
    "Can_min30n = ip.load_json(\"analysis-ipyrad/Canarium-min30-nout.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}