Project Mobile Games A.ipynb

{"cells":[{"metadata":{"dc":{"key":"3"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 1. Of cats and cookies\n<p><a href=\"https://www.facebook.com/cookiecatsgame\">Cookie Cats</a> is a hugely popular mobile puzzle game developed by <a href=\"http://tactile.dk\">Tactile Entertainment</a>. It's a classic \"connect three\"-style puzzle game where the player must connect tiles of the same color to clear the board and win the level. It also features singing cats. We're not kidding! Check out this short demo:</p>\n<p><a href=\"https://youtu.be/GaP5f0jVTWE\"><img src=\"https://assets.datacamp.com/production/project_184/img/cookie_cats_video.jpeg\" style=\"width: 500px\"></a></p>\n<p>As players progress through the levels of the game, they will occasionally encounter gates that force them to wait a non-trivial amount of time or make an in-app purchase to progress. In addition to driving in-app purchases, these gates serve the important purpose of giving players an enforced break from playing the game, hopefully resulting in that the player's enjoyment of the game being increased and prolonged.</p>\n<p><img src=\"https://assets.datacamp.com/production/project_184/img/cc_gates.png\" alt></p>\n<p>But where should the gates be placed? Initially the first gate was placed at level 30, but in this notebook we're going to analyze an AB-test where we moved the first gate in Cookie Cats from level 30 to level 40. In particular, we will look at the impact on player retention. But before we get to that, a key step before undertaking any analysis is understanding the data. So let's load it in and take a look!</p>"},{"metadata":{"dc":{"key":"3"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# Importing pandas\nimport pandas as pd\n\n# Reading in the data\ndf = pd.read_csv('datasets/cookie_cats.csv')\n\n# Showing the first few rows\ndf.head()","execution_count":2,"outputs":[{"output_type":"execute_result","execution_count":2,"data":{"text/plain":"   userid  version  sum_gamerounds  retention_1  retention_7\n0     116  gate_30               3        False        False\n1     337  gate_30              38         True        False\n2     377  gate_40             165         True        False\n3     483  gate_40               1        False        False\n4     488  gate_40             179         True         True","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>userid</th>\n      <th>version</th>\n      <th>sum_gamerounds</th>\n      <th>retention_1</th>\n      <th>retention_7</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>116</td>\n      <td>gate_30</td>\n      <td>3</td>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>337</td>\n      <td>gate_30</td>\n      <td>38</td>\n      <td>True</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>377</td>\n      <td>gate_40</td>\n      <td>165</td>\n      <td>True</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>483</td>\n      <td>gate_40</td>\n      <td>1</td>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>488</td>\n      <td>gate_40</td>\n      <td>179</td>\n      <td>True</td>\n      <td>True</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"dc":{"key":"10"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 2. The AB-test data\n<p>The data we have is from 90,189 players that installed the game while the AB-test was running. The variables are:</p>\n<ul>\n<li><code>userid</code> - a unique number that identifies each player.</li>\n<li><code>version</code> - whether the player was put in the control group (<code>gate_30</code> - a gate at level 30) or the group with the moved gate (<code>gate_40</code> - a gate at level 40).</li>\n<li><code>sum_gamerounds</code> - the number of game rounds played by the player during the first 14 days after install.</li>\n<li><code>retention_1</code> - did the player come back and play <strong>1 day</strong> after installing?</li>\n<li><code>retention_7</code> - did the player come back and play <strong>7 days</strong> after installing?</li>\n</ul>\n<p>When a player installed the game, he or she was randomly assigned to either <code>gate_30</code> or <code>gate_40</code>. As a sanity check, let's see if there are roughly the same number of players in each AB group. </p>"},{"metadata":{"dc":{"key":"10"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# Counting the number of players in each AB group.\ndf.groupby('version')['userid'].count()","execution_count":4,"outputs":[{"output_type":"execute_result","execution_count":4,"data":{"text/plain":"version\ngate_30    44700\ngate_40    45489\nName: userid, dtype: int64"},"metadata":{}}]},{"metadata":{"dc":{"key":"17"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 3. The distribution of game rounds\n<p><img src=\"https://assets.datacamp.com/production/project_184/img/mr_waffles_smiling.png\" style=\"width:200px; float:left\"> </p>\n<p>It looks like there is roughly the same number of players in each group, nice!</p>\n<p>The focus of this analysis will be on how the gate placement affects player retention, but just for fun: Let's plot the distribution of the number of game rounds players played during their first week playing the game.</p>"},{"metadata":{"dc":{"key":"17"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# This command makes plots appear in the notebook\n%matplotlib inline\n\n# Counting the number of players for each number of game rounds \nplot_df = df.groupby('sum_gamerounds')['userid'].count()\n\n# Plotting the distribution of players that played 0 to 100 game rounds\nax = plot_df.head(n=100).plot(x=\"sum_gamerounds\", y=\"userid\", kind=\"hist\")\nax.set_xlabel(\"Game Rounds\")\nax.set_ylabel(\"User Count\")","execution_count":6,"outputs":[{"output_type":"execute_result","execution_count":6,"data":{"text/plain":"Text(0,0.5,'User Count')"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<matplotlib.figure.Figure at 0x7fed55e08a58>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAFkZJREFUeJzt3WuwZWV95/HvTy7ioAitJ10UaBpiR4MpRTw6KoyjIMbbCJlxGBxKOw4zXUmMl2HGBEPNlFbNC5xKRTFxZHrUpJ1SEREC0fFCGuKlxqCHi3KTcBEmMEC3lxYlGRX4z4v9HNm0ffrsbnqd85yzv5+qXXutZ6291v/ZtU//eq299rNSVUiS1JvHLHcBkiTtjAElSeqSASVJ6pIBJUnqkgElSeqSASVJ6pIBJUnqkgElSeqSASVJ6tK+y13AJJ785CfXunXrlrsMSdJecOWVV363qmYWW29FBNS6deuYm5tb7jIkSXtBkjsmWc9TfJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC6tiJEk9oZ1Z352uUsA4PazX73cJUjSiuARlCSpSwaUJKlLgwVUkqcnuWbscV+StydZk+TSJDe350OGqkGStHINFlBVdVNVHV1VRwPPBf4euAg4E9hSVeuBLW1ekqRHWKpTfCcAt1bVHcBJwObWvhk4eYlqkCStIEsVUKcCn2jTa6vq7jZ9D7B2Zy9IsjHJXJK5bdu2LUWNkqSODB5QSfYHXgt8asdlVVVA7ex1VbWpqmaranZmZtEbL0qSVpmlOIJ6JXBVVd3b5u9NcihAe966BDVIklaYpQio1/Pw6T2AS4ANbXoDcPES1CBJWmEGDagkBwInAheONZ8NnJjkZuBlbV6SpEcYdKijqrofeNIObd9jdFWfJEkLciQJSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXBg2oJAcnuSDJt5PcmOSFSdYkuTTJze35kCFrkCStTEMfQZ0DfL6qngE8G7gROBPYUlXrgS1tXpKkRxgsoJI8EXgx8GGAqvppVW0HTgI2t9U2AycPVYMkaeUa8gjqCGAb8GdJrk7yoSQHAmur6u62zj3A2p29OMnGJHNJ5rZt2zZgmZKkHg0ZUPsCxwAfrKrnAPezw+m8qiqgdvbiqtpUVbNVNTszMzNgmZKkHg0ZUHcCd1bVFW3+AkaBdW+SQwHa89YBa5AkrVCDBVRV3QP8XZKnt6YTgBuAS4ANrW0DcPFQNUiSVq59B97+W4CPJdkfuA14E6NQPD/J6cAdwCkD1yBJWoEGDaiqugaY3cmiE4bcryRp5XMkCUlSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpf2HXLjSW4HfgQ8CDxQVbNJ1gCfBNYBtwOnVNUPhqxDkrTyLMUR1Eur6uiqmm3zZwJbqmo9sKXNS5L0CMtxiu8kYHOb3gycvAw1SJI6N3RAFfDFJFcm2dja1lbV3W36HmDtzl6YZGOSuSRz27ZtG7hMSVJvBv0OCjiuqu5K8kvApUm+Pb6wqipJ7eyFVbUJ2AQwOzu703UkSavXoEdQVXVXe94KXAQ8H7g3yaEA7XnrkDVIklamwQIqyYFJnjA/DbwcuA64BNjQVtsAXDxUDZKklWvIU3xrgYuSzO/n41X1+STfAM5PcjpwB3DKgDVIklaowQKqqm4Dnr2T9u8BJwy1X0nS6uBIEpKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuLRpQSY6dpE2SpL1pkiOoP5mwTZKkvWbB30EleSHwImAmyRljiw4C9hm6MEnSdNvVD3X3Bx7f1nnCWPt9wOuGLEqSpAUDqqq+BHwpyZ9X1R1LWJMkSRMNdfTYJJsY3aL95+tX1fFDFSVJ0iQB9SngXOBDwIPDliNJ0sgkAfVAVX1w8EokSRozyWXmf5nkd5McmmTN/GPwyiRJU22SI6j5mwu+Y6ytgCP3fjmSJI0sGlBVdcRSFCJJ0rhFAyrJG3fWXlUf3fvlSJI0MskpvueNTR/A6G64VwEGlCRpMJOc4nvL+HySg4HzBqtIkiT27HYb9wN+LyVJGtQk30H9JaOr9mA0SOyvAecPWZQkSZN8B/VHY9MPAHdU1Z0D1SNJEjDBKb42aOy3GY1ofgjw093ZQZJ9klyd5DNt/ogkVyS5Jcknk+y/J4VLkla3Se6oewrwdeBfAqcAVyTZndttvA24cWz+PcB7q+ppwA+A03djW5KkKTHJRRJnAc+rqg1V9Ubg+cB/mmTjSQ4HXs1ooFmSBDgeuKCtshk4eXeLliStfpME1GOqauvY/PcmfB3A+4DfBx5q808CtlfVA23+TuCwnb0wycYkc0nmtm3bNuHuJEmrxSRB8/kkX0jyW0l+C/gs8LnFXpTkNcDWqrpyTwqrqk1VNVtVszMzM3uyCUnSCjbJD3XfkeSfA8e1pk1VddEE2z4WeG2SVzEageIg4Bzg4CT7tqOow4G79qx0SdJqtuARVJKnJTkWoKourKozquoMYFuSX1lsw1X1zqo6vKrWAacCl1XVacDlwPxFFhuAix9tJyRJq8+uTvG9D7hvJ+0/bMv21B8AZyS5hdF3Uh9+FNuSJK1SuzrFt7aqrt2xsaquTbJud3ZSVX8N/HWbvo3RlYCSJC1oV0dQB+9i2eP2diGSJI3bVUDNJfl3OzYm+bfAHl2ZJ0nSpHZ1iu/twEVJTuPhQJoF9gd+c+jCJEnTbcGAqqp7gRcleSnw6635s1V12ZJUJkmaapP8DupyRpeGS5K0ZPbkhoWSJA3OgJIkdWmXAdXu5eTpPUnSkttlQFXVg8BDSZ64RPVIkgRMdsv3HwPXJrkUuH++sareOlhVkqSpN0lAXdgekiQtmUkuM9+c5HHAU6vqpiWoSZKkxa/iS/LPgGuAz7f5o5NcMnRhkqTpNsll5u9iNPr4doCqugY4csCaJEmaKKB+VlU/3KHtoSGKkSRp3iQXSVyf5F8D+yRZD7wV+N/DliVJmnaTHEG9BXgm8BPgE4zusvv2IYuSJGmSq/j+HjgLOCvJPsCBVfX/Bq9MkjTVJrmK7+NJDkpyIHAtcEOSdwxfmiRpmk1yiu+oqroPOBn4HHAE8IZBq5IkTb1JAmq/JPsxCqhLqupnQA1bliRp2k0SUP8duB04EPhykl9mdKGEJEmDWTSgqur9VXVYVb2qqgr4P8BLF3tdkgOSfD3JN5Ncn+Tdrf2IJFckuSXJJ5Ps/+i7IUlabRa8ii/JGTs0FfBd4KtV9Z0Jtv0T4Piq+nE7RfjVJJ8DzgDeW1XnJTkXOB344J6VL0larXZ1BPWEHR4HAbPA55KcutiGa+THbXa/9ijgeOCC1r6Z0XdbkiQ9woJHUFX17p21J1kD/BVw3mIbb7+buhJ4GvAB4FZge1U90Fa5EzhsgdduBDYCPPWpT11sV5KkVWaSiyQeoaq+D2TCdR+sqqOBwxkNOPuM3djPpqqararZmZmZ3S1TkrTC7XZAJXkp8IPdeU1VbQcuB14IHJxk/sjtcOCu3a1BkrT67eoiiWv5xd87rQH+L/DGxTacZIbRSOjb2w0PTwTewyioXsfoFOEG4OI9K12StJrtaiy+1+wwX8D3qur+Cbd9KLC5fQ/1GOD8qvpMkhuA85L8F+Bq4MO7W7QkafXb1UUSdzyaDVfVt4Dn7KT9NkbfR0mStKDd/g5KkqSlYEBJkrpkQEmSumRASZK6ZEBJkrpkQEmSumRASZK6ZEBJkrpkQEmSumRASZK6ZEBJkrpkQEmSumRASZK6ZEBJkrpkQEmSumRASZK6ZEBJkrpkQEmSumRASZK6ZEBJkrpkQEmSujRYQCV5SpLLk9yQ5Pokb2vta5JcmuTm9nzIUDVIklauIY+gHgD+Q1UdBbwAeHOSo4AzgS1VtR7Y0uYlSXqEwQKqqu6uqqva9I+AG4HDgJOAzW21zcDJQ9UgSVq5luQ7qCTrgOcAVwBrq+rutugeYO0Cr9mYZC7J3LZt25aiTElSRwYPqCSPBz4NvL2q7htfVlUF1M5eV1Wbqmq2qmZnZmaGLlOS1JlBAyrJfozC6WNVdWFrvjfJoW35ocDWIWuQJK1MQ17FF+DDwI1V9cdjiy4BNrTpDcDFQ9UgSVq59h1w28cCbwCuTXJNa/tD4Gzg/CSnA3cApwxYgyRphRosoKrqq0AWWHzCUPuVJK0OjiQhSeqSASVJ6pIBJUnqkgElSeqSASVJ6pIBJUnqkgElSeqSASVJ6pIBJUnqkgElSeqSASVJ6pIBJUnqkgElSeqSASVJ6pIBJUnqkgElSeqSASVJ6pIBJUnqkgElSeqSASVJ6pIBJUnqkgElSerSYAGV5CNJtia5bqxtTZJLk9zcng8Zav+SpJVtyCOoPwdesUPbmcCWqloPbGnzkiT9gsECqqq+DHx/h+aTgM1tejNw8lD7lyStbEv9HdTaqrq7Td8DrF1oxSQbk8wlmdu2bdvSVCdJ6sayXSRRVQXULpZvqqrZqpqdmZlZwsokST1Y6oC6N8mhAO156xLvX5K0Quy7xPu7BNgAnN2eL17i/S+7dWd+drlLAOD2s1+93CVI0i4NeZn5J4CvAU9PcmeS0xkF04lJbgZe1uYlSfoFgx1BVdXrF1h0wlD7lCStHkt9ik+d8FSjpN451JEkqUsGlCSpSwaUJKlLBpQkqUsGlCSpSwaUJKlLBpQkqUsGlCSpSwaUJKlLBpQkqUsGlCSpSwaUJKlLBpQkqUsGlCSpSwaUJKlL3g9Ky8r7UklaiEdQkqQuGVCSpC55ik/qhKc7H8n3Qx5BSZK6ZEBJkrqUqlr6nSavAM4B9gE+VFVn72r92dnZmpube1T77OV0gSStZHvjlGeSK6tqdrH1lvwIKsk+wAeAVwJHAa9PctRS1yFJ6ttynOJ7PnBLVd1WVT8FzgNOWoY6JEkdW46r+A4D/m5s/k7gH++4UpKNwMY2++MkNy2y3ScD390rFa4809r3ae032Pdp7HsX/c579spmfnmSlbq9zLyqNgGbJl0/ydwk5zRXo2nt+7T2G+z7NPZ9Gvu9HKf47gKeMjZ/eGuTJOnnliOgvgGsT3JEkv2BU4FLlqEOSVLHlvwUX1U9kOT3gC8wusz8I1V1/V7Y9MSnA1ehae37tPYb7Ps0mrp+L8vvoCRJWowjSUiSumRASZK6tOIDKskrktyU5JYkZy53PXtDko8k2ZrkurG2NUkuTXJzez6ktSfJ+1v/v5XkmLHXbGjr35xkw3L0ZXckeUqSy5PckOT6JG9r7dPQ9wOSfD3JN1vf393aj0hyRevjJ9uFRSR5bJu/pS1fN7atd7b2m5L8xvL0aPcl2SfJ1Uk+0+anou9Jbk9ybZJrksy1tlX/mZ9IVa3YB6OLLG4FjgT2B74JHLXcde2Ffr0YOAa4bqztvwJntukzgfe06VcBnwMCvAC4orWvAW5rz4e06UOWu2+L9PtQ4Jg2/QTgbxkNhzUNfQ/w+Da9H3BF69P5wKmt/Vzgd9r07wLntulTgU+26aPa38FjgSPa38c+y92/Cd+DM4CPA59p81PRd+B24Mk7tK36z/wkj5V+BLUqh02qqi8D39+h+SRgc5veDJw81v7RGvkb4OAkhwK/AVxaVd+vqh8AlwKvGL76PVdVd1fVVW36R8CNjEYemYa+V1X9uM3u1x4FHA9c0Np37Pv8e3IBcEKStPbzquonVfUd4BZGfyddS3I48GrgQ20+TEnfF7DqP/OTWOkBtbNhkw5bplqGtraq7m7T9wBr2/RC78GKfm/aaZvnMDqSmIq+t1Nc1wBbGf0DcyuwvaoeaKuM9+PnfWzLfwg8iRXad+B9wO8DD7X5JzE9fS/gi0muzGiIN5iSz/xiuh3qSAurqkqyan8fkOTxwKeBt1fVfaP/HI+s5r5X1YPA0UkOBi4CnrHMJS2JJK8BtlbVlUlestz1LIPjququJL8EXJrk2+MLV/NnfjEr/QhqmoZNurcdytOet7b2hd6DFfneJNmPUTh9rKoubM1T0fd5VbUduBx4IaNTOPP/kRzvx8/72JY/EfgeK7PvxwKvTXI7o9P0xzO6X9w09J2quqs9b2X0H5PnM2Wf+YWs9ICapmGTLgHmr8zZAFw81v7GdnXPC4AftlMDXwBenuSQdgXQy1tbt9r3CB8GbqyqPx5bNA19n2lHTiR5HHAio+/gLgde11bbse/z78nrgMtq9G35JcCp7Uq3I4D1wNeXphd7pqreWVWHV9U6Rn/Dl1XVaUxB35McmOQJ89OMPqvXMQWf+Yks91Uaj/bB6KqWv2V0vv6s5a5nL/XpE8DdwM8YnUs+ndE59i3AzcBfAWvaumF0A8hbgWuB2bHt/BtGXxTfArxpufs1Qb+PY3Q+/lvANe3xqinp+7OAq1vfrwP+c2s/ktE/srcAnwIe29oPaPO3tOVHjm3rrPae3AS8crn7tpvvw0t4+Cq+Vd/31sdvtsf18/+GTcNnfpKHQx1Jkrq00k/xSZJWKQNKktQlA0qS1CUDSpLUJQNKktQlA0pTL8naJB9PclsbbuZrSX5zCff/kiQ/bKNZfzvJHy3BPtdlbLR8qUcGlKZa+3HwXwBfrqojq+q5jH4sevgSl/KVqjqa0fiDr0ly7BLvX+qOAaVpdzzw06o6d76hqu6oqj+Bnx9pfCXJVe3xotb+kiRfSnJxO/I6O8lpGd3T6dokv9LWm0ny6STfaI9dBk9V/QOjHygf1l6/JslftHv//E2SZ7X2dyX5j/OvS3Jdq3VdkhuT/I+M7iv1xTYyBUmem9H9pr4JvHnstc9sdV/T9rN+77y10qNjQGnaPRO4ahfLtwInVtUxwL8C3j+27NnAbwO/BrwB+NWqej6jW0a8pa1zDvDeqnoe8C/asgW1YWrWA19uTe8Grq6qZwF/CHx0gj6tBz5QVc8Etrf9AvwZ8JaqevYO6/82cE47gptlNHqJtOwczVwak+QDjIZc+mkLlf2AP01yNPAg8Ktjq3+j2i0RktwKfLG1Xwu8tE2/DDhqbET2g5I8vh6+99O8f9KObNYD76uqe1r7cbSAqarLkjwpyUGLdOM7VXVNm74SWNfG+Tu4RvcaA/ifwCvb9NeAszK6J9OFVXXzItuXloRHUJp21zO6ezEAVfVm4ARgpjX9e+BeRkdLs4zu3DzvJ2PTD43NP8TD//l7DPCCqjq6PQ7bSTjB6DuoZzM6oju9BeKuPMAj/34PWKCuB1nkP6JV9XHgtcA/AP8ryfGL7FtaEgaUpt1lwAFJfmes7R+NTT8RuLuqHmJ0Gm+f3dz+F3n4dB+LBU+N7gR7NvAHrekrwGnttS8BvltV9zG6Tfgxrf0YRrc439V2twPbkxzXmk4bq+lI4Laqej+jUbOfNVnXpGEZUJpqNRot+WTgnyb5TpKvM7rF9nxA/DdgQzv99gzg/t3cxVuB2XbxwQ2Mvu9ZzLnAizO6q/C7gOcm+Raj4Jq/BcOngTVJrgd+j9GI/ot5E/CBjO7am7H2U4DrWvuvM9n3XNLgHM1cktQlj6AkSV0yoCRJXTKgJEldMqAkSV0yoCRJXTKgJEldMqAkSV36/4YJLCA5uQf6AAAAAElFTkSuQmCC\n"},"metadata":{"needs_background":"light"}}]},{"metadata":{"dc":{"key":"24"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 4. Overall 1-day retention\n<p>In the plot above we can see that some players install the game but then never play it (0 game rounds), some players just play a couple of game rounds in their first week, and some get really hooked!</p>\n<p>What we want is for players to like the game and to get hooked. A common metric in the video gaming industry for how fun and engaging a game is <em>1-day retention</em>: The percentage of players that comes back and plays the game <em>one day</em> after they have installed it.  The higher 1-day retention is, the easier it is to retain players and build a large player base. </p>\n<p>As a first step, let's look at what 1-day retention is overall.</p>"},{"metadata":{"dc":{"key":"24"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# The % of users that came back the day after they installed\ndf['retention_1'].sum() / df['retention_1'].count()","execution_count":8,"outputs":[{"output_type":"execute_result","execution_count":8,"data":{"text/plain":"0.4452095044850259"},"metadata":{}}]},{"metadata":{"dc":{"key":"31"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 5. 1-day retention by AB-group\n<p><img src=\"https://assets.datacamp.com/production/project_184/img/belle_cookie.png\" style=\"width:200px; float:right\"> </p>\n<p>So, a little less than half of the players come back one day after installing the game. Now that we have a benchmark, let's look at how 1-day retention differs between the two AB-groups.</p>"},{"metadata":{"dc":{"key":"31"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# Calculating 1-day retention for each AB-group\ndf.groupby('version')['retention_1'].sum() / df.groupby('version')['userid'].count()","execution_count":10,"outputs":[{"output_type":"execute_result","execution_count":10,"data":{"text/plain":"version\ngate_30    0.448188\ngate_40    0.442283\ndtype: float64"},"metadata":{}}]},{"metadata":{"dc":{"key":"38"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 6. Should we be confident in the difference?\n<p>It appears that there was a slight decrease in 1-day retention when the gate was moved to level 40 (44.2%) compared to the control when it was at level 30 (44.8%). It's a small change, but even small changes in retention can have a large impact. But while we are certain of the difference in the data, how certain should we be that a gate at level 40 will be worse in the future?</p>\n<p>There are a couple of ways we can get at the certainty of these retention numbers. Here we will use bootstrapping: We will repeatedly re-sample our dataset (with replacement) and calculate 1-day retention for those samples. The variation in 1-day retention will give us an indication of how uncertain the retention numbers are.</p>"},{"metadata":{"dc":{"key":"38"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# Creating an list with bootstrapped means for each AB-group\nboot_1d = []\nfor i in range(500):\n    boot_mean = df.sample(frac=1, replace=True).groupby('version')['retention_1'].mean()\n    boot_1d.append(boot_mean)\n    \n# Transforming the list to a DataFrame\nboot_1d = pd.DataFrame(boot_1d)\n    \n# A Kernel Density Estimate plot of the bootstrap distributions\nboot_1d.plot(kind='kde')","execution_count":12,"outputs":[{"output_type":"execute_result","execution_count":12,"data":{"text/plain":"<matplotlib.axes._subplots.AxesSubplot at 0x7fed55bec4a8>"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<matplotlib.figure.Figure at 0x7fed55bd86d8>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl8XHW9+P/XO3uatc3epG3SJk23tOkGBVpFdgSqsuMKLhUX9OrXey9eF7iK16si/vTqFeGCoIKIIFDZF2UtFLrvbZI2bZJmT7M363x+f5yZkqZZJpnlzJm+n4/HPCZz5sznvDNN5z3n83mfz0eMMSillFKhJsLuAJRSSqmRaIJSSikVkjRBKaWUCkmaoJRSSoUkTVBKKaVCkiYopZRSIUkTlFJKqZCkCUoppVRI0gSllFIqJEXZHYAv0tPTTX5+vt1hKKWUmoDNmzc3GWMyxtvP0QkqPz+fTZs22R2GUkqpCRCRw97sp118SimlQpImKKWUUiFJE5RSSqmQ5OgxKKWUCrb+/n6qq6vp6emxO5SQFxcXR15eHtHR0ZN6vSYopZSagOrqapKSksjPz0dE7A4nZBljaG5uprq6moKCgkm1oV18Sik1AT09PaSlpWlyGoeIkJaW5tOZpiYopZSaIE1O3vH1fdIEpZRSKiRpglJKjeuf+xv45qPbeGZHrd2hKC98//vf5+WXX7Y7DJ8FrEhCRO4HLgcajDGL3Nv+AhS7d0kFWo0xpSKSD+wF9rufe8cYc3OgYlNKee/V/Q189oH3iBThb1tqMCzl8sXT7Q7rtGeMwRhDRMSp5xk/+MEPbIjI/wJ5BvUAcMnQDcaY64wxpcaYUuBx4G9Dnq7wPKfJSanQ0NM/yHef3MWcjES2fP9CFuelcMfTe+npH7Q7tLBx66238pvf/ObE49tvv50777yTn/3sZ6xcuZLFixdz2223AVBZWUlxcTGf/vSnWbRoEVVVVdx4440sWrSIkpISfvGLXwBw44038thjjwHwyiuvsHTpUkpKSvjsZz9Lb28vYE0Vd9ttt7Fs2TJKSkrYt29fkH/z8QUsQRljXgdaRnpOrJGza4E/B+r4SinfPbuzlupjx/nuZfNJjovmWxcVU9fewwu76+wOLWxcd911PProoyceP/roo2RkZFBWVsa7777Ltm3b2Lx5M6+//joAZWVlfPnLX2b37t00NTVRU1PDrl272LlzJzfddNNJbff09HDjjTfyl7/8hZ07dzIwMMBvf/vbE8+np6ezZcsWvvSlL3HnnXcG5xeeALvGoNYA9caYsiHbCkRkq4i8JiJrRnuhiKwTkU0isqmxsTHwkSo1nr4ueOk2eOBy+MePoLfT7oj85o/vHGZ2RgIfnGtNPL26MJ3c1Hie2Fpjc2ThY+nSpTQ0NHD06FG2b9/O1KlT2blzJy+++CJLly5l2bJl7Nu3j7Iy6+Ny1qxZrFq1CoDZs2dz8OBBbrnlFp5//nmSk5NPanv//v0UFBQwd+5cAD7zmc+cSHQAV155JQDLly+nsrIyCL/txNiVoG7g5LOnWmCmMWYp8E3gYRFJHumFxph7jDErjDErMjLGna1dqcByDcLD18Fbv4SeNnj9p3DfRdDp/C9P++s62HqklU+cOetEuXBEhLC2dDpvlDXRdrzf5gjDxzXXXMNjjz3GX/7yF6677jqMMXz7299m27ZtbNu2jfLycj73uc8BkJCQcOJ1U6dOZfv27Zx77rncfffdfP7zn5/QcWNjYwGIjIxkYGDAf7+QnwQ9QYlIFHAl8BfPNmNMrzGm2f3zZqACmBvs2JSasE33Q+UbsPZ/4OY34BOPQ8tBePRTMOjsD/Dnd9UhAlcszjlp+3nzMhl0GTaUN9kUWfi57rrreOSRR3jssce45ppruPjii7n//vvp7LTOxmtqamhoaDjldU1NTbhcLq666iruuOMOtmzZctLzxcXFVFZWUl5eDsAf//hHPvjBDwb+F/ITO6Y6ugDYZ4yp9mwQkQygxRgzKCKzgSLgoA2xKeW9wX7rzGnGKlj6SWtb0QVWsvrb5+GfP4ILbrczQp+8uKeOpTNSyUyOO2l76YxUkmKjeO1AI5eW5IzyajURCxcupKOjg9zcXHJycsjJyWHv3r2cddZZACQmJvKnP/2JyMjIk15XU1PDTTfdhMvlAuDHP/7xSc/HxcXx+9//nmuuuYaBgQFWrlzJzTc7pwZNjDGBaVjkz8C5QDpQD9xmjLlPRB7AKiO/e8i+VwE/APoBl3vfv493jBUrVhhdsFDZZs9T8Oin4YZHoPjSk5976iuw7c+w7lXIWWxHdD6paulmzU//ybcvnccXPzjnlOe/8IdNHKjv4LV//ZAN0dlr7969zJ8/3+4wHGOk90tENhtjVoz32oCdQRljbhhl+40jbHscq+xcKefY/QRMSYfCC0997qI74MCL8PevwedfgYjIU/cJYa/srQfg4oXZIz6/fNZUXtpTT1NnL+mJscEMTZ1GdCYJpSajrxsOvAAL1kLkCN/z4qfCxT+Co1thx6OnPh/i3ixvYlbaFPLTE0Z8fvmsqQBsPdIazLDUaUYTlFKTceRt6O+G4stG32fR1TB9KfzjDug/HrzYfDQw6GLjwRbOnpM+6j4luSlERQibDx8LYmTqdKMJSqnJqHwDIqJg5qrR94mIgAt/AO3VsPHu0fcLMTtq2ujoHWB14egJKi46koXTk9lyRBOUChxNUEpNxqE3IHcFxCaOvV/BB6DoYnjjF3DcGd1hnvLxs+akjbnf4rxU9hxtx+UKTKGVUpqglJqo/uPW2NKss73b/7zvQG8bvHdvYOPykzfLm1iQk8y0hJgx91swPZnO3gGqjnUHKTJ1utEEpdRE1e8GMwi5y7zbP2cJFF0Eb/+vNS1SCDveN8iWw62cUzj22RPAghxrspc9R9sDHZY6TWmCUmqijm617qcv9f41a74Fx1tg8wMBCclfNh1uoW/QxdljjD95FGcnERkh7KnVBBXqnnzySfbs2TOp1959992UlJRQWlrK6tWrT2rnxz/+MYWFhRQXF/PCCy/4K9wT7JhJQilnO7rNuv4pOdf718w8E/LXwIb/gTPWQWR04OLzwVvlzURFCGfkTxt337joSOZkJJzWZ1D/+ffdfv/9F0xP5rYrFvq1zSeffJLLL7+cBQsWTPi1H//4x0/MPrF+/Xq++c1v8vzzz7Nnzx4eeeQRdu/ezdGjR7ngggs4cODAKbNd+ELPoJSaqNptML0U3BOoeu2sr0JHLex7OjBx+cGGiiaWzkwlIda7764LcpL1DMomP/zhDykuLmb16tXccMMN3Hnnndx7772sXLmSJUuWcNVVV9Hd3c2GDRtYv349//qv/0ppaSkVFRVUVFRwySWXsHz5ctasWTPmWlBDZ0jv6uo6MXHwU089xfXXX09sbCwFBQUUFhby7rvv+vV31DMopSaivwca9sLcS8bfd7iiCyF1Jrx3Hyz8mP9j81Fbdz87a9r42nlFXr9mwfRkntx2lJauvnGLKsKRv890vPXee+/x+OOPs337dvr7+1m2bBnLly/nyiuv5Atf+AIA3/3ud7nvvvu45ZZbWLt2LZdffjlXX301AOeffz533303RUVFbNy4kS9/+cv84x//GPV4v/nNb7jrrrvo6+s7sV9NTc2JZT8A8vLyqKnx7zIsegal1EQ0HbAKJLIm8cEUEQkrPmtdQ9Ww1/+x+ejtg00YA6uLxh9/8piblQTAgfqOQIWlRvDWW2/xkY98hLi4OJKSkrjiiisA2LVrF2vWrKGkpISHHnqI3bt3n/Lazs5ONmzYwDXXXENpaSlf/OIXqa2tHfN4X/nKV6ioqOAnP/kJd9xxR0B+p5FoglJqIpoOWPcZ8yb3+qWfsi7w3faw/2Lyk7fKm5kSE8mSvFSvX+NJUGWaoELCjTfeyK9//Wt27tzJbbfdRk9Pzyn7uFwuUlNTT6w1tW3bNvbu9e4L0/XXX8+TTz4JQG5uLlVVVSeeq66uJjd3AuOyXtAEpdRENO4DiYC0U2f49kpCOsw5H3Y9Du4lEkLFWxVNnFEwjZgo7z8WclLiSIqN4kB9+Kwi7ATnnHMOf//73+np6aGzs5Onn7bGNTs6OsjJyaG/v5+HHnroxP5JSUl0dFhfIpKTkykoKOCvf/0rAMYYtm/fPuqxPCv5AjzzzDMUFVldwGvXruWRRx6ht7eXQ4cOUVZWxhlnnOHX31MTlFIT0bgfps2GKB9m8C65Btpr4MgG/8Xlo/r2Hg42dnH2OLNHDCciFGYlahdfkK1cuZK1a9eyePFiLr30UkpKSkhJSeGHP/whZ555Jueccw7z5r1/ln/99dfzs5/9jKVLl1JRUcFDDz3Efffdx5IlS1i4cCFPPfXUqMf69a9/zcKFCyktLeWuu+7iwQcfBKw1rK699loWLFjAJZdcwm9+8xu/VvBBANeDCgZdD0oF3W/OhGlz4AYfuuj6uuBnhVaiWvsr/8Xmg6e21fD1R7bx9C2rWZSbMqHX/vtjO3hpbz1bvjfCsiNhKFTWg+rs7CQxMZHu7m4+8IEPcM8997BsmZcXjweRL+tB6RmUUt4a7Ifmcsgo9q2dmASYezHsfzZkuvneOdhMUlwU83OSx995mKKsRFq6+mjq7A1AZGo069ato7S0lGXLlnHVVVeFZHLylZaZK+WtlkPgGvA9QQHMvdRa8LB2K+Qu9709H71d0cyZBdOIjJjgtV2cXMmnixcGz8MP+7fQ5kc/+tGJcSmPa665hu985zt+Pc5EaIJSylstB637aZMskBiq8AKr2OLAC7YnqNq241Q2d/PJVbMm9fr3K/k6x1xDSoW273znO7Ymo5FoF59S3jp2yLqfVuB7WwlpkLcSDjzve1s+eudgMwCrZk+sQMIjKzmWpLgoLZRQfqcJSilvtRyCmCSYMrkP8lPMvRhqt0NHnX/am6R3KlpInuT4E1iVfHOzkijTUnPlZ5qglPLWsUMwLX/ic/CNZs551n3lm/5pb5LeOdTMGQVpkxp/8piblciBhg6cXBWsQo8mKKW8dawSpub7r73sxRCbbE19ZJOjrcc53Nw97uq54ynKTKK1u59GreRTfhSwBCUi94tIg4jsGrLtdhGpEZFt7tuHhzz3bREpF5H9InJxoOJSalJcLjh2GKb6YfzJIyLSWpXXxjOo98efxl9eYyxDCyVU6PFlPSiPxx9/HBFh6LWnTl4P6gHg18Afhm3/hTHmzqEbRGQBcD2wEJgOvCwic40xgwGMTynvdRyFwV7/FEgMlb/aKpToqIOkbP+27YV3DjaTEh/N/OzJjT95zM1OBGB/XQfneLHYYdh47lao2+nfNrNL4NL/9muTvqwHBdYUSr/85S8588wzT2xz9HpQxpjXgRYvd/8I8IgxptcYcwgoB/w7qZNSvmhxV/D58wwKrAQFtp1FvX3Quv4pwofxJ4CMxFimTommrEEr+YIlWOtBAXzve9/j3//934mLizuxLVzXg/qqiHwa2AT8P2PMMSAXeGfIPtXubUqFBk+JuT/HoMAah4pJgsMboORq/7Y9jqOtx6lqOc6NZ/uedD2VfPvrTrME5eczHW8Fcz2oLVu2UFVVxWWXXcbPfvazE9uDsR5UsBPUb4EfAsZ9/3PgsxNpQETWAesAZs6c6e/4lBrZsUprmYyUGf5tNyIScpdCzWb/tuuFTYePAXi1vLs3irOT+NuWGowxJ1ZdVYExdD2ouLi4k9aD+u53v0trayudnZ1cfPGpw/lD14Py6O0dubjF5XLxzW9+kwceeCAgv8d4gpqgjDH1np9F5F7As/Z1DTD0f36ee9tIbdwD3APWZLGBiVSpYVqPQHIuRAbgv0zuCtjwK+g/DtHx/m9/FJsrW5gSE8n8nCS/tDc3K4nO3gGOtvWQmxq830O978Ybb+TJJ59kyZIlPPDAA7z66qun7DN0PajxdHR0sGvXLs4991wA6urqWLt2LevXrw+/9aBEJGfIw48Bngq/9cD1IhIrIgVAEeDfzkylfNFW7f+zJ4/c5dYcf7U7AtP+KDYfOUbpjFSiIv3zMVCc7Z6T73Tr5rNBsNaDSklJoampicrKSiorK1m1ahXr169nxYoVzl4PSkT+DLwNFItItYh8DvipiOwUkR3Ah4BvABhjdgOPAnuA54GvaAWfCilt1ZCSF5i289yrDtQEb+mYrt4B9tZ2sHzWVL+1OTfTSlD7dcqjgAvmelCj0fWgxqHrQamgcA3CDzNg9Tfg/O8F5hh3LYSZZ8LV9wem/WHeKm/iE/+3kQduWsm5xZl+a3fVf73C2XPSuOu6Ur+1GWp0PaiJ8WU9KJ3NXKnxdNSBGYSUABaW5i6D6uB92dp8+BgisHSm/86gAOZmJ+kZVJCsW7eOPXv20NPTw2c+85mQTE6+0gSl1Hjaqq37QI1BgdXNt3c9dDVBQuAvdN10+BhzM5NIiY/2a7vFWYk8eLCZQZfxaW4/NT5dD0opBW3uSqVAjUEBTHd/+z26FYoCu3S6MYatR45x+eLpfm97blYSfQMuDjd3MTsj0e/th4pwLKUPxHpQvg4h6WSxSo3HcwaVHMAuvuwS674u8JV8h5u76egZoHRGit/bPlHJF8bdfHFxcTQ3N+vM7eMwxtDc3HzS7BMTpWdQSo2nvQZiUyDOt/nqxhSfCqkz/T+v2wh21LQBsCjX/wmqMDMREdhf18kli/zefEjIy8ujurqaxsZGu0MJeXFxceTlTb7nQROUUuMJZIn5UNmLg5KgdtW0ERMVcWIGcn+aEhPFzGlTwvoMKjo6moICP8/JqEakXXxKjaetKngJqrkCegO7ZMWO6lbm5yQT7acLdIebm6WVfMo/NEEpNZ62msCWmHtklwAG6ncH7BAul2F3TTsluYHrrizOSuJQUxc9/XqtvfKNJiilxtLXBcdbgnQGFfhCicrmLjp6B1icmxqwYyzKTWbQZU6/mc2V32mCUmosbe45iwN5DZRHSh7EpQZ0HGpnAAskPBZOt9redbQtYMdQpwdNUEqNxXMNVCBLzD1EICewhRI7q60CiaKswF2jlDc1nuS4KHbVtAfsGOr0oAlKqbG0e86ggtDFB1ahRMMeGBwISPN7atuZn50UsAIJsBYvXJSbwm49g1I+0gSl1FjaqgGBZP/PujCi7BIY6IHmsoA0X9bQGZDy8uEW5aawr66D/kFXwI+lwpcmKKXG0lYNSTkQ6d8560blKZQIwNpQbd39NHb0UpgZ+CmIFk5Ppm/ARXlDYEvmVXjTBKXUWNqqglNi7pE+FyJjoN7/41DljVZVXSDHnzxOFErUaDefmjxNUEqNpa0meONPYJ2pZc6Hul3j7ztBZfXW2UxhRuC7+ArSE5gSE8nuo1oooSZPE5RSozEmeNMcDZVVYlXy+Xky0vKGTmKjIsidGu/XdkcSGSEsyEnWQgnlE01QSo2mqwkGeyE5yAkquwS6m6Cz3q/Nljd2MicjMWjrNFmVfO0MunTWbzU5mqCUGk27Z6HCYCco9zTgfr4eqqy+MygFEh4luSl09w1S0aiFEmpyNEEpNZo2mxJUlv8TVHffADWtx4OaoEpnWtMpbT1yLGjHVOFFE5RSo7ErQcWnQop/14Y62NgFQFEQE1RBWgIp8dFsq2oN2jFVeNEEpdRo2qohKg6mpAX/2NklUO+/Sr6yBqvEPJhnUBERwpIZqWw9oglKTY4mKKVG01ZtzcEnwSkqOEn2Imguh75uvzRX3tBJVIQwKy3BL+15a+mMVA7Ud9DZG5ipm1R4C1iCEpH7RaRBRHYN2fYzEdknIjtE5AkRSXVvzxeR4yKyzX27O1BxKeU1O0rMPbJLwLigYa9fmiur72RW2hRiooL7nbR0ZiouYy2SqNREBfKv9QHgkmHbXgIWGWMWAweAbw95rsIYU+q+3RzAuJTyTlt1cJbZGMmJQgn/THlU3hjcCj6P0jyrUELHodRkBCxBGWNeB1qGbXvRGOM5138HsOnrqVLjGOizrkOy6wwqdRbEJvtlHKpvwMXh5m6KMgM/g8RwUxNiKEhPYJuOQ6lJsHMM6rPAc0MeF4jIVhF5TUTWjPYiEVknIptEZFNjY2Pgo1Snp46jgAnuPHxDRURA1kK/VPJVNncx6DK2nEEBlM5IZWtVK8bPM2Oo8GdLghKR7wADwEPuTbXATGPMUuCbwMMikjzSa40x9xhjVhhjVmRkZAQnYHX6aQvyOlAjyS6B+t3g8m3JCs+M4nYmqMaOXo629dhyfOVcQU9QInIjcDnwCeP+SmWM6TXGNLt/3gxUAHODHZtSJ3iugQr2NEdDZS2Cvk44dsinZsobOhGBORn2JKilesGumqSgJigRuQT4N2CtMaZ7yPYMEYl0/zwbKAIOBjM2pU5yYpojm7r44P21oXwchypr6CQ3NZ74mEg/BDVx87KTiYmK0HEoNWGBLDP/M/A2UCwi1SLyOeDXQBLw0rBy8g8AO0RkG/AYcLMxpmXEhpUKhrZqiJ8GMcG9bugkmfNBInwehypv6AzqDBLDxURFUJKbwlat5FMTFBWoho0xN4yw+b5R9n0ceDxQsSg1YW019p49AUTHWwsY+rA21KDLUNHYyepCG2bDGGJJXioPbTxM/6CL6EidH0B5R/9SlBqJnddADZW1yKczqOpj3fQNuGwrkPBYMiOF3gEXB+o7bI1DOYsmKKVG0u6e5shu2SVWLN2T6/F+v4Iv+NdADVU6wyqU2F6lCxgq72mCUmq4nnboabO3xNzDx0KJMptLzD1mTptC6pRotus4lJoATVBKDdceAtdAeWQvtu6PbpvUy8sbOslMiiUlPtqPQU2ciLAkL5XtOiefmgBNUEoNFwoX6XokZlhrQx3dMqmXlzXYMwffSJa4Zzbv7tOZzZV3NEEpNVxblXUfCgkKIHcZ1Gye8MuMMVSEUoLKS8FlYFdNu92hKIfQBKXUcO011vVHidl2R2LJXQ6tR6CraUIvq2/vpbN3wNZroIZanOcplNBuPuUdTVBKDddWDUnTITJglwlOTO4y675mYt18nlV054RIgspIiiU3NZ5tOg6lvKQJSqnh2qrtv0h3qJxS64xugt18dk8SO5LSGal6BqW8pglKqeHsXEl3JLGJkDFvwoUS5Q2dpMRHk5EYG6DAJm7JjBSqjx2nubPX7lCUA2iCUmoo12DozCIxlKdQYgJrKnkq+EQkgIFNzBL3ONSOar1gV41PE5RSQ3XUgqsfps6yO5KT5S6H7mZoPez1SypsniR2JItyUxDRBKW8owlKqaGOuRNAaoglqOmeQgnvxqFauvpo7uoLqfEngITYKPLTEthbq6XmanyaoJQaynOGMjXf1jBOkbUQouK8ruTzFEiESgXfUAtyktmjCUp5QROUUkO1HgEktIokACKjIWcJVG/yandPggq1Lj6ABdOTOdLSTXtPv92hqBCnCUqpoY4dhqQciAqdyrcT8lZC7TYY6Bt317KGDuKjI5meEh+EwCZmQU4yAPtqdekNNTZNUEoN1Xo49AokPPJWwECPVzOblzd0MiczgYiI0Kng81gw3UpQe45qoYQamyYopYY6djj0CiQ88s6w7qvfG3dXq4LP3jWgRpOZFEtaQgx79QxKjUMTlFIeA33WPHyhegaVkmtNwTROgursHeBoW0/IVfB5iAjztVBCeUETlFIebVWACd0zKLC6+cZJUBUhOMXRcAumJ7O/voP+QZfdoagQpglKKQ9PiXnqTHvjGEveSjhWCZ2No+4SKqvojmVBTjJ9Ay4ONnbZHYoKYZqglPLwXKQbql18YCUogJrRy83LGzqJjhRmTZsSpKAmrjjbGh/bX6/jUGp0AU1QInK/iDSIyK4h26aJyEsiUua+n+reLiLyKxEpF5EdIrIskLEpdYrWwxARBckhNJP5cNNLrRir3h11l/KGTgrSE4iKDN3vn7MzEogQKNMEpcYQ6L/gB4BLhm27FXjFGFMEvOJ+DHApUOS+rQN+G+DYlDpZyyGrey8i0u5IRhcdD9klY45DlTd0hHT3HkBsVCT5aQkc0ASlxuBVghKRv4nIZSIyoYRmjHkdaBm2+SPAg+6fHwQ+OmT7H4zlHSBVRHImcjylfNJcAWmFdkcxvryV1pRHrsFTnurpH+RISzeFIVpiPlRRViJl9Z12h6FCmLcJ53+BjwNlIvLfIlLswzGzjDG17p/rgCz3z7lA1ZD9qt3blAo8Y6ClAqbNsTuS8eWthP4uaNh7ylMVjZ24DMzNCu0zKIC5WUlUNnfRO3BqolUKvExQxpiXjTGfAJYBlcDLIrJBRG4SkejJHtwYYwDvF7gBRGSdiGwSkU2NjaNXMik1IR210N8NaU5IUCus+xG6+TxnJHOznHAGlYTLoJV8alRed9mJSBpwI/B5YCvwS6yE9dIEj1nv6bpz3ze4t9cAQ1eJy3NvO4kx5h5jzApjzIqMjIwJHlqpUTRXWPdOSFBTC2BK2ogJ6kB9B1ERQn5agg2BTYznLE/HodRovB2DegJ4A5gCXGGMWWuM+Ysx5hZgon0J64HPuH/+DPDUkO2fdlfzrQLahnQFKhVYzeXWvRPGoESsbr4RE1Qn+ekJxESFbgWfR0F6ApERouNQalRRXu53rzHm2aEbRCTWGNNrjFkx2otE5M/AuUC6iFQDtwH/DTwqIp8DDgPXund/FvgwUA50AzdN5BdRyictFRAZC8khtszGaPJWwoHn4fgxiJ96YnNZQwcL3ZOxhjqrkm+KnkGpUXmboO7ASiBDvY3VxTcqY8wNozx1/gj7GuArXsajlH81H4RpsyEi9M88gCEX7G6GwgsAON5nVfB9tNQ5tUVzs5LYV6cJSo1szP+NIpItIsuBeBFZKiLL3Ldzsbr7lAoPzeXOGH/yyF0GyEkLGFY0dmKMMwokPIqykjjc3EVPv1byqVONdwZ1MVZhRB5w15DtHcB/BCgmpYJrcACOHYLi4deUh7DYJMhccNKMEp6uMieUmHvMzUrEZazkunB6it3hqBAzZoIyxjwIPCgiVxljHg9STEoF17FDMNgHGfPsjmRi8lbAnifB5YKICA7UW3Pw5aeHfgWfh2fNqvIGTVDqVGMmKBH5pDHmT0C+iHxz+PPGmLtGeJlSzuK54NVxCWolbHmZnrtCAAAgAElEQVTQ6p7MmEtZfQcF6QlEh/AcfMPlp08hQqwEpdRw4/0le76KJQJJI9yUcr7GfdZ9hi8TpNhgxskr7JaF8Cq6o4mNimRWWoImKDWi8br4fue+/8/ghKOUDRr2WosUxjinawyAtCKITYHq9zi+8HqqjnVz5TLnVPB5FGYmaoJSI/L2Qt2fikiyiESLyCsi0iginwx0cEoFRcNeyJxvdxQTFxEBecuh+j3KG5xXwedRmJnIoaYuXV1XncLbzuqLjDHtwOVYc/EVAv8aqKCUCprBfvcYjsPGnzzyVkLDHsqqrElX5mU7MEFlJDLgMhxu7rY7FBVivE1Qnq7Ay4C/GmPaAhSPUsHVXAGufmeeQYGVoIyL9oPvEh9tjec4jWftKu3mU8N5m6CeFpF9wHLgFRHJAHoCF5ZSQdLo0Ao+j9zlAMTUbWZeThKREWJzQBM3x52gKho1QamTebvcxq3A2cAKY0w/0IW1wKBSzla3EyTSuQlqyjRMWhE5HTuZn+OMOfiGS4yNYnpKnJ5BqVN4OxcfwDys66GGvuYPfo5HqeCq3WF170XH2R3JpHWnL6K46TVqHJqgwDqLKmvQOfnUybyt4vsjcCewGljpvo06i7lSjmAM1G6D7MV2R+KTmthCpksLi6Y6dz67wsxEKhq6cLkmtH6pCnPenkGtABa4ZxxXKjx01EFXI+QssTsSn+wenMVcoJhKYLbN0UxOYWYix/sHOdp2nLypOg+1snhbJLELyA5kIEoFXd0O697hCeqtziwA4lv22hzJ5BVmaCWfOpW3Z1DpwB4ReRfo9Ww0xqwNSFRKBUPtdkAge5Hdkfjk3cZoWiPTSK3baXcok1aU9f6ksecWZ9ocjQoV3iao2wMZhFK2qN1urQEV67yLWz06evo50tJNW04xqXW77A5n0qYlxDAtIUbPoNRJvC0zfw1rBolo98/vAVsCGJdSgWWMtRptTqndkfjEsxqtyV5kTXo70GdzRJNXmKFz8qmTeVvF9wXgMeB37k25wJOBCkqpgGuvgY7a92cEd6id1dakLqkFy6wZMTwzszvQnMxEyhs70Vos5eFtkcRXgHOAdgBjTBmgHcXKuTwr0eattDcOH+2obiUrOZbUAmtGCeqd281XmJlIa3c/zV3OPQtU/uVtguo1xpz4q3FfrKtfc5RzVb8HUXGQXWJ3JD7ZUdNGSW6qNZYWFW/NjOFQRTonnxrG2wT1moj8BxAvIhcCfwX+HriwlAqwqndh+lKIjLY7kklr7+nnYGMXS/JSICISshY4OkF5Jo0t0wSl3LxNULcCjcBO4IvAs8B3AxWUUgHV32NV8Dm8e29XjTX+VJKXYm3ILrESlEPHcHJS4kiIiaRCE5Ry86rM3BjjEpEngSeNMY2+HFBEioG/DNk0G/g+kAp8ASsRAvyHMeZZX46l1IjqdlgFBWFSILE4L9XakLUINj9gzZCRnGNfYJMkIlahhCYo5TbmGZRYbheRJmA/sN+9mu73J3tAY8x+Y0ypMaYUa/mObuAJ99O/8DynyUkFTNVG697hZ1A7qtvImxrPtIQYa4NnRvaGPfYF5SMtNVdDjdfF9w2s6r2VxphpxphpwJnAOSLyDT8c/3ygwhhz2A9tKeWdyjdh2hxIcvbsXTtqWlns6d6D9xddbHDwlEdZidS199DR0293KCoEjJegPgXcYIw55NlgjDkIfBL4tB+Ofz3w5yGPvyoiO0TkfhGZOtILRGSdiGwSkU2NjT71NqrTkWsQDm+AgjV2R+KTlq4+qlqOv9+9B5CQDgmZzk5QOiefGmK8BBVtjGkavtE9DuVT+ZOIxABrsSoCAX4LzAFKgVrg5yO9zhhzjzFmhTFmRUZGhi8hqNNR7XbobYd8ZyeoHdWtACzOTTn5icz5768S7EC6/LsaarwENdYVc75eTXcpsMUYUw9gjKk3xgwaY1zAvYCzR7BVaKp807rPX21vHD7aeqQVkSEVfB6Z86FhH7hc9gTmo5nTphATGUG5Lv+uGL+Kb4mItI+wXQBflyC9gSHdeyKSY4ypdT/8GNYSH0r5V+WbkFbk+PGnLUeOUZyVRFLcsI6MzPnQ3wVtR2Bqvi2x+SIqMoL89Claaq6AcRKUMSYyEAcVkQTgQqxrqjx+KiKlWDNUVA57TinfDQ5Y408lV9sdiU8GXYZtR1pZWzr91CczF1j3DXsdmaAAijKT2H20ze4wVAjwdrkNvzLGdAFpw7Z9yo5Y1Gmkbjv0dTi+QKKsoYOO3gGWzRyhjuhEqfleKL40uIH5yZzMRJ7bVUtP/yBx0QH5jqwcwtuZJJRyvkNvWPeznD3+tPnwMQCWzxohQcUlQ3Kesyv5MhNxGTjU1GV3KMpmmqDU6ePQ65BeDElZdkfiky2HW0lLiGFW2pSRd8ic7+wEpaXmyk0TlDo99PdY409zPmR3JD7bcuQYy2ZNRURG3iFzPjTtt8bcHGh2RgIimqCUJih1ujiyAQaOw5zz7Y7EJ82dvRxq6hp5/MkjcwEM9sGxQ6PvE8LioiOZOW2KlporTVDqNFHxD4iMgfxz7I7EJ1uPWBfojjj+5HFiyiOHz8lXrwnqdKcJSp0eyv8BM1dBTILdkfhk85FjREXIyXPwDZc+FxBnj0NlJnKoqYuBQWdecKz8QxOUCn/ttdCw2/HdewDvHWphYW7K2OXXMVNgWoGjz6DmZCbSN+ii6thxu0NRNtIEpcLfwX9a93POszcOHx3vG2R7dSurZk8bf+fMBdaURw6lc/Ip0ASlTgflr1izfGctsjsSn2w9coz+QcOqgrTxd86cD83lMNAb+MACQBOUAk1QKty5XNYZ1JwPQYSz/9zfOdRChMCK/DEKJDwy5oEZhKaywAcWAMlx0WQlx1LW0GF3KMpGzv4fq9R46rZDd3NYjD+9c7CZhdNTTp0gdiQn5uRz7jhUYWaiThp7mtMEpcJb+SvWvcMv0O3pH2RbVStnFngx/gSQVggRUY5OUEWZSZQ3dOJyGbtDUTbRBKXCW8U/IbsEEjPtjsQn26pa6RtwceZsL8afAKJirHLzeucmqAXTk+nqG+RQs87Jd7rSBKXCV28HVL0TFt17Gw+2IAJn5Ht5BgXuSj7nJqgS92rBu2p06Y3TlSYoFb4OvQGuAceXlwNsPNTM/OxkUqZ4Mf7kkbUA2qqgZ6Q1R0NfUWYisVER7KzWBHW60gSlwlfZixCTCDPPsjsSn/QNuNhy5BhnenP901CZC617h84oERUZwfycZHboGdRpSxOUCk/GWAlq9rnWeIyDba9upaffxZneXP80VJankm+3/4MKkpLcFPYcbddCidOUJigVnhr2QHsNFF1kdyQ+e7OsiQiBs7wtkPBImQExSY4ulCjJTaGzd0ALJU5TmqBUeDrwgnUfDgmqvImSvNSJjT8BiLgXL3RuglqkhRKnNU1QKjyVvWiVlyfn2B2JTzp6+tlW1crqwgmePXlkLYD63VaXpwMVZSUSFx1xYpkRdXrRBKXCz/FjULURii62OxKfbTzYwqDLcE5h+uQayFwIPa3QUeffwIIkOjKCJXmpbD58zO5QlA00QanwU/4KGFfYdO/FRUeMvUDhWMKgUGJl/jT21LbT1evMJezV5NmWoESkUkR2isg2Ednk3jZNRF4SkTL3/ST/V6rTWtlLED8V8lbYHYnP3ixv4oyCNGKjxlj/aSyeOfkcXCixPH8qgy7Dtirt5jvd2H0G9SFjTKkxxvNJcivwijGmCHjF/Vgp77kGofwlKLwAIib5oR4iatuOU97QOfnxJ4Ap0yAx29GFEstmTkUENlVqN9/pxu4ENdxHgAfdPz8IfNTGWJQTHd1qzV4eBuNPb5U3A7C6MMO3hjyFEg6VEh9NcVYSmw632B2KCjI7E5QBXhSRzSKyzr0tyxhT6/65DsiyJzTlWAdeAImAQufPv/dWeRNpCTHMy07yraHMBdC4HwadO4azfNZUth5pZWDQZXcoKojsTFCrjTHLgEuBr4jIB4Y+aYwxWEnsJCKyTkQ2icimxsbGIIWqHKPsRchbaXVtOZgxhjfLmzi7MJ2ICPGtsayFMNgLLQf9E5wNVs1Oo7N3QKc9Os3YlqCMMTXu+wbgCeAMoF5EcgDc9w0jvO4eY8wKY8yKjAwfuz5UeOmog9ptUHSh3ZH47EB9J40dvayZbHn5UJnzrXsHj0OdU5iOCLxxoMnuUFQQ2ZKgRCRBRJI8PwMXAbuA9cBn3Lt9BnjKjviUQ5W/bN2HwfjTG2VW78A5RX5IUBnzrG5PByeoaQkxlOSmnHhf1OnBrjOoLOBNEdkOvAs8Y4x5Hvhv4EIRKQMucD9WyjsHXoCkHGsGCYd7q7yJgvQEclPjfW8sOh6mzXZ0oQTAmqJ0tla10tHTb3coKkhsSVDGmIPGmCXu20JjzI/c25uNMecbY4qMMRcYY7RsR3lnsN9aPbfoQmsOOgfr6R/k7YPNrPHH2ZNH9mKo2+G/9mywujCDQZfh7Ypmu0NRQRJqZeZKTc6Rt6GvIyxmj3ivsoWefhfnFvtxjDVnMbQegW7nfudbPmsqU2IiefWAdvOdLjRBqfBQ9iJERFvrPzncq/sbiYmMYNVEl9cYS84S697BZ1ExURF8cG4GL++p1/WhThOaoFR4OPAizDobYn28ZigEvLq/gTNnT2NKTJT/Gs12J6ha5yYogIsWZtHQ0cu2ap326HSgCUo537FKaNoPc51fvVfV0k1FYxcfnOvnSygS0qwFDGu3+7fdIDuvOIuoCOGF3c6cnV1NjCYo5Xz7n7fuw6C8/DX3+Mq5xZn+bzx7seMTVMqUaM6ak8aLu+sxDl3jSnlPE5RyvgPPQfpcSC+0OxKfvbq/kdzUeOZkJPi/8Zwl0FwOvZ3+bzuILlqYzaGmLsobnP17qPFpglLO1tMGlW/C3EvsjsRnfQMuNlQ0cW5xBhKIUvmcJYCB+l3+bzuILl6QhQg8vaN2/J2Vo2mCUs5W/gq4BqD4w3ZH4rNNlS109w0GpnsPrFJzcHw3X2ZyHKsK0vj79qPazRfmNEEpZzvwPMRPgxln2B2Jz1490Eh0pHDWHD+Wlw+VlAMJGY5PUAAfKZ3OwaYudtW02x2KCiBNUMq5Bges6Y3mXuz4xQnBKi9fmT+NxFg/lpcPJWJ18zm81Bzg0kU5xERG8NS2GrtDUQGkCUo5V9VG6GkNi/Gno63HOVDf6d/ZI0aSU2pNGtvXHdjjBFjKlGjOLc5g/fajDOpFu2FLE5Ryrv3PQmRMWCxO+MreegDOmxfgNTrzVoIZtJYlcbiPlObS0NHLxoM6N1+40gSlnOvA85C/Jixmj3hpbwMF6QmBKS8fKm+FdV/9XmCPEwTnz88kISaSp7YdtTsUFSCaoJQzNZVZ1/QUX2p3JD7r6Onn7YomLpifGZjy8qES0mFqQVgkqLjoSC5elM2zu2rp6R+0OxwVAJqglDPtf866D4Pxp9cPNNE/aLhwQXZwDpi3EqregzAo0b5yaR4dPQO8sveUxbdVGNAEpZzpwPOQVQKpM+yOxGcv761n6pRols1MDc4B81ZCZx20VQfneAF01pw0spPj+NsW5/8u6lSaoJTzdLdY6z8VO//saWDQxT/2NXDevCyiIoP03zGMxqEiI4SPLs3l1QONNHX22h2O8jNNUMp59j0DxgXzLrc7Ep+9V3mMtuP9XLggQLNHjCRrEUTFQfWm4B0zgK5alsugy2ixRBjSBKWcZ+96SJ35/iJ8DvbC7jpioiJYUxTg65+GioqB6Uuh+t3gHTOAirKSWJyXot18YUgTlHKWnjao+CfMX2vNjOBggy7Dsztr+VBxBgmBmj1iNHkrrSmPHH7BrseVS3PZfbSdfXU69VE40QSlnOXAC+DqhwUfsTsSn71X2UJDRy+XL54e/IPnr4HBvrAYhwK4Ysl0oiKEJ7bo1EfhRBOUcpY9T1mTnuausDsSnz2zo5a46AjOmxfE8SePmatAIqylSsJAWmIs5xZn8sTWGp36KIxoglLO0dsJ5S/D/Csgwtl/ugODLp7bVcv587KC370HEJdszcsXJgkK4Orl1tRHb5Y32R2K8pOg/y8XkRki8k8R2SMiu0Xk6+7tt4tIjYhsc9+cv8CP8q/yl2Cgxxp/crh3DrbQ1NnHZYtz7AsifzXUbAqbcagPzcskJT5aiyXCiB1fQweA/2eMWQCsAr4iIgvcz/3CGFPqvj1rQ2wqlO1+Eqakw6yz7Y7EZ3/dXEVyXJQ93XseYTYOFRsVyRVLcnhhdx0dPf12h6P8IOgJyhhTa4zZ4v65A9gL5AY7DuUwPe3W7BELP+b4tZ/auvt5blcdH12aS1y0jb/LiXGoN+yLwc+uXJZHT7+L53bV2R2K8gNbO/JFJB9YCmx0b/qqiOwQkftFZOoor1knIptEZFNjY2OQIlW22/t3q3tv8bV2R+Kz9TuO0jfg4toVNk/TFJdsFZuUv2JvHH60dEYqs9MTeHyzdvOFA9sSlIgkAo8D/2KMaQd+C8wBSoFa4Ocjvc4Yc48xZoUxZkVGRhAvblT22vkoTM23rt9xuL9uqmJ+TjILpyfbHQoUXQRHt0BneHzZExGuXJbLxkMtVLWEx9ja6cyWBCUi0VjJ6SFjzN8AjDH1xphBY4wLuBc4w47YVAhqr4VDr0PJtY6/OHdHdSs7qtu4dkVe4JfW8EbRBdZ9RficRX10qTVi8ORWvSbK6eyo4hPgPmCvMeauIduHljN9DNgV7NhUiNr1uDX3Xhh079335iESY6O4enme3aFYspdAQiaUvWh3JH6TN3UKq2ZP47Et1bj0mihHs+MM6hzgU8B5w0rKfyoiO0VkB/Ah4Bs2xKZC0Y6/WNfspBfZHYlP6tp6eGZHLdeumEFSXLTd4VgiIqDwAmscanDA7mj85oYzZnK4uZvXy8Kj6/J0ZUcV35vGGDHGLB5aUm6M+ZQxpsS9fa0xpjbYsakQdHQr1O2A0k/YHYnPHthQicsYbjon3+5QTlZ0IfS0hs3ksQCXLsohPTGWP7592O5QlA+cfTm+Cn+bH4SoeMd377V09fHHtyv5cEkOM6ZNsTuckxVdCJGx1jRSYSImKoIbzpjBP/Y3aLGEg2mCUqGrtxN2/tW69ik+SKvNBsg9rx+ku3+Qr58fgt2UsUlWktrzFLhcdkfjNx8/cyYRIvzxHT2LcipNUCp07Xoc+jph+Y12R+KTxo5e/vB2JWuXTKcoK8nucEa28GPQUQtVG8ff1yFyUuL5cEkOD288Qlu3zizhRJqgVGgyBjbdBxnzYYazrzj46fP76B908S8XzLU7lNHNvdjq5tv1uN2R+NWXz51DZ+8AD2yotDsUNQmaoFRoOrzBWlDvzHWOvvZp65Fj/HVzNZ9dXUBBeoLd4YwuNgnmXWZ1qfb32B2N38zPSeaC+Znc/9YhOnvDp0rxdKEJSoWmt38N8dNg8fV2RzJpgy7D7et3k5kUyy3nheDY03DLP2NV8+39u92R+NUt5xXRdryf375abncoaoI0QanQ01QO+5+DlZ+HmBCreJuAe14/yPbqNr57+QIS7VjzaaLyP2BNJ7XlQbsj8aslM1L5aOl07n3jENXHtKLPSTRBqdDzzm8gMtpKUA61v66DX7x0gEsXZXOFnWs+TUREBCz7tDW7ef1uu6Pxq3+7ZB4RAj96Zq/doagJ0ASlQktbNWz9Eyy5HpKy7I5mUo73DfL1R7aSFBfFHR9dFBpz7nlr+U0QnQBv/dLuSPxqemo8t5xXxHO76vj79qN2h6O8pAlKhZY3fm5V8K35lt2RTIoxhu88sZP99R38/NolpCXG2h3SxEyZZpX173wMWo/YHY1fffEDs1kyI5XvPbWL+vbwKQQJZ5qgVOhoPQJb/gjLPgVTZ9kdzaT8aeMR/ra1hq+fX8S5xTauluuLs75iLQr56n/bHYlfRUVG8PNrltA34OKLf9xMT/+g3SGpcWiCUqHjlR9YK7yu+X92RzIpr+5v4Pb1u/lQcQZfc0LV3mhScuHML8K2h61S/zBSmJnIz69ZwraqVv7jiZ0623mI0wSlQsORd6xrcM6+BVJCZCmKCdhV08aXH9pCcVYS//PxZUREOGjcaSRrvmV19z13a1hNfwRwaUkO37hgLn/bUsMPnt6DMZqkQpUmKGU/1yA892+QNB3WfNPuaCbsYGMnN/7+PaZOieGBm1Y6o6R8PPGpcOEP4MgG2Hi33dH43dfOL+Rzqwt4YEMl//XsXj2TClFh8D9JOd7bv7a6kq66D2JCeLaFERxs7OT6e97BGMODn11JZnKc3SH5T+knYO/T8PLtULAGskvsjshvRITvXjafgUEX975xiMaOXn569RJiovQ7eyjRfw1lr/o98I87YN7lsOgqu6OZkENNXdxw7zsMugx/XreKwswQnQh2skRg7a9gSho8fD101NsdkV+JCLevXci3LprLk9uOctMD79LS1Wd3WGoITVDKPr2d8PjnITYZrvilo+bc21fXznW/e5uBQcPDX1jF3FCdpdxXiZnw8UfgeAv86UroarI7Ir8SEb56XhF3XrOE9w4d44r/eZNtVa12h6XcNEEpe7hc8OSXoHEvXPk7SEi3OyKvvXuohWvufhsR+PO6VRRnh2ly8shZAtf9CZrL4YHLoT38LnS9enkej33pLACuuXsD975+kEEdl7KdJigVfMbAs9+CveutgfjCC+yOyGvP76rjk/dtJCMplse/dHb4njkNV3g+fPxRaKuC333Qmm0+zCzOS+WZr63m3OJMfvTsXq6+ewPlDR12h3Va0wSlgmuwH57+F2utp7O/Bmd91e6IvOJyGX71Shk3/2kzC3KSeezms8mb6tyJbCdl9gfh869YS3M8eAX888cwEF5jNqlTYrjnU8v55fWlHGrq4tJfvsEdT++htTu8fk+nECdfA7BixQqzadMmu8NQ3mo/Ck/cDIdeg9XfgPNvc8S4U2t3H//++A5e2F3PlUtz+a8rS4iLjrQ7LPv0tMEz34Kdj0LmArjs5zDrbLuj8rvGjl7ufGE/j26uIjkumi9+cDafXDWL5Lhou0NzPBHZbIxZMe5+mqBUwA0OwNY/wsu3WWdQH74Tln7C7qi88s/9Ddz6+A6aO/u49dJ5fG51gbMmfw2kAy/A09+A9hoovgwuuA0yiu2Oyu/21rbzk+f38er+RpJio/j4qpl8+qx8clPj7Q7NsRyboETkEuCXQCTwf8aYUScE0wQV4nrarW/Zb/8vtFTAzLPhI7+GtDl2Rzau8oYOfvL8fl7aU8/crETuuraURbkpdocVevq64Z3/hTf/P+jrhLmXwFlfhvw1jjg7nohdNW3c/VoFz+6sxQCrC9O5dsUMLlyQdXqfUU+CIxOUiEQCB4ALgWrgPeAGY8yekfbXBOWjgV5rgtaWQ3Cs0rodPwYDxwGBuBTrlpgJiVnWLSnbuo9NOvUDaLDfqvSq2ghlL0HFP6C/G7IXw7nfhuJLQ/pDq6Onn9cPNPHopipeO9BIYmwUXzp3Dp9bXaAfQOPpaoKNv7PGFrubYWoBLPwYFH/YqgKMirE7Qr+pPtbNXzdV89jmampajxMfHck5helcMD+TlQXTKEhLcP5UVwHm1AR1FnC7MeZi9+NvAxhjfjzS/j4nqI46GOixqsoAcN8Pf0/GfH74Nj89HnGfSRyzvwuOt1qJp7vZSkieZNR+9OTjRU+BKekQFQvGBb3t1mtd/QzniopnMCGLwSkZyGA/Eb1tRHXUIC5rMLk/MZfu/PNpL76W3sxSkPdDOhHliV/LvP/cCNuGsl5vhr3e85w5pW1G2NcYKxk1d/XR0tXHwcZO9tZ2sK+unf5BQ2ZSLJ84cxafXDXTectl2K3/OOz6G+x6DA6+BmYQouJheql15jxtjvWFJy7Vmk4pKt6aOT0iylqkMiLK/SUmBD7gx/ky5XIZtlS18saBRt4qb6auw1rCIyk2ijkZieSkTiE5ezaZybEkx0WTFBdNQmwkURERREbISbcIAQmF33kUw9+KrOQ4n2bdcGqCuhq4xBjzeffjTwFnGmNGLPXyOUH9/sNw+K3Jv96JknKsZb2n5kPqLJhWYH3bnVYACRmn/iUaAz2t1iwCnXX8y/+9QKYcI1NayZRW0mmjnyhaSaTWpLHPNYOdpoAKM52Q+JDxQlpCDPNzkinJS+FDxZksm5lKVKQWuPqsq9lanffIO1C7DZoroKvB7qiCppkUlvf81u4wAuK5r69hfk7ypF/vbYJy3Fx8IrIOWAcwc+ZM3xpb/U1Y+klPy54DePn4pKAm2cYojyfVBiM/H51gfVONn2p9a51oV4uI9dr4qZA5j3OunIOI9V2vH6gbcrhshGzgQzK8ife/G8qJcGXIz8N+pRGfG6GNIW0NfxtGO6bncWJsFNMSYkhLjGFKjOP+GzhDQhos/Kh18+jthO4m68y8p9UqU3f1g2vAfRu0brbz/Yt7WlQse+deQnNXLx09A3T0DNDVO8CAyzDouRnDoMsV0hPGj/RO5KQEZ87JUPufWQPMGPI4z73tBGPMPcA9YJ1B+XS0IudcIBoqrlkxY/ydlBpNbKJ1m2p3IMERD+TFnGbXy/lRqPVjvAcUiUiBiMQA1wPrbY5JKaWUDULqDMoYMyAiXwVewCozv98Ys9vmsJRSStkgpBIUgDHmWeBZu+NQSillr1Dr4lNKKaUATVBKKaVClCYopZRSIUkTlFJKqZCkCUoppVRI0gSllFIqJIXUXHwTJSKNwOExdkkHmoIUjj9p3MGlcQePE2MGjdvfZhljMsbbydEJajwissmbCQlDjcYdXBp38DgxZtC47aJdfEoppUKSJiillFIhKdwT1D12BzBJGndwadzB48SYQeO2RViPQSmllHKucD+DUkop5VCaoJRSSoUkxyQoEblERPaLSLmI3DrGfleJiBGRFe7HF4rIZhHZ6b4/b8i+r7rb3Oa+ZYZQ3GcMiWu7iHxsom2GYNyV7l2CnngAAAQrSURBVH+HbSKyKZTiHrJ9poh0isi3JtpmCMYdsu+3iOSLyPEhfyt3D9l3uTvuchH5lYiIQ+IO2c8T97bFIvK2iOx2v79x7u0Bf78nzRgT8jesxQsrgNlADLAdWDDCfknA68A7wAr3tqXAdPfPi4CaIfu/6tkvBOOeAkS5f84BGrDW7/KqzVCL2/24EkgPxfd7yHOPAX8FvjWRNkMt7lB/v4F8YNco7b4LrAIEeA641CFxvzr83yWE4o4CdgBL3I/TgMhgvN++3JxyBnUGUG6MOWiM6QMeAT4ywn4/BH4C9Hg2GGO2GmOOuh/uBuJFJDbQAbv5Ene3MWbA/TAO8FSzeNtmqMUdDJOOG0BEPgocwvo7mWiboRZ3MPgU90hEJAdINsa8Y6xPzz8AH/VjzBCAuIPEl7gvAnYYY7YDGGOajTGDQXq/J80pCSoXqBryuNq97QQRWQbMMMY8M0Y7VwFbjDG9Q7b93n06/r0AnNr6FLeInCkiu4GdwM3uD/5x2wzRuMFKVi+K1dW6zs8x+xS3iCQC/w7850Tb9INAxA0h/H67FYjIVhF5TUTWDGmzeqw2/SAQcXuE6ufJXMCIyAsiskVE/m1Im4F+vyct5JZ8nwwRiQDuAm4cY5+FWN8qLhqy+RPGmBoRSQIeBz6F9Q0iKMaL2xizEVgoIvOBB0XkuWDFNpbJxG2M6QFWu9/vTOAlEdlnjHk9ROK+HfiFMaYzlLrgwae4Q/n9rgVmGmOaRWQ58KT7/6jtJhO3Maad0P48iQJWAyuBbuAVEdkMtAUrvslwyhlUDTBjyOM89zaPJKzxpVdFpBKrP3X9kIHNPOAJ4NPGmArPi4wxNe77DuBhrFPokIl7SJx7gU73vuO1GapxD32/G7D+PULp/T4T+Kl7+78A/yEiX/WizVCNO6Tfb2NMrzGm2R3fZqyxlbnu1+eN0Waoxh3qnyfVwOvGmCZjTDfwLLCM4Lzfk2f3IJg3N6zsfxAo4P3BwYVj7P8q7w8Oprr3v3KENtPdP0djDTLfHEJxF/B+ccEs4CjWzMQTajOE4k4AktzbE4ANwCWhEvew7bfzfpFESL/fY8Qd0u83kMH7g/SzsT4Up7kfDx+0/3Cox03of55MBbbgLmICXgYuC8b77cvNEV18xpgB97fCF7AqWe43xuwWkR8Am4wx68d4+VeBQuD7IvJ997aLgC7gBRGJdrf5MnBvCMW9GrhVRPoBF/BlY0wTwEhthnrcIjIbeMLdDRUFPGyMeT6E4p5Qm6EeN5BFaL/fHwB+MOTv5GZjTIv7uS8DDwDxWB+Yfu3aDkTcIpJACH+eGGOOichdwHtYY5PPmvfHqQL6fvtCpzpSSikVkpwyBqWUUuo0owlKKaVUSNIEpZRSKiRpglJKKRWSNEEppZQKSZqglFJKhSRNUEoppULS/w9u8/TkNsLgIwAAAABJRU5ErkJggg==\n"},"metadata":{"needs_background":"light"}}]},{"metadata":{"dc":{"key":"46"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 7. Zooming in on the difference\n<p>These two distributions above represent the bootstrap uncertainty over what the underlying 1-day retention could be for the two AB-groups. Just eyeballing this plot, we can see that there seems to be some evidence of a difference, albeit small. Let's zoom in on the difference in 1-day retention</p>\n<p>(<em>Note that in this notebook we have limited the number of bootstrap replication to 500 to keep the calculations quick. In \"production\" we would likely increase this to a much larger number, say, 10 000.</em>)</p>"},{"metadata":{"dc":{"key":"46"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# Adding a column with the % difference between the two AB-groups\nboot_1d['diff'] = (boot_1d['gate_30'] - boot_1d['gate_40']) /  boot_1d['gate_40'] * 100\n\n# Ploting the bootstrap % difference\nax = boot_1d['diff'].plot(kind = 'kde')\nax.set_xlabel(\"% difference in means\")","execution_count":14,"outputs":[{"output_type":"execute_result","execution_count":14,"data":{"text/plain":"Text(0.5,0,'% difference in means')"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<matplotlib.figure.Figure at 0x7fed1a504710>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl4XWW99vHvL3ObZmjTdEjSNp3bpC2lLWUeRJCKWkDwCIrKe1QcQNBzjr4oCgh4HI/D68EBFQ8eEVRAKZNMIjIJHeiUjkmnpFPSIU3SNPPv/WPvQIgZ2+ysvXfuz3Xl6t47a6/cpSR31lrPeh5zd0RERKJNQtABREREuqKCEhGRqKSCEhGRqKSCEhGRqKSCEhGRqKSCEhGRqKSCEhGRqKSCEhGRqKSCEhGRqJQUdID+Gj16tBcWFgYdQ0REjtPKlSsPuHtub9vFXEEVFhayYsWKoGOIiMhxMrOdfdlOp/hERCQqqaBERCQqqaBERCQqqaBERCQqqaBERCQqqaBERCQqqaBERCQqxdx9UCKxqqmljZI9R6htaGHG2AzGZaUFHUkkqqmgRCLM3fnjigq+9/RmKmsbATCDC2aP5balxeRnDws4oUh0iugpPjNbYmabzazUzG7q4vPXmFmVma0Of3wiknlEBpu7c+uyEr700Fom5Qznrg8t4PfXnsbnzp/Oy6UHuPhHL7Jix6GgY4pEpYgdQZlZInAXcCFQASw3s2XuvqHTpr939+sjlUMkSD94Zgu/eXUnnzhrMl++eDaJCQbAqVNyuHxBPtf8ejkfved1/vjp0ynOywo4rUh0ieQR1GKg1N23uXsT8ABwSQS/nkhUeWnrAX78fClXLCzg5ve8VU7tJuWk8/trTyN7WDIf/58VVNY0BJRUJDpFsqDygfIOzyvCr3V2uZmtNbMHzWxCVzsys2vNbIWZraiqqopEVpEB1dDcypf/tJbJo9O545I5mFmX243JTONX15xC9bEmvvjgWtx9kJOKRK+gh5k/ChS6+zzgGeDerjZy97vdfZG7L8rN7XWGdpHA3fPydsoPHeP2pXMYlpLY47azx2dy88WzeWFLFb99bdcgJRSJfpEsqN1AxyOigvBrb3L3g+7eGH76S2BhBPOIDIoj9c3c9ddSLpg9lrOmj+7Te64+bRLnzMjlPx/fSMXh+ggnFIkNkSyo5cB0M5tsZinAlcCyjhuY2fgOT5cCGyOYR2RQ/Pa1nRxtauXf3zWjz+8xM775/rkAfP3RzuOIRIamiBWUu7cA1wNPESqeP7h7iZndbmZLw5vdYGYlZrYGuAG4JlJ5RAZDQ3Mrv355B+fOyGX2+Mx+vTc/exg3XjCdZzbs59kN+yOUUCR2RPRGXXd/Anii02u3dHj8ZeDLkcwgMpgeX7uXA3WNXHvOlON6/8fPmsxDKyu47dESzpw2utfrVyLxLOhBEiJx5Q8rypk8Op0zpuYc1/uTExO489I5VBw+xo//unWA04nEFhWUyADZdbCe17Yf4oqFBd0OK++LU6fkcMXCAu7++zY27q0ZwIQisUUFJTJAHlxVgRlcdnJXt/v1z80XzyZrWDI3PbyO1jbdGyVDkwpKZIA8tnYPp0/JIW8AJn8dmZ7CLe8rYk15Nfe+suPEw4nEIBWUyAAoraxlW9VR3j1n3IDtc+lJeZw7I5fvPb2Zsqq6AduvSKxQQYkMgL+s3wfAhUUDV1Bmxrcun0tqUgLX3beKY02tA7ZvkVigghIZAH8p2cfJE7MHfBHC8VnD+MEH57N5fy2fu38Vza1tA7p/kWimghI5QftrGli/u4YLi8ZGZP/nzRzD7UuLeXZjJZ+9bxV1jS0R+Toi0UYr6oqcoJe2HgDgnOmRm8j4I6cX0trm3P7YBi76wd/57Dumcva0XNJSEth3pIE3dlWzpryaDXtr2HbgKIlmFOdlcvVpk7hkft4JDXsXCYoKSuQEvVR6gJz0FIr6ObVRf11z5mSK8rK4bVkJN/9p/T99PjcjlTl5mZwzI5fm1jZe2nqAz/9+NU+s28uPrjxZs1JIzFFBiZwAd+el0gOcMW00CQmRP0pZPHkUj99wFhv21rB+9xGaWtoYPSKVeROyyctKe9uRUlub8+tXdnDn4xv43P2r+NnVC0lK1Fl9iR0qKJETsHl/LVW1jZw9rW/LagwEM6M4L6vXJeITEoyPnzWZ5ETjlkdK+OVL2/n0uVMHKaXIidOvUyInoP36U1/XfQrCR06bxJLicXz/mS1sP3A06DgifaaCEjkBL5ceYMro9AGZPSJSzIzbLy0mKcH4r6c3Bx1HpM9UUCLHqa3NWbnzMIsnjwo6Sq/GZKTxr2dO5rG1ezUBrcQMFZTIcSqtqqOmoYWFk0YGHaVPPnn2FIYlJ3LPS9uDjiLSJyookeO0YsdhABYVRv8RFEDW8GTevyCfR9bs4UBdY9BxRHqlghI5Tit2HiInPYXCnOFBR+mzj51RSFNLG4+s3hN0FJFeqaBEjtPKnYdZOGlkTM3SMGNsBnPyM/nzG7uDjiLSKxWUyHGoqm1k58F6FhXGxvWnji6dn8+63UcorawNOopIj1RQIsdh5c7Q9aeFk2Lj+lNHS+fnkWDw5zd0mk+imwpK5DisragmKSE0IWusGZORxqmTc3h6w76go4j0SAUlchzW7T7CzHEZpCXH5gSsFxSNZcv+OnYdrA86iki3VFAi/eTurNt9hHkFPc+FF80umD0GgGc37g84iUj3VFAi/VRx+BjV9c3MyY/dgpqUk870MSNUUBLVVFAi/bS24ggA8/KzA05yYt45eyyvbz/EUa3QK1FKBSXST+t2HyE50ZgxbkTQUU7IWdNG09LmLN9xKOgoIl1SQYn007rd1cwal0lqUmwOkGi3cNJIUhITeKXsYNBRRLqkghLpB3dnXcUR5sbwAIl2w1ISWTApm1fKDgQdRaRLKiiRfqg4fIyahpaYvP+pK2dMHU3Jnhqq65uCjiLyT1RQIv2waV9oeqDZ4+OjoM6cloM7vKrTfBKFVFAi/bApvNjfzLEZAScZGPMKsklPSdR1KIlKES0oM1tiZpvNrNTMbuphu8vNzM1sUSTziJyoTftqmZQznPTUpKCjDIjkxAQWTBrJivDcgiLRJGIFZWaJwF3Au4Ei4CozK+piuwzgRuC1SGURGSib9tXEzdFTu4WTRrJ5Xw21Dc1BRxF5m0geQS0GSt19m7s3AQ8Al3Sx3R3At4GGCGYROWENza1sP3CUWXFy/andwkkjaXNYXV4ddBSRt4lkQeUD5R2eV4Rfe5OZLQAmuPvjEcwhMiC27q+jzWH2uPg6gpo/IZsEe2sJEZFoEdggCTNLAL4P/Hsftr3WzFaY2YqqqqrIhxPpwqZ9oQES8XYElZGWzMxxmSooiTqRLKjdwIQOzwvCr7XLAOYAfzOzHcBpwLKuBkq4+93uvsjdF+Xm5kYwskj3Nu2rJS05gYmjhgcdZcAtnJTNG7uqaW3zoKOIvCmSBbUcmG5mk80sBbgSWNb+SXc/4u6j3b3Q3QuBfwBL3X1FBDOJHLf2ARKJCRZ0lAG3aNIo6hpb2LxPy8BL9IhYQbl7C3A98BSwEfiDu5eY2e1mtjRSX1ckUjbvq2XWuPg6vdduwcSRgAZKSHSJ6M0c7v4E8ESn127pZtvzIplF5ERU1TZyoK6JWePja4BEuwmjhpE9PJm1FdV86NSJQccRATSThEiftA+QmBlnI/jamRlz87NYE17rSiQaqKBE+qD92ky8nuIDOKkgmy37aznW1Bp0FBFABSXSJ2VVdeSkpzAqPSXoKBEzryCL1jZnQ3i+QZGgqaBE+qCs8ihTc2N7Bd3ezCsILWG/tkIDJSQ6qKBE+qCsqo6pY9KDjhFR47LSGJORylpdh5IooYIS6cXho00cPNoU90dQEDqKWqMjKIkSKiiRXpRV1QEMkYLKYlvVUc1sLlFBBSXSi/aCmjZmaBQUwLrdOs0nwVNBifSirOooqUkJ5GUPCzpKxM3NDxVUyW6N5JPgqaBEelFWWcfk0elxOQdfZzkjUhmXmUbJHh1BSfBUUCK9KK2qY+oQOL3XrigvU/dCSVRQQYn0oKG5lfJD9UNigES74rxMyqqO0tCsGSUkWCookR7sPFhPmw+NARLtivMyaW1zNmnpDQmYCkqkB28NMY/vm3Q7Ks4LDZTYsEen+SRYKiiRHpRWhgpqyuihcwRVMHIYGWlJGighgVNBifSgrKqO/OxhDEtJDDrKoDEzisZnUqIjKAmYCkqkB2VDbARfu+K8LDbtq6G1zYOOIkOYCkqkG21tTlnlUaYNoRF87YryMmlobmP7gbqgo8gQpoIS6cbemgaONbfG/SzmXSnOCy3MqNN8EiQVlEg3yiqHziSxnU0bM4KUpASN5JNAqaBEujGUZjHvLDkxgZljM3QEJYFSQYl0o6yqjsy0JEaPiN9l3nsSGsl3BHcNlJBgqKBEulFWeZRpY0ZgFv+TxHalOD+Tw/XN7KtpCDqKDFEqKJFulFbVDcnTe+3eHCihpTckICookS4cOdZMVW3jkLwHqt2scZmYaSSfBEcFJdKFbUN4gES79NQkJueks2GvpjySYKigRLpQVnUUGFqTxHalKE9THklwVFAiXSitrCM50Zg4anjQUQJVlJdJxeFjHKlvDjqKDEEqKJEulFXVUZiTTlLi0P4WeXPpDa2wKwEY2t99It0oG+Ij+NoVjW+f8kjXoWTwqaBEOmlubWPXwfohOQdfZ7kZqYzJSNURlARCBSXSyc6DR2lpcx1BhRXnZWpOPgmECkqkk9LK0Ai+aUP4HqiOivIy2VpZR0Nza9BRZIiJaEGZ2RIz22xmpWZ2Uxef/7SZrTOz1Wb2kpkVRTKPSF+0TxI7RUdQQGigRGubs3W/1oaSwRWxgjKzROAu4N1AEXBVFwX0O3ef6+7zge8A349UHpG+KquqY1xmGiNSk4KOEhU0UEKCEskjqMVAqbtvc/cm4AHgko4buHvHE9vpgKZNlsCVVdZpgEQHE0cNZ0RqkgZKyKCLZEHlA+UdnleEX3sbM7vOzMoIHUHd0NWOzOxaM1thZiuqqqoiElYEwN0pqzqqARIdJCRYeOkNFZQMrsAHSbj7Xe4+Ffi/wFe72eZud1/k7otyc3MHN6AMKZW1jdQ1tmiARCdFeZls3FtDa5tOcsjgiWRB7QYmdHheEH6tOw8Al0Ywj0ivhvIy7z0pysukvqmVnQePBh1FhpA+FZSZPWxm7zGz/hTacmC6mU02sxTgSmBZp/1O7/D0PcDWfuxfZMAN5WXee/LWQAmd5pPB09fC+QnwIWCrmX3LzGb29gZ3bwGuB54CNgJ/cPcSM7vdzJaGN7vezErMbDXwb8DH+v9XEBk4pZV1pKckMjYzNegoUWXG2AySE00FJYOqT+No3f1Z4FkzywKuCj8uB34B/Nbdu5zq2N2fAJ7o9NotHR7feLzBRSKhrOooU4fwMu/dSUlKYPqYDI3kk0HV51N2ZpYDXAN8AngD+BGwAHgmIslEAlBWVcc0nd7rUlFeJhv2HMFdAyVkcPT1GtSfgBeB4cD73H2pu//e3T8H6LtZ4kJdYwt7jzQM6WXee1Kcl8mBuiaqahuDjiJDRF9vlf9F+HTdm8ws1d0b3X1RBHKJDLq3lnnXTbpd6ThQYkxmWsBpZCjo6ym+O7t47dWBDCIStNLwEHPdA9W1ojxNeSSDq8cjKDMbR2j2h2FmdjLQfuU4k9DpPpG4UVpZR2KCMXGUjqC6kpGWzKSc4RooIYOmt1N8FxEaGFHA2ydyrQW+EqFMIoEoraxjUs5wUpICn2AlamnKIxlMPRaUu98L3Gtml7v7Q4OUSSQQpRrB16vivEyeXL+PmoZmMtOSg44jca63U3xXu/tvgUIz+7fOn3d3LY8hcaF9mfclxeOCjhLV2q9Dbdpby+LJowJOI/Gut3MZ7SfjRwAZXXyIxIX2Zd41QKJnxXlZAKzbrYESEnm9neL7efjPrw9OHJFgaARf34zNTGNcZhprK6qDjiJDQF9v1P2OmWWaWbKZPWdmVWZ2daTDiQyWUs1i3mcnTchiTbkKSiKvr8OV3hVe/fa9wA5gGvDFSIUSGWyllXXkZaWRrmXee3XShGx2HKynur4p6CgS5/paUO3fte8B/ujuOgEtcaW0qk5THPXR/AnZAKzWUZREWF8L6jEz2wQsBJ4zs1ygIXKxRAZPW5tTVqll3vtqbn4WZrCmXL+nSmT1qaDc/SbgDGBReGmNo8AlkQwmMlj21jRwrLlVAyT6KCMtmeljRrBGAyUkwvpzwn0WofuhOr7nNwOcR2TQaQRf/51UkM1zmypxd62dJRHT11F8/wt8DzgLOCX8oVnMJS6ooPrvpAnZHDraRMXhY0FHkTjW1yOoRUCRa6UyiUOllXVkD08mJz0l6Cgxo+NAiQmjNG+0REZfB0msBzQHjMSlssrQHHw6VdV3M8dlkJqUoPuhJKL6egQ1GthgZq8Dby6n6e5LI5JKZBCVVtXxrqKxQceIKcmJCczNz+INFZREUF8L6rZIhhAJyqGjTRw62qQh5sdhYeFI7nlpOw3NraQlJwYdR+JQX4eZv0BoBonk8OPlwKoI5hIZFG8OkBirguqvxYWjaG513bArEdPXUXyfBB4Efh5+KR/4c6RCiQyWzftCi+/NHKvJ+ftr0aRRmMHr2w8FHUXiVF8HSVwHnAnUALj7VmBMpEKJDJbN+2vJSEtifFZa0FFiTtbwZGaOzWD5DhWUREZfC6rR3d+cGTJ8s66GnEvM27KvjpljMzSC7zgtnjyKlTsP09LaFnQUiUN9LagXzOwrwDAzuxD4I/Bo5GKJRJ67s3l/LTPG6fTe8Vo8eRT1Ta2U7KkJOorEob4W1E1AFbAO+BTwBPDVSIUSGQz7axo5cqyZWSqo47a4MLTsu07zSST0dRRfG6FBEZ919yvc/ReaVUJi3eb9tQDM0ACJ4zYmM43CnOH8Y5sKSgZejwVlIbeZ2QFgM7A5vJruLYMTTyRytuxTQQ2E06bk8Nr2g7oOJQOutyOoLxAavXeKu49y91HAqcCZZvaFiKcTiaBN+2rJzUhllObgOyFnT8+ltqGFNRVaH0oGVm8F9RHgKnff3v6Cu28DrgY+GslgIpG2ZX+trj8NgDOn5WAGL26tCjqKxJneCirZ3Q90ftHdq4DkyEQSibzWNmdrZa1O7w2A7OEpzCvI5sWt//SjQuSE9FZQTcf5OQDMbImZbTazUjO7qYvP/5uZbTCztWb2nJlN6m2fIgNh16F6GprbNIPEADln+mhWl1dz5Fhz0FEkjvRWUCeZWU0XH7XA3J7eaGaJwF3Au4Ei4CozK+q02RuElpGfR2gqpe8c319DpH82hwdIzNQpvgFx9vRcWtucV8t0FCUDp8eCcvdEd8/s4iPD3Xs7xbcYKHX3beFZKB4ALum0/+fdvT789B9AwfH+RUT6Y0t4iPl0TRI7IE6emE1GWhLPbawMOorEkb7eqHs88oHyDs8rwq915+PAk119wsyuNbMVZraiqkoXYuXEbdhTQ2HOcIan9HXFGelJcmIC588aw7Mb92u4uQyYSBZUn5nZ1YSWlf9uV59397vdfZG7L8rNzR3ccBKXSvYeoTgvK+gYceWi4nEcrm9mxc7DQUeROBHJgtoNTOjwvCD82tuY2QXAzcBSd2/s/HmRgXbkWDPlh45RlJcZdJS4cu6MXFKSEni6ZH/QUSRORLKglgPTzWyymaUAVwLLOm5gZicTWmNqqbvr5LUMio17QxObqqAGVnpqEmdPG81TJfvQTGgyECJWUO7eAlwPPAVsBP7g7iVmdruZLQ1v9l1gBPBHM1ttZsu62Z3IgGmfebtYBTXgLioex+7qY6zVrBIyACJ6hdjdnyA083nH127p8PiCSH59ka6U7DlCbkYqYzK0SOFAu6h4HF99ZD1/emM3J03IDjqOxLioGCQhMpg27KnR0VOEZA1P5oLZY1i2Zg/NGs0nJ0gFJUNKQ3MrWyvrVFAR9P6TCzh0tIkXNuuWEDkxKigZUrbur6O1zSkaryHmkXLuzFxGpafw8BsVQUeRGKeCkiGlZE/o4r2OoCInOTGBS+fn88yG/VTWNgQdR2KYCkqGlPV7jjAiNYmJo4YHHSWufeT0STS3Or97bVfQUSSGqaBkSFlTfoR5BVkkJFjQUeLa5NHpnDczl/te20VTiwZLyPFRQcmQ0dDcysa9NRr+PEg+dkYhVbWNPLl+b9BRJEapoGTIKNlTQ0ubM18FNSjOnZ7L1Nx0fvq3MtraNLOE9J8KSoaMNeXVACqoQZKQYFx//jQ27avlmY2an0/6TwUlQ8bq8mrGZaYxNlMzSAyW983LozBnOD/+61bNzyf9poKSIWNNRbWOngZZUmIC171jGut31/BUyb6g40iMUUHJkHDoaBM7D9ZrgEQALjs5n+ljRvCtJzdpRJ/0iwpKhoQ1Fbr+FJSkxAS+8p7Z7DhYz2//sTPoOBJDVFAyJKzeVY0ZzC3QFEdBOG9GLmdNG82PnttKdX1T0HEkRqigZEhYufMwM8dmMCI1oivMSDfMjJvfM5vahma+89TmoONIjFBBSdxrbm1j5c7DnDp5VNBRhrTZ4zP5P2dO5v7Xd7Fq1+Gg40gMUEFJ3CvZU8Ox5lZOUUEF7gsXzmBsRho3/2k9LVovSnqhgpK49/r2gwAsLlRBBW1EahK3vq+IjXtruPdVDZiQnqmgJO69vv0wk0enM0Y36EaFJXPG8Y6ZuXz/6c3sPXIs6DgSxVRQEtfa2pzlOw5xSuHIoKNImJnx9aVzaGlz7nhsQ9BxJIqpoCSuba2s48ixZhZPzgk6inQwMWc4nzt/Gk+s28fzmyuDjiNRSgUlce3l0gMAGsEXhT55zhSm5KZz6yMlNDS3Bh1HopAKSuLai1urmDw6nQlaQTfqpCYlcuelc9h1qJ67ni8NOo5EIRWUxK3Gllb+se0QZ08fHXQU6cYZU0dz2cn5/OyFMsqq6oKOI1FGBSVxa9XOao41t3L29Nygo0gPvnLxbIYlJ/K1P6/XkhzyNiooiVsvbq0iMcE4bYquP0Wz3IxUvrhkFq+UHWTZmj1Bx5EoooKSuPXi1gMsmJhNRlpy0FGkFx9aPJGTJmRzx2MbqW1oDjqORAkVlMSlypoG1u0+wjk6vRcTEhOMOy4p5kBdIz/5W1nQcSRKqKAkLj29YT8A7yoeF3AS6at5BdlcvqCAX724nfJD9UHHkSiggpK49PSG/RTmDGfG2BFBR5F++NKSmSQmGN98cmPQUSQKqKAk7tQ0NPNq2QHeVTwOMws6jvTD2Mw0PnveVJ5Yt4/Xth0MOo4ETAUlcef5TZU0tzoXFY8NOooch0+eM4W8rDRuf2wDrW0adj6UqaAk7jy5bh+jR6Ry8gRNEBuL0pITueni2ZTsqeGhVRVBx5EARbSgzGyJmW02s1Izu6mLz59jZqvMrMXMrohkFhkaquub+OumSpaelEdCgk7vxar3zRvPgonZfPepzdQ1tgQdRwISsYIys0TgLuDdQBFwlZkVddpsF3AN8LtI5ZCh5bG1e2lqbeP9C/KDjiInwMy45X3FVNU28tO/aZ6+oSqSR1CLgVJ33+buTcADwCUdN3D3He6+FtDazzIgHlpVwcyxGRTnZQYdRU7Q/AnZXDo/j1+8uJ2Kwxp2PhRFsqDygfIOzyvCr/WbmV1rZivMbEVVVdWAhJP4s62qjjd2VfP+BfkavRcnvrRkFgkG3/7L5qCjSABiYpCEu9/t7ovcfVFurmYGkK7d99oukhKMS0/W6b14kZc9jGvPnsKja/awcufhoOPIIItkQe0GJnR4XhB+TWTAHW1s4Q/Ly3n33PGMzUwLOo4MoE+dO5UxGanc8dgG2jTsfEiJZEEtB6ab2WQzSwGuBJZF8OvJEPbwqgpqG1u45ozCoKPIAEtPTeKLF81kdXk1j67VbOdDScQKyt1bgOuBp4CNwB/cvcTMbjezpQBmdoqZVQAfAH5uZiWRyiPxq7m1jbtf3MZJE7JZMDE76DgSAZcvKGBOfibffnITx5q0PPxQEdFrUO7+hLvPcPep7v6N8Gu3uPuy8OPl7l7g7ununuPuxZHMI/HpT6t2U37oGDecP02DI+JUQoLxtfcUsedIA798cVvQcWSQxMQgCZHuNLa08uPntzI3P4vzZ40JOo5E0KlTclhSPI6fvlDG/pqGoOPIIFBBSUz79cs7KD90jP+4aKaOnoaAL188i5ZW53tPadj5UKCCkpi1v6aBHz+3lQtmj+XcGbr9YCiYlJPONWcW8uCqCtbvPhJ0HIkwFZTEJHfnKw+vo6XN+dp7ZwcdRwbR9edPY+TwFO54bAPuGnYez1RQEpMeWF7Oc5sq+b9LZjEpJz3oODKIMtOS+bcLZ/Da9kM8tnZv0HEkglRQEnNWl1dz67ISzpo2Wvc9DVFXLZ7I3Pws7nhsA7UNzUHHkQhRQUlM2XnwKNf+ZgVjMlL5f1edrCU1hqjEBOPOS+dQVdfI95/ZEnQciRAVlMSM8kP1XHX3P2hubeNXHzuFUekpQUeSAJ00IZsPnzqRe1/ZQckeDZiIRyooiQlv7DrM+3/6CvXNrfz2E6cyc1xG0JEkCnzxXbMYlZ7CV/+8XvP0xSEVlES1tjbnf1/dwQfv/gdpyQn8/trTKc7LCjqWRIms4cl85eLZvLGrmt+vKO/9DRJTVFAStbZV1fGRe17ja4+UcOrkUTxy3Vk6cpJ/ctnJ+SyePIpvPbmJA3WNQceRAaSCkqhTfqieLz24hgt/8HdW76rmPy+by2/+dbGuOUmXzIxvXDqHY02t3LZM803Hk6SgA4i021/TwH//tZQHlu/CzPjY6YV85ryp5GakBh1Notz0sRlcf/40vv/MFpaetI93FY8LOpIMABWUBO5gXSM/e6GM37y6k9Y254OnTOD686cxPmtY0NEkhnz63Kk8sW4vX3tkPadOySFrWHLQkeQEqaAkMHWNLdz992388sVtNDS3ctnJBdz4zulMzBkedDSJQSlJCXzninlcetfLfOvJjXzz/fOCjiQnSAUlg665tY37X9/F/3tuKwfqmnjP3PF84cIZTBszIuhoEuPmFWTzybOn8PO/b+N98/JVsD0kAAAOiUlEQVQ4Y9rooCPJCVBByaBaXV7NTQ+tZdO+Wk6dPIpffmw28ydoFVwZOJ+/YAZPlezjpofX8ZfPn83wFP2Yi1UaxSeDoqG5ldsf3cBlP3mZ6vpmfv6RhTxw7WkqJxlww1IS+dbl8yg/XM83Ht8YdBw5AfrVQiKurKqO6+5bxaZ9tVx92kS+tGQWmWm6gC2Rc9qUHD5x1mR+8eJ23jl7DOfPGht0JDkOOoKSiHp0zR6W/vgl9tc08Ov/cwp3XjpX5SSD4j8umsmscRl86cF1HNQNvDFJBSUR4e786NmtfO7+N5g1PpPHbzibd8wcE3QsGUJSkxL54ZXzqTnWzE0Pr9PihjFIBSUDrqmljf/441p+8OwWLl9QwP2fPI28bN3TJINv1rhMvnjRTJ7ZsJ/7X9dcfbFG16BkQB051sxnfruSV8oO8oULZnDDO6dhpjWbJDgfP2syf99axW2PljA3P4u5BZpsOFboCEoGTMXheq746Sss33GI//rASdx4wXSVkwQuIcH44QfnMzo9hc/ct5Lq+qagI0kfqaBkQKytqOayn7zCvpoG7v3XxVy+sCDoSCJvyhmRyl0fXsD+mgZufGA1rVo7KiaooOSEPV2yj3/5+aukJCbw8GfO4Iypuntfos/JE0dy6/uKeWFLle6PihG6BiXHzd255+Ud3Pn4BuYVZPPLjy7SzOMS1T586kRKK+u45+Xt5GWn8YmzpwQdSXqggpLj0tgSmhnivtd2cVHxWH74wZMZlpIYdCyRHpkZX3tvEZW1Ddz5+EZGpCZx5eKJQceSbqigpN/KD9Vz3e9WsbbiCJ86dwpfumgWiQkaDCGxITHB+P6/zKe+aSU3PbyOxpY2PnZGYdCxpAsqKOkzd+fBlRXc8dgG3OFnVy9kyRwtDCexJy05kZ9/ZCHX/+4Nbl1Wwp4jx/SLVhRSQUmfbD9wlFuXlfD3LVWcUjiS733gJCblpAcdS+S4pSYl8pMPL+DWZSX8/IVtbNhTw3evOIlxWWlBR5MwFZT0qPxQPT99oYzfLy8nNSmBry8t5iOnTSJBv2lKHEhOTOA/L5vLnLwsbn+shAt/8AJfumgmVy6eSHKiBjkHzSI5P5WZLQF+BCQCv3T3b3X6fCrwG2AhcBD4oLvv6GmfixYt8hUrVkQmsABwtLGFF7ZU8fCq3Ty3aT9JCcaHFk/k+vOna5SexK0dB45y08Nr+ce2Q0wcNZxrzijk/QvyyR6eEnS0uGNmK919Ua/bRaqgzCwR2AJcCFQAy4Gr3H1Dh20+C8xz90+b2ZXAZe7+wZ72q4IaWI0treypbqC0so415dWsLq/m9R2HaGppY/SIFK5aPJEPnTqR8VmaS0/in7vz/OZKfvRcKWvKq0lJSuDUyaM4a9po5hVkM2tcBiPTVVgnqq8FFclTfIuBUnffFg70AHAJsKHDNpcAt4UfPwj8t5mZR/Cw7t5XdlBd34wT+hLtX+nNLxh+oWOAt7bp+j2dP9/xk29t0/N7u9r/W1/f+/yef97mrc8fa26lrqGF2oYWahtbOHS0kcraxje3TUwwZo7N4OpTJ3FR8VgWThpJkk5zyBBiZpw/ayznzxrLhj01PLyqghe2VPHNJze9uU1mWhK5GankZqSSNSyZtORE0pISSUtOIDU5EQvvx4zwY0gww0Jf4M3Xwq/EpOveMXVQfjZEsqDygY7TB1cAp3a3jbu3mNkRIAc40HEjM7sWuBZg4sQTu2fh1y9vZ8fB+h63aZ8+zt72mr3ttbe2efvGb3/P27fpvN/O++xqP29u083rXWWim6+XlpxIRloSGWlJ5GcPY05eJgUjh1MwchiFo4dTND5L9zKJhBXlZVKUV8RXgcraBjburWXzvhp2Hz5GVV0jlTWN7DhQT0NLKw3NrTQ0t9HY0op76BdCx8N/hn7JjKfZlT517hSSBuFHRUwMknD3u4G7IXSK70T29dy/n/fPJaMJTUWkB2My0hiTkca5M3JPeF/u/k9nSWLNYP3IjGRB7QYmdHheEH6tq20qzCwJyCI0WCJidJ+DiASp/fSf9C6SJxGXA9PNbLKZpQBXAss6bbMM+Fj48RXAXyN5/UlERGJHxI6gwteUrgeeIjTM/B53LzGz24EV7r4M+BXwv2ZWChwiVGIiIiKRvQbl7k8AT3R67ZYOjxuAD0Qyg4iIxCaNIRYRkaikghIRkaikghIRkaikghIRkaikghIRkagU0dnMI8HMqoCdvWw2mk7TJUWxWMkaKzkhdrLGSk6InayxkhNiJ2skck5y916n5Yi5guoLM1vRl5lyo0GsZI2VnBA7WWMlJ8RO1ljJCbGTNcicOsUnIiJRSQUlIiJRKV4L6u6gA/RDrGSNlZwQO1ljJSfETtZYyQmxkzWwnHF5DUpERGJfvB5BiYhIjFNBiYhIVIrbgjKzO8xsrZmtNrOnzSwv6ExdMbPvmtmmcNY/mVl20Jm6Y2YfMLMSM2szs6gbHmtmS8xss5mVmtlNQefpjpndY2aVZrY+6Cw9MbMJZva8mW0I/7vfGHSm7phZmpm9bmZrwlm/HnSmnphZopm9YWaPBZ2lJ2a2w8zWhX+Orhjsrx+3BQV8193nuft84DHglt7eEJBngDnuPg/YAnw54Dw9WQ+8H/h70EE6M7NE4C7g3UARcJWZFQWbqlv/AywJOkQftAD/7u5FwGnAdVH837QRON/dTwLmA0vM7LSAM/XkRmBj0CH66B3uPj+Ie6HitqDcvabD03QgKkeDuPvT7t4SfvoPoCDIPD1x943uvjnoHN1YDJS6+zZ3bwIeAC4JOFOX3P3vhBbojGruvtfdV4Uf1xL6gZofbKqueUhd+Gly+CMqv+fNrAB4D/DLoLNEu7gtKAAz+4aZlQMfJnqPoDr6V+DJoEPEqHygvMPzCqL0h2ksMrNC4GTgtWCTdC982mw1UAk84+7RmvWHwJeAtqCD9IEDT5vZSjO7drC/eEwXlJk9a2bru/i4BMDdb3b3CcB9wPXRmjO8zc2ETqncF1TOcI5es8rQYmYjgIeAz3c6MxFV3L01fEq/AFhsZnOCztSZmb0XqHT3lUFn6aOz3H0BoVPn15nZOYP5xSO65HukufsFfdz0PkJLz98awTjd6i2nmV0DvBd4pwd8Y1o//ptGm93AhA7PC8KvyQkws2RC5XSfuz8cdJ6+cPdqM3ue0HW+aBuIciaw1MwuBtKATDP7rbtfHXCuLrn77vCflWb2J0Kn0gftGnRMH0H1xMymd3h6CbApqCw9MbMlhA73l7p7fdB5YthyYLqZTTazFOBKYFnAmWKamRnwK2Cju38/6Dw9MbPc9hGwZjYMuJAo/J539y+7e4G7FxL6f/Sv0VpOZpZuZhntj4F3MciFH7cFBXwrfGpqLaH/sNE6RPa/gQzgmfBQzp8FHag7ZnaZmVUApwOPm9lTQWdqFx5ocj3wFKGL+X9w95JgU3XNzO4HXgVmmlmFmX086EzdOBP4CHB++P/N1eHf/KPReOD58Pf7ckLXoKJ6CHcMGAu8ZGZrgNeBx939L4MZQFMdiYhIVIrnIygREYlhKigREYlKKigREYlKKigREYlKKigREYlKKiiJOeF7Xl4K30ZwaYfXH+nrrPVmVhf+M8/MHuzw+v3hmeW/YGazwkOr3zCzqQP/NzlxZvZpM/to0DlEIkHDzCXmmNkNhCZbfRh4wt3PM7P3AQvd/bY+7qPO3Ud0em0c8JK7Tws/vwlIcvc7+5Et0d1b+7q9iHRPR1ASi5qB4UAq0GpmScDnge9094bwDBOvhte2ubPD64Ud1mV6GsgPHzXdGt7nZ8LT5mBmV4fXHFptZj8PL/GBmdWZ2X+Fb2g83cwWmtkL4Qk2nzKz8eHt/mZm3w7vY4uZnR1+PdHMvtd+Y7mZfS78epf76fT3us3M/qOn/Xfa/rzwPh8xs21m9i0z+3D4PevajxTDR6kPmdny8MeZ4dcXh/87vmFmr5jZzPDr15jZw2b2FzPbambf6fB3+5/w322dmX2hr//IIri7PvQRUx9AFvA4sAJ4J3ADcE0v71kGfDT8+DqgLvy4EFjf+XH4+W3Af4QfzwYeBZLDz3/SYX8O/Ev4cTLwCpAbfv5B4J7w478B/xV+fDHwbPjxZ4AHCR2tAYzqaT+d/l4dM3a5/07bnwdUE5p5IZXQfIVfD3/uRuCH4ce/IzRRKMBEQtMdAWR2yHkB8FD48TXAtvC/TRqwk9DciAsJzerQ/vWzg/7/Rx+x8xHTk8XK0OTuRwitp4OZjQRuAi4zs18AIwn9kH6109vOBC4PP/5f4Nv9/LLvJPTDdnloijqGEVrWAaCV0ISqADOBOYSmrgJIBPZ22E/7hKsrCRUihH7Q/8zD64K5+yELzcTd036609X+O1vu7nsBzKyM0JEjwDrgHR0yFYW/NoQmNR1BqIDutdBcl06oSNs9F/63wcw2AJOAEmCKmf2Y0C8VTyPSRyooiXVfA74BXAW8ROhI5GHgoi62PZELrgbc6+5drXjc4G9ddzKgxN1P72Y/jeE/W+n5+6+3/XSnL/tv7PC4rcPztg7vSQBOc/eGt4Uy+2/geXe/zEJrRP2tm/22EjrSOmxmJxH69/g08C+E1j0T6ZWuQUnMCv8WX+DufyN0TaqNUAkN62LzlwnNHg2hBSz76zngCjMbE/7ao8xsUhfbbQZyzez08HbJZlbcy76fAT4VvpaGmY06zv0MpKeBz7U/MbP54YdZvLWMyTW97cTMRgMJ7v4Q8FVgwcDGlHimgpJY9g3g5vDj+wldy1kO/KiLbW8ktODaOo5jpV1330DoB+zTFpox+xlC13E6b9cEXAF8OzxoYjVwRi+7/yWwC1gbfs+HjnM/A+kGYFF40MYGQkc/EBqI8k0ze4O+nYHJB/5moZVufwt0dQQq0iUNMxcRkaikIygREYlKKigREYlKKigREYlKKigREYlKKigREYlKKigREYlKKigREYlK/x8mAOe8Fo/l4AAAAABJRU5ErkJggg==\n"},"metadata":{"needs_background":"light"}}]},{"metadata":{"dc":{"key":"53"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 8. The probability of a difference\n<p><img src=\"https://assets.datacamp.com/production/project_184/img/ziggy_smiling.png\" style=\"width:200px; float:left\"> </p>\n<p>From this chart, we can see that the most likely % difference is around 1% - 2%, and that most of the distribution is above 0%, in favor of a gate at level 30. But what is the <em>probability</em> that the difference is above 0%? Let's calculate that as well.</p>"},{"metadata":{"dc":{"key":"53"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# Calculating the probability that 1-day retention \n# is greater when the gate is at level 30.\nprob = (boot_1d['diff'] > 0).sum() / len(boot_1d)\n\n# Pretty printing the probability\n'{:.1%}'.format(prob)","execution_count":16,"outputs":[{"output_type":"execute_result","execution_count":16,"data":{"text/plain":"'95.8%'"},"metadata":{}}]},{"metadata":{"dc":{"key":"60"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 9. 7-day retention by AB-group\n<p>The bootstrap analysis tells us that there is a high probability that 1-day retention is better when the gate is at level 30. However, since players have only been playing the game for one day, it is likely that most players haven't reached level 30 yet. That is, many players won't have been affected by the gate, even if it's as early as level 30. </p>\n<p>But after having played for a week, more players should have reached level 40, and therefore it makes sense to also look at 7-day retention. That is: What percentage of the people that installed the game also showed up a week later to play the game again.</p>\n<p>Let's start by calculating 7-day retention for the two AB-groups.</p>"},{"metadata":{"dc":{"key":"60"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# Calculating 7-day retention for both AB-groups\ndf.groupby('version')['retention_7'].sum() / df.groupby('version')['userid'].count()","execution_count":18,"outputs":[{"output_type":"execute_result","execution_count":18,"data":{"text/plain":"version\ngate_30    0.190201\ngate_40    0.182000\ndtype: float64"},"metadata":{}}]},{"metadata":{"dc":{"key":"67"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 10. Bootstrapping the difference again\n<p>Like with 1-day retention, we see that 7-day retention is slightly lower (18.2%) when the gate is at level 40 than when the gate is at level 30 (19.0%). This difference is also larger than for 1-day retention, presumably because more players have had time to hit the first gate. We also see that the <em>overall</em> 7-day retention is lower than the <em>overall</em> 1-day retention; fewer people play a game a week after installing than a day after installing.</p>\n<p>But as before, let's use bootstrap analysis to figure out how certain we should be of the difference between the AB-groups.</p>"},{"metadata":{"dc":{"key":"67"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# Creating a list with bootstrapped means for each AB-group\nboot_7d = []\nfor i in range(500):\n    boot_mean = df.sample(frac=1, replace=True).groupby('version')['retention_7'].mean()\n    boot_7d.append(boot_mean)\n    \n# Transforming the list to a DataFrame\nboot_7d = pd.DataFrame(boot_7d)\n\n# Adding a column with the % difference between the two AB-groups\nboot_7d['diff'] = (boot_7d['gate_30'] - boot_7d['gate_40']) /  boot_7d['gate_30'] * 100\n\n# Ploting the bootstrap % difference\nax = boot_7d['diff'].plot(kind = 'kde')\nax.set_xlabel(\"% difference in means\")\n\n# Calculating the probability that 7-day retention is greater when the gate is at level 30\nprob = (boot_7d['diff'] > 0).sum() / len(boot_7d)\n\n# Pretty printing the probability\n'{:.1%}'.format(prob)","execution_count":20,"outputs":[{"output_type":"execute_result","execution_count":20,"data":{"text/plain":"'100.0%'"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<matplotlib.figure.Figure at 0x7fed1a426fd0>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl8XXW97//XJ3PTDG06pU3apqVzC21paGUQEWRWkCMqdQKPihzh4HDO/f1wuA5cvcfhHJUrTqhcOSogIGpVFMpQZeyY0tI5ndJ0TJomTaekST73j70Cm5A2acnea+2d9/Px2I+svfZaa7/bJPuT71rf9f2auyMiIhI1GWEHEBER6Y4KlIiIRJIKlIiIRJIKlIiIRJIKlIiIRJIKlIiIRJIKlIiIRJIKlIiIRJIKlIiIRFJW2AH6ytChQ72ioiLsGCIi0oPly5fXu/uwnrZLmwJVUVHBsmXLwo4hIiI9MLPtvdlOp/hERCSSVKBERCSSVKBERCSSVKBERCSSVKBERCSSVKBERCSSVKBERCSS0uY+KJH+oPFIK6tqmyjIy+KssmKyMvU3pqQvFSiRFNDW3sH3n9zEz57dQktbBwBjSvL5j386k/MnDA05nUhi6M8vkYhr73Buu7+Ku5+p5ooZpTzwibfwf+bPJivTuPHeJTy2enfYEUUSQi0okYj7zuMb+NuaPXzp6ql8/K3jX13/9snD+Oj/XcpnfruSsUPymT6qOMSUIn1PLSiRCFu+vYGf/mMz8+eOfl1xAijMy+YnH57D4PxsPv3gSo63d4SUUiQxElqgzOwKM9tgZtVmdkc3r99iZqvNbKWZPWdm0+Je+3yw3wYzuzyROUWiqKPD+fIf1zCqeABfvHpat9sMLcjlG+8+k+p9h/jl89uSG1AkwRJWoMwsE/ghcCUwDZgfX4AC97v7me4+C/g28N1g32nADcB04ArgR8HxRPqNv63Zw5pdB/m3yyZRkHvis/GXTB3O2yYN4+5nqjnU0pbEhCKJlcgW1Fyg2t23uHsr8CBwbfwG7n4w7ulAwIPla4EH3b3F3bcC1cHxRPqFjg7nv57YwMThBVw7q+yk25oZn710Ek1Hj/OrF3s1i4FISkhkgSoDdsQ9rw3WvY6Z3Wpmm4m1oG4/xX1vNrNlZrasrq6uz4KLhO3vm+rYXHeY2y6eQGaG9bj9rNGDuGDCUH75wlbadC1K0kTonSTc/Yfufgbw/wNfOsV973H3SnevHDasx8kZRVLGfS9sY1hhLlfOGNnrfT587lj2Hmxh0Qb9sSbpIZEFaicwOu55ebDuRB4E3n2a+4qkjW31h1m0oY4PzhtDTlbvf0UvnjKc4YW5PLCkJoHpRJInkQVqKTDRzMaZWQ6xTg8L4jcws4lxT68GNgXLC4AbzCzXzMYBE4ElCcwqEhmPrqjFDG44Z8wp7ZedmcF7K8t5ZsM+9h48lqB0IsmTsALl7m3AbcDjwDrgIXdfY2Z3mtk1wWa3mdkaM1sJfA64Mdh3DfAQsBb4G3Cru7cnKqtIVLg7f3x5F+edMYTS4rxT3v+62WV0ODy+Zk8C0okkV0JHknD3x4DHuqz7ctzyp0+y7zeAbyQunUj0rNzRyPb9R7j1ogmntf+E4YVMHF7AY6t385FzK/o2nEiShd5JQkRe88eVu8jJyuCKM0tP+xhXnjmSJVsbqGtu6cNkIsmnAiUSEe0dzp9X7ebiycMpyss+7eNcdWYpHQ4L1+7tw3QiyacCJRIRK3c0Un+ohavO6n3X8u5MHlFI2aAB/H3jvj5KJhIOFSiRiHhy3V6yMoy3TXpz9/SZGRdOGsbz1fs1gKykNBUokYh4cu1e5o4roXjA6Z/e6/S2ScM41NJGVU1jHyQTCYcKlEgEbN9/mE37DvGOqSP65HjnTRhCZobpNJ+kNBUokQh4cl2skPRVgSrKy+bsMYN4blN9nxxPJAwqUCIRsGjDPiYML2DMkPw+O+a8cUN4ZddBDmsKDklRKlAiITt2vJ0lWxt468ShfXrcc8aV0N7hug4lKUsFSiRky7YdoKWtgwsn9u2I/HPGDibDYMm2hj49rkiyqECJhOzZ6jqyM41540v69LgFuVlMH1XMkq37+/S4IsmiAiUSsmc31nP2mMHk5/T90JjnVJRQVdNIa5vuh5LUowIlEqL6Qy2s3X2QC9/kzbknMndcCS1tHaze2ZSQ44skkgqUSIier451A79gQt92kOh0TsVgAJZv13UoST0qUCIher66nuIB2cwoK07I8YcU5FI+eAAv16oFJalHBUokRIu3NjB3XAmZGZaw95g5ehAv71BXc0k9KlAiIdnTdIzt+48wb1zf9t7ramZ5MbUHjrL/kOaHktSiAiUSksVB9+9544Yk9H1mlg8CYJVO80mKUYESCcmSrQ0U5GYxdWRhQt9nRlkxGRabb0oklahAiYRk8dYGKisGk5WZ2F/DgblZTBxeyMu1KlCSWlSgREJQf6iF6n2HmJvg60+dziovZlVtE+6elPcT6QsqUCIhWLo1dl9Soq8/dTpr9CAaDreys/FoUt5PpC+oQImEYPHWBvKyMzgzQfc/dTVtZBEAa3cdTMr7ifQFFSiRECze2sCcsYPJyUrOr+DUkYWYwdrdKlCSOlSgRJKs6chx1u85yNyK5JzeA8jPyWLc0IFqQUlKSWiBMrMrzGyDmVWb2R3dvP45M1trZqvM7CkzGxv3WruZrQweCxKZUySZlm5rwJ0+n16jJ9NGFqkFJSklYQXKzDKBHwJXAtOA+WY2rctmVUClu58FPAJ8O+61o+4+K3hck6icIsm2dFsDOZkZzBo9KKnvO21UEbUHjtJ09HhS31fkdCWyBTUXqHb3Le7eCjwIXBu/gbs/4+5HgqcvAeUJzCMSCStqDjCjrIi87Mykvq86SkiqSWSBKgN2xD2vDdadyMeAv8Y9zzOzZWb2kpm9u7sdzOzmYJtldXV1bz6xSIIdb+9gVW0Ts8cMTvp7TxsVFCid5pMU0fdTeJ4GM/sQUAm8LW71WHffaWbjgafNbLW7b47fz93vAe4BqKys1B2IEnnrdzfT0tbB7DHJPb0HMLwwj2GFuWpBScpIZAtqJzA67nl5sO51zOwdwBeBa9z91eGW3X1n8HULsAiYncCsIklRteMAQCgtKFBHCUktiSxQS4GJZjbOzHKAG4DX9cYzs9nAT4kVp31x6webWW6wPBQ4H1ibwKwiSbFi+wGGF+YyqjgvlPefNqqITXubaWlrD+X9RU5FwgqUu7cBtwGPA+uAh9x9jZndaWadvfK+AxQAD3fpTj4VWGZmLwPPAN90dxUoSXlVOxqZPWYQZomboPBkppQW0tbhbK0/HMr7i5yKhF6DcvfHgMe6rPty3PI7TrDfC8CZicwmkmz7D7Wwff8RPjB3TGgZJpfGpvbYsKeZKaVFoeUQ6Q2NJCGSJJ3zMYV1/Qlg/NACsjKM9XuaQ8sg0lsqUCJJUlXTSGaGJW2A2O7kZGVwxrACNqpASQpQgRJJkqodB5g6spABOcm9QberSaWFakFJSlCBEkmC9g7n5R1NzB4d3um9TlNKC9nZeJTmYxrySKJNBUokCar3HeJQS1soN+h2NWlErKPExr2HQk4icnIqUCJJUFUT7g268aaUdhYoneaTaFOBEkmCFTUHGJSfTcWQ/LCjUDZoAPk5mWzQdSiJOBUokSSoqmlk9ujwbtCNl5FhTBpRqAIlkacCJZJgTUePs2nfIc6OwOm9TlNKC9mwtxl3jbEs0aUCJZJgq2rDv0G3q0kjCmk43Er9odawo4ickAqUSIJV1TRiBmeNDu8G3a6mxA15JBJVKlAiCVZVc4CJwwsoyssOO8qrJgUFav0eTb0h0aUCJZJA7h4bwTwCN+jGG1qQy9CCHHU1l0hTgRJJoG37j9B45HgkbtDtanKpevJJtKlAiSRQlG7Q7WrSiEI27j1ER4d68kk0qUCJJNCKmgMU5GYxYXhB2FHeYEppIUePt7PjwJGwo4h0SwVKJIGqahqZObqYzIzwb9DtqnNMPo1sLlGlAiWSIEda21i/pzlSN+jG6yxQug4lUaUCJZIgq2ubaO/wSHaQABiYm8WYknwVKIksFSiRBKkKpnifFbEu5vGmlBbqXiiJLBUokQSpqjlAxZB8SgbmhB3lhKaUFrK1/jDHjreHHUXkDVSgRBLA3VlR0xjJ7uXxJpcW0eGxCRVFokYFSiQBdjUdo665JbLXnzpNLlVPPokuFSiRBHj1Bt0IX38CqBiST05WBht0HUoiSAVKJAFWbG8kNyuDKSMLw45yUlmZGUwaUaAWlERSQguUmV1hZhvMrNrM7ujm9c+Z2VozW2VmT5nZ2LjXbjSzTcHjxkTmFOlrVTsOcFZ5MdmZ0f8bcPKIIhUoiaSE/faYWSbwQ+BKYBow38ymddmsCqh097OAR4BvB/uWAF8B5gFzga+YWbTPlYgEWtraWbPzYGRv0O1qSmkhdc0tNBzW5IUSLYn8824uUO3uW9y9FXgQuDZ+A3d/xt07BwJ7CSgPli8HFrp7g7sfABYCVyQwq0ifWbvrIK3tHZHvINFpsuaGkohKZIEqA3bEPa8N1p3Ix4C/nsq+ZnazmS0zs2V1dXVvMq5I36iqid4U7yfTeZ1MI0pI1ETiBLmZfQioBL5zKvu5+z3uXunulcOGDUtMOJFTVLWjkVHFeYwoygs7Sq8MK8ilZGAO63erQEm0JLJA7QRGxz0vD9a9jpm9A/gicI27t5zKviJRVFVzIGVaTwBmxuQRhazX7LoSMYksUEuBiWY2zsxygBuABfEbmNls4KfEitO+uJceBy4zs8FB54jLgnUikbav+Ri1B46mzPWnTpNLC9m0t1mTF0qkJKxAuXsbcBuxwrIOeMjd15jZnWZ2TbDZd4AC4GEzW2lmC4J9G4D/RazILQXuDNaJRNpr159Sq0BNKS3kSKsmL5RoyUrkwd39MeCxLuu+HLf8jpPsey9wb+LSifS9qppGsjON6aOKw45ySqaMLAJg3e5mxg4ZGHIakZhIdJIQSRdVNQeYNrKIvOzMsKOckkkjCjBTTz6JFhUokT7S1t7BqtqmlOog0Sk/J4uxJfm6F0oiRQVKpI9s2NvM0ePtKXf9qdP0UcWs2aUCJdGhAiXSRzo7SKTKEEddTS8roqbhCE1HjocdRQRQgRLpM1U1jQwtyKF88ICwo5yWGUHHjjW7m0JOIhKjAiXSR6p2HGDW6MGYWdhRTsv0UbGefGt26jSfRIMKlEgfaDzSypa6wyl7/QlgSEEuo4rzeGWXWlASDSpQIn2gakdq3qDb1fSyYl7ZqQIl0aACJdIHqmoayTA4qzy1C9SMUcVsqT/M4Za2sKOI9K5AmdmjZna1mamgiXSjquYAk0YUUpCb0MFZEm76qCLcYd1uXYeS8PW24PwI+ACwycy+aWaTE5hJJKV0dDgrdzSm5A26Xc0oi/Xk02k+iYJeFSh3f9LdPwicDWwDnjSzF8zso2aWnciAIlFXXXeI5mNtzBmb+gVqRFEuQwtyeEU37EoE9PqUnZkNAW4CPg5UAXcRK1gLE5JMJEWs2H4AgLNTvIMExOaGmj5KHSUkGnp1wtzMfg9MBn4FvMvddwcv/dbMliUqnEgqWFFzgMH52Ywbmh6jgJ9ZVsxz1fUcbW1nQE5qDXor6aW3V3R/Fkyd8Sozy3X3FnevTEAukZSxfHtsBt1UvUG3q1mjB9He4byyq4lzKkrCjiP9WG9P8X29m3Uv9mUQkVTUeKSVzXWH0+L0XqfOe7k6T12KhOWkLSgzKwXKgAHB9OydfyIWAfkJziYSeZ036J6dBh0kOg0pyGXskPxXB78VCUtPp/guJ9Yxohz4btz6ZuALCcokkjKqth8gw2Bmit+g29Xs0YN4YfN+3D1tTl1K6jlpgXL3+4D7zOw97v67JGUSSRkrahqZUlrEwBS/Qber2WMG84eVu9jddIxRg1JzdHZJfT2d4vuQu/8aqDCzz3V93d2/281uIv1Ce3CD7rtnjwo7Sp/rvA5VVdOoAiWh6amTRGe/2QKgsJuHSL+1cW8zh1raUnaCwpOZOrKI3KwMqmrUUULC09Mpvp8GX7+WnDgiqWNF8OGdDiNIdJWdmcFZ5cWvdgIRCUNvB4v9tpkVmVm2mT1lZnVm9qFEhxOJshXbGxkyMIcxJenZoXX2mMGsrm3i2PH2sKNIP9Xb+6Auc/eDwDuJjcU3AfgfiQolkgqqatLrBt2u5o0robW9g5VqRUlIelugOk8FXg087O4aqEv6tQOHW9lSf5izx6ZX9/J4lRUlZBi8tGV/2FGkn+ptgfqzma0H5gBPmdkw4FhPO5nZFWa2wcyqzeyObl6/0MxWmFmbmV3f5bV2M1sZPBb0MqdIUizd1gBA5dj0HQqoeEA200cVq0BJaHo73cYdwHlApbsfBw4D155sHzPLBH4IXAlMA+ab2bQum9UQuxH4/m4OcdTdZwWPa3qTUyRZlmxtICcrg5mji8OOklDzxpWwoqZR16EkFKcyQ+4U4P1m9hHgeuCyHrafC1S7+xZ3bwUepEtRc/dt7r4K6DiFHCKhW7qtgVmjB5Gbld6jfb9l/BBa23QdSsLR2158vwL+E7gAOCd49DSKeRmwI+55bbCut/LMbJmZvWRm7z5BrpuDbZbV1dWdwqFFTt+hljZe2XWQeePS9/Rep3PGlWAGi7c0hB1F+qHejs9SCUxzd09kmC7GuvtOMxsPPG1mq919c/wG7n4PcA9AZWVlMrNJP7Zi+wHaO5y5/aBAFQ/IZtrIIl7asp9PMzHsONLP9PYU3ytA6SkeeycwOu55ebCuV9x9Z/B1C7AImH2K7y+SEEu2NpCZYWk5gkR3zh0/hOU1BzjaqutQkly9LVBDgbVm9riZLeh89LDPUmCimY0zsxzgBqBXvfHMbLCZ5QbLQ4HzgbW9zCqSUEu2NjBjVPoNEHsiF04aRmtbh3rzSdL19jfsq6d6YHdvM7PbgMeBTOBed19jZncCy9x9gZmdA/weGAy8y8y+5u7TganAT82sg1gR/aa7q0BJ6I4db2dlbSM3njs27ChJM3dcCQOyM1m0YR9vnzI87DjSj/SqQLn7381sLDDR3Z80s3xiRaen/R4DHuuy7stxy0uJnfrrut8LwJm9ySaSTKtqm2ht62DuuCFhR0mavOxMzj1jCIs2qiOSJFdve/F9AngE+Gmwqgz4Q6JCiUTVkq2x01znVPSP60+dLpo8jO37j7C1/nDYUaQf6e01qFuJXQc6CODumwC19aXfWby1gSmlhQzKzwk7SlJdNCn2675ow76Qk0h/0tsC1RLcbAuAmWUB6tYt/UpLWztLtzX0i/ufuhozJJ9xQweyaINO80ny9LZA/d3MvgAMMLNLgYeBPyUulkj0rNjeyLHjHVwwcVjYUUJx0eRhvLhlP4db2sKOIv1EbwvUHUAdsBr4JLGOD19KVCiRKHq+up7MDGPe+P7XggK4YnoprW0dPKPTfJIkve3F12FmfwD+4O5q40u/9Fx1PTPLiynKyw47SigqK0oYWpDDX1/ZwzvPGhV2HOkHTtqCspivmlk9sAHYEMym++WT7SeSbpqOHmdVbSMXTBgadpTQZGYYl00v5Zn1+zS6uSRFT6f4Pkus99457l7i7iXAPOB8M/tswtOJRMRLW/bT4XB+Py5QAFfOKOVIazv/0D1RkgQ9FagPA/PdfWvnimBsvA8BH0lkMJEoeb66ngHZmczuJ+Pvnchbxg+heEA2f3tlT9hRpB/oqUBlu3t915XBdaj+eSJe+qXnquuZN76EnKxTmUIt/WRnZnDptBEsXLtXp/kk4Xr6bWs9zddE0sbOxqNsqTvcr68/xbt21iiaW9p4ap1680li9VSgZprZwW4ezWisPOknnlkf+yC+aHL/vP+pq/POGMqIolx+X1UbdhRJcyftZu7u6T2ftUgvPLN+H6NLBnDGsIKwo0RCZoZx7awy7n1uK/sPtTCkIDfsSJKm+vcJdZEeHDvezvOb67l48nDMLOw4kXHd7DLaOpw/r9oddhRJYypQIifx4pb9HDvewcVTR4QdJVKmjixiSmkhj1b1epJskVOmAiVyEs+s38eA7Mx+OUBsT66fU87LOxpZt/tg2FEkTalAiZyAu/P0+n2cP2Eoedm6HNvV9XPKycnK4P7FNWFHkTSlAiVyAuv3NFN74CiXTNXUZ90ZlJ/D1WeO5PdVOzXCuSSECpTICfz1lT1kGFw6TdefTuQD88ZwqKWNP728K+wokoZUoERO4K+rdzN3XAlD1Y36hCrHDmbSiALuX6LTfNL3VKBEulG9r5lN+w5x5YyRYUeJNDPjA3PHsKq2iVW1jWHHkTSjAiXSjb+ujg2Gevn00pCTRN8/zSlnYE4m9z63teeNRU6BCpRINx57ZQ9zxg6mtDgv7CiRV5SXzfvPGcOfV+1mT9OxsONIGlGBEuliW/1h1u0+yJUz1HrqrY+eX0GHO/e9uC3sKJJGVKBEuvjL6tjwPTq913ujS/K5fHop9y+u4UirupxL30hogTKzK8xsg5lVm9kd3bx+oZmtMLM2M7u+y2s3mtmm4HFjInOKdHJ3Hl1RyzkVgxldkh92nJTysQvG0XT0OL9brlHOpW8krECZWSbwQ+BKYBow38ymddmsBrgJuL/LviXAV4hNLz8X+IqZ9e+pTCUpVu9sYnPdYa6bXR52lJQzZ+xgZpYXc+/z2+jo8LDjSBpIZAtqLlDt7lvcvRV4ELg2fgN33+buq4COLvteDix09wZ3PwAsBK5IYFYRAH5ftZOczAyuPlPdy0+VmfGxt45na/1hFq7bG3YcSQOJLFBlwI6457XBuj7b18xuNrNlZrasrq7utIOKALS1d/Cnl3dxydThFOdnhx0nJV01o5QxJfn8aNFm3NWKkjcnpTtJuPs97l7p7pXDhmm2U3lznt1UT/2hVq6b3du/o6SrrMwMbnnbGby8o5EXNu8PO46kuEQWqJ3A6Ljn5cG6RO8rclp+t6KWwfnZXDRZg8O+Ge+ZU8bwwlzufro67CiS4hJZoJYCE81snJnlADcAC3q57+PAZWY2OOgccVmwTiQhGg638sSavVw7q4ycrJQ+sRC63KxMbr5wPC9u2c/y7QfCjiMpLGG/ie7eBtxGrLCsAx5y9zVmdqeZXQNgZueYWS3wXuCnZrYm2LcB+F/EitxS4M5gnUhCPLqiltb2DubPHRN2lLQwf+4YBuVn8+NFakXJ6ctK5MHd/THgsS7rvhy3vJTY6bvu9r0XuDeR+UQgdu/T/UtqmDN2MJNLC8OOkxYG5mbx0fPG8b0nN7Ju90GmjiwKO5KkIJ3LkH5vydYGttQdVuupj9143lgG5mTyo0Wbw44iKUoFSvq9B5bUUJiXpXuf+tig/Bw+dO5Y/rJqF9X7DoUdR1KQCpT0awcOt/LYK3v4p9llDMjJDDtO2vnEW8eTm5XJ3U9vCjuKpCAVKOnXHq3aSWtbBzfo9F5CDC3I5SPnjmXBy2pFyalTgZJ+q6PD+c3i7cwaPUgX8RPoExeqFSWnRwVK+q1nq+vZUneYm86rCDtKWotvRW2uUytKek8FSvqt+17YxtCCXK5S54iEe60VpfuipPdUoKRf2lZ/mGc27OOD88Zo5IgkGFqQy4fPHcsfV+5UK0p6Tb+Z0i/994vbyTTjg/PUOSJZbr5wPDlZGWpFSa+pQEm/c7iljYeX7eCqM0cyvCgv7Dj9RuxaVIVaUdJrKlDS7zy6opbmljZuOr8i7Cj9jlpRcipUoKRfcXd++cI2ziovZvboQWHH6XfiW1Fb1IqSHqhASb/yXHU9m+sOc+O5FZhZ2HH6JbWipLdUoKRfiXUtz+GdM9W1PCydrag/6FqU9EAFSvqNmv1HeGr9PubPHUNulsbdC9PNwX1R/+cpjS4hJ6YCJf3Gf7+4LehaPjbsKP3e0IJcbjyvggUv72Lj3uaw40hEqUBJv3C4pY2Hlu3g8hmllBara3kUfPLC8eRnZ3LXk2pFSfdUoKRfeGR5LQePtfGxC8aFHUUCgwfm8M8XjOMvq3ezdtfBsONIBKlASdpr73B+8dxWzh4ziLPHDA47jsT5+AXjKczL4vtPbgw7ikSQCpSkvYVr91LTcISPv3V82FGki+L8bD5+wXieWLuX1bVNYceRiFGBkrT3i+e2UD54AJdNGxF2FOnGP19QQfGAbL6nVpR0oQIlaW3ljkaWbjvAR88fR1amftyjqDAvm5svHM/T6/exouZA2HEkQvQbK2ntF89tpTA3i/dVlocdRU7ipvMqKBmYw/cWqhUlr1GBkrS1s/Eoj63ezQ1zR1OYlx12HDmJgblZ3PK28Ty7qZ6l2xrCjiMRoQIlaeu+F7YBcKOmdE8JH35LBUMLcvnuE2pFSUxCC5SZXWFmG8ys2szu6Ob1XDP7bfD6YjOrCNZXmNlRM1sZPH6SyJySfg61tPHA4hqunFFK+eD8sONILwzIyeRTF53Bi1v288Lm+rDjSAQkrECZWSbwQ+BKYBow38ymddnsY8ABd58AfA/4Vtxrm919VvC4JVE5JT09sLiG5pY2dS1PMR+YN4YRRbl8b+FG3D3sOBKyRLag5gLV7r7F3VuBB4Fru2xzLXBfsPwIcIlpDgR5k1ra2vnZs1s474whzNKcTyklLzuT294+gaXbDvDsJrWi+rtEFqgyYEfc89pgXbfbuHsb0AQMCV4bZ2ZVZvZ3M3trd29gZjeb2TIzW1ZXV9e36SVl/W75TvY1t3Dr2yeEHUVOw/vOGU3ZoAF8V62ofi+qnSR2A2PcfTbwOeB+MyvqupG73+Pule5eOWzYsKSHlOhpa+/gJ3/fzMzyYs47Y0jPO0jk5GZlctvFE1i5o5FFG/SHZ3+WyAK1Exgd97w8WNftNmaWBRQD+929xd33A7j7cmAzMCmBWSVN/GX1bmoajvCpt0/QjLkp7Po55YwuUSuqv0tkgVoKTDSzcWaWA9wALOiyzQLgxmD5euBpd3czGxZ0ssDMxgMTgS0JzCppoKPD+dEzm5k4vIBLp2pYo1SWnZnB7RdPZPXOJhau3Rt2HAlJwgpUcE3pNuBxYB3wkLuvMbM7zeyaYLNfAEPMrJrYqbzOrugXAqvMbCWxzhO2dv+ZAAAQ/ElEQVS3uLvu3pOTWrhuLxv2NnPL284gI0Otp1R33ewyxg0dyHcXbqSjQ62o/sjSpflcWVnpy5YtCzuGhKS9w7nqrmc53t7BE5+9UOPupYk/VO3kM79dyQ/mz+ZdM0eFHUf6iJktd/fKnrbTb7GkhT+v2sWGvc185tJJKk5p5F0zRzGltJDvPL6Blrb2sONIkuk3WVJeW3sH339yE1NKC3nnmSPDjiN9KDPD+MJVU6lpOMKvXtwedhxJMhUoSXm/W1HL1vrDfO7SSbr2lIYunDSMCycN4wdPV9N4pDXsOJJEKlCS0o62tnPXk5uYOXoQl2pCwrT1haum0HzsOHc/XR12FEkiFShJaff8Ywu7mo7x+Sun6L6nNDaltIj3zhnNfS9uY/v+w2HHkSRRgZKUtavxKD/+ezVXnzmSt4zXqBHp7nOXTSInM4OvLlijm3f7CRUoSVnf+tt6OhzuuHJK2FEkCUYU5fHZSyfxzIY6Hl+jm3f7AxUoSUmLt+znjyt38ckLxzO6RPM99Rc3nlfBlNJCvvanNRxuaQs7jiSYCpSknGPH27nj0dWMLhnAv1x0RthxJImyMzP4+rtnsLvpGN9bqJl3050KlKScu57axNb6w/zHdWeRn5MVdhxJssqKEj44bwy/eH4ri7fsDzuOJJAKlKSUl3c0cs8/tvDeOeVcMHFo2HEkJF+4aipjSvL5t4dfpvnY8bDjSIKoQEnKaD52nNsfrGJEYS5funpa2HEkRANzs/ju+2ayq/Eod/5pbdhxJEFUoCQluDv/8w+vsKPhCHfNn01xfnbYkSRkc8aW8C8XncHDy2t5eNmOnneQlKMCJSnh1y9t5w8rd3H7JRM5p6Ik7DgSEZ99xyTOO2MIX/zDK7yysynsONLHVKAk8p6vruerf1rLxVOG868XTww7jkRIVmYGP5g/m6EDc/jkr5az9+CxsCNJH1KBkkjbuLeZT/1mBWcMG8hdN8wiU4PBShdDCnK55yOVNB5p5cZ7l9B0VJ0m0oUKlETW5rpDfOBni8nNyuAXN55DYZ6uO0n3ZpQV85MPz2Fz3SE+cd8y3cSbJlSgJJI27m3mgz9bDDj3f+ItGi1CevTWicP47vtmsbzmAB/+xWK1pNKACpREzgvV9bznxy/Q7s6vPz6PCcMLwo4kKeJdM0dx9/zZrN7ZxAd//hL1h1rCjiRvggqUREZ7h/OjRdV85N4ljCzO4/efOo8ppUVhx5IUc+WZI7nnw5VU7zvEtXc/z9pdB8OOJKdJBUoiYcOeZub/7CW+/bcNXD69lIdvOY/ywTqtJ6fn7VOG8/Anz6O9w3nPj19gwcu7wo4kp0EDmUmoqvc187N/bOXh5TsoyM3iP987k/ecXabJB+VNO7O8mAW3nc8tv17O7Q9UsWj9Pr527XR1tkkhKlCSVO7OjoajPL1+L4+v2cuLW/aTk5nBTeeN4/ZLJjAoPyfsiJJGhhfl8dtPnssPnq7m7qc3sWRbA19/9wwumjw87GjSC5YuM1NWVlb6smXLwo4hcRoOt7JxbzOb9jazYW8zG/ceYtPeZg4cifWuGj90IO+ZU877zxnN0ILckNNKulu+vYF/f3gVW+sPc9m0EXzx6qmMHTIw7Fj9kpktd/fKHrdTgZK+cOBwK6t2NrFqR2Psa20jew++1oOqMDeLiSMKmDSikKkji3jbpGFUDNWHgyRXS1s7P392K3c/XU1rewfXzS7j1rdPYJx+FpMqEgXKzK4A7gIygZ+7+ze7vJ4L/DcwB9gPvN/dtwWvfR74GNAO3O7uj5/svVSgkudQSxuvBEXo5domVtc2UdNw5NXXxw8dyFnlxUwfVcyk0kImjSigtChP15UkMvYePMZP/r6Z+xfX0NLWwXlnDOH954zm4inDdY0qCUIvUGaWCWwELgVqgaXAfHdfG7fNp4Cz3P0WM7sBuM7d329m04AHgLnAKOBJYJK7t5/o/VSg+t6x4+3sajzK9oYjrNt9kLW7DrJu90G21B+m88embNAAziov5qzyQcwsL2Z6WTHFA/QLLqlhX/MxfrtkB79dtoPaA0fJyjDOqShh7rgSZpQVM21UEaVFeRpiq49FoUCdC3zV3S8Pnn8ewN3/I26bx4NtXjSzLGAPMAy4I37b+O1O9H5vtkA9tHQHu5uO4cT+P+L/W15dDFb6658G616/38m26fIl2K73+3f9lsV/D7vu53HvcuJjO0db22k+1sbBY8dpPtZG/aHWN9zkWD54ANNGFjFtVBEzywdxZnmxrh1JWujocJZua+CZDXUs2rCPjXub6Qh+PzIzjBGFuYwozqMwL5uC3EwG5mSRn5NJRoaRaUZmhmFmZGZApsWW0+mEgfH6f8zVZ5UyYXjh6R+vlwUqkb34yoD4SVpqgXkn2sbd28ysCRgSrH+py75lXd/AzG4GbgYYM2bMmwr74NIaVtQ09rhd5w+dvfr8tW+cvWGbLhufbJuTHPt1Pxon2qbb93jj/q9t9/r9BmRnUpiXRWFeFqNL8plZPoiywQMoGzSA0SX5TB5RqDmYJG1lZBjzxg9h3vgh3HHlFI60trFu90HW72lmV+NRdjceY2/zMZqOHmdX41GOtLRx5Hg77R1OR4fT7k5HB7S7096RHtf1T2ZyacGbKlC9ldLdzN39HuAeiLWg3syxHrnlvFeXXy0U6fQnkIj0Wn5OFnPGljBn7OnNPZYunc/gjWdsgKS1DhNZoHYCo+OelwfrutumNjjFV0yss0Rv9u1TGTrHLCJ9JJ3+uA3zn5LIoY6WAhPNbJyZ5QA3AAu6bLMAuDFYvh542mN/eiwAbjCzXDMbB0wEliQwq4iIREzCWlDBNaXbgMeJdTO/193XmNmdwDJ3XwD8AviVmVUDDcSKGMF2DwFrgTbg1pP14BMRkfSjG3VFRCSpetuLT6OZi4hIJKlAiYhIJKlAiYhIJKlAiYhIJKlAiYhIJKVNLz4zqwO2n8auQ4H6Po7T11IhI6RGzlTICKmRUxn7Tirk7MuMY919WE8bpU2BOl1mtqw33R3DlAoZITVypkJGSI2cyth3UiFnGBl1ik9ERCJJBUpERCJJBSoYDT3iUiEjpEbOVMgIqZFTGftOKuRMesZ+fw1KRESiSS0oERGJJBUoERGJJBWogJn9m5m5mQ0NO0t3zOw7ZrbezFaZ2e/NbFDYmTqZ2RVmtsHMqs3sjrDzdMfMRpvZM2a21szWmNmnw850ImaWaWZVZvbnsLOciJkNMrNHgp/JdWZ2btiZujKzzwbf61fM7AEzyws7E4CZ3Wtm+8zslbh1JWa20Mw2BV8HRzBj0j+DVKCIfXgBlwE1YWc5iYXADHc/C9gIfD7kPEDswxT4IXAlMA2Yb2bTwk3VrTbg39x9GvAW4NaI5gT4NLAu7BA9uAv4m7tPAWYSsbxmVgbcDlS6+wxic9LdEG6qV/0SuKLLujuAp9x9IvBU8DxMv+SNGZP+GaQCFfM94P8DIttjxN2fcPe24OlLQHmYeeLMBardfYu7twIPAteGnOkN3H23u68IlpuJfaCWhZvqjcysHLga+HnYWU7EzIqBC4lNOIq7t7p7Y7ipupUFDDCzLCAf2BVyHgDc/R/EJmiNdy1wX7B8H/DupIbqoruMYXwG9fsCZWbXAjvd/eWws5yCfwb+GnaIQBmwI+55LRH84I9nZhXAbGBxuEm69X1ifyx1hB3kJMYBdcD/DU5F/tzMBoYdKp677wT+k9hZkd1Ak7s/EW6qkxrh7ruD5T3AiDDD9EJSPoP6RYEysyeD89BdH9cCXwC+HHZG6DFn5zZfJHa66jfhJU1dZlYA/A74jLsfDDtPPDN7J7DP3ZeHnaUHWcDZwI/dfTZwmPBPSb1OcA3nWmLFdBQw0Mw+FG6q3vHYvT+RPZuTzM+grES/QRS4+zu6W29mZxL7AX7ZzCDWZF1hZnPdfU8SIwInztnJzG4C3glc4tG5gW0nMDrueXmwLnLMLJtYcfqNuz8adp5unA9cY2ZXAXlAkZn92t2j9sFaC9S6e2cL9BEiVqCAdwBb3b0OwMweBc4Dfh1qqhPba2Yj3X23mY0E9oUdqDvJ/gzqFy2oE3H31e4+3N0r3L2C2C/e2WEUp56Y2RXETv1c4+5Hws4TZykw0czGmVkOsQvRC0LO9AYW+wvkF8A6d/9u2Hm64+6fd/fy4GfxBuDpCBYngt+PHWY2OVh1CbA2xEjdqQHeYmb5wff+EiLWkaOLBcCNwfKNwB9DzNKtMD6D+nWBSjF3A4XAQjNbaWY/CTsQQHDR9DbgcWIfAA+5+5pwU3XrfODDwMXB/9/KoKUip+dfgd+Y2SpgFvC/Q87zOkHr7hFgBbCa2GddJIYTMrMHgBeByWZWa2YfA74JXGpmm4i1/r4ZwYxJ/wzSUEciIhJJakGJiEgkqUCJiEgkqUCJiEgkqUCJiEgkqUCJiEgkqUBJSjOzYWb2XDDixrvj1v/RzEb18hiHgq+jzOyRuPUPBCM3f9bMpgRda6vM7Iy+/5e8eWZ2i5l9JOwcIn1F3cwlpZnZ7cQGtXwUeMzdLzKzdwFz3P2rvTzGIXcv6LKuFHjO3ScEz+8Astz966eQLdPd23u7vYi8nlpQkuqOExupOhdoD0au/gzw7RPtEIx68aKZrTazr8etr4ib/+YJoCxoNX0lOOa/mNkzwbYfMrMlwes/DaYdwcwOmdl/mdnLwLlmNsfM/m5my83s8WAYG8xskZl9KzjGRjN7a7A+08z+M2gRrjKzfw3Wd3ucLv+ur5rZv5/s+F22vyg45h/NbIuZfdPMPhjss7qzpRi0Un9nZkuDx/nB+rnB/2OVmb3QObKEmd1kZo+a2d8sNr/Rt+P+bb8M/m2rzeyzvf0mSz/l7nrokbIPoBj4C7CM2HA2twM39bDPAuAjwfKtwKFguQJ4pety8PyrwL8Hy1OBPwHZwfMfxR3PgfcFy9nAC8Cw4Pn7gXuD5UXAfwXLVwFPBsv/QmwEhKzgecnJjtPl3xWfsdvjd9n+IqARGEmswO8Evha89mng+8Hy/cAFwfIYYsNFARTF5XwH8Ltg+SZgS/C9yQO2ExuvcQ6wMO79B4X986NHtB/9YrBYSV/u3kRs/qTOEazvAK4zs58Bg4l9SL/YZbfzgfcEy78CvnWKb3sJsQ/bpbFh3hjAa4N7thMbkBZgMjCD2NAwEJs0b3fccToHrF1OrCBC7IP+Jx7Mu+PuDWY2o4fjnEh3x+9qqQfTPJjZZmItR4gND/T2uEzTgveG2CC2BcQK0H1mNpFYYc6OO+5TwfcGM1sLjAXWAOPN7AfE/qiI8vQXEgEqUJJO/ifwDWA+8ByxlsijwOXdbPtmLr4acJ+7dzej6DF/7bqTAWvc/UTTobcEX9s5+e9iT8c5kd4cvyVuuSPueUfcPhnAW9z92OtCmd0NPOPu11lsjq1FJzhuO7GW1gEzm0ns+3EL8D5i8wqJdEvXoCQtBH/Fl7v7ImLXpDqIFaEB3Wz+PK9N//3B03i7p4DrzWx48N4lZja2m+02AMPM7Nxgu2wzm97DsRcCnwyupWFmJad5nL70BLHBYQnef1awWMxrU6vc1NNBzGwokOHuvwO+RGxOKZETUoGSdPEN4IvB8gPEruUsBe7qZttPA7ea2WpOY/Zfd19L7AP2CYuN5r2Q2HWcrtu1AtcD3wo6TawkNifRyfyc2FQRq4J9PnCax+lLtwOVQaeNtcRaPxDriPIfZlZF787GlAGLzGwlsXmZumuBirxK3cxFRCSS1IISEZFIUoESEZFIUoESEZFIUoESEZFIUoESEZFIUoESEZFIUoESEZFI+n84qQJzBlKBogAAAABJRU5ErkJggg==\n"},"metadata":{"needs_background":"light"}}]},{"metadata":{"dc":{"key":"74"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 11.  The conclusion\n<p>The bootstrap result tells us that there is strong evidence that 7-day retention is higher when the gate is at level 30 than when it is at level 40. The conclusion is: If we want to keep retention high — both 1-day and 7-day retention — we should <strong>not</strong> move the gate from level 30 to level 40. There are, of course, other metrics we could look at, like the number of game rounds played or how much in-game purchases are made by the two AB-groups. But retention <em>is</em> one of the most important metrics. If we don't retain our player base, it doesn't matter how much money they spend in-game.</p>\n<p><img src=\"https://assets.datacamp.com/production/project_184/img/cookie_yellow.png\" style=\"width:100px; float:center\"> </p>\n<p>So, why is retention higher when the gate is positioned earlier? One could expect the opposite: The later the obstacle, the longer people are going to engage with the game. But this is not what the data tells us. The theory of <em>hedonic adaptation</em> can give one explanation for this. In short, hedonic adaptation is the tendency for people to get less and less enjoyment out of a fun activity over time if that activity is undertaken continuously. By forcing players to take a break when they reach a gate, their enjoyment of the game is prolonged. But when the gate is moved to level 40, fewer players make it far enough, and they are more likely to quit the game because they simply got bored of it. </p>"},{"metadata":{"dc":{"key":"74"},"tags":["sample_code"],"trusted":true,"collapsed":true},"cell_type":"code","source":"# So, given the data and the bootstrap analysis\n# Should we move the gate from level 30 to level 40 ?\nmove_to_level_40 = False # True or False ?","execution_count":22,"outputs":[]}],"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.6.7","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":2}