Skip to content

Commit 75c0acd

Browse files
trallardtrallard
committed
Add property based testing
1 parent 81ae85b commit 75c0acd

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

59 files changed

+414
-80
lines changed

‎Property-based-testing.ipynb

+372
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,372 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Property based testing and data integrity testing\n",
8+
"\n",
9+
"## Property based testing with Hypothesis\n",
10+
"\n",
11+
"So far all the tests we have written/write are done at a basic level in which you take 1 test and write test cases for which ti should pass or fail. But what about edge cases? How do you know when you have actually found a bug and not an edge case for your model/algorithm or else?\n",
12+
"\n",
13+
"The [Hypothesis package](https://hypothesis.readthedocs.org/en/latest/) is very useful in these situtations as it performs property based testing. So instead of taking let's say 1 or 2 test cases for a single function hypothesis tests a whole range of parameters for each test case. \n",
14+
"\n",
15+
"Not clear? Ok imagine we do not know about the existence of the `modullo` operator so we have instead writen a function that tests divisibility by 11 based on the following property: \n",
16+
"> A number is divisible by 11 if and only if the alternating (in sign) sum of the number’s digits is 0\n",
17+
"\n",
18+
"Let's create a `main.py` script with our function:"
19+
]
20+
},
21+
{
22+
"cell_type": "code",
23+
"execution_count": 1,
24+
"metadata": {},
25+
"outputs": [],
26+
"source": [
27+
"def divisible_by_11(number):\n",
28+
" \"\"\"Uses above criterion to check if number is divisible by 11\"\"\"\n",
29+
" string_number = str(number)\n",
30+
" alternating_sum = sum([(-1) ** i * int(d) for i, d\n",
31+
" in enumerate(string_number)])\n",
32+
" return alternating_sum == 0"
33+
]
34+
},
35+
{
36+
"cell_type": "markdown",
37+
"metadata": {},
38+
"source": [
39+
"And we can demonstrate that this works by running some examples"
40+
]
41+
},
42+
{
43+
"cell_type": "code",
44+
"execution_count": null,
45+
"metadata": {},
46+
"outputs": [],
47+
"source": [
48+
"for k in range(10):\n",
49+
" print(divisible_by_11(11 * k))"
50+
]
51+
},
52+
{
53+
"cell_type": "markdown",
54+
"metadata": {},
55+
"source": [
56+
"Now let's write a test case for the above example! And we are going to save this in `test_main.py`"
57+
]
58+
},
59+
{
60+
"cell_type": "code",
61+
"execution_count": 21,
62+
"metadata": {},
63+
"outputs": [],
64+
"source": [
65+
"import pytest\n",
66+
"\n",
67+
"\n",
68+
"def test_divisible_by_11():\n",
69+
" for k in range(10):\n",
70+
" assert divisible_by_11(11*k)\n",
71+
" assert divisible_by_11(121)\n",
72+
" assert divisible_by_11(12122)"
73+
]
74+
},
75+
{
76+
"cell_type": "code",
77+
"execution_count": 11,
78+
"metadata": {},
79+
"outputs": [],
80+
"source": [
81+
"test_divisible_by_11()"
82+
]
83+
},
84+
{
85+
"cell_type": "markdown",
86+
"metadata": {},
87+
"source": [
88+
"The tests passed so no errors were raised at all. The above tests the first 10 numbers divisible by 11 and also some specific tests (121 and 12122).\n",
89+
"\n",
90+
"\n",
91+
"At this point we could be very happy and proud of ourselves: we have tested well written software that can be shipped and used by researchers world wide to test the divisibility of a number by 11!!!\n",
92+
"\n",
93+
"![](assets/not.png)\n",
94+
"\n",
95+
"That is how mathematics breaks... so let's use hypothesis to demonstrate. Let's create a `test_property_main.py` script:"
96+
]
97+
},
98+
{
99+
"cell_type": "code",
100+
"execution_count": 22,
101+
"metadata": {},
102+
"outputs": [],
103+
"source": [
104+
"from hypothesis import given # define the inputs\n",
105+
"import hypothesis.strategies as st\n",
106+
"\n",
107+
"@given(k=st.integers(min_value=1)) # main decorator\n",
108+
"def test_divisible_by_11(k):\n",
109+
" assert divisible_by_11(11*k)"
110+
]
111+
},
112+
{
113+
"cell_type": "markdown",
114+
"metadata": {},
115+
"source": [
116+
"And we can run the test:\n",
117+
"```\n",
118+
"$ python -m pytest test_main.py\n",
119+
"============================= test session starts ================\n",
120+
"=============\n",
121+
"platform win32 -- Python 3.6.5, pytest-3.5.1, py-1.5.3, pluggy-0.6\n",
122+
".0\n",
123+
"plugins: nbval-0.9.0, hypothesis-3.58.1\n",
124+
"collecting 0 items\n",
125+
"collecting 1 item\n",
126+
"collected 1 item\n",
127+
"\n",
128+
"\n",
129+
"test_property_main.py F\n",
130+
" [100%]\n",
131+
"\n",
132+
"================================== FAILURES ======================\n",
133+
"=============\n",
134+
"____________________________ test_divisible_by_11 ________________\n",
135+
"_____________\n",
136+
"\n",
137+
" @given(k=st.integers(min_value=1)) # main decorator\n",
138+
"> def test_divisible_by_11(k):\n",
139+
"\n",
140+
"test_main.py:8:\n",
141+
"_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\n",
142+
"_ _ _ _ _ _ _\n",
143+
"C:\\Users\\Tania\\Anaconda3\\envs\\reproPython\\lib\\site-packages\\hypoth\n",
144+
"esis\\core.py:584: in execute\n",
145+
" result = self.test_runner(data, run)\n",
146+
"C:\\Users\\Tania\\Anaconda3\\envs\\reproPython\\lib\\site-packages\\hypoth\n",
147+
"esis\\executors.py:58: in default_new_style_executor\n",
148+
" return function(data)\n",
149+
"C:\\Users\\Tania\\Anaconda3\\envs\\reproPython\\lib\\site-packages\\hypoth\n",
150+
"esis\\core.py:576: in run\n",
151+
" return test(*args, **kwargs)\n",
152+
"test_main.py:8: in test_divisible_by_11\n",
153+
" def test_divisible_by_11(k):\n",
154+
"C:\\Users\\Tania\\Anaconda3\\envs\\reproPython\\lib\\site-packages\\hypoth\n",
155+
"esis\\core.py:524: in test\n",
156+
" result = self.test(*args, **kwargs)\n",
157+
"_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\n",
158+
"_ _ _ _ _ _ _\n",
159+
"\n",
160+
"k = 28\n",
161+
"\n",
162+
" @given(k=st.integers(min_value=1)) # main decorator\n",
163+
" def test_divisible_by_11(k):\n",
164+
"> assert main.divisible_by_11(11*k)\n",
165+
"E assert False\n",
166+
"E + where False = <function divisible_by_11 at 0x0000014FE\n",
167+
"5CA4D90>((11 * 28))\n",
168+
"E + where <function divisible_by_11 at 0x0000014FE5CA4D9\n",
169+
"0> = main.divisible_by_11\n",
170+
"\n",
171+
"test_main.py:9: AssertionError\n",
172+
"--------------------------------- Hypothesis ---------------------\n",
173+
"-------------\n",
174+
"Falsifying example: test_divisible_by_11(k=28)\n",
175+
"========================== 1 failed in 0.60 seconds ==============\n",
176+
"=============\n",
177+
"\n",
178+
"\n",
179+
"```\n",
180+
"We get an error! An right at the top we get the `Falsifying example` so we see that our function fails for `k=28`. The number resultign from $11x28$ is not divisible by 11 by construction. \n",
181+
"\n",
182+
"At this point we can think of the following correct definition for a number to be divisuble by 11:\n",
183+
"\n",
184+
"> A number is divisible by 11 if and only if the alternating (in sign) sum of the number’s digits is divisible by 11.\n",
185+
"\n",
186+
"So we can now modify `main.py` :"
187+
]
188+
},
189+
{
190+
"cell_type": "code",
191+
"execution_count": 23,
192+
"metadata": {},
193+
"outputs": [],
194+
"source": [
195+
"def divisible_by_11(number):\n",
196+
" \"\"\"Uses above criterion to check if number is divisible by 11\"\"\"\n",
197+
" string_number = str(number)\n",
198+
" # Using abs as the order of the alternating sum doesn't matter.\n",
199+
" alternating_sum = abs(sum([(-1) ** i * int(d) for i, d\n",
200+
" in enumerate(string_number)]))\n",
201+
" # Recursively calling the function\n",
202+
" return (alternating_sum in [0, 11]) or divisible_by_11(alternating_sum)"
203+
]
204+
},
205+
{
206+
"cell_type": "markdown",
207+
"metadata": {},
208+
"source": [
209+
"Running the tests:\n",
210+
"\n",
211+
"```\n",
212+
"$ python -m pytest test_property_main\n",
213+
".\n",
214+
"----------------------------------------------------------------------\n",
215+
"Ran 1 test in 0.043s\n",
216+
"\n",
217+
"OK\n",
218+
"```"
219+
]
220+
},
221+
{
222+
"cell_type": "code",
223+
"execution_count": 4,
224+
"metadata": {},
225+
"outputs": [
226+
{
227+
"data": {
228+
"text/html": [
229+
"<link href=\"https://fonts.googleapis.com/css?family=Didact+Gothic|Dosis:400,500,700\" rel=\"stylesheet\"><style>\n",
230+
"@font-face {\n",
231+
" font-family: \"Computer Modern\";\n",
232+
" src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');\n",
233+
"}\n",
234+
"/* div.cell{\n",
235+
"width:800px;\n",
236+
"margin-left:16% !important;\n",
237+
"margin-right:auto;\n",
238+
"} */\n",
239+
"h1 {\n",
240+
" font-family: 'Dosis', \"Helvetica Neue\", Arial, sans-serif;\n",
241+
" color: #0B132B;\n",
242+
"}\n",
243+
"h2 {\n",
244+
" font-family: 'Dosis', sans-serif;\n",
245+
" color: #1C2541;\n",
246+
"}\n",
247+
"h3{\n",
248+
" font-family: 'Dosis', sans-serif;\n",
249+
" margin-top:12px;\n",
250+
" margin-bottom: 3px;\n",
251+
" color: #40a8a6;\n",
252+
"}\n",
253+
"h4{\n",
254+
" font-family: 'Dosis', sans-serif;\n",
255+
" color: #40a8a6;\n",
256+
"}\n",
257+
"h5 {\n",
258+
" font-family: 'Dosis', sans-serif;\n",
259+
" color: #40a8a6;\n",
260+
"}\n",
261+
"div.text_cell_render{\n",
262+
" font-family: 'Didact Gothic',Computer Modern, \"Helvetica Neue\", Arial, Helvetica,\n",
263+
" Geneva, sans-serif;\n",
264+
" line-height: 130%;\n",
265+
" font-size: 110%;\n",
266+
" /* width:600px; */\n",
267+
" /* margin-left:auto;\n",
268+
" margin-right:auto; */\n",
269+
"}\n",
270+
"\n",
271+
".text_cell_render h1 {\n",
272+
" font-weight: 200;\n",
273+
" font-size: 30pt;\n",
274+
" /* font-size: 50pt */\n",
275+
" line-height: 100%;\n",
276+
" color:#0B132B;\n",
277+
" margin-bottom: 0.5em;\n",
278+
" margin-top: 0.5em;\n",
279+
" display: block;\n",
280+
"}\n",
281+
"\n",
282+
".text_cell_render h2{\n",
283+
" font-weight: 500;\n",
284+
"}\n",
285+
"\n",
286+
".text_cell_render h3{\n",
287+
" font-weight: 500;\n",
288+
"}\n",
289+
"\n",
290+
"\n",
291+
".warning{\n",
292+
" color: rgb( 240, 20, 20 )\n",
293+
"}\n",
294+
"\n",
295+
"div.warn {\n",
296+
" background-color: #FF5A5F;\n",
297+
" border-color: #FF5A5F;\n",
298+
" border-left: 5px solid #C81D25;\n",
299+
" padding: 0.5em;\n",
300+
"\n",
301+
" color: #fff;\n",
302+
" opacity: 0.8;\n",
303+
"}\n",
304+
"\n",
305+
"div.info {\n",
306+
" background-color: #087E8B;\n",
307+
" border-color: #087E8B;\n",
308+
" border-left: 5px solid #0B3954;\n",
309+
" padding: 0.5em;\n",
310+
" color: #fff;\n",
311+
" opacity: 0.8;\n",
312+
"}\n",
313+
"\n",
314+
"</style>\n",
315+
"<script>\n",
316+
"MathJax.Hub.Config({\n",
317+
" TeX: {\n",
318+
" extensions: [\"AMSmath.js\"]\n",
319+
" },\n",
320+
" tex2jax: {\n",
321+
" inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
322+
" displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
323+
" },\n",
324+
" displayAlign: 'center', // Change this to 'center' to center equations.\n",
325+
" \"HTML-CSS\": {\n",
326+
" styles: {'.MathJax_Display': {\"margin\": 4}}\n",
327+
" }\n",
328+
" });\n",
329+
" </script>\n"
330+
],
331+
"text/plain": [
332+
"<IPython.core.display.HTML object>"
333+
]
334+
},
335+
"execution_count": 4,
336+
"metadata": {},
337+
"output_type": "execute_result"
338+
}
339+
],
340+
"source": [
341+
"from IPython.core.display import HTML\n",
342+
"\n",
343+
"\n",
344+
"def css_styling():\n",
345+
" styles = open(\"styles/custom.css\", \"r\").read()\n",
346+
" return HTML(styles)\n",
347+
"css_styling()"
348+
]
349+
}
350+
],
351+
"metadata": {
352+
"kernelspec": {
353+
"display_name": "Python [default]",
354+
"language": "python",
355+
"name": "python3"
356+
},
357+
"language_info": {
358+
"codemirror_mode": {
359+
"name": "ipython",
360+
"version": 3
361+
},
362+
"file_extension": ".py",
363+
"mimetype": "text/x-python",
364+
"name": "python",
365+
"nbconvert_exporter": "python",
366+
"pygments_lexer": "ipython3",
367+
"version": "3.6.5"
368+
}
369+
},
370+
"nbformat": 4,
371+
"nbformat_minor": 2
372+
}

0 commit comments

Comments
 (0)