|
| 1 | +## Definition |
| 2 | +Criteria used to measure the progress of a testing/fuzzing campaign. |
| 3 | +## Black Box Coverage |
| 4 | +Based on a specification (e.g. behaviour of a public API) without knowledge of the programs internal structure. |
| 5 | +- Works when code is unavailable (black box) |
| 6 | +- Can uncover unimplemented parts of the specification (e.g. postconditions on public library function incorrect) |
| 7 | +## White Box Coverage |
| 8 | +Based on a the code / internal knowledge (control & data flow, logic). |
| 9 | +```C |
| 10 | +int foo(int x, int y) { |
| 11 | + if (x > 0) |
| 12 | + printf("A"); |
| 13 | + else |
| 14 | + printf("B"); |
| 15 | + if (y < 10 || y > 20) |
| 16 | + printf("C") |
| 17 | + else |
| 18 | + printf("D") |
| 19 | +} |
| 20 | +``` |
| 21 | +
|
| 22 | +| Coverage Measure | Description | 100% Coverage | Tests for Example 100% Coverage | |
| 23 | +| ------------------------------ | ----------------------------------------------------------------------------------------- | ------------------------------- | ------------------------------------------------------------------ | |
| 24 | +| Function | How many/which functions are called | No unreachable functions | Any test (1) | |
| 25 | +| Instruction / Statement / Line | Instructions hit during execution | No unreachable code | `(10,0)` and `(-10, 15)` to get `A,C` and `B, D` | |
| 26 | +| Branch / Decision | How many branches covered (with each side of the conditional) | No dead branches | *Same as instruction coverage* | |
| 27 | +| Path Coverage | How many possible paths are covered (can be bounded on loops, e.g. 0, 1 or >1 iterations) | verification (full, or bounded) | `(10,0)`, `(-10, 15)`, `(10, 15)`, `(-10, 0)` (all possible paths) | |
| 28 | +*Note: Cost increasing in descending order* |
| 29 | +### Condition Coverage |
| 30 | +A condition is an atomic clause in a logical predicate. Condition coverage is the number of condition outcomes exercised. |
| 31 | +```C |
| 32 | +if ( (x > y) || ( (x < 0) && (y > 0) ) ) // x > y, x < 0, y > 0 are atoms |
| 33 | +
|
| 34 | +// Given the tests |
| 35 | +{.x=1, .y=0} |
| 36 | +{.x=0, .y=1} |
| 37 | +``` |
| 38 | + |
| 39 | +| Short Circuiting | Conditions | Coverage | |
| 40 | +| ---------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------ | |
| 41 | +| On | $\top \lor (\bot \land \bot)$, $\bot \lor(\bot \land \top)$ | $\cfrac{5}{6}$ as $x>y$ and $y>0$ both get $\top/\bot$ and $x < 0$ gets $\bot$ only. | |
| 42 | +| Off | $\top \lor (\_ \land \_ )$, $\bot \lor (\bot \land \_)$ | $\cfrac{3}{6}$ as $x>y$ gets \bot/\top$ and $x < 0$ gets $\bot$ | |
| 43 | +### Multiple Condition Coverage |
| 44 | +Take all possible combinations (in general $\approx 2^N$ for $N$ conditions). |
| 45 | + |
| 46 | +| Short Circuiting | Conditions | Coverage | |
| 47 | +| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | |
| 48 | +| On | $\top \lor (\_ \land \_)$, $\bot \lor (\bot \land \_)$, $\bot \lor (\top \land \top)$, $\bot \lor (\top \land \bot)$ | $\cfrac{2}{4}$ | |
| 49 | +| Off | $\bot \lor (\bot \land \bot)$, $\bot \lor (\bot \land \top)$, $\bot \lor (\top \land \bot)$, $\bot \lor (\top \land \top)$,$\top \lor (\bot \land \bot)$, $\top \lor (\bot \land \top)$, $\top \lor (\top \land \bot)$, $\top \lor (\top \land \top)$ | $\cfrac{2}{8}$ | |
| 50 | +### Modified Condition/Decision Coverage |
| 51 | +Decision coverage $+$ Condition Coverage $+$ Must show each atomic condition independently influences the branch outcome. |
| 52 | +$$\text{MC/DC Coverage} = \text{Percentage of conditions that meet MCDC criteria}$$ |
| 53 | +In general $n+1$ tests are required for $n$ atoms, to pairs of test cases where only one atom changes, and the result of the condition changes. |
| 54 | + |
| 55 | +| Test | $x$ | $y$ | $x>y$ | $x < 0$ | $y > 0$ | Branch | |
| 56 | +| ---- | --- | --- | ------ | ------- | ------- | ------ | |
| 57 | +| T1 | 1 | 1 | $\bot$ | $\bot$ | $\top$ | $\bot$ | |
| 58 | +| T2 | 2 | 1 | $\top$ | $\bot$ | $\top$ | $\top$ | |
| 59 | +| T3 | -1 | 1 | $\bot$ | $\top$ | $\top$ | $\top$ | |
| 60 | +| T4 | -1 | 0 | $\bot$ | $\top$ | $\bot$ | $\bot$ | |
| 61 | +The test cases demonstrate that each condition does independently affect the branch outcome. |
| 62 | +- T1 $\to$ T2 shows $x > y$ changes branch outcome |
| 63 | +- T3 $\to$ T4 shows $y > 0$ changes branch outcome |
| 64 | +- T1 $\to$ T3 shows $x < 0$ changes the branch outcome |
| 65 | +### Data Flow Coverage |
| 66 | +```C |
| 67 | +int foo(int x /* def(x, 1) */) { |
| 68 | + while (X > 0) { // use(x, 1) |
| 69 | + x = x - 1; // def(x, 2), use(x, 2) |
| 70 | + } |
| 71 | +} |
| 72 | +``` |
| 73 | +We have the pairs: |
| 74 | +$$\begin{split} |
| 75 | +P1 : & \ def_1(x) \to use_1(x) \\ |
| 76 | +P2 : & \ def_1(x) \to use_2(x) \\ |
| 77 | +P3: & \ def_2(x) \to use_1(x) \\ |
| 78 | +P4: & \ def_2(x) \to use_2(x) \\ |
| 79 | +\end{split}$$ |
| 80 | +When $x \geq 2$ we get all data flows, when $x = 1$ we only get $P1, P2, P3$, and when $x = 0$ we only get $P1$. |
| 81 | +### Mutation-Based Coverage |
| 82 | +We use a [[Mutation-Based Fuzzer]] and apply small transformations to the program. |
| 83 | +- Aim to generate a test that has different behaviour between the original and mutated programs |
| 84 | +- Can apply many mutations, on top of mutations (Higher Order Mutants) |
| 85 | +Mutants are *killed* when a test results in differing behaviour to the original. |
| 86 | +
|
| 87 | +| Type | Description | |
| 88 | +| --------------- | -------------------------------------------------------------------------------------------------- | |
| 89 | +| Strongly Killed | Different program outputs | |
| 90 | +| Weakly Killed | Different program states (an internal component computes a different value, but outputs unchanged) | |
| 91 | +$$\text{Mutation Coverage} = \cfrac{\text{Killed Mutants}}{\text{Total Mutants}}$$ |
| 92 | +Mutants not killed are *surviving mutants*. |
| 93 | +
|
| 94 | +Main issues |
| 95 | +- *Equivalent Mutants* (Syntactically different, semantically identical) to the original, can never be killed. Determining if it is identical is undecidable in the general case. |
| 96 | +- *Indistinguishable Mutants* (equivalent mutants, but to another mutant rather than the original), as many can be killed, but are actually identical, this can inflate the mutation score above what it would be if *Indistinguishable Mutants* could be eliminated. |
0 commit comments