Skip to content

Commit 3dfd1cb

Browse files
OliverKillaneOliverKillane
committed
70024: Added fuzzing lectures
1 parent 78ea9ca commit 3dfd1cb

14 files changed

+162
-0
lines changed

‎70024 - Software Reliability/American Fuzzy Lop (AFL).md

+38
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
## Definition
2+
A [[Mutation-Based Fuzzer|mutation-based]], [[Dumb Fuzzing|dumb]], [[Grey-Box Fuzzing|grey-box]] fuzzer.
3+
- General purpose and does not know input format
4+
```python
5+
files = get_user_provided_files()
6+
prev_behaviour = {}
7+
while keep_fuzzing():
8+
next = files.pop()
9+
10+
# AFL measures branch coverage
11+
behaviour = fuzz_with(next)
12+
prev_benhaviour.add(behaviour)
13+
14+
# trim the test case to the smallest size with the
15+
# same behaviour as the original
16+
fuzz_and_trim(behaviour, next)
17+
18+
# Mutate the file using a variety of traditional methods
19+
new_files = mutate(next)
20+
21+
# If any have different behaviour, add to queue
22+
for mutated in new_files:
23+
if fuzz_with(mutated) not in prev_behaviour:
24+
files.append(mutated)
25+
```
26+
27+
The mutation strategies used are:
28+
29+
| Strategy | Description |
30+
| ---- | ---- |
31+
| Walking Bit-Flips | Walk input with 1-bit stride, flipping 1-4 consecutive bits |
32+
| Walking byte-flips | walk input with 1-byte stride, flipping 8, 16 or 32 consecutive bits |
33+
| Increment/Decrement | Changed integer values in the input file (by default in range -34 to 35) |
34+
| Insert known integers | -1, 256, 1024, MAX_INT-1 |
35+
| Miscellaneous Random Tweaks | deleting or mem-setting parts of the input file |
36+
| Splicing | Concatenate parts of existing input |
37+
## [Github Repository](https://github.com/google/AFL)
38+
## [Creator's Website](https://lcamtuf.coredump.cx/afl/)

‎70024 - Software Reliability/Black-Box Fuzzing.md

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
## Definition
2+
The [[SUT]] is executed in an unmodified form (no sanitizers or coverage added).
3+
### Advantages
4+
- Can be applied to closed source [[SUT]]
5+
- [[SUT]] can run at full speed (optimized binary), enabling a higher rate of fuzzing
6+
### Disadvantages
7+
- Feedback directed fuzzing can only make use of externally-visible [[SUT]] behaviour (cannot augment [[SUT]] with coverage)

‎70024 - Software Reliability/Dumb Fuzzing.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
## Definition
2+
[[Fuzzing]] that generates or mutates input without knowledge of the [[SUT]]'s input domain.
3+
## Examples
4+
- Generating random strings for a compiler, without knowledge of the grammar or semantics.
5+
- Sending random bytes to a server to fuzz a protocol

‎70024 - Software Reliability/Feedback-Directed Fuzzing.md

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
## Definition
2+
A [[Fuzzing|fuzzer]]'s input generator can observe the behaviour of the [[SUT]] to inform new inputs.
3+
- Currently the state of the art for fuzzing.
4+
5+
*Interesting inputs can*
6+
- Log a previously unseen message
7+
- Cover some previously uncovered code
8+
- Consume more memory than usual
9+
- Do some previously unseen IO (e.g. system call)
10+
11+
| Fuzzing | To Take Advantage |
12+
| ---- | ---- |
13+
| [[Generation-Based Fuzzer]] | Attempt to generate inputs with similar properties. |
14+
| [[Mutation-Based Fuzzer]] | Prioritise the used templates as candidates for the next inputs. |

‎70024 - Software Reliability/Fuzzing.md

+43
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
## Definition
2+
Testing a system ([[SUT]]) using input that are wholly or partially randomly-generated.
3+
4+
The goal is to get the interesting behaviour to aide in finding bugs.
5+
- *interesting* is application dependent, most often it means crashing the system.
6+
### Fuzzer Types
7+
- [[Generation-Based Fuzzer]]s and [[Mutation-Based Fuzzer]]s as well as combinations of both.
8+
- [[Dumb Fuzzing]] versus [[Smart Fuzzing]]
9+
- [[Feedback-Directed Fuzzing]]
10+
- [[Black-Box Fuzzing]] vs [[Grey-Box Fuzzing]] vs [[White-Box Fuzzing]]
11+
## Input Types
12+
| Input | For Compiler Fuzzing |
13+
| ---- | ---- |
14+
| Totally Invalid | Random invalid strings to check invalid inputs are not acceptable. |
15+
| Malformed Inputs | Sequences of tokens structurally correct, but invalid. |
16+
| Inputs with high validity | Token sequences that are in the language's grammar, but are ensured to be not semantically valid |
17+
| High Integrity | Well formed programs free from undefined behaviour. |
18+
Whenever using random inputs, it is important to consider their distribution (e.g. generating numbers, what should be distribution be like to get maximal coverage?). One way to improve this is with [[Swarm Testing]].
19+
### Minimal Requirements
20+
| Component | Description |
21+
| ---- | ---- |
22+
| [[SUT]] | The system to provide input to |
23+
| [[Oracle]] | To determine which behaviours are *interesting* |
24+
### Advantages
25+
- Effective in finding edge-case inputs missed by human written test suites
26+
- Can automatically increase coverage of a codebase
27+
*Note: Can be used to find exploitable defects in programs. We consider it an advantage for the developer to find them first!*
28+
## Early Days
29+
> *_We didn't call it fuzzing back in the 1950s, but it was our standard practice to test programs by inputting decks of punch cards taken from the trash. We also used decks of random number punch cards. We weren't networked in those days, so we weren't much worried about security, but our random/trash decks often turned up undesirable behavior. Every programmer I knew (and there weren't many of us back then, so I knew a great proportion of them) used the trash-deck technique._*
30+
> **- [Gerald Weinberg's Secrets of Writing and Consulting: Fuzz Testing and Fuzz History (secretsofconsulting.blogspot.com)](http://secretsofconsulting.blogspot.com/2017/02/fuzz-testing-and-fuzz-history.html)
31+
32+
### Introduction of *Fuzzing*
33+
In 1990 the term *fuzzing*was introduced by a paper testing linux utilities with random data.
34+
- The paper: [An empirical study of the reliability of UNIX utilities | Communications of the ACM](https://dl.acm.org/doi/10.1145/96267.96279)
35+
- Tested if random inputs could cause the utilities to crash.
36+
- This was a [[Generation-Based Fuzzer|generating]] [[Dumb Fuzzing|dumb]] fuzzer
37+
-
38+
39+
40+
## Example
41+
### GLFuzz
42+
Generates OpenGL shader programs by mutation (transforms existing programs), but also can insert some randomly generated code fragments (generation).
43+
### [[American Fuzzy Lop (AFL)]]

‎70024 - Software Reliability/Generation-Based Fuzzer.md

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
## Definition
2+
A [[fuzzer|Fuzzing]] that produces inputs from nothing (purely random).

‎70024 - Software Reliability/Grammar-Based Fuzzing.md

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
## Definition
2+
Using a grammar for a text input to generate valid random inputs.
3+
4+
Taking some grammar:
5+
```text
6+
Expr ::= Number | Expr Expr Op
7+
Op ::= '+' | '-' | '*' | '/'
8+
Number ::= <32-bit signed integer>
9+
```
10+
Then randomly traversing the grammar from a start symbol:
11+
- Randomly pick the non-terminal symbol and jump to its production rule.
12+
- We need to consider the distribution of outputs (e.g. with integers, to bias towards edge values? To bias towards complexity rather than simple/shallow trees?) (See [[Swarm Testing]])

‎70024 - Software Reliability/Grey-Box Fuzzing.md

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
## Definition
2+
Instrumentation is applied to the [[SUT]] to provide information about internals (e.g. coverage, [[Compiler Sanitizers]] for behaviour).
3+
- Can be done at compile time, or binaries with debug & relocation information can be instrumented
4+
### Advantages
5+
- More feedback for [[Feedback-Directed Fuzzing]]
6+
### Disadvantages
7+
- Instrumentation decreases binary performance
8+
- Compile-time instrumentation requires the [[SUT]] source

‎70024 - Software Reliability/Mutation-Based Fuzzer.md

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
## Definition
2+
3+
A [[fuzzer|Fuzzing]] that produces inputs by modifying or combining existing inputs.

‎70024 - Software Reliability/Oracle.md

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
## Definition
2+
Used to determine which behaviours are *interesting*/flaggable when [[Fuzzing]] a system.
3+
4+
For example:
5+
- Does the [[SUT]] crash
6+
- Is new code covered/used by the input that was not previously
7+
- Does a dynamic analysis (e.g. a [[Compiler Sanitizers]]) report an issue
8+
- Does an assertion fail (often causing a system crash)
9+
- Does the [[SUT]] behave correctly / output is valid?

‎70024 - Software Reliability/SUT.md

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
## Definition
2+
Term for *System Under Test*

‎70024 - Software Reliability/Smart Fuzzing.md

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
## Definition
2+
[[Fuzzing]] that uses information about the expected input format.
3+
- One type is [[Grammar-Based Fuzzing]]
4+
## Examples
5+
- Grammar based fuzzing (syntactically correct inputs to fuzz a semantic analyser, or to fuzz a parser for invalid rejection)
6+
- Protocol fuzzing
7+
- Model-based fuzzing

‎70024 - Software Reliability/Swarm Testing.md

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
## Definition
2+
Instead of running a single [[Fuzzing]] configuration for time budget $T$, run $n$ configurations for $\cfrac{T}{n}$.
3+
Each configuration omits some features, and failure based on configuration can provide more details as to the bug.
4+
## [Paper: A Targeted Fuzzing Technique Based on Neural Networks and Particle Swarm Optimization](https://link.springer.com/chapter/10.1007/978-3-030-50399-4_36)

‎70024 - Software Reliability/White-Box Fuzzing.md

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
## Definition
2+
[[Program Analysis]] using constraint solving to generate inputs that provable cover different parts of the [[SUT]].
3+
- Called [[Symbolic Execution]], and is only loosely definable as [[Fuzzing]]
4+
### Advantages
5+
Constraint solving ensures a minimal set of inputs (fast to run, slow to generate) that can perfectly cover entire [[SUT]], including inputs that are extremely unlikely to occur under random testing.
6+
### Disadvantages
7+
- Limited scalability due to solving overhead
8+
- Requires specialized toolchain

0 commit comments

Comments
 (0)