Skip to content

Commit 87ac39c

Browse files
ehussehuss
committed
Add a chapter on the test harness.
1 parent e7c23ab commit 87ac39c

File tree

4 files changed

+288
-9
lines changed

4 files changed

+288
-9
lines changed

‎library/test/src/cli.rs

+2-8
Original file line numberDiff line numberDiff line change
@@ -51,14 +51,8 @@ fn optgroups() -> getopts::Options {
5151
.optflag("", "test", "Run tests and not benchmarks")
5252
.optflag("", "bench", "Run benchmarks instead of tests")
5353
.optflag("", "list", "List all tests and benchmarks")
54-
.optflag("h", "help", "Display this message (longer with --help)")
55-
.optopt(
56-
"",
57-
"logfile",
58-
"Write logs to the specified file instead \
59-
of stdout",
60-
"PATH",
61-
)
54+
.optflag("h", "help", "Display this message")
55+
.optopt("", "logfile", "Write logs to the specified file", "PATH")
6256
.optflag(
6357
"",
6458
"nocapture",

‎src/doc/rustc/src/SUMMARY.md

+1
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
- [Deny-by-default lints](lints/listing/deny-by-default.md)
1212
- [Codegen options](codegen-options/index.md)
1313
- [JSON Output](json.md)
14+
- [Tests](tests/index.md)
1415
- [Platform Support](platform-support.md)
1516
- [Targets](targets/index.md)
1617
- [Built-in Targets](targets/built-in.md)

‎src/doc/rustc/src/command-line-arguments.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -204,7 +204,8 @@ out a longer explanation of a given error.
204204
## `--test`: build a test harness
205205

206206
When compiling this crate, `rustc` will ignore your `main` function
207-
and instead produce a test harness.
207+
and instead produce a test harness. See the [Tests chapter](tests/index.md)
208+
for more information about tests.
208209

209210
<a id="option-target"></a>
210211
## `--target`: select a target triple to build

‎src/doc/rustc/src/tests/index.md

+283
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,283 @@
1+
# Tests
2+
3+
`rustc` has a built-in facility for building and running tests for a crate.
4+
More information about writing and running tests may be found in the [Testing
5+
Chapter] of the Rust Programming Language book.
6+
7+
Tests are written as free functions with the [`#[test]`
8+
attribute][attribute-test]. For example:
9+
10+
```rust
11+
#[test]
12+
fn it_works() {
13+
assert_eq!(2 + 2, 4);
14+
}
15+
```
16+
17+
Tests "pass" if they return without an error. They "fail" if they [panic], or
18+
return a [`Result`] with an error.
19+
20+
By passing the [`--test` option] to `rustc`, the compiler will build the crate
21+
in a special mode to construct an executable that will run the tests in the
22+
crate. The `--test` flag will make the following changes:
23+
24+
* The crate will be built as a `bin` [crate type], forcing it to be an
25+
executable.
26+
* Links the executable with [`libtest`], the test harness that is part of the
27+
standard library, which handles running the tests.
28+
* Synthesizes a [`main` function] which will process command-line arguments
29+
and run the tests. If the crate already has a `main` function, it will be
30+
replaced.
31+
* Enables the [`test` cfg option], which allows your code to use conditional
32+
compilation to detect if it is being built as a test.
33+
* Enables building of functions annotated with the [`test`][attribute-test]
34+
and [`bench`](#benchmarks) attributes, which will be run by the test
35+
harness.
36+
37+
After the executable is created, you can run it to execute the tests and
38+
receive a report on what passes and fails. If you are using [Cargo] to manage
39+
your project, it has a built-in [`cargo test`] command which handles all of
40+
this automatically. An example of the output looks like this:
41+
42+
```text
43+
running 4 tests
44+
test it_works ... ok
45+
test check_valid_args ... ok
46+
test invalid_characters ... ok
47+
test walks_the_dog ... ok
48+
49+
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
50+
```
51+
52+
> **Note**: Tests must be built with the [`unwind` panic
53+
> strategy][panic-strategy]. This is because all tests run in the same
54+
> process, and they are intended to catch panics, which is not possible with
55+
> the `abort` strategy. See the unstable [`-Z panic-abort-tests`] option for
56+
> experimental support of the `abort` strategy by spawning tests in separate
57+
> processes.
58+
59+
## Test attributes
60+
61+
Tests are indicated using attributes on free functions. The following
62+
attributes are used for testing, see the linked documentation for more
63+
details:
64+
65+
* [`#[test]`][attribute-test] — Indicates a function is a test to be run.
66+
* `#[bench]` — Indicates a function is a benchmark to be
67+
run. Benchmarks are currently unstable and only available in the nightly
68+
channel, see the [unstable docs][bench-docs] for more details.
69+
* [`#[should_panic]`][attribute-should_panic] — Indicates that the test
70+
function will only pass if the function [panics][panic].
71+
* [`#[ignore]`][attribute-ignore] — Indicates that the test function will be
72+
compiled, but not run by default. See the [`--ignored`](#--ignored) and
73+
[`--include-ignored`](#--include-ignored) options to run these tests.
74+
75+
## CLI arguments
76+
77+
The libtest harness has several command-line arguments to control its
78+
behavior.
79+
80+
> Note: When running with [`cargo test`], the libtest CLI arguments must be
81+
> passed after the `--` argument to differentiate between flags for Cargo and
82+
> those for the harness. For example: `cargo test -- --nocapture`
83+
84+
### Filters
85+
86+
Bare arguments (those without a `-` prefix) are treated as filters which will
87+
only run tests whose name matches one of those strings. The filter will match
88+
any substring found in the full path of the test function. For example, if the
89+
test function `it_works` is located in the module `utils::paths::tests`, then
90+
any of the filters `works`, `path`, `utils::`, or
91+
`utils::paths::tests::it_works` will match that test.
92+
93+
See [Selection options](#selection-options) for more options to control which
94+
tests are run.
95+
96+
### Action options
97+
98+
The following options perform different actions other than running tests.
99+
100+
#### `--list`
101+
102+
Prints a list of all tests and benchmarks. Does not run any of the tests.
103+
[Filters](#filters) can be used to list only matching tests.
104+
105+
#### `-h`, `--help`
106+
107+
Displays usage information and command-line options.
108+
109+
### Selection options
110+
111+
The following options change how tests are selected.
112+
113+
#### `--test`
114+
115+
This is the default mode where all tests will be run as well as running all
116+
benchmarks with only a single iteration (to ensure the benchmark works,
117+
without taking the time to actually perform benchmarking). This can be
118+
combined with the `--bench` flag to run both tests and perform full
119+
benchmarking.
120+
121+
#### `--bench`
122+
123+
This runs in a mode where tests are ignored, and only runs benchmarks. This
124+
can be combined with `--test` to run both benchmarks and tests.
125+
126+
#### `--exact`
127+
128+
This forces [filters](#filters) to match the full path of the test exactly.
129+
For example, if the test `it_works` is in the module `utils::paths::tests`,
130+
then only the string `utils::paths::tests::it_works` will match that test.
131+
132+
#### `--skip` _FILTER_
133+
134+
Skips any tests whose name contains the given _FILTER_ string. This flag may
135+
be passed multiple times.
136+
137+
#### `--ignored`
138+
139+
Runs only tests that are marked with the [`ignore`
140+
attribute][attribute-ignore].
141+
142+
#### `--include-ignored`
143+
144+
Runs both [ignored](#--ignored) and non-ignored tests.
145+
146+
#### `--exclude-should-panic`
147+
148+
Excludes tests marked with the [`should_panic`
149+
attribute][attribute-should_panic].
150+
151+
⚠️ 🚧 This option is [unstable](#unstable-options), and requires the `-Z
152+
unstable-options` flag. See [tracking issue
153+
#82348](https://github.com/rust-lang/rust/issues/82348) for more information.
154+
155+
### Execution options
156+
157+
The following options affect how tests are executed.
158+
159+
#### `--test-threads` _NUM_THREADS_
160+
161+
Sets the number of threads to use for running tests in parallel. By default,
162+
uses the amount of concurrency available on the hardware as indicated by
163+
[`available_concurrency`].
164+
165+
This can also be specified with the `RUST_TEST_THREADS` environment variable.
166+
167+
#### `--force-run-in-process`
168+
169+
Forces the tests to run in a single process when using the [`abort` panic
170+
strategy][panic-strategy].
171+
172+
⚠️ 🚧 This only works with the unstable [`-Z panic-abort-tests`] option, and
173+
requires the `-Z unstable-options` flag. See [tracking issue
174+
#67650](https://github.com/rust-lang/rust/issues/67650) for more information.
175+
176+
#### `--ensure-time`
177+
178+
⚠️ 🚧 This option is [unstable](#unstable-options), and requires the `-Z
179+
unstable-options` flag. See [tracking issue
180+
#64888](https://github.com/rust-lang/rust/issues/64888) and the [unstable
181+
docs](../../unstable-book/compiler-flags/report-time.html) for more information.
182+
183+
### Output options
184+
185+
The following options affect the output behavior.
186+
187+
#### `-q`, `--quiet`
188+
189+
Displays one character per test instead of one line per test. This is an alias
190+
for [`--format=terse`](#--format-format).
191+
192+
#### `--nocapture`
193+
194+
Does not capture the stdout and stderr of the test, and allows tests to print
195+
to the console. Usually the output is captured, and only displayed if the test
196+
fails.
197+
198+
This may also be specified by setting the `RUST_TEST_NOCAPTURE` environment
199+
variable set to anything but `0`.
200+
201+
#### `--show-output`
202+
203+
Displays the stdout and stderr of successful tests after all tests have run.
204+
205+
Contrast this with [`--nocapture`](#--nocapture) which allows tests to print
206+
*while they are running*, which can cause interleaved output if there are
207+
multiple tests running in parallel, `--show-output` ensures the output is
208+
contiguous, but requires waiting for all tests to finish.
209+
210+
#### `--color` _COLOR_
211+
212+
Control when colored terminal output is used. Valid options:
213+
214+
* `auto`: Colorize if stdout is a tty and [`--nocapture`](#--nocapture) is not
215+
used. This is the default.
216+
* `always`: Always colorize the output.
217+
* `never`: Never colorize the output.
218+
219+
#### `--format` _FORMAT_
220+
221+
Controls the format of the output. Valid options:
222+
223+
* `pretty`: This is the default format, with one line per test.
224+
* `terse`: Displays only a single character per test. [`--quiet`](#-q---quiet)
225+
is an alias for this option.
226+
* `json`: Emits JSON objects, one per line. ⚠️ 🚧 This option is
227+
[unstable](#unstable-options), and requires the `-Z unstable-options` flag.
228+
See [tracking issue #49359](https://github.com/rust-lang/rust/issues/49359)
229+
for more information.
230+
231+
#### `--logfile` _PATH_
232+
233+
Writes the results to the tests to the given file.
234+
235+
#### `--report-time` _FORMAT_
236+
237+
⚠️ 🚧 This option is [unstable](#unstable-options), and requires the `-Z
238+
unstable-options` flag. See [tracking issue
239+
#64888](https://github.com/rust-lang/rust/issues/64888) and the [unstable
240+
docs](../../unstable-book/compiler-flags/report-time.html) for more information.
241+
242+
### Unstable options
243+
244+
Some CLI options are added in an "unstable" state, where they are intended for
245+
experimentation and testing to determine if the option works correctly, has
246+
the right design, and is useful. The option may not work correctly, break, or
247+
change at at any time. To signal that you acknowledge that you are using an
248+
unstable option, they require passing the `-Z unstable-options` command-line
249+
flag.
250+
251+
## Benchmarks
252+
253+
The libtest harness supports running benchmarks for functions annotated with
254+
the `#[bench]` attribute. Benchmarks are currently unstable, and only
255+
available on the [nightly channel]. More information may be found in the
256+
[unstable book][bench-docs].
257+
258+
## Custom test frameworks
259+
260+
Experimental support for using custom test harnesses is available on the
261+
[nightly channel]. See [tracking issue
262+
#50297](https://github.com/rust-lang/rust/issues/50297) and the
263+
[custom_test_frameworks documentation] for more information.
264+
265+
[`--test` option]: ../command-line-arguments.md#option-test
266+
[`-Z panic-abort-tests`]: https://github.com/rust-lang/rust/issues/67650
267+
[`available_concurrency`]: ../../std/thread/fn.available_concurrency.html
268+
[`cargo test`]: ../../cargo/commands/cargo-test.html
269+
[`libtest`]: ../../test/index.html
270+
[`main` function]: ../../reference/crates-and-source-files.html#main-functions
271+
[`Result`]: ../../std/result/index.html
272+
[`test` cfg option]: ../../reference/conditional-compilation.html#test
273+
[attribute-ignore]: ../../reference/attributes/testing.html#the-ignore-attribute
274+
[attribute-should_panic]: ../../reference/attributes/testing.html#the-should_panic-attribute
275+
[attribute-test]: ../../reference/attributes/testing.html#the-test-attribute
276+
[bench-docs]: ../../unstable-book/library-features/test.html
277+
[Cargo]: ../../cargo/index.html
278+
[crate type]: ../../reference/linkage.html
279+
[custom_test_frameworks documentation]: ../../unstable-book/language-features/custom-test-frameworks.html
280+
[nightly channel]: ../../book/appendix-07-nightly-rust.html
281+
[panic-strategy]: ../../book/ch09-01-unrecoverable-errors-with-panic.html
282+
[panic]: ../../book/ch09-01-unrecoverable-errors-with-panic.html
283+
[Testing Chapter]: ../../book/ch11-00-testing.html

0 commit comments

Comments
 (0)