From d1fd1d3a6d794df04b885e6c33eee708011d2292 Mon Sep 17 00:00:00 2001 From: Zalathar Date: Wed, 8 Nov 2023 17:59:27 +1100 Subject: [PATCH] Update documentation for coverage tests --- src/llvm-coverage-instrumentation.md | 24 +++++++------- src/tests/compiletest.md | 47 +++++++++++++++++++--------- 2 files changed, 46 insertions(+), 25 deletions(-) diff --git a/src/llvm-coverage-instrumentation.md b/src/llvm-coverage-instrumentation.md index fd215e3e9..a21254946 100644 --- a/src/llvm-coverage-instrumentation.md +++ b/src/llvm-coverage-instrumentation.md @@ -273,20 +273,22 @@ since it will not be called), and adds a new `FunctionCoverage`, with ## Testing LLVM Coverage -[(See also the compiletest documentation for the `tests/coverage-map` and -`tests/run-coverage` test suites.)](./tests/compiletest.md#coverage-tests) +[(See also the compiletest documentation for the `tests/coverage` +test suite.)](./tests/compiletest.md#coverage-tests) Coverage instrumentation in the MIR is validated by a `mir-opt` test: [`tests/mir-opt/instrument_coverage.rs`]. -Coverage instrumentation in LLVM IR is validated by the [`tests/coverage-map`] -test suite. These tests compile a test program to LLVM IR assembly, and then +Coverage instrumentation in LLVM IR is validated by the [`tests/coverage`] +test suite in `coverage-map` mode. +These tests compile a test program to LLVM IR assembly, and then use the [`src/tools/coverage-dump`] tool to extract and pretty-print the coverage mappings that would be embedded in the final binary. End-to-end testing of coverage instrumentation and coverage reporting is -performed by the [`tests/run-coverage`] and [`tests/run-coverage-rustdoc`] -test suites. These tests compile and run a test program with coverage +performed by the [`tests/coverage`] test suite in `coverage-run` mode, +and by the [`tests/coverage-run-rustdoc`] test suite. +These tests compile and run a test program with coverage instrumentation, then use LLVM tools to convert the coverage data into a human-readable coverage report. @@ -296,19 +298,19 @@ expected LLVM IR instructions and structured data for a coverage-enabled program, including various checks for Coverage Map-related metadata and the LLVM intrinsic calls to increment the runtime counters. -Expected results for the `coverage-map`, `run-coverage`, `run-coverage-rustdoc`, +Expected results for the `coverage`, `coverage-run-rustdoc`, and `mir-opt` tests can be refreshed by running: ```shell -./x test tests/*coverage* --bless +./x test coverage --bless +./x test coverage-run-rustdoc --bless ./x test tests/mir-opt --bless ``` [`tests/mir-opt/instrument_coverage.rs`]: https://github.com/rust-lang/rust/blob/master/tests/mir-opt/instrument_coverage.rs -[`tests/coverage-map`]: https://github.com/rust-lang/rust/tree/master/tests/coverage-map +[`tests/coverage`]: https://github.com/rust-lang/rust/tree/master/tests/coverage [`src/tools/coverage-dump`]: https://github.com/rust-lang/rust/tree/master/src/tools/coverage-dump -[`tests/run-coverage`]: https://github.com/rust-lang/rust/tree/master/tests/run-coverage -[`tests/run-coverage-rustdoc`]: https://github.com/rust-lang/rust/tree/master/tests/run-coverage-rustdoc +[`tests/coverage-run-rustdoc`]: https://github.com/rust-lang/rust/tree/master/tests/coverage-run-rustdoc [`coverage-llvmir`]: https://github.com/rust-lang/rust/tree/master/tests/run-make/coverage-llvmir ## Implementation Details of the `InstrumentCoverage` MIR Pass diff --git a/src/tests/compiletest.md b/src/tests/compiletest.md index 5cacb3013..7b6ed435d 100644 --- a/src/tests/compiletest.md +++ b/src/tests/compiletest.md @@ -67,11 +67,8 @@ The following test suites are available, with links for more information: - `run-make-fulldeps` — `run-make` tests which require a linkable build of `rustc`, or the rust demangler - [`run-pass-valgrind`](#valgrind-tests) — tests run with Valgrind -- [`coverage-map`](#coverage-tests) - tests for coverage maps produced by - coverage instrumentation -- [`run-coverage`](#coverage-tests) - tests that run an instrumented program - and check its coverage report -- [`run-coverage-rustdoc`](#coverage-tests) - coverage tests that also run +- [`coverage`](#coverage-tests) - tests for coverage instrumentation +- [`coverage-run-rustdoc`](#coverage-tests) - coverage tests that also run instrumented doctests - [Rustdoc tests](../rustdoc.md#tests): - `rustdoc` — tests for rustdoc, making sure that the generated files @@ -402,7 +399,28 @@ These may be removed in the future. ### Coverage tests -The tests in [`tests/coverage-map`] test the mappings between source code +The tests in [`tests/coverage`] are shared by multiple test modes that test +coverage instrumentation in different ways. +Running the `coverage` test suite will automatically run each test in all of +the different coverage modes. + +Each mode also has an alias to run the coverage tests in just that mode: + +```bash +./x test coverage # runs all of tests/coverage in all coverage modes +./x test tests/coverage # same as above + +./x test tests/coverage/if.rs # runs the specified test in all coverage modes + +./x test coverage-map # runs all of tests/coverage in "coverage-map" mode only +./x test coverage-run # runs all of tests/coverage in "coverage-run" mode only + +./x test coverage-map -- tests/coverage/if.rs # runs the specified test in "coverage-map" mode only +``` + +--- + +In `coverage-map` mode, these tests verify the mappings between source code regions and coverage counters that are emitted by LLVM. They compile the test with `--emit=llvm-ir`, then use a custom tool ([`src/tools/coverage-dump`]) @@ -416,18 +434,18 @@ coverage reports. As a rule of thumb, any PR that doesn't change coverage-specific code should **feel free to re-bless** the `coverage-map` tests as necessary, -without worrying about the actual changes, as long as the `run-coverage` tests +without worrying about the actual changes, as long as the `coverage-run` tests still pass. --- -The tests in [`tests/run-coverage`] perform an end-to-end test of coverage reporting. +In `coverage-run` mode, these tests perform an end-to-end test of coverage reporting. They compile a test program with coverage instrumentation, run that program to produce raw coverage data, and then use LLVM tools to process that data into a human-readable code coverage report. Instrumented binaries need to be linked against the LLVM profiler runtime, -so `run-coverage` tests are **automatically skipped** +so `coverage-run` tests are **automatically skipped** unless the profiler runtime is enabled in `config.toml`: ```toml @@ -439,14 +457,15 @@ profiler = true This also means that they typically don't run in PR CI jobs, though they do run as part of the full set of CI jobs used for merging. -The tests in [`tests/run-coverage-rustdoc`] also run instrumented doctests and +--- + +The tests in [`tests/coverage-run-rustdoc`] also run instrumented doctests and include them in the coverage report. This avoids having to build rustdoc when -only running the main `run-coverage` suite. +only running the main `coverage` suite. -[`tests/coverage-map`]: https://github.com/rust-lang/rust/tree/master/tests/coverage-map +[`tests/coverage`]: https://github.com/rust-lang/rust/tree/master/tests/coverage [`src/tools/coverage-dump`]: https://github.com/rust-lang/rust/tree/master/src/tools/coverage-dump -[`tests/run-coverage`]: https://github.com/rust-lang/rust/tree/master/tests/run-coverage -[`tests/run-coverage-rustdoc`]: https://github.com/rust-lang/rust/tree/master/tests/run-coverage-rustdoc +[`tests/coverage-run-rustdoc`]: https://github.com/rust-lang/rust/tree/master/tests/coverage-run-rustdoc ## Building auxiliary crates