Skip to content

Commit 91a7174

Browse files
BruceForstallBruceForstall
authored
Update SuperPMI readme (dotnet#723)
* Update SuperPMI readme Convert it to Markdown and increase the amount of detail describing the collection process. * Fix relative link * Update links, add statement about publishing resultant files. * More updates
1 parent dd60d56 commit 91a7174

File tree

3 files changed

+300
-101
lines changed

3 files changed

+300
-101
lines changed

‎src/coreclr/scripts/superpmi.md

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,19 @@
1-
### An overview of using superpmi.py
2-
-------------------------
1+
# An overview of using superpmi.py
32

4-
General information on [SuperPMI](https://github.com/dotnet/runtime/blob/master/src/coreclr/ToolBox/superpmi/readme.txt)
3+
General information on [SuperPMI](../src/ToolBox/superpmi/readme.md)
54

6-
------------------------
5+
## Overview
76

8-
#### Overview
9-
10-
Although SuperPMI has many uses, setup and use of SuperPMI is not always trivial. superpmi.py is a tool to help automate the use of SuperPMI, augmenting its usefulness. The tool has three different modes, collect, replay, and asmdiffs. Below you will find more specific information on each of the different modes.
7+
Although SuperPMI has many uses, setup and use of SuperPMI is not always trivial.
8+
superpmi.py is a tool to help automate the use of SuperPMI, augmenting its usefulness.
9+
The tool has three different modes: collect, replay, and asmdiffs.
10+
Below you will find more specific information on each of the different modes.
1111

1212
**SuperPMI has a large limitation, the replay/asmdiff functionality is sensitive to jitinterface changes. Therefore if there has been an interface change, the new JIT will not load and SuperPMI will fail.**
1313

1414
**At the time of writing collections are done manually on all platforms that support libcoredistools. See before for a full list of supported platforms and where the .mch collection exists.**
1515

16-
#### Supported Platforms
16+
## Supported Platforms
1717

1818
| OS | Arch | Replay | AsmDiffs | mch location |
1919
| --- | --- | --- |--- | --- |
@@ -26,7 +26,7 @@ Although SuperPMI has many uses, setup and use of SuperPMI is not always trivial
2626
| Ubuntu | arm32 | <ul><li>- [ ] </li></ul> | <ul><li>- [ ] </li></ul> | N/A |
2727
| Ubuntu | arm64 | <ul><li>- [ ] </li></ul> | <ul><li>- [ ] </li></ul> | N/A |
2828

29-
### Default Collections
29+
## Default Collections
3030

3131
See the table above for locations of default collections that exist. If there is an mch file that exists, then SuperPMI will automatically download and setup the mch using that location. Please note that, it is possible that the collection is out of date, or there is a jitinterface change which makes the collection invalid. If this is the case, then in order to use the tool a collection will have to be done manually. In order to reproduce the default collections, please see below for what command the default collections are done with.
3232

@@ -36,7 +36,7 @@ See the table above for locations of default collections that exist. If there is
3636

3737
`/Users/jashoo/runtime/src/coreclr/scripts/superpmi.py collect bash "/Users/jashoo/runtime/src/coreclr/tests/runtest.sh x64 checked" --skip-cleanup`
3838

39-
**Collect**
39+
## Collect
4040

4141
Given a specific command collect over all of the managed code called by the child process. Note that this allows many different invocations of any managed code. Although it does specifically require that any managed code run by the child process to handle the complus variables set by SuperPMI and defer them to the later. These are below:
4242

@@ -55,11 +55,11 @@ Note that collection will try to run as much managed code as possible. In order
5555

5656
Also note that collection generates gigabytes of data, most of this data will be removed when the collection is finished and de-dupped into a single mch file. That being said, it is worth mentioning that this process will use 3x the size of the unclean mch file, which to give an example of the size, a collection of the coreclr `priority=1` tests uses roughly `200gb` of disk space. Most of this space will be used in a temp directory, which on Windows will default to `C:\Users\blah\AppData\Temp\...`. It is recommended to set the temp variable to a different location before running collect to avoid running out of disk space. This can be done by simply running `set TEMP=D:\TEMP`.
5757

58-
**Replay**
58+
## Replay
5959

6060
SuperPMI replay supports faster assertion checking over a collection than running the tests individually. This is useful if the collection includes a larger corpus of data that can reasonably be run against by executing the actual code. Note that this is similar to the PMI tool, with the same limitation, that runtime issues will not be caught by SuperPMI replay only assertions.
6161

62-
**AsmDiffs**
62+
## Asm Diffs
6363

6464
SuperPMI will take two different JITs, a baseline and diff JIT and run the compiler accross all the methods in the mch file. It uses coredistools to do a binary difference of the two different outputs. Note that sometimes the binary will differ, and SuperPMI will be run once again dumping the asm that was output in text format. Then the text will be diffed, if there are differences, you should look for text differences. If there are some then it is worth investigating the asm differences.
6565

‎src/coreclr/src/ToolBox/superpmi/readme.md

Lines changed: 288 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,288 @@
1+
# Overview
2+
3+
This directory (`src/coreclr/src/ToolBox/superpmi` in the GitHub
4+
https://github.com/dotnet/runtime repository) contains the SuperPMI
5+
tool used for testing the .NET just-in-time (JIT) compiler.
6+
7+
## Purpose
8+
9+
SuperPMI has two primary uses:
10+
1. Verification that a JIT code change doesn't cause any asserts.
11+
2. Finding test code where two JIT compilers generate different code, or
12+
verifying that the two compilers generate the same code.
13+
14+
Case #1 is useful for doing quick regression checking when making a source
15+
code change to the JIT compiler. The process is: (a) make a JIT source code
16+
change, (b) run that newly built JIT through a SuperPMI run to verify no
17+
asserts have been introduced.
18+
19+
Case #2 is useful for generating assembly language diffs, to help analyze the
20+
impact of a JIT code change.
21+
22+
## SuperPMI architecture
23+
24+
SuperPMI works in two phases: collection and playback.
25+
26+
In the collection phase, the system is configured to collect SuperPMI data.
27+
Then, run any set of .NET managed programs. When these managed programs invoke the JIT
28+
compiler, SuperPMI gathers and captures all information passed between the
29+
JIT and its .NET host. This data is post-processed to remove essentially
30+
duplicate function information, and is collected into one or just a few
31+
files.
32+
33+
In the playback phase, SuperPMI loads the JIT directly, and causes it to
34+
compile all the functions that were previously compiled in the collection
35+
phase, but using the collected data to provide answers to various questions
36+
that the JIT needs to ask. The .NET execution engine (EE) is not invoked at all.
37+
When doing playback for assertion checking, only a single JIT is loaded and
38+
used for compilation. When doing playback to check for assembly diffs, both
39+
a "baseline" and a "diff" compiler are loaded. Each JIT is asked to compile
40+
each recorded function. The generated results are compared with a built-in
41+
SuperPMI "near differ", which depends on an external disassembler component
42+
called `coredistools`. Typically, scripting has SuperPMI generate a list of
43+
functions with differences, then the script re-invokes each JIT to generate
44+
disassembly output (using `COMPlus_JitDisasm`) for each differing function.
45+
These are then compared either visually, or with the jitutils tool
46+
`jit-analyze`, or both.
47+
48+
49+
# Tools
50+
51+
There are two native executable tools: `superpmi` and `mcs`. To do the collection,
52+
there is the `superpmi-shim-collector` binary (`.dll` or `.so` or `.dylib`).
53+
54+
To harness collection, there is a .NET Core C# program that is built as
55+
part of the coreclr tests build called superpmicollect.exe
56+
(source: src/coreclr/tests/src/JIT/superpmi in https://github.com/dotnet/runtime repository).
57+
This tool also functions as a SuperPMI collection and playback unit test.
58+
59+
The superpmicollect tool is also being moved to the jitutils repository
60+
(https://github.com/dotnet/jitutils).
61+
62+
Each tool will show a help screen if passed `-?`.
63+
64+
Finally, there is a Python script that harnesses SuperPMI collection,
65+
playback, and other functions, such as download of existing SuperPMI
66+
collections from well-known locations. This tool is `superpmi.py`,
67+
found in `src/corclr/scripts` in the dotnet/runtime repository. See
68+
[superpmi.md](../../../scripts/superpmi.md) for more details.
69+
70+
71+
# Collection
72+
73+
To manually do a collection (not using the `superpmi.py` script or
74+
`superpmicollect.exe` tool), follow the following steps.
75+
76+
## Overall collection process
77+
78+
First, build the `dotnet/runtime` repo, which builds the `superpmi`, `mcs`,
79+
and `superpmi-shim-collector` programs, along with the rest of coreclr,
80+
and places them in the same native code directory as the JIT and the rest
81+
of coreclr, e.g., `f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\superpmi.exe`
82+
for a `dotnet/runtime` repo rooted at the `f:\gh\runtime` directory, and
83+
built on Windows for the x64 Checked architecture / build flavor combination.
84+
85+
The SuperPMI collection process requires a lot of disk space. How much will
86+
depend on the number, size, and complexity of the functions the JIT needs to
87+
compile. Make sure that the disk where the collected data will be collected
88+
is sufficiently large. It is also best if the data is written on an SSD, to
89+
speed up the disk operations.
90+
91+
These are the general steps that are followed when doing a SuperPMI collection:
92+
93+
1. Collect .MC files. Set up for collection, then cause the JIT to be invoked
94+
by either running a scenario, running tests, crossgen compiling assemblies,
95+
or using PMI to force the JIT to compile functions in an assembly.
96+
During collection, the data for each JIT compiled function is stored
97+
in a uniquely named file with a ".MC" filename extension (for "method context").
98+
2. Merge .MC files to .MCH file. We want all the generated data to be merged to a
99+
single file (or some smaller set of files), to both be more manageable, and
100+
to collect the data for multiple compiled functions into one place for easier
101+
replay. (MCH stands for "method context hive".)
102+
3. Remove duplicates in the .MCH file. Many compiled functions are essentially
103+
equivalent, such as trivial class constructors, or if some functions are
104+
compiled multiple times in different scenarios or tests. We filter out the
105+
duplicates, which makes playback much faster, and the resultant MCH file much
106+
smaller.
107+
4. Create a "clean" .MCH with no SuperPMI failures. The original collected MCH
108+
file might not replay cleanly. This is generally due to existing, un-investigated
109+
SuperPMI bugs or limitations. We don't want to see these during normal playback,
110+
so filter out the failing replays at this point so the "baseline" replay is clean.
111+
5. Create a table of contents (TOC) file. This creates an index for the generated
112+
MCH file that greatly speeds up certain operations.
113+
6. Test final .MCH file is clean. This is purely to check that the resultant
114+
MCH file is in a good state for future use.
115+
116+
117+
## Collect .MC files
118+
119+
Set the following environment variables:
120+
121+
```
122+
SuperPMIShimLogPath=<full path to an existing, empty temporary directory>
123+
SuperPMIShimPath=<full path to clrjit.dll, the "standalone" JIT>
124+
COMPlus_AltJit=*
125+
COMPlus_AltJitNgen=*
126+
COMPlus_AltJitName=superpmi-shim-collector.dll
127+
```
128+
129+
for example, on Windows:
130+
131+
```
132+
mkdir f:\spmi\temp
133+
set SuperPMIShimLogPath=f:\spmi\temp
134+
set SuperPMIShimPath=f:\gh\runtime\artifacts\tests\coreclr\Windows_NT.x64.Checked\Tests\Core_Root\clrjit.dll
135+
set COMPlus_AltJit=*
136+
set COMPlus_AltJitNgen=*
137+
set COMPlus_AltJitName=superpmi-shim-collector.dll
138+
```
139+
140+
(On Linux, use `libclrjit.so` and `libsuperpmi-shim-collector.so`.
141+
On Mac, use `libclrjit.dylib` and `libsuperpmi-shim-collector.dylib`.)
142+
143+
Note that the `superpmi-shim-collector.dll` must live in the same directory as the `coreclr.dll`
144+
(or libcoreclr.so on Linux) that will be invoked when running .NET Core. This will normally be
145+
a `Core_Root` directory, since you must create such a directory to be able to run
146+
.NET Core applications (such as by using the `corerun` tool).
147+
148+
If you want to collect using an official .NET Core build, you will need to build a matching set
149+
of superpmi binaries, and copy `superpmi-shim-collector.dll` to the correct directory. This
150+
option has not been tested (as far as I know).
151+
152+
Then, cause the JIT to compile some code. Do one or more of:
153+
1. Run a managed scenario.
154+
2. Run managed code tests.
155+
3. Crossgen some assemblies.
156+
4. Run PMI over some assemblies (see https://github.com/dotnet/jitutils for details on PMI)
157+
158+
When done running programs, un-set these environment variables.
159+
160+
Now, you will have a large number of .MC files in the specified temporary
161+
directory (specified by `SuperPMIShimLogPath`).
162+
163+
164+
## Merge .MC files to .MCH file
165+
166+
Merge the generated .MC files using the `mcs` tool:
167+
168+
```
169+
mcs -merge base.mch *.mc -recursive
170+
```
171+
172+
This assumes the current directory is the root directory where the .MC files
173+
were placed, namely the directory specified as `SuperPMIShimLogPath` above.
174+
You can also specify a directory prefix to the file system regular expression.
175+
The `-recursive` flag is only necessary if .MC files also exist in subdirectories
176+
of this, and you want those also added to the resultant, collected `base.mch`
177+
file. So, for the example above, you might use:
178+
179+
```
180+
f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\mcs.exe -merge f:\spmi\base.mch f:\spmi\temp\*.mc -recursive
181+
```
182+
183+
Note that `mcs -merge` is literally just a file concatenation of many files into
184+
one, so this step will double the required disk space.
185+
186+
After this step, you can remove all the individual .MC files unless you want
187+
to keep them to debug the SuperPMI collection process itself.
188+
189+
190+
## Remove duplicates in the .MCH file
191+
192+
One benefit of SuperPMI is the ability to remove duplicated compilations, so
193+
on replay only unique functions are compiled. Use the following to create a
194+
"unique" set of functions:
195+
196+
```
197+
mcs -removeDup -thin base.mch unique.mch
198+
```
199+
200+
Note that `-thin` is not required. However, it will delete all the compilation
201+
results collected during the collection phase, which makes the resulting MCH
202+
file smaller. Those compilation results are not required for playback for
203+
the ways in which we normally use SuperPMI.
204+
205+
For the continuing example, you might use:
206+
207+
```
208+
f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\mcs.exe -removeDup -thin f:\spmi\base.mch f:\spmi\unique.mch
209+
```
210+
211+
After this step, you can remove the base.mch file (unless you want to debug
212+
the SuperPMI collection process itself).
213+
214+
215+
## Create a "clean" .MCH with no SuperPMI failures
216+
217+
As stated above, due to various bugs or otherwise uninvestigated issues, a SuperPMI
218+
replay of the unique.mch file might contain errors. We don't want that, so we filter
219+
out those errors in a "baseline" run, as follows.
220+
221+
```
222+
superpmi -p -f basefail.mcl unique.mch clrjit.dll
223+
mcs.exe -strip basefail.mcl unique.mch final.mch
224+
```
225+
226+
Or, continuing the example above, giving full paths, we have:
227+
228+
```
229+
f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\superpmi.exe -p -f f:\spmi\basefail.mcl f:\spmi\unique.mch f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\clrjit.dll
230+
f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\mcs.exe -strip f:\spmi\basefail.mcl f:\spmi\unique.mch f:\spmi\final.mch
231+
```
232+
233+
234+
## Create a table of contents (TOC) file
235+
236+
This is
237+
238+
```
239+
mcs -toc final.mch
240+
```
241+
242+
or, using the full paths from above:
243+
244+
```
245+
f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\mcs.exe -toc f:\spmi\final.mch
246+
```
247+
248+
249+
## Test final .MCH file is clean
250+
251+
This is done using a replay just like for the "create clean .MCH file" step:
252+
253+
```
254+
superpmi -p -f finalfail.mcl final.mch clrjit.dll
255+
```
256+
257+
Or, continuing the example above, giving full paths, we have:
258+
259+
```
260+
f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\superpmi.exe -p -f f:\spmi\finalfail.mcl f:\spmi\final.mch f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\clrjit.dll
261+
```
262+
263+
In this case, if `finalfail.mcl` is not empty, there was a failure in the final "check" replay.
264+
This will lead to the same failure in all future replays, if the resultant final.mch file is
265+
published for general use. This is annoying, but not necessarily a fatal problem.
266+
It's possible to cycle the process, stripping this failure from the final MCH,
267+
testing for "clean-ness" again, and repeating. It should not be necessary, however.
268+
269+
270+
## Publishing the SuperPMI collection
271+
272+
The files that should be published for consumption either by yourself or by a group
273+
are `final.mch` and `final.mch.mct` (the TOC). All other files described above are
274+
intermediate files in the collection process, and are not needed afterwards.
275+
276+
277+
# Playback
278+
279+
Once you have a merged, de-duplicated MCH collection, you can play it back
280+
using:
281+
282+
superpmi -p final.mch clrjit.dll
283+
284+
The `-p` switch says to utilize all the processors on your machine,
285+
and replay in parallel. You can omit this if you wish to replay
286+
without using parallelism (this is usually done when replaying
287+
a single compilation, whereas `-p` is usually used when replaying
288+
a full MCH file set of compilations).

0 commit comments

Comments
 (0)