Skip to content

Latest commit

 

History

History
106 lines (104 loc) · 6.88 KB

PERFORMANCE.md

File metadata and controls

106 lines (104 loc) · 6.88 KB

Benchmarks: General setup

System

  • Windows 11
  • AMD Ryzen 5 5500
  • 16 GB RAM

Compilers

  • MSVC 19.43 (Release /DWIN32 /D_WINDOWS /W3 /GR /EHsc /MD /O2 /Ob2 /DNDEBUG -std:c++latest)
  • llvm-MinGW 18.1.6 (-O3 -DNDEBUG -std=gnu++23 -fansi-escape-codes -fcolor-diagnostics)
  • MinGW 13 (-O3 -DNDEBUG -std=gnu++23 -fdiagnostics-color=always)
  • MinGW 14 (-O3 -DNDEBUG -std=gnu++23 -fdiagnostics-color=always)

If you think it's unfair to compare MSVC's O2 with O3 from other compilers...that's what I get when I configure "Release build" with C++23

Benchmark comparing std::ranges::views with classic raw loops

Test description

Input is filled with numbers 1..size (std::iota)
Intermediate values are generated by creating an iota again for each number:
1 -> 1
2 -> 1,2
3 -> 1,2,3
etc.
This is done with 3 levels of nesting, so
3 -> 1,2,3 -> 1,1,2,1,2,3 -> 1,1,1,2,1,1,2,1,2,3
etc.
This blows up the intermediate range nicely in a fashion that is not deterministic in order (because the original input is random_shuffle'd every time) but that is deterministic in which values exist.
Then, the intermediate values are folded to a single result, the results collected, summed up and averaged, and the final result printed. The point of the exercise is to prevent the compiler from optimizing away the whole loop (which might happen if the results were never used, for example).
Only the segment which calculates the per-iteration result from the initial input is benchmarked.

Test implementations

  1. Classic loop to build intermediate vector, loop to accumulate result
  2. Same as (1), but with a pre-allocated vector
  3. Classic loop with integrated accumulation of final result
  4. Create view on intermediate range, loop to accumulate result
  5. Like (1), but use a view to build the initial vector

Test setup

  • 50 items in initial range
  • 1000 Iterations

Results

Milliseconds for total run, per compiler

|                                             | MSVC 19.43 | llvm-MinGW 18.1.6 | MinGW 13 | MinGW 14 |
| ------------------------------------------- | ---------- | ----------------- | -------- | -------- |
| Calculate result in loop                    |         61 |                50 |       73 |       73 |
| Build view→Reduce to result                 |        307 |               662 |      169 |      178 |
| Build vector in loop→Reduce                 |        893 |              1240 |      782 |      883 |
| Like above, but with preallocated vector    |        557 |               406 |      378 |      367 |
| View→Vector→Reduce                          |       1130 |              1880 |      829 |      954 |

Relative speed (Baseline = 1.0, greater is slower)

|                                             | MSVC 19.43 | llvm-MinGW 18.1.6 | MinGW 13 | MinGW 14 |
| ------------------------------------------- | ---------- | ----------------- | -------- | -------- |
| Calculate result in loop                    | Baseline   |                   |          |          |
| Build view→Reduce to result                 |       5,03 |             13,24 |     2,32 |     2,44 |
| Build vector in loop→Reduce                 |      14,64 |             24,80 |    10,71 |    12,10 |
| Like above, but with preallocated vector    |       9,13 |              8,12 |     5,18 |     5,03 |
| View→Vector→Reduce                          |      18,52 |             37,60 |    11,36 |    13,07 |

Speed in nanoseconds per item in intermediate range

|                                             | MSVC 19.43 | llvm-MinGW 18.1.6 | MinGW 13 | MinGW 14 |
| ------------------------------------------- | ---------- | ----------------- | -------- | -------- |
| Calculate result in loop                    |       0,21 |              0,17 |     0,25 |     0,25 |
| Build view→Reduce to result                 |       1,05 |              2,26 |     0,58 |     0,61 |
| Build vector in loop→Reduce                 |       3,05 |              4,23 |     2,67 |     3,02 |
| Like above, but with preallocated vector    |       1,90 |              1,39 |     1,29 |     1,25 |
| View→Vector→Reduce                          |       3,86 |              6,42 |     2,83 |     3,26 |

Benchmark comparing std::optional and std::expected with other ways of error handling

Test description

An input value is passed into a function with 5 gates.
Each gate accepts everything BUT a certain value:

  • Gate 1 rejects '1' but accepts any other number
  • Gate 2 rejects '2' but accepts any other number

...and so on.
If a value is accepted, it is forwarded to the next gate. This is repeated for a number of input values and iterations. Because of the way the gates are configured, you can create a random sequence of numbers with a statistically clear rejection rate: Just choose the desired ration between numbers 1 to 5 (rejected) and numbers above 5 (accepted).
So, an all happy-path run will have (different) numbers 6 or higher. And all-sad-path run will have numbers 1 to 5. A 50:50 happy/sad path run will have an equal distribution of numbers 1 to 10.

Test implementations

  1. Baseline (only simulated work, no error handling)
  2. Value-based: bool returned from function indicates success or failure
  3. std::expected with and_then
  4. std::optional with and_then
  5. Exceptions

Test setup

  • Input length: 10000 (each number passing through up to 5 gates, depending on rejection rate)
  • Number of iterations: 100
  • Rejection rates: 0%, 2%, 100%

Results in microseconds for whole test run

|                   |               | 0% rejection | 2% rejection | 100% rejection |
| ----------------- | ------------- | ------------ | ------------ | -------------- |
| MinGW 14          | Value-based   |         2130 |         2130 |           2130 |
| llvm-MinGW 18.1.6 | Value-based   |         2840 |         2840 |           2840 |
| MSVC 19.43        | Value-based   |         9020 |         9580 |           7660 |
|                   |               |              |              |                |
| MinGW 14          | std::optional |         2130 |         2450 |           5900 |
| llvm-MinGW 18.1.6 | std::optional |         3110 |         3750 |           3070 |
| MSVC 19.43        | std::optional |         9860 |         1060 |          12250 |
|                   |               |              |              |                |
| MinGW 14          | std::expected |         2140 |         2430 |           5610 |
| llvm-MinGW 18.1.6 | std::expected |         4400 |         4060 |           7020 |
| MSVC 19.43        | std::expected |        10240 |        10820 |          11610 |
|                   |               |              |              |                |
| MinGW 14          | Exceptions    |         3930 |        28660 |        1320000 |
| llvm-MinGW 18.1.6 | Exceptions    |         3070 |         4157 |        1920000 |
| MSVC 19.43        | Exceptions    |         9270 |        52920 |        2230000 |