Skip to content

Commit

Permalink
Add performance & benchmarking parts
Browse files Browse the repository at this point in the history
* Add Performance main part to hold findings about profiling and
  benchmarks
* Add an article about benchmarking rust apps and libraries, explaining
  the two main method for benchmarking, their pros and cons and their
  supporting crates
  • Loading branch information
AmmarAbouZor committed Aug 28, 2024
1 parent 14dd139 commit 344df32
Show file tree
Hide file tree
Showing 3 changed files with 41 additions and 0 deletions.
2 changes: 2 additions & 0 deletions developer/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
[Introduction](./README.md)

- [Architecture](architecture/README.md)
- [Performance](performance/README.md)
- [Benchmarking](performance/benchmarking.md)
- [Plugin System](plugin/README.md)
- [Life Sessions](plugin/life_sessions.md)
- [Decisions Context](plugin/decisions/decisions-context.md)
Expand Down
4 changes: 4 additions & 0 deletions developer/src/performance/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Performance

This section contains various performance findings for Chipmunk, including benchmarks and profiling results.
Please refer to the child parts for detailed information and insights.
35 changes: 35 additions & 0 deletions developer/src/performance/benchmarking.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Benchmarking


## Benchmarking main methods

* In general we have two main methods to do benchmarks, the first measures the execution time and the second estimates CPU cycles.
* Rust has official support for benchmarking that is used to bench the standard-library. However, this support is available currently on Rust nightly only.
* Rust Eco-system has crates supporting the two methods, providing a similar interface to the `Bencher` from the standard-library which make it easier to write benchmarks for both of them together, or migrate from one to the other (Or to the `Bencher` once it has support on stable rust)


### Wall Clock Time Method

* This method measures the elapsed time between starting and finishing a task using the system wall clock time.
* It must run the tests many times to get more accurate results, and the results will vary between hosts' environment.
* This method isn't suitable for CI by saving bench results of master branch and compare it with different branch, because the virtual machine state can change between each run. To get best accuracy, we need to run the benches on the master and on the branch in the same run to compare them in the environment.
* The crate [criterion](https://github.com/bheisler/criterion.rs) provides a similar interface for `Bencher` from standard-library while it can run rust stable, providing more features like graphs and more control over the benchmarks themselves.
* Benchmarks for specific parts of the app can be written with mocks if needed similar to the unit tests, providing a way to bench specific functions in isolated way.
* This method doesn't demand and external support and can run cross-platform


### CPU Cycles Method

* This method estimates and compares the CPU cycles using [Valgrind Callgrind](https://valgrind.org/docs/manual/cl-manual.html).
* Each test will run once only and have results that are independent form the host state, which makes it reliable in CI pipelines
* It provides high-precision measurements since it's counting the CPU cycles which make any difference noticeable with clear numbers.
* Currently, the crate [Iai-Callgrind](https://github.com/iai-callgrind/iai-callgrind/tree/main?tab=readme-ov-file) has support for this kind of benchmarks in Rust.

#### Cons:

* This method can run only where [Valgrind](https://valgrind.org/) runs (Currently Linux and MacOS)
* CPU cycles don't match necessarily how much time the process took and it will still need timed benchmarks along side.
* `Iai-Callgrind` doesn't have currently a lot of recognition and support from Rust community compared to `Criterion`, however they have great documentation and the library is active development.



0 comments on commit 344df32

Please sign in to comment.