Make assertions thread-safe

[jeremy-rifkin]

Description

This PR makes Catch2 assertion and logging macros thread-safe, allowing threads spawned in test cases to perform checks. It does this by introducing a global lock and locking at entry points called from these macros. This was easier and safer than trying to track down every use of static storage in the library and locking only around those, however, at some point that might be desirable.

A std::recursive_mutex is used for the global lock. There is some overhead associated with taking out the lock, however, it's extremely minimal in the uncontested (single-thread) case. In a benchmark locally I found that for a debug build on linux the overhead of the lock is around 300ns on my machine, meaning that a user would have to perform a million assertions in order to add a second of run-time overhead to their program.

benchmark name                       samples       iterations    est run time
                                     mean          low mean      high mean
                                     std dev       low std dev   high std dev
-------------------------------------------------------------------------------
no lock                                        100          1604     5.9348 ms 
                                        37.1058 ns     36.915 ns    37.4472 ns 
                                        1.26649 ns    0.80055 ns    1.98121 ns 
                                                                               
with std::recursive_mutex                      100           203     6.0088 ms 
                                        303.559 ns    299.777 ns    313.273 ns 
                                        28.6615 ns    11.6045 ns    56.3121 ns 

On release the overhead is far smaller:

benchmark name                       samples       iterations    est run time
                                     mean          low mean      high mean
                                     std dev       low std dev   high std dev
-------------------------------------------------------------------------------
no lock                                        100         20405     2.0405 ms 
                                        1.29613 ns    1.28875 ns    1.31015 ns 
                                      0.0501252 ns  0.0300847 ns  0.0787845 ns 
                                                                               
with std::recursive_mutex                      100           226      2.712 ms 
                                        137.018 ns    128.907 ns    150.877 ns 
                                        53.1962 ns    37.1166 ns    85.6079 ns 

MSVC debug:

benchmark name                       samples       iterations    est run time
                                     mean          low mean      high mean
                                     std dev       low std dev   high std dev
-------------------------------------------------------------------------------
no lock                                        100          6510      1.302 ms 
                                        2.07343 ns     2.0573 ns    2.10123 ns 
                                       0.105615 ns  0.0698496 ns    0.14992 ns

with std::recursive_mutex                      100           567     1.5309 ms 
                                        25.3474 ns    24.8342 ns    27.2275 ns 
                                        4.45937 ns    1.13563 ns    10.2695 ns

If this impact is deemed too high I have ideas for reducing the overhead in debug mode and optimizing for the uncontested case.

GitHub Issues

#99
#246
#875
#1043
#1169
#1252
#1302
#1714
#1904
#2641
And probably others

[codecov]

Codecov Report

Attention: Patch coverage is 97.05882% with 1 line in your changes missing coverage. Please review.

Project coverage is 90.99%. Comparing base (914aeec) to head (58137e5).

Additional details and impacted files

@@            Coverage Diff             @@
##            devel    #2948      +/-   ##
==========================================
- Coverage   91.01%   90.99%   -0.02%     
==========================================
  Files         198      200       +2     
  Lines        8599     8626      +27     
==========================================
+ Hits         7826     7849      +23     
- Misses        773      777       +4     

[shahsb]

The sentence “at some point that might be desirable” is a bit vague. It would be more helpful to either clarify what you mean or remove it. If you're suggesting the possibility of refining the approach in the future (e.g., focusing on static storage), consider expanding on that idea.

[shahsb]

Clarify the environment under which these benchmarks were taken (e.g., specific hardware, OS, and configuration), as this can significantly affect the results. Also, mention if these benchmarks are reflective of real-world usage or more of a worst-case scenario.

[shahsb]

Clarify the impact of this change on larger test suites, as that could be relevant for performance-sensitive environments.

[shahsb]

Codecov is complaining for this line, please check:

Check warning on line 90 in src/catch2/internal/catch_assertion_handler.cpp


Codecov
/ codecov/patch
src/catch2/internal/catch_assertion_handler.cpp#L90

Added line #L90 was not covered by tests

[shahsb]

Explain the rationale behind choosing std::recursive_mutex over other types of mutexes.

[jeremy-rifkin]

Hi @shahsb, I am awaiting further feedback from the maintainer who I have been in contact with regarding this PR over discord. I'm a bit confused by your reviews, are you involved with the Catch2 project?