Up to date load testing

[siladu]

We haven't done much testing beyond 10K validators, however users are asking about (or already running in some cases?) 20K or more validators.

Main things to test would be:

1. Key Loading/startup time
2. Large numbers of signing requests (e.g. looking for database contention issues)

[siladu]

Note, we do appear to have a performance test which might be easy to tweak to cover this. Not sure if current test implementation would reveal any database contention issues, it's a constant rate of 100 requests per second...

https://github.com/Consensys/web3signer/blob/be75f366056348f5f7a36be90eb1096e6e97c86c/acceptance-tests/src/gatling/scala/Eth2SignSimulation.scala

• 1000 keys

SubProcess | 2023-09-07 16:15:31.714+10:00 | pool-2-thread-1 | INFO  | SignerLoader | Total Artifact Signer loaded via configuration files: 1000
SubProcess | Error count 0
SubProcess | Time Taken: 00:00:00.204.

================================================================================
---- Global Information --------------------------------------------------------
> request count                                       6000 (OK=6000   KO=0     )
> min response time                                      2 (OK=2      KO=-     )
> max response time                                    273 (OK=273    KO=-     )
> mean response time                                     6 (OK=6      KO=-     )
> std deviation                                         12 (OK=12     KO=-     )
> response time 50th percentile                          5 (OK=5      KO=-     )
> response time 75th percentile                          6 (OK=6      KO=-     )
> response time 95th percentile                         12 (OK=12     KO=-     )
> response time 99th percentile                         31 (OK=31     KO=-     )
> mean requests/sec                                    100 (OK=100    KO=-     )
---- Response Time Distribution ------------------------------------------------
> t < 800 ms                                          6000 (100%)
> 800 ms <= t < 1200 ms                                  0 (  0%)
> t >= 1200 ms                                           0 (  0%)
> failed                                                 0 (  0%)
================================================================================

• 10,000 keys

SubProcess | 2023-09-07 17:00:33.328+10:00 | pool-2-thread-1 | INFO  | SignerLoader | Total Artifact Signer loaded via configuration files: 10000
SubProcess | Error count 0
SubProcess | Time Taken: 00:00:00.463.

================================================================================
---- Global Information --------------------------------------------------------
> request count                                       6000 (OK=6000   KO=0     )
> min response time                                      2 (OK=2      KO=-     )
> max response time                                    144 (OK=144    KO=-     )
> mean response time                                     6 (OK=6      KO=-     )
> std deviation                                          6 (OK=6      KO=-     )
> response time 50th percentile                          5 (OK=5      KO=-     )
> response time 75th percentile                          6 (OK=6      KO=-     )
> response time 95th percentile                         10 (OK=10     KO=-     )
> response time 99th percentile                         17 (OK=17     KO=-     )
> mean requests/sec                                    100 (OK=100    KO=-     )
---- Response Time Distribution ------------------------------------------------
> t < 800 ms                                          6000 (100%)
> 800 ms <= t < 1200 ms                                  0 (  0%)
> t >= 1200 ms                                           0 (  0%)
> failed                                                 0 (  0%)
================================================================================

• 20,000 keys

SubProcess | 2023-09-07 16:14:23.047+10:00 | pool-2-thread-1 | INFO  | SignerLoader | Total Artifact Signer loaded via configuration files: 20000
SubProcess | Error count 0
SubProcess | Time Taken: 00:00:00.834.


================================================================================
---- Global Information --------------------------------------------------------
> request count                                       6000 (OK=6000   KO=0     )
> min response time                                      2 (OK=2      KO=-     )
> max response time                                    166 (OK=166    KO=-     )
> mean response time                                     6 (OK=6      KO=-     )
> std deviation                                          7 (OK=7      KO=-     )
> response time 50th percentile                          5 (OK=5      KO=-     )
> response time 75th percentile                          6 (OK=6      KO=-     )
> response time 95th percentile                         10 (OK=10     KO=-     )
> response time 99th percentile                         18 (OK=18     KO=-     )
> mean requests/sec                                    100 (OK=100    KO=-     )
---- Response Time Distribution ------------------------------------------------
> t < 800 ms                                          6000 (100%)
> 800 ms <= t < 1200 ms                                  0 (  0%)
> t >= 1200 ms                                           0 (  0%)
> failed                                                 0 (  0%)
================================================================================

• 100,000 keys

SubProcess | 2023-09-07 16:28:41.799+10:00 | pool-2-thread-1 | INFO  | SignerLoader | Total Artifact Signer loaded via configuration files: 100000
SubProcess | Error count 0
SubProcess | Time Taken: 00:00:03.053.

================================================================================
---- Global Information --------------------------------------------------------
> request count                                       6000 (OK=6000   KO=0     )
> min response time                                      2 (OK=2      KO=-     )
> max response time                                    139 (OK=139    KO=-     )
> mean response time                                     5 (OK=5      KO=-     )
> std deviation                                          5 (OK=5      KO=-     )
> response time 50th percentile                          4 (OK=4      KO=-     )
> response time 75th percentile                          5 (OK=5      KO=-     )
> response time 95th percentile                          8 (OK=8      KO=-     )
> response time 99th percentile                         15 (OK=15     KO=-     )
> mean requests/sec                                    100 (OK=100    KO=-     )
---- Response Time Distribution ------------------------------------------------
> t < 800 ms                                          6000 (100%)
> 800 ms <= t < 1200 ms                                  0 (  0%)
> t >= 1200 ms                                           0 (  0%)
> failed                                                 0 (  0%)
================================================================================

• 1,000,000 keys

SubProcess | 2023-09-07 16:55:24.697+10:00 | pool-2-thread-1 | INFO  | SignerLoader | Total Artifact Signer loaded via configuration files: 1000000
SubProcess | Error count 0
SubProcess | Time Taken: 00:00:22.572.

================================================================================
---- Global Information --------------------------------------------------------
> request count                                       6000 (OK=6000   KO=0     )
> min response time                                      2 (OK=2      KO=-     )
> max response time                                    219 (OK=219    KO=-     )
> mean response time                                     6 (OK=6      KO=-     )
> std deviation                                         11 (OK=11     KO=-     )
> response time 50th percentile                          4 (OK=4      KO=-     )
> response time 75th percentile                          6 (OK=6      KO=-     )
> response time 95th percentile                         14 (OK=14     KO=-     )
> response time 99th percentile                         30 (OK=30     KO=-     )
> mean requests/sec                                    100 (OK=100    KO=-     )
---- Response Time Distribution ------------------------------------------------
> t < 800 ms                                          6000 (100%)
> 800 ms <= t < 1200 ms                                  0 (  0%)
> t >= 1200 ms                                           0 (  0%)
> failed                                                 0 (  0%)
================================================================================

[non-fungible-nelson]

good candidate for holesky validators?

[jframe]

Definitely, using the holesky validators could be a good way of testing this