Add shorthand version of opensearch-benchmark invocation, osb

[IanHoang]

Description

Adding shorthand version of "opensearch-benchmark". Users can invoke "osb" and "osbd" (latter for running distributed workload generation on several nodes)

Issues Resolved

#324 #744 #639

Testing

• New functionality includes testing

Ran geonames test in test mode as well as other commonly used commands

opensearch-benchmark % osb execute-test --target-hosts=https://endpoint.es.amazonaws.com --client-options=basic_auth_user:'xxxxx',basic_auth_password:'x' --workloadxxxxx=geonames --kill-running-processes --test-mode

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/

[INFO] [Test Execution ID]: 5a2de279-6004-45e5-bad1-52e1610006ae
[INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds.
[INFO] Executing test with workload [geonames], test_procedure [append-no-conflicts] and provision_config_instance ['external'] with version [2.5.0].

[WARNING] merges_total_time is 4279983 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] merges_total_throttled_time is 1681595 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] indexing_total_time is 21328064 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] refresh_total_time is 1252834 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] flush_total_time is 1666964 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running check-cluster-health                                                   [100% done]
Running index-append                                                           [100% done]
Running refresh-after-index                                                    [100% done]
Running force-merge                                                            [100% done]
Running refresh-after-force-merge                                              [100% done]
Running wait-until-merges-finish                                               [100% done]
Running index-stats                                                            [100% done]
Running node-stats                                                             [100% done]
Running default                                                                [100% done]
Running term                                                                   [100% done]
Running phrase                                                                 [100% done]
Running country_agg_uncached                                                   [100% done]
Running country_agg_cached                                                     [100% done]
Running scroll                                                                 [100% done]
Running expression                                                             [100% done]
Running painless_static                                                        [100% done]
Running painless_dynamic                                                       [100% done]
Running decay_geo_gauss_function_score                                         [100% done]
Running decay_geo_gauss_script_score                                           [100% done]
Running field_value_function_score                                             [100% done]
Running field_value_script_score                                               [100% done]
Running large_terms                                                            [100% done]
Running large_filtered_terms                                                   [100% done]
Running large_prohibited_terms                                                 [100% done]
Running desc_sort_population                                                   [100% done]
Running asc_sort_population                                                    [100% done]
Running asc_sort_with_after_population                                         [100% done]
Running desc_sort_geonameid                                                    [100% done]
Running desc_sort_with_after_geonameid                                         [100% done]
Running asc_sort_geonameid                                                     [100% done]
Running asc_sort_with_after_geonameid                                          [100% done]

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------

|                                                         Metric |                           Task |       Value |    Unit |
|---------------------------------------------------------------:|-------------------------------:|------------:|--------:|
|                     Cumulative indexing time of primary shards |                                |     355.468 |     min |
|             Min cumulative indexing time across primary shards |                                |           0 |     min |
|          Median cumulative indexing time across primary shards |                                |     0.06475 |     min |
|             Max cumulative indexing time across primary shards |                                |     92.7309 |     min |
|            Cumulative indexing throttle time of primary shards |                                |           0 |     min |
|    Min cumulative indexing throttle time across primary shards |                                |           0 |     min |
| Median cumulative indexing throttle time across primary shards |                                |           0 |     min |
|    Max cumulative indexing throttle time across primary shards |                                |           0 |     min |
|                        Cumulative merge time of primary shards |                                |     71.3331 |     min |
|                       Cumulative merge count of primary shards |                                |          64 |         |
|                Min cumulative merge time across primary shards |                                |           0 |     min |
|             Median cumulative merge time across primary shards |                                |           0 |     min |
|                Max cumulative merge time across primary shards |                                |      52.735 |     min |
|               Cumulative merge throttle time of primary shards |                                |     28.0266 |     min |
|       Min cumulative merge throttle time across primary shards |                                |           0 |     min |
|    Median cumulative merge throttle time across primary shards |                                |           0 |     min |
|       Max cumulative merge throttle time across primary shards |                                |     23.7529 |     min |
|                      Cumulative refresh time of primary shards |                                |     20.8797 |     min |
|                     Cumulative refresh count of primary shards |                                |        2972 |         |
|              Min cumulative refresh time across primary shards |                                |           0 |     min |
|           Median cumulative refresh time across primary shards |                                |   0.0180083 |     min |
|              Max cumulative refresh time across primary shards |                                |     9.97653 |     min |
|                        Cumulative flush time of primary shards |                                |     27.7827 |     min |
|                       Cumulative flush count of primary shards |                                |         182 |         |
|                Min cumulative flush time across primary shards |                                |           0 |     min |
|             Median cumulative flush time across primary shards |                                | 0.000283333 |     min |
|                Max cumulative flush time across primary shards |                                |     4.86025 |     min |
|                                        Total Young Gen GC time |                                |           0 |       s |
|                                       Total Young Gen GC count |                                |           0 |         |
|                                          Total Old Gen GC time |                                |           0 |       s |
|                                         Total Old Gen GC count |                                |           0 |         |
|                                                     Store size |                                |     32.2862 |      GB |
|                                                  Translog size |                                | 5.78817e-06 |      GB |
|                                         Heap used for segments |                                |           0 |      MB |
|                                       Heap used for doc values |                                |           0 |      MB |
|                                            Heap used for terms |                                |           0 |      MB |
|                                            Heap used for norms |                                |           0 |      MB |
|                                           Heap used for points |                                |           0 |      MB |
|                                    Heap used for stored fields |                                |           0 |      MB |
|                                                  Segment count |                                |         370 |         |
|                                                 Min Throughput |                   index-append |      3249.7 |  docs/s |
|                                                Mean Throughput |                   index-append |      3249.7 |  docs/s |
|                                              Median Throughput |                   index-append |      3249.7 |  docs/s |
|                                                 Max Throughput |                   index-append |      3249.7 |  docs/s |
|                                        50th percentile latency |                   index-append |     301.358 |      ms |
|                                       100th percentile latency |                   index-append |       331.2 |      ms |
|                                   50th percentile service time |                   index-append |     301.358 |      ms |
|                                  100th percentile service time |                   index-append |       331.2 |      ms |
|                                                     error rate |                   index-append |           0 |       % |
|                                                 Min Throughput |       wait-until-merges-finish |        3.89 |   ops/s |
|                                                Mean Throughput |       wait-until-merges-finish |        3.89 |   ops/s |
|                                              Median Throughput |       wait-until-merges-finish |        3.89 |   ops/s |
|                                                 Max Throughput |       wait-until-merges-finish |        3.89 |   ops/s |
|                                       100th percentile latency |       wait-until-merges-finish |     235.891 |      ms |
|                                  100th percentile service time |       wait-until-merges-finish |     235.891 |      ms |
|                                                     error rate |       wait-until-merges-finish |           0 |       % |
|                                                 Min Throughput |                    index-stats |        4.17 |   ops/s |
|                                                Mean Throughput |                    index-stats |        4.17 |   ops/s |
|                                              Median Throughput |                    index-stats |        4.17 |   ops/s |
|                                                 Max Throughput |                    index-stats |        4.17 |   ops/s |
|                                       100th percentile latency |                    index-stats |     305.784 |      ms |
|                                  100th percentile service time |                    index-stats |      63.668 |      ms |
|                                                     error rate |                    index-stats |           0 |       % |
|                                                 Min Throughput |                     node-stats |        3.93 |   ops/s |
|                                                Mean Throughput |                     node-stats |        3.93 |   ops/s |
|                                              Median Throughput |                     node-stats |        3.93 |   ops/s |
|                                                 Max Throughput |                     node-stats |        3.93 |   ops/s |
|                                       100th percentile latency |                     node-stats |     327.391 |      ms |
|                                  100th percentile service time |                     node-stats |     69.5392 |      ms |
|                                                     error rate |                     node-stats |           0 |       % |
|                                                 Min Throughput |                        default |         4.3 |   ops/s |
|                                                Mean Throughput |                        default |         4.3 |   ops/s |
|                                              Median Throughput |                        default |         4.3 |   ops/s |
|                                                 Max Throughput |                        default |         4.3 |   ops/s |
|                                       100th percentile latency |                        default |     454.288 |      ms |
|                                  100th percentile service time |                        default |     220.921 |      ms |
|                                                     error rate |                        default |           0 |       % |
|                                                 Min Throughput |                           term |        4.07 |   ops/s |
|                                                Mean Throughput |                           term |        4.07 |   ops/s |
|                                              Median Throughput |                           term |        4.07 |   ops/s |
|                                                 Max Throughput |                           term |        4.07 |   ops/s |
|                                       100th percentile latency |                           term |       527.2 |      ms |
|                                  100th percentile service time |                           term |     280.355 |      ms |
|                                                     error rate |                           term |           0 |       % |
|                                                 Min Throughput |                         phrase |        4.18 |   ops/s |
|                                                Mean Throughput |                         phrase |        4.18 |   ops/s |
|                                              Median Throughput |                         phrase |        4.18 |   ops/s |
|                                                 Max Throughput |                         phrase |        4.18 |   ops/s |
|                                       100th percentile latency |                         phrase |     451.871 |      ms |
|                                  100th percentile service time |                         phrase |     211.991 |      ms |
|                                                     error rate |                         phrase |           0 |       % |
|                                                 Min Throughput |           country_agg_uncached |        3.63 |   ops/s |
|                                                Mean Throughput |           country_agg_uncached |        3.63 |   ops/s |
|                                              Median Throughput |           country_agg_uncached |        3.63 |   ops/s |
|                                                 Max Throughput |           country_agg_uncached |        3.63 |   ops/s |
|                                       100th percentile latency |           country_agg_uncached |     497.172 |      ms |
|                                  100th percentile service time |           country_agg_uncached |     221.354 |      ms |
|                                                     error rate |           country_agg_uncached |           0 |       % |
|                                                 Min Throughput |             country_agg_cached |        4.21 |   ops/s |
|                                                Mean Throughput |             country_agg_cached |        4.21 |   ops/s |
|                                              Median Throughput |             country_agg_cached |        4.21 |   ops/s |
|                                                 Max Throughput |             country_agg_cached |        4.21 |   ops/s |
|                                       100th percentile latency |             country_agg_cached |     455.772 |      ms |
|                                  100th percentile service time |             country_agg_cached |     217.238 |      ms |
|                                                     error rate |             country_agg_cached |           0 |       % |
|                                                 Min Throughput |                         scroll |        3.03 | pages/s |
|                                                Mean Throughput |                         scroll |        3.03 | pages/s |
|                                              Median Throughput |                         scroll |        3.03 | pages/s |
|                                                 Max Throughput |                         scroll |        3.03 | pages/s |
|                                       100th percentile latency |                         scroll |     1121.94 |      ms |
|                                  100th percentile service time |                         scroll |     350.866 |      ms |
|                                                     error rate |                         scroll |           0 |       % |
|                                                 Min Throughput |                     expression |        3.95 |   ops/s |
|                                                Mean Throughput |                     expression |        3.95 |   ops/s |
|                                              Median Throughput |                     expression |        3.95 |   ops/s |
|                                                 Max Throughput |                     expression |        3.95 |   ops/s |
|                                       100th percentile latency |                     expression |      472.08 |      ms |
|                                  100th percentile service time |                     expression |     218.309 |      ms |
|                                                     error rate |                     expression |           0 |       % |
|                                                 Min Throughput |                painless_static |        4.05 |   ops/s |
|                                                Mean Throughput |                painless_static |        4.05 |   ops/s |
|                                              Median Throughput |                painless_static |        4.05 |   ops/s |
|                                                 Max Throughput |                painless_static |        4.05 |   ops/s |
|                                       100th percentile latency |                painless_static |     462.137 |      ms |
|                                  100th percentile service time |                painless_static |     214.138 |      ms |
|                                                     error rate |                painless_static |           0 |       % |
|                                                 Min Throughput |               painless_dynamic |        4.23 |   ops/s |
|                                                Mean Throughput |               painless_dynamic |        4.23 |   ops/s |
|                                              Median Throughput |               painless_dynamic |        4.23 |   ops/s |
|                                                 Max Throughput |               painless_dynamic |        4.23 |   ops/s |
|                                       100th percentile latency |               painless_dynamic |     458.541 |      ms |
|                                  100th percentile service time |               painless_dynamic |     221.108 |      ms |
|                                                     error rate |               painless_dynamic |           0 |       % |
|                                                 Min Throughput | decay_geo_gauss_function_score |         3.8 |   ops/s |
|                                                Mean Throughput | decay_geo_gauss_function_score |         3.8 |   ops/s |
|                                              Median Throughput | decay_geo_gauss_function_score |         3.8 |   ops/s |
|                                                 Max Throughput | decay_geo_gauss_function_score |         3.8 |   ops/s |
|                                       100th percentile latency | decay_geo_gauss_function_score |     483.261 |      ms |
|                                  100th percentile service time | decay_geo_gauss_function_score |       219.2 |      ms |
|                                                     error rate | decay_geo_gauss_function_score |           0 |       % |
|                                                 Min Throughput |   decay_geo_gauss_script_score |        3.38 |   ops/s |
|                                                Mean Throughput |   decay_geo_gauss_script_score |        3.38 |   ops/s |
|                                              Median Throughput |   decay_geo_gauss_script_score |        3.38 |   ops/s |
|                                                 Max Throughput |   decay_geo_gauss_script_score |        3.38 |   ops/s |
|                                       100th percentile latency |   decay_geo_gauss_script_score |     532.316 |      ms |
|                                  100th percentile service time |   decay_geo_gauss_script_score |      235.93 |      ms |
|                                                     error rate |   decay_geo_gauss_script_score |           0 |       % |
|                                                 Min Throughput |     field_value_function_score |        3.78 |   ops/s |
|                                                Mean Throughput |     field_value_function_score |        3.78 |   ops/s |
|                                              Median Throughput |     field_value_function_score |        3.78 |   ops/s |
|                                                 Max Throughput |     field_value_function_score |        3.78 |   ops/s |
|                                       100th percentile latency |     field_value_function_score |     489.103 |      ms |
|                                  100th percentile service time |     field_value_function_score |     222.749 |      ms |
|                                                     error rate |     field_value_function_score |           0 |       % |
|                                                 Min Throughput |       field_value_script_score |        4.03 |   ops/s |
|                                                Mean Throughput |       field_value_script_score |        4.03 |   ops/s |
|                                              Median Throughput |       field_value_script_score |        4.03 |   ops/s |
|                                                 Max Throughput |       field_value_script_score |        4.03 |   ops/s |
|                                       100th percentile latency |       field_value_script_score |     475.456 |      ms |
|                                  100th percentile service time |       field_value_script_score |     225.937 |      ms |
|                                                     error rate |       field_value_script_score |           0 |       % |
|                                                 Min Throughput |                    large_terms |        0.93 |   ops/s |
|                                                Mean Throughput |                    large_terms |        0.93 |   ops/s |
|                                              Median Throughput |                    large_terms |        0.93 |   ops/s |
|                                                 Max Throughput |                    large_terms |        0.93 |   ops/s |
|                                       100th percentile latency |                    large_terms |     2093.27 |      ms |
|                                  100th percentile service time |                    large_terms |     1006.56 |      ms |
|                                                     error rate |                    large_terms |           0 |       % |
|                                                 Min Throughput |           large_filtered_terms |        1.08 |   ops/s |
|                                                Mean Throughput |           large_filtered_terms |        1.08 |   ops/s |
|                                              Median Throughput |           large_filtered_terms |        1.08 |   ops/s |
|                                                 Max Throughput |           large_filtered_terms |        1.08 |   ops/s |
|                                       100th percentile latency |           large_filtered_terms |     1824.24 |      ms |
|                                  100th percentile service time |           large_filtered_terms |     883.446 |      ms |
|                                                     error rate |           large_filtered_terms |           0 |       % |
|                                                 Min Throughput |         large_prohibited_terms |        1.02 |   ops/s |
|                                                Mean Throughput |         large_prohibited_terms |        1.02 |   ops/s |
|                                              Median Throughput |         large_prohibited_terms |        1.02 |   ops/s |
|                                                 Max Throughput |         large_prohibited_terms |        1.02 |   ops/s |
|                                       100th percentile latency |         large_prohibited_terms |     2020.56 |      ms |
|                                  100th percentile service time |         large_prohibited_terms |     1018.81 |      ms |
|                                                     error rate |         large_prohibited_terms |           0 |       % |
|                                                 Min Throughput |           desc_sort_population |         4.1 |   ops/s |
|                                                Mean Throughput |           desc_sort_population |         4.1 |   ops/s |
|                                              Median Throughput |           desc_sort_population |         4.1 |   ops/s |
|                                                 Max Throughput |           desc_sort_population |         4.1 |   ops/s |
|                                       100th percentile latency |           desc_sort_population |     717.234 |      ms |
|                                  100th percentile service time |           desc_sort_population |     472.626 |      ms |
|                                                     error rate |           desc_sort_population |           0 |       % |
|                                                 Min Throughput |            asc_sort_population |        4.12 |   ops/s |
|                                                Mean Throughput |            asc_sort_population |        4.12 |   ops/s |
|                                              Median Throughput |            asc_sort_population |        4.12 |   ops/s |
|                                                 Max Throughput |            asc_sort_population |        4.12 |   ops/s |
|                                       100th percentile latency |            asc_sort_population |     465.375 |      ms |
|                                  100th percentile service time |            asc_sort_population |      221.66 |      ms |
|                                                     error rate |            asc_sort_population |           0 |       % |
|                                                 Min Throughput | asc_sort_with_after_population |        3.19 |   ops/s |
|                                                Mean Throughput | asc_sort_with_after_population |        3.19 |   ops/s |
|                                              Median Throughput | asc_sort_with_after_population |        3.19 |   ops/s |
|                                                 Max Throughput | asc_sort_with_after_population |        3.19 |   ops/s |
|                                       100th percentile latency | asc_sort_with_after_population |     540.402 |      ms |
|                                  100th percentile service time | asc_sort_with_after_population |     225.843 |      ms |
|                                                     error rate | asc_sort_with_after_population |           0 |       % |
|                                                 Min Throughput |            desc_sort_geonameid |        4.01 |   ops/s |
|                                                Mean Throughput |            desc_sort_geonameid |        4.01 |   ops/s |
|                                              Median Throughput |            desc_sort_geonameid |        4.01 |   ops/s |
|                                                 Max Throughput |            desc_sort_geonameid |        4.01 |   ops/s |
|                                       100th percentile latency |            desc_sort_geonameid |     479.082 |      ms |
|                                  100th percentile service time |            desc_sort_geonameid |     228.808 |      ms |
|                                                     error rate |            desc_sort_geonameid |           0 |       % |
|                                                 Min Throughput | desc_sort_with_after_geonameid |        4.03 |   ops/s |
|                                                Mean Throughput | desc_sort_with_after_geonameid |        4.03 |   ops/s |
|                                              Median Throughput | desc_sort_with_after_geonameid |        4.03 |   ops/s |
|                                                 Max Throughput | desc_sort_with_after_geonameid |        4.03 |   ops/s |
|                                       100th percentile latency | desc_sort_with_after_geonameid |     471.864 |      ms |
|                                  100th percentile service time | desc_sort_with_after_geonameid |       222.8 |      ms |
|                                                     error rate | desc_sort_with_after_geonameid |           0 |       % |
|                                                 Min Throughput |             asc_sort_geonameid |        4.12 |   ops/s |
|                                                Mean Throughput |             asc_sort_geonameid |        4.12 |   ops/s |
|                                              Median Throughput |             asc_sort_geonameid |        4.12 |   ops/s |
|                                                 Max Throughput |             asc_sort_geonameid |        4.12 |   ops/s |
|                                       100th percentile latency |             asc_sort_geonameid |     463.471 |      ms |
|                                  100th percentile service time |             asc_sort_geonameid |     219.815 |      ms |
|                                                     error rate |             asc_sort_geonameid |           0 |       % |
|                                                 Min Throughput |  asc_sort_with_after_geonameid |        4.02 |   ops/s |
|                                                Mean Throughput |  asc_sort_with_after_geonameid |        4.02 |   ops/s |
|                                              Median Throughput |  asc_sort_with_after_geonameid |        4.02 |   ops/s |
|                                                 Max Throughput |  asc_sort_with_after_geonameid |        4.02 |   ops/s |
|                                       100th percentile latency |  asc_sort_with_after_geonameid |      465.72 |      ms |
|                                  100th percentile service time |  asc_sort_with_after_geonameid |     215.846 |      ms |
|                                                     error rate |  asc_sort_with_after_geonameid |           0 |       % |


--------------------------------
[INFO] SUCCESS (took 35 seconds)
--------------------------------

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[OVI3D0]

LGTM. Is it worth adding this shorthand to some of the tests?