Skip to content

Latest commit

 

History

History
479 lines (355 loc) · 12 KB

IDEAS.adoc

File metadata and controls

479 lines (355 loc) · 12 KB

Developer Notes

Table of Contents

1. [DONE] Virtual Nanosecond Clock

Rework the rate governor logic to use a virtual nanosecond clock. Each virtual clock tick should advance the virtual clock by

(1,000,000,000.0 / tps_rate)

nanoseconds.

2. [DONE] External Config via JSON File

config.json
{
    "json_filename": "json_results.txt",
    "contexts": [
        {
            "name": "Scenario-1",
            "num_users": 16,
            "num_threads": 2
        },
        {
            "name": "Scenario-2",
            "num_users": 32,
            "num_threads": 4
        }
    ],
    "benchmarks": [
        {
            "name": "Test0 (Initialize)",
            "enabled": true,
            "time": {
                "startup_duration": 0,
                "warmup_duration": 0,
                "main_duration": 0,
                "main_duration_repeat_count": 1
            },
            "throughput_rate": 0.0,
            "work_in_progress": 1,
            "actions": [
                {
                    "id": 0
                },
                {
                    "id": 7
                }
            ]
        },
        {
            "name": "Test1 (Throughput Test - Max)",
            "enabled": false,
            "time": {
                "startup_duration": 60,
                "warmup_duration": 60,
                "main_duration": 60,
                "main_duration_repeat_count": 1
            },
            "throughput_rate": 0.0,
            "work_in_progress": -1,
            "actions": [
                {
                    "id": 8
                }
            ]
        },
        {
            "name": "Test2 (Throughput Test - Fixed)",
            "enabled": true,
            "time": {
                "startup_duration": 15,
                "warmup_duration": 15,
                "main_duration": 60,
                "main_duration_repeat_count": 4
            },
            "throughput_rate": 100.0,
            "work_in_progress": 0,
            "actions": [
                {
                    "id": 1,
                    "weight": 25
                },
                {
                    "id": 2,
                    "weight": 75
                }
            ]
        }
    ]
}

3. [DONE] Runtime Configuration

Add --config parameter to specify which config.jsonc file to use.

4. [DONE] HttpClient vs OkHttp

  1. Use HttpClient from Java 21, and remove support for OkHttp

  2. Remove unused and optional JAR dependencies

    • http4k

    • …​.

5. Tulip Documentation

Create a user guide for Tulip with Antora

6. [DROPPED] Pkl Config Support

Write a config.pkl file to generate config.json

7. GraalVM native application

Build a native (exe) using GraalVM of a Tulip benchmark application

$ ./gradlew nativeCompile

$ ./build/native/nativeCompile/tulip -c ./config.jsonc

8. Docker Support

Create a Docker container of a Tulip benchmark application using Docker Compose

9. [DONE] Tulip Runtime Library - local Maven

Create a tulip-runtime-jvm.jar library and publish it to Maven local.

10. Tulip Runtime Library - Maven Central

Create a Maven Central hosted tulip-core.jar runtime library that can be imported by benchmark applications

<dependency>
    <groupId>io.github.wfouche</groupId>
    <artifactId>tulip-core</artifactId>
    <version>0.8.1</version>
</dependency>

11. [DONE] Java Benchmark Support

Allow benchmark user class to be written in Java or other JVM compatible languages. Add support for:

  • Kotlin

  • Java

12. [DROPPED] Reimplement JSON input (config.json)

Use Kotlin Serialization instead of GSON:

  • Support JSON5 format

  • Support GraalVM

13. [DROPPED] Reimplement JSON output

Re-implement how the json_results.txt file is created. Only use a hierarchy of data classes and GSON to create the JSON output, or kotlinx

import kotlinx.serialization.Serializable
@Serializable
data class Car(val type: String, @EncodeDefault val color: String = "Blue")

val car = Car("Ford")
val jsonString = Json.encodeToString(car)
assertEquals("{\"type\":\"Ford\",\"color\":\"Blue\"}", jsonString)

14. [DONE] Micrometer Support

15. [DONE] Amper Support

16. [DONE] Remove Glowroot support

Remove folder tulip/runtime/glowroot.

17. [DONE] Add user_actions and user_class to config.json

{
    "user_class": "user.UserHttp",
    "user_actions": {
        "0": "start",
        "1": "DELAY-6ms",
        "2": "DELAY-14ms",
        "3": "REST-posts",
        "4": "REST-comments",
        "5": "REST-albums",
        "6": "REST-photos",
        "7": "REST-todos",
        "8": "login",
        "99": "stop"
    }
}

18. [DONE] Add user_params to config.json

{
    ....
    "user_params": {
        "url": "https://jsonplaceholder.typicode.com",
        ....
    },
    ....
}

19. [DONE] Re-write tulip_user.py in Kotlin Script

  • tulip_user.py

  • tulip_user.kts

20. [DROPPED] Re-write json_print_asciidoc.py in Kotlin Script

  • json_print_asciidoc.py

  • json_print_asciidoc.kts

21. [DONE] Benchmark Summary

Display a summary of benchmark results at the end of the benchmark:

Benchmark1

  • Name

  • Average TPS

  • Average response time

  • 90th percentile

  • Max response time

  • Num-failed nnn (%xyz)

Benchmark2

  • Name

  • Average TPS

  • Average response time

  • 90th percentile

  • Max response time

  • Num-failed nnn (%xyz)

Benchmark…​

  • Name

  • Average TPS

  • Average response time

  • 90th percentile

  • Max response time

  • Num-failed nnn (%xyz)

22. [DROPPED] Performance Requirements

{
    "performance_requirements": {
        "avg-tps": "12 tps",
        "avg-tps-variance": "10 percent",
        ...
    }
}

23. [DROPPED] HdrHistogram

Use HdrHistogram to replace Tulip’s own log-linear quantization logic.

HdrHistogram is a standard used by several load testing tools.

///usr/bin/env jbang "$0" "$@" ; exit $?
//DEPS org.hdrhistogram:HdrHistogram:2.2.2

import org.HdrHistogram.Histogram;

import java.util.concurrent.ThreadLocalRandom;

public class test_hdrhistogram {
    public static void main(String[] args) {
        //Histogram histogram = new Histogram(3600*1000*1000L, 3);
        Histogram histogram = new Histogram(3);

        // 6 ms delay (average) with 25% of values
        for (int i=0; i != 250000; i++) {
            histogram.recordValue(ThreadLocalRandom.current().nextLong(12 + 1));
        }

        // 14 ms delay (average) with 75% of values
        for (int i=0; i != 750000; i++) {
            histogram.recordValue(ThreadLocalRandom.current().nextLong(28 + 1));
        }
        // histogram.getMean() = 12.0

        System.out.println(histogram.getTotalCount());
        histogram.outputPercentileDistribution(System.out,1.0);
        System.out.println(histogram.getMean());
        System.out.println(histogram.getStdDeviation());
        System.out.println(histogram.getMaxValue());
        System.out.println(histogram.getValueAtPercentile(50.0));
        System.out.println(histogram.getValueAtPercentile(90.0));
        System.out.println(histogram.getValueAtPercentile(95.0));
        System.out.println(histogram.getValueAtPercentile(99.0));
        System.out.println(histogram.getValueAtPercentile(99.9));
    }
}

24. [DONE] Summary Statistics

Implemented HTML reports: full and summary. See reports folder.

Hyperfoil as reference
PHASE    METRIC  THROUGHPUT    ACTIONS  MEAN      STD_DEV  p50       p90       p99       p99.9     MAX        SUCCESS   FAILED
example  test    29,41 req/s         1  17,37 ms     0 ms  17,43 ms  17,43 ms  17,43 ms  17,43 ms  17,43 ms         1        0

25. JCTools Concurrent Queues

Replace queues in JC queues.

26. [DONE] kscript

Remove kscript, kotlin 1.9.24 and use jbang

27. [DONE] Total Time Blocked

Add a counter that records the total time that the main thread is blocked waiting to assigned tasks to worker threads.

28. [DONE] Add histogram_rt data to results file

{
    "histogram_rt": {
      "10000": 839,
      "20000": 745,
      "3000": 284,
      "30000": 175,
      "15000": 792,
      "10": 95,
      "5000": 247,
      "25000": 815,
      "2000": 242,
      "15": 121,
      "8000": 259,
      "9000": 277,
      "7000": 247,
      "4000": 273,
      "6000": 261,
      "1000": 267,
      "5": 3,
      "50": 1,
      "20": 33,
      "6500": 1,
      "9": 12,
      "1500": 1,
      "8": 4,
      "6": 1,
      "7": 3,
      "9500": 3,
      "3500": 1,
      "25": 3,
      "2500": 1,
      "8500": 1,
      "4500": 1,
      "5500": 1,
      "7500": 1,
      "4": 1,
      "30": 1
    }
}

29. [DONE] Coordinaton Ommision Problem

Measure the wait time for each action, and add it to the service time.

30. Virtual Threads

Experiment with assigning one virtual thread to each user object when using Java 21 or above:

31. Glances

Measure resource utilization via Glances:

32. Grazie Pro

Use Grazie Pro to write better documentation:

35. w3m vs lynx

Currently using w3m to display HTML report to text console. Consider using lynx instead,

36. Dokka and Mermaid

Enable mermaid diagram plugin for Dokka