Skip to content

Commit abc2118

Browse files
KirillKurdyukovcursoragentclaude
authored
Expand SLO workload suite (#64)
Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 80d8013 commit abc2118

55 files changed

Lines changed: 2543 additions & 2527 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

pom.xml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,6 @@
3030
<module>url-shortener-demo</module>
3131
<module>jdbc</module>
3232
<module>project-course</module>
33-
<module>slo</module>
3433
<module>slo-workload</module>
3534
</modules>
3635

slo-workload/README.md

Lines changed: 29 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -5,14 +5,20 @@ reliability of YDB Java clients under load and chaos using the
55
[YDB SLO action](https://github.com/ydb-platform/ydb-slo-action).
66

77
Each submodule is a self-contained, runnable workload that follows the same
8-
contract as the SDK SLO workload in [`../slo`](../slo): it reads its
9-
configuration from environment variables, runs setup/run/teardown phases, and
10-
pushes OpenTelemetry (OTLP) metrics that the action scrapes and compares
11-
between the current PR run and a baseline run.
8+
contract: it reads its configuration from environment variables, runs
9+
setup/run/teardown phases, and pushes OpenTelemetry (OTLP) metrics that the
10+
action scrapes and compares between the current PR run and a baseline run.
11+
12+
Shared harness code lives in [`core`](core) (`Config`, `Metrics`, KV row
13+
model, rate-limited runner). Every workload plugs a `KvClient` adapter into
14+
that runner so all of them emit the same metric contract.
1215

1316
| Module | Component under test | Description |
1417
| --- | --- | --- |
18+
| [`query`](query) | `ydb-java-sdk` (query client) | Native SDK KV workload |
1519
| [`jdbc`](jdbc) | `ydb-jdbc-driver` | Plain JDBC KV workload (no framework) |
20+
| [`spring-data-jdbc`](spring-data-jdbc) | `ydb-jdbc-driver` + `spring-data-jdbc-ydb` + `spring-ydb-retry` | Spring Data JDBC KV workload |
21+
| [`spring-data-jpa`](spring-data-jpa) | `ydb-jdbc-driver` + Hibernate 6 + `spring-ydb-retry` | Spring Data JPA KV workload |
1622

1723
## How a workload behaves
1824

@@ -76,11 +82,27 @@ KV tunables are passed on the command line and parsed by JCommander:
7682
--partition-size <int> Auto-partitioning partition size in MB (default 1)
7783
--min-partition-count <int> Minimum number of table partitions (default 6)
7884
--max-partition-count <int> Maximum number of table partitions (default 1000)
79-
--duration <int> Override WORKLOAD_DURATION when > 0
85+
--duration / --time <int> Override WORKLOAD_DURATION when > 0
86+
--shutdown-time <int> Extra grace seconds for in-flight ops on shutdown (default 30)
87+
--max-attempts <int> Per-operation attempt cap, initial + retries (default 10)
88+
--max-workers <int> Hard cap on workers per operation type (default 64)
8089
```
8190

82-
Unknown flags are ignored, so a workload accepts command strings designed for
83-
other SDKs without erroring.
91+
Unknown flags are rejected — a typo in the ydb-slo-action invocation should
92+
fail loudly rather than silently fall back to defaults.
93+
94+
The Spring-backed workloads expose one more knob via the `SLO_HIKARI_POOL_SIZE`
95+
environment variable (default `130`, sized for `2 × max-workers` plus headroom).
96+
Raise it together with `--max-workers` or the workload measures Hikari
97+
contention rather than the JDBC driver.
98+
99+
### Cross-implementation comparability
100+
101+
Every implementation derives the primary-key `hash` column from `id` with the
102+
same client-side mix (`RowGenerator.numericHash`). A table written by the
103+
`query` workload is therefore byte-compatible with the `jdbc` and Spring-Data
104+
workloads — useful when one prefills the table and another reads from it
105+
during cross-driver experiments.
84106

85107
## How CI uses this module
86108

slo/pom.xml renamed to slo-workload/core/pom.xml

Lines changed: 19 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -6,83 +6,66 @@
66

77
<parent>
88
<groupId>tech.ydb.examples</groupId>
9-
<artifactId>ydb-sdk-examples</artifactId>
9+
<artifactId>slo-workload</artifactId>
1010
<version>1.1.0-SNAPSHOT</version>
11+
<relativePath>../pom.xml</relativePath>
1112
</parent>
1213

13-
<artifactId>ydb-slo-workload</artifactId>
14-
<name>YDB SLO workload</name>
15-
<description>SLO workload application for testing YDB Java SDK reliability under load and chaos</description>
16-
17-
<properties>
18-
<jcommander.version>1.82</jcommander.version>
19-
<opentelemetry.version>1.59.0</opentelemetry.version>
20-
<hdrhistogram.version>2.2.2</hdrhistogram.version>
21-
</properties>
14+
<artifactId>slo-workload-core</artifactId>
15+
<name>YDB SLO workload core</name>
16+
<description>
17+
Driver-agnostic core of the YDB SLO workloads: OTLP metrics, env config,
18+
the KV row model and the load-generating runner. Every concrete workload
19+
(native query client, plain JDBC, Spring Data) plugs a KvClient into this
20+
runner so all of them emit the exact same metric contract.
21+
</description>
2222

2323
<dependencies>
24-
<dependency>
25-
<groupId>tech.ydb</groupId>
26-
<artifactId>ydb-sdk-query</artifactId>
27-
</dependency>
28-
2924
<dependency>
3025
<groupId>com.beust</groupId>
3126
<artifactId>jcommander</artifactId>
32-
<version>${jcommander.version}</version>
3327
</dependency>
3428

3529
<dependency>
3630
<groupId>org.hdrhistogram</groupId>
3731
<artifactId>HdrHistogram</artifactId>
38-
<version>${hdrhistogram.version}</version>
32+
</dependency>
33+
34+
<dependency>
35+
<groupId>com.google.guava</groupId>
36+
<artifactId>guava</artifactId>
3937
</dependency>
4038

4139
<dependency>
4240
<groupId>io.opentelemetry</groupId>
4341
<artifactId>opentelemetry-api</artifactId>
44-
<version>${opentelemetry.version}</version>
4542
</dependency>
4643
<dependency>
4744
<groupId>io.opentelemetry</groupId>
4845
<artifactId>opentelemetry-sdk</artifactId>
49-
<version>${opentelemetry.version}</version>
5046
</dependency>
5147
<dependency>
5248
<groupId>io.opentelemetry</groupId>
5349
<artifactId>opentelemetry-sdk-metrics</artifactId>
54-
<version>${opentelemetry.version}</version>
5550
</dependency>
5651
<dependency>
5752
<groupId>io.opentelemetry</groupId>
5853
<artifactId>opentelemetry-exporter-otlp</artifactId>
59-
<version>${opentelemetry.version}</version>
6054
</dependency>
6155

6256
<dependency>
63-
<groupId>org.apache.logging.log4j</groupId>
64-
<artifactId>log4j-slf4j2-impl</artifactId>
57+
<groupId>org.slf4j</groupId>
58+
<artifactId>slf4j-api</artifactId>
6559
</dependency>
6660
</dependencies>
6761

6862
<build>
69-
<finalName>ydb-slo-workload</finalName>
7063
<plugins>
7164
<plugin>
7265
<groupId>org.apache.maven.plugins</groupId>
73-
<artifactId>maven-dependency-plugin</artifactId>
74-
</plugin>
75-
<plugin>
76-
<groupId>org.apache.maven.plugins</groupId>
77-
<artifactId>maven-jar-plugin</artifactId>
66+
<artifactId>maven-compiler-plugin</artifactId>
7867
<configuration>
79-
<archive>
80-
<manifest>
81-
<addClasspath>true</addClasspath>
82-
<classpathPrefix>libs/</classpathPrefix>
83-
<mainClass>tech.ydb.slo.Main</mainClass>
84-
</manifest>
85-
</archive>
68+
<release>${maven.compiler.release}</release>
8669
</configuration>
8770
</plugin>
8871
</plugins>

slo/src/main/java/tech/ydb/slo/Config.java renamed to slo-workload/core/src/main/java/tech/ydb/slo/core/Config.java

Lines changed: 55 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,8 @@
1-
package tech.ydb.slo;
2-
3-
/**
4-
* Configuration for the SLO workload, populated from environment variables
5-
* provided by the YDB SLO action runtime.
6-
*
7-
* <p>The action sets these variables on the workload container:
8-
* <ul>
9-
* <li>{@code YDB_CONNECTION_STRING} or {@code YDB_ENDPOINT} + {@code YDB_DATABASE} — YDB connection</li>
10-
* <li>{@code WORKLOAD_REF} — value used as the {@code ref} label on all metrics</li>
11-
* <li>{@code WORKLOAD_NAME} — workload name (also used as part of the table path)</li>
12-
* <li>{@code WORKLOAD_DURATION} — workload run duration in seconds (0 = unlimited)</li>
13-
* <li>{@code OTEL_EXPORTER_OTLP_ENDPOINT} — OTLP endpoint for pushing metrics</li>
14-
* </ul>
15-
*/
1+
package tech.ydb.slo.core;
2+
163
public final class Config {
174
private final String connectionString;
5+
private final String jdbcUrl;
186
private final String token;
197
private final String ref;
208
private final String workloadName;
@@ -23,24 +11,34 @@ public final class Config {
2311

2412
private Config(
2513
String connectionString,
14+
String jdbcUrl,
2615
String token,
2716
String ref,
2817
String workloadName,
2918
int durationSeconds,
3019
String otlpEndpoint
3120
) {
3221
this.connectionString = connectionString;
22+
this.jdbcUrl = jdbcUrl;
3323
this.token = token;
3424
this.ref = ref;
3525
this.workloadName = workloadName;
3626
this.durationSeconds = durationSeconds;
3727
this.otlpEndpoint = otlpEndpoint;
3828
}
3929

30+
31+
4032
public String connectionString() {
4133
return connectionString;
4234
}
4335

36+
37+
38+
public String jdbcUrl() {
39+
return jdbcUrl;
40+
}
41+
4442
public String token() {
4543
return token;
4644
}
@@ -61,43 +59,72 @@ public String otlpEndpoint() {
6159
return otlpEndpoint;
6260
}
6361

64-
/**
65-
* Loads configuration from environment variables.
66-
*
67-
* @return configuration instance
68-
* @throws IllegalStateException if required variables are missing or invalid
69-
*/
70-
public static Config fromEnv() {
62+
63+
64+
public static Config fromEnv(String defaultWorkloadName) {
7165
String connectionString = resolveConnectionString();
7266
if (connectionString == null || connectionString.isEmpty()) {
7367
throw new IllegalStateException(
74-
"YDB connection is not configured: set YDB_CONNECTION_STRING or YDB_ENDPOINT + YDB_DATABASE"
68+
"YDB connection is not configured: set YDB_CONNECTION_STRING, "
69+
+ "YDB_JDBC_URL or YDB_ENDPOINT + YDB_DATABASE"
7570
);
7671
}
7772

7873
String token = envOrDefault("YDB_TOKEN", "");
7974
String ref = envOrDefault("WORKLOAD_REF", "unknown");
80-
String workloadName = envOrDefault("WORKLOAD_NAME", "java-slo-workload");
75+
String workloadName = envOrDefault("WORKLOAD_NAME", defaultWorkloadName);
8176
int durationSeconds = parseInt(envOrDefault("WORKLOAD_DURATION", "600"), 600);
8277
String otlpEndpoint = envOrDefault("OTEL_EXPORTER_OTLP_ENDPOINT", "");
8378

84-
return new Config(connectionString, token, ref, workloadName, durationSeconds, otlpEndpoint);
79+
return new Config(
80+
connectionString,
81+
toJdbcUrl(connectionString),
82+
token,
83+
ref,
84+
workloadName,
85+
durationSeconds,
86+
otlpEndpoint
87+
);
8588
}
8689

90+
91+
8792
private static String resolveConnectionString() {
93+
String jdbc = System.getenv("YDB_JDBC_URL");
94+
if (jdbc != null && !jdbc.isEmpty()) {
95+
return stripJdbcPrefix(jdbc);
96+
}
97+
8898
String cs = System.getenv("YDB_CONNECTION_STRING");
8999
if (cs != null && !cs.isEmpty()) {
90-
return cs;
100+
return stripJdbcPrefix(cs);
91101
}
92102

93103
String endpoint = System.getenv("YDB_ENDPOINT");
94104
String database = System.getenv("YDB_DATABASE");
95105
if (endpoint == null || endpoint.isEmpty() || database == null || database.isEmpty()) {
96106
return null;
97107
}
108+
return composeConnectionString(endpoint, database);
109+
}
110+
111+
private static String stripJdbcPrefix(String value) {
112+
if (value.startsWith("jdbc:ydb:")) {
113+
return value.substring("jdbc:ydb:".length());
114+
}
115+
return value;
116+
}
117+
118+
119+
120+
private static String toJdbcUrl(String connectionString) {
121+
if (connectionString.startsWith("jdbc:")) {
122+
return connectionString;
123+
}
124+
return "jdbc:ydb:" + connectionString;
125+
}
98126

99-
// Compose connection string in the form expected by GrpcTransport.forConnectionString:
100-
// grpc://host:port/database
127+
private static String composeConnectionString(String endpoint, String database) {
101128
if (endpoint.endsWith("/") && database.startsWith("/")) {
102129
return endpoint + database.substring(1);
103130
}

0 commit comments

Comments
 (0)