feat: Integration test for End to End tracing #3691

manu2 · 2025-03-18T08:53:42Z

With End to End tracing, customers are able to view server side traces. The new integration tests aims to verify this from a client side pov.
Note: Currently the InegrationTestEnv sets up default interceptor which in turns creates a default GlobalOpenTelementry sdk. This sdk is then injected to the gRPC connection builder and stud. So a new opentelemetry sdk has been created in IntegrationTestEnv that sets up the required config for End to End Testing and will be used by default for all integration tests.

…on Test only.

manu2 · 2025-03-19T04:31:04Z

Looking through the details of the failure in Kokoro - Test: Integration with Directpath using the following target log

Many of the tests seem to be facing "com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: The caller does not have permission" when on trace exporter.
However in case of ITEndToEndTracingTest, we have a assertion on whether the traces are present in cloud trace, which is causing an assertion failure.

So the underlying issue regarding permission denied for trace export is not related to this change but is causing failure in the new test introduced.

…rect path build

sakthivelmanii · 2025-03-19T06:04:22Z

google-cloud-spanner/src/test/java/com/google/cloud/spanner/IntegrationTestEnv.java

+            traceConfigurationBuilder.setProjectId(options.getProjectId()).build());
+
+    String serviceName =
+        "java-spanner-jdbc-integration-tests-" + ThreadLocalRandom.current().nextInt();


Suggested change

"java-spanner-jdbc-integration-tests-" + ThreadLocalRandom.current().nextInt();

"java-spanner-integration-tests-" + ThreadLocalRandom.current().nextInt();

sakthivelmanii · 2025-03-19T06:10:06Z

google-cloud-spanner/src/test/java/com/google/cloud/spanner/it/ITEndToEndTracingTest.java

+
+  static {
+    SpannerOptionsHelper.resetActiveTracingFramework();
+    SpannerOptions.enableOpenTelemetryMetrics();


why do we need to enable metrics?

sakthivelmanii · 2025-03-19T06:15:13Z

google-cloud-spanner/pom.xml

+    </dependency>
+    <dependency>
+      <groupId>com.google.api.grpc</groupId>
+      <artifactId>proto-google-cloud-trace-v1</artifactId>


why are we adding this?

We query trace using trace client which requires this dependency.
" trace = client.getTrace(env.getTestHelper().getInstanceId().getProject(), traceId)"

sakthivelmanii · 2025-03-19T06:17:48Z

google-cloud-spanner/src/test/java/com/google/cloud/spanner/it/ITEndToEndTracingTest.java

+
+  @Before
+  public void initSelectValueQuery() {
+    selectValueQuery = "SELECT @p1 + @p1 ";


can you initialize selectValueQuery as a static variable?

private static final String selectValueQuery = "SELECT @p1 + @p1";

sakthivelmanii · 2025-03-19T06:21:05Z

google-cloud-spanner/src/test/java/com/google/cloud/spanner/it/ITEndToEndTracingTest.java

+                .build();
+    try (TraceServiceClient client = TraceServiceClient.create(settings)) {
+      // It can take a few seconds before the trace is visible.
+      Thread.sleep(5000L);


Is there any better way to know that trace is exported? I just wanted to make sure this is not flakky

Do a forceflush(), instead of adding a sleep.

sdkTracerProvider.forceFlush();

we need to wait for sever side traces which are collected from request on Span FE and asynchronously exported later.

surbhigarg92 · 2025-03-19T07:01:44Z

google-cloud-spanner/src/test/java/com/google/cloud/spanner/it/ITEndToEndTracingTest.java

+              clientTrace.getSpansList().stream()
+                  .anyMatch(
+                      span ->
+                          "CloudSpannerOperation.ExecuteStreamingQuery".equals(span.getName())));


We are only asserting on Client Trace and not on server trace?

surbhigarg92 · 2025-03-19T07:05:14Z

google-cloud-spanner/src/test/java/com/google/cloud/spanner/it/ITEndToEndTracingTest.java

+                .build();
+    try (TraceServiceClient client = TraceServiceClient.create(settings)) {
+      // It can take a few seconds before the trace is visible.
+      Thread.sleep(5000L);


Do a forceflush(), instead of adding a sleep.

sdkTracerProvider.forceFlush();

surbhigarg92 · 2025-03-19T07:08:20Z

google-cloud-spanner/src/test/java/com/google/cloud/spanner/IntegrationTestEnv.java

+        OpenTelemetrySdk.builder()
+            .setTracerProvider(sdkTracerProvider)
+            .setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
+            .buildAndRegisterGlobal();


Do not register the tracerProvider at global, instead inject OpenTelemerty object via spanner options.

If we register it globally, we might start seeing traces coming from other parallel integration tests as enableOpenTelemetryTraces is a static method.

google-cloud-spanner/src/test/java/com/google/cloud/spanner/IntegrationTestEnv.java

surbhigarg92 · 2025-03-19T07:44:24Z

google-cloud-spanner/src/test/java/com/google/cloud/spanner/it/ITEndToEndTracingTest.java

+      boolean foundTrace = false;
+      for (int attempts = 0; attempts < 2; attempts++) {
+        try {
+          Trace clientTrace =


Create the traceClient in "before" block

TraceClient is created before the block. This is actually the trace, not the client. Updated naming in recent commit to avoid confusion.

surbhigarg92 · 2025-03-19T07:45:02Z

@manu2 I don't understand the need for this Integration test. Do we want to verify server traces as I don't see that in the assert?
Client traces could be verified with InMemoryExporter

manu2 · 2025-03-25T15:31:14Z

@manu2 I don't understand the need for this Integration test. Do we want to verify server traces as I don't see that in the assert? Client traces could be verified with InMemoryExporter

The test intents to verify the generation of server side traces when end to end tracing is enabled. Have updated the assertion to validate server side trace.

surbhigarg92 · 2025-03-26T10:07:16Z

google-cloud-spanner/src/test/java/com/google/cloud/spanner/it/ITEndToEndTracingTest.java

+        try {
+          Trace trace = client.getTrace(env.getTestHelper().getInstanceId().getProject(), traceId);
+          // Assert Spanner Frontend Trace is present
+          assertTrue(


assertTrue will result into error if the trace is not found and the test will be marked as failed, so the for loop will never actually work. Instead of this , we could try something like below and also remove the 10 second sleep before the loop.

Trace trace = client.getTrace(env.getTestHelper().getInstanceId().getProject(), traceId); while (trace.getSpansList().stream() .anyMatch(span -> "Spanner.ExecuteStreamingSql".equals(span.getName()) && metricsPollingStopwatch.elapsed(TimeUnit.SECONDS) < 30) { // Try every 5 seconds Thread.sleep(5000); } assertTrue(trace.getSpansList().stream() .anyMatch(span -> "Spanner.ExecuteStreamingSql".equals(span.getName());

Makes sense. If the getTrace calls fails, it throws an APIException. Have kept the catch block to handle it but changed the logic to poll upto 30 sec and sleep for 5 sec if not found.

(cherry picked from commit f47f01a834c80fadf754ed1ceee5179c1bd00deb)

surbhigarg92 · 2025-03-28T07:27:10Z

google-cloud-spanner/pom.xml

+    <dependency>
+      <groupId>com.google.cloud.opentelemetry</groupId>
+      <artifactId>exporter-trace</artifactId>
+      <version>0.33.0</version>


Please remove the version from here. It should either fetch from bom or the latest version

…dency" This reverts commit 66856a8.

| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [com.google.cloud:google-cloud-spanner](https://github.com/googleapis/java-spanner) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `6.89.0` -> `6.90.0` | --- ### Release Notes <details> <summary>googleapis/java-spanner (com.google.cloud:google-cloud-spanner)</summary> ### [`v6.90.0`](https://github.com/googleapis/java-spanner/blob/HEAD/CHANGELOG.md#6900-2025-03-31) ##### Features - Add default_isolation_level connection property ([#3702](googleapis/java-spanner#3702)) ([9472d23](googleapis/java-spanner@9472d23)) - Adds support for Interval datatype in Java client ([#3416](googleapis/java-spanner#3416)) ([8be8f5e](googleapis/java-spanner@8be8f5e)) - Integration test for End to End tracing ([#3691](googleapis/java-spanner#3691)) ([bf1a07a](googleapis/java-spanner@bf1a07a)) - Specify isolation level per transaction ([#3704](googleapis/java-spanner#3704)) ([868f30f](googleapis/java-spanner@868f30f)) - Support PostgreSQL isolation level statements ([#3706](googleapis/java-spanner#3706)) ([dda2e1d](googleapis/java-spanner@dda2e1d)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: a63885015974acb6061bae5ca3b9d61136991376

manu2 and others added 5 commits March 12, 2025 09:19

Enable End to End tracing in Integration Test

6d54c9d

Integration test for End to End tracing

a0806ee

Integration test for End to End tracing

00a1735

Merge branch 'googleapis:main' into main

a45ca7e

Integration test for End to End tracing

680ce7f

manu2 requested review from a team as code owners March 18, 2025 08:53

product-auto-label bot added size: m Pull request size is medium. api: spanner Issues related to the googleapis/java-spanner API. labels Mar 18, 2025

manu2 added 3 commits March 18, 2025 09:03

Integration test for End to End tracing

11baac3

Fix errors in Integration test for End to End tracing

8f12e88

Fix errors in Integration test for End to End tracing

8bc559d

manu2 changed the title ~~Integration test for End to End tracing~~ feat: Integration test for End to End tracing Mar 18, 2025

Add TestEnvOptions to enable End to End tracing for the new Integrati…

9f81dbb

…on Test only.

product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Mar 18, 2025

Fix ITEndToEndTracingTest error on direct path build

19eee1c

product-auto-label bot added size: m Pull request size is medium. and removed size: l Pull request size is large. labels Mar 18, 2025

Skip ITEndToEndTracingTest on permission denied error for error in di…

646a6b9

…rect path build

product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Mar 19, 2025

manu2 requested review from sakthivelmanii and olavloite March 19, 2025 05:29

sakthivelmanii reviewed Mar 19, 2025

View reviewed changes

surbhigarg92 reviewed Mar 19, 2025

View reviewed changes

Assert server side span name

d752823

manu2 added 2 commits March 25, 2025 09:23

Increase delay before trace fetch.

74b901d

Remove update to GlobalTelemetry. Use opentelemetrysdk created in test.

9c1cd08

product-auto-label bot added size: m Pull request size is medium. and removed size: l Pull request size is large. labels Mar 25, 2025

Merge branch 'googleapis:main' into main

8ec9109

surbhigarg92 reviewed Mar 26, 2025

View reviewed changes

manu2 and others added 2 commits March 27, 2025 10:04

Update retry logic for trace retrieval.

eb25e67

(cherry picked from commit f47f01a834c80fadf754ed1ceee5179c1bd00deb)

Merge branch 'main' into main

0611d07

surbhigarg92 approved these changes Mar 28, 2025

View reviewed changes

surbhigarg92 reviewed Mar 28, 2025

View reviewed changes

manu2 and others added 4 commits March 28, 2025 07:44

Remove version number from opentelemetry exporter-trace dependency

66856a8

Merge branch 'googleapis:main' into main

9f18bf7

Revert "Remove version number from opentelemetry exporter-trace depen…

e7f3109

…dency" This reverts commit 66856a8.

Merge branch 'googleapis:main' into main

7e42c8c

sakthivelmanii approved these changes Mar 28, 2025

View reviewed changes

manu2 merged commit bf1a07a into googleapis:main Mar 28, 2025
31 of 32 checks passed

release-please bot mentioned this pull request Mar 28, 2025

chore(main): release 6.90.0 #3714

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Integration test for End to End tracing #3691

feat: Integration test for End to End tracing #3691

manu2 commented Mar 18, 2025

manu2 commented Mar 19, 2025

sakthivelmanii Mar 19, 2025

manu2 Mar 25, 2025

sakthivelmanii Mar 19, 2025

manu2 Mar 25, 2025

sakthivelmanii Mar 19, 2025

manu2 Mar 28, 2025

sakthivelmanii Mar 19, 2025

manu2 Mar 25, 2025

sakthivelmanii Mar 19, 2025

surbhigarg92 Mar 19, 2025

manu2 Mar 25, 2025

surbhigarg92 Mar 19, 2025

manu2 Mar 25, 2025

surbhigarg92 Mar 19, 2025

surbhigarg92 Mar 19, 2025

manu2 Mar 25, 2025

surbhigarg92 Mar 19, 2025

manu2 Mar 25, 2025

surbhigarg92 commented Mar 19, 2025

manu2 commented Mar 25, 2025

surbhigarg92 Mar 26, 2025

manu2 Mar 27, 2025

surbhigarg92 Mar 28, 2025

	"java-spanner-jdbc-integration-tests-" + ThreadLocalRandom.current().nextInt();
	"java-spanner-integration-tests-" + ThreadLocalRandom.current().nextInt();

feat: Integration test for End to End tracing #3691

feat: Integration test for End to End tracing #3691

Conversation

manu2 commented Mar 18, 2025

manu2 commented Mar 19, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

surbhigarg92 commented Mar 19, 2025

manu2 commented Mar 25, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment