[docker] Add fast client support to Docker quickstart #2466

sushantmane · 2026-02-09T05:48:03Z

Summary

Adds Venice fast client (D2-based) support to the Docker quickstart, which directly reads data from storage servers bypassing the router. This addresses issue #2150 (D2Server integration).

D2 announcement: Router now announces itself and the server to ZooKeeper for D2 service discovery, gated by router.d2.announce.enabled property (default false)
D2ConfigUtils: New production-ready utility extracted from test D2TestUtils for setting up D2 config and server announcers
FastClientQueryTool: CLI tool for querying stores via the fast client with D2 (java -jar venice-client-all.jar <store> <key> <zk_address>)
SSL scheme fix: ServerReadMetadataRepository now uses the server's actual SSL configuration for metadata response URLs instead of hardcoding HTTPS. Added HelixUtils.instanceIdToUrl(instanceId, https) overload
Docker integration: Added fast-client-fetch.sh script, shadow JAR build for venice-client, and updated Dockerfile/build script

Test plan

Unit tests pass: HelixUtilsTest, ServerReadMetadataRepositoryTest (3/3 tests)
Gradle build succeeds for all modified modules
Docker quickstart end-to-end verification:
- All containers start healthy
- Store created, data pushed (100 records)
- Thin client reads still work (fetch.sh)
- Fast client reads work for multiple keys: 1→val1, 25→val25, 50→val50, 100→val100
- Non-existent key returns null: 999→null

Closes #2150

Copilot

Pull request overview

Adds D2-based Venice fast client support to the Docker quickstart so users can query storage nodes directly (bypassing the router), and fixes server metadata URL scheme generation to respect the server’s actual SSL configuration.

Changes:

Add D2 announcement support (router + optional server URI) for quickstart D2 service discovery.
Introduce D2ConfigUtils (prod utility) and FastClientQueryTool (CLI) plus build/docker packaging for a venice-client-all.jar.
Fix server-based metadata to return http:// vs https:// URLs based on server SSL enablement.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
services/venice-router/src/main/java/com/linkedin/venice/router/RouterServer.java	Adds D2 config setup + service discovery announcers during router startup
internal/venice-common/src/main/java/com/linkedin/venice/d2/D2ConfigUtils.java	New utility to configure D2 in ZK and create announcers
clients/venice-client/src/main/java/com/linkedin/venice/fastclient/FastClientQueryTool.java	New CLI for fast-client reads using D2
clients/venice-client/build.gradle	Adds shadow JAR config + sets CLI main class manifest
services/venice-server/src/main/java/com/linkedin/venice/server/VeniceServer.java	Passes SSL-enabled flag to server metadata repository
services/venice-server/src/main/java/com/linkedin/venice/listener/ServerReadMetadataRepository.java	Uses SSL-enabled flag to generate correct routing/Helix URLs
internal/venice-common/src/main/java/com/linkedin/venice/utils/HelixUtils.java	Adds `instanceIdToUrl(instanceId, https)` overload
docker/venice-router/single-dc-configs/router.properties	Enables D2 announcement + adds server D2 announce config for quickstart
docker/venice-client/fast-client-fetch.sh	New quickstart helper script to run the fast-client CLI
docker/venice-client/Dockerfile	Packages the new `venice-client-all.jar` + helper script into the client container
docker/build-venice-docker-images.sh	Copies/removes the new client fat JAR into docker build context

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docker/venice-client/fast-client-fetch.sh

services/venice-router/src/main/java/com/linkedin/venice/router/RouterServer.java

clients/venice-client/src/main/java/com/linkedin/venice/fastclient/FastClientQueryTool.java

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

services/venice-router/src/main/java/com/linkedin/venice/router/RouterServer.java

clients/venice-client/src/main/java/com/linkedin/venice/fastclient/FastClientQueryTool.java

internal/venice-common/src/main/java/com/linkedin/venice/d2/D2ConfigUtils.java

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

clients/venice-client/src/main/java/com/linkedin/venice/fastclient/FastClientQueryTool.java

internal/venice-common/src/main/java/com/linkedin/venice/d2/D2ConfigUtils.java

docker/venice-client/fast-client-fetch.sh

services/venice-router/src/main/java/com/linkedin/venice/router/RouterServer.java

services/venice-server/src/main/java/com/linkedin/venice/server/VeniceServer.java

...s/venice-server/src/main/java/com/linkedin/venice/listener/ServerReadMetadataRepository.java

Copilot

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

services/venice-router/src/main/java/com/linkedin/venice/router/RouterServer.java

Copilot

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

clients/venice-client/src/main/java/com/linkedin/venice/fastclient/FastClientQueryTool.java

services/venice-server/src/main/java/com/linkedin/venice/server/VeniceServer.java

services/venice-router/src/main/java/com/linkedin/venice/router/RouterServer.java

Add D2-based Venice fast client to the Docker quickstart, enabling direct data reads to storage servers bypassing the router. Changes: - Add D2ConfigUtils: production-ready D2 setup utility extracted from test code (setupD2Config, createD2Server, getD2Servers) - Add D2 self-announcement to RouterServer.main() for cluster discovery services, gated by router.d2.announce.enabled (default false) - Add D2 self-announcement to VeniceServer.run() for server D2 service, gated by server.d2.announce.enabled (default false) - Add FastClientQueryTool: CLI for fast-client reads using D2, with optional --insecure flag for trust-all SSL - Add shadow JAR config to venice-client build.gradle - Fix ServerReadMetadataRepository to respect server SSL config when generating instance URLs (was hardcoded to HTTPS) - Add HelixUtils.instanceIdToUrl(instanceId, https) overload - Add fast-client-fetch.sh wrapper script and Docker packaging - Update quickstart docs with fast client usage (Option B) - Add unit tests for HelixUtils, FastClientQueryTool, and ServerReadMetadataRepository SSL-aware URL generation

xunyin8

Overall looks good, just some NIT.

xunyin8 · 2026-02-10T20:58:49Z

docker/venice-controller/single-dc-configs/controller.properties

 controller.parent.mode=false
 controller.system.schema.cluster.name=venice-cluster0
 cluster.to.d2=venice-cluster0:venice-discovery
+cluster.to.server.d2=venice-cluster0:venice-server-d2


Should we include the cluster name in the corresponding server d2 too? e.g. venice-cluster0:venice-server-cluster0-d2. Otherwise it's going to be confusing when we try to run with multiple clusters.

xunyin8 · 2026-02-10T23:08:01Z

services/venice-router/src/main/java/com/linkedin/venice/router/RouterServer.java

+        d2Servers.addAll(D2ConfigUtils.getD2Servers(zkAddress, d2ClusterName, localUri));
+      }
+
+      // Announce the global discovery service only if not already covered by cluster.to.d2


This is confusing, why would cluster.to.d2 ever have DEFAULT_CLUSTER_DISCOVERY_D2_SERVICE_NAME as a value? The intention of cluster.to.d2 is a mapping of venice cluster to corresponding d2 service/cluster name.

Copilot AI review requested due to automatic review settings February 9, 2026 05:48

Copilot started reviewing on behalf of sushantmane February 9, 2026 05:48 View session

sushantmane requested a review from xunyin8 February 9, 2026 05:50

Copilot AI reviewed Feb 9, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings February 9, 2026 06:10

sushantmane force-pushed the fast-client-docker-quickstart branch from e6e2ee3 to 1ef6f9a Compare February 9, 2026 06:10

Copilot started reviewing on behalf of sushantmane February 9, 2026 06:10 View session

Copilot AI reviewed Feb 9, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings February 9, 2026 06:33

Copilot started reviewing on behalf of sushantmane February 9, 2026 06:34 View session

sushantmane force-pushed the fast-client-docker-quickstart branch from 25338e4 to f890481 Compare February 9, 2026 06:35

Copilot AI reviewed Feb 9, 2026

View reviewed changes

sushantmane force-pushed the fast-client-docker-quickstart branch from f890481 to 25a354b Compare February 9, 2026 07:17

Copilot AI review requested due to automatic review settings February 9, 2026 09:06

sushantmane force-pushed the fast-client-docker-quickstart branch from 25a354b to 6cdae17 Compare February 9, 2026 09:06

Copilot started reviewing on behalf of sushantmane February 9, 2026 09:06 View session

Copilot AI reviewed Feb 9, 2026

View reviewed changes

services/venice-router/src/main/java/com/linkedin/venice/router/RouterServer.java Show resolved Hide resolved

sushantmane force-pushed the fast-client-docker-quickstart branch from 6cdae17 to 73043ab Compare February 9, 2026 09:58

Copilot AI review requested due to automatic review settings February 9, 2026 10:14

sushantmane force-pushed the fast-client-docker-quickstart branch from 73043ab to dfa2ee0 Compare February 9, 2026 10:14

Copilot started reviewing on behalf of sushantmane February 9, 2026 10:14 View session

Copilot AI reviewed Feb 9, 2026

View reviewed changes

sushantmane force-pushed the fast-client-docker-quickstart branch from dfa2ee0 to 450517e Compare February 9, 2026 18:25

xunyin8 reviewed Feb 10, 2026

View reviewed changes

[docker] Add fast client support to Docker quickstart #2466

Are you sure you want to change the base?

[docker] Add fast client support to Docker quickstart #2466

Conversation

sushantmane commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xunyin8 left a comment

Choose a reason for hiding this comment

Uh oh!

xunyin8 Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

xunyin8 Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sushantmane commented Feb 9, 2026 •

edited

Loading