Skip to content

implementation to get dynamic config during runtime #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

implementation to get dynamic config during runtime #7

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6ccd439
implementing to get dynamic config during runtime
shashank11p Apr 14, 2021
e886774
adding test
shashank11p Apr 15, 2021
1212cbf
updating dependency
shashank11p Apr 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion javaagent-core/build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import com.google.protobuf.gradle.*
plugins {
`java-library`
idea
id("com.google.protobuf") version "0.8.13"
id("com.google.protobuf") version "0.8.15"
id("org.hypertrace.publish-maven-central-plugin")
}

Expand All @@ -12,7 +12,18 @@ protobuf {
// The artifact spec for the Protobuf Compiler
artifact = "com.google.protobuf:protoc:3.11.4"
}
plugins {
id("grpc") {
artifact = "io.grpc:protoc-gen-grpc-java:1.37.0"
}
}
generateProtoTasks {
ofSourceSet("main").forEach {
it.plugins {
// Apply the "grpc" plugin whose spec is defined above, without options.
id("grpc")
}
}
}
}

Expand All @@ -31,6 +42,13 @@ dependencies {

api("com.google.protobuf:protobuf-java:3.11.4")
api("com.google.protobuf:protobuf-java-util:3.11.4")
api("io.grpc:grpc-stub:1.37.0")
api("io.grpc:grpc-protobuf:1.37.0")
if (JavaVersion.current().isJava9Compatible) {
// Workaround for @javax.annotation.Generated
// see: https://github.com/grpc/grpc-java/issues/3633
api("javax.annotation:javax.annotation-api:1.3.1")
}
// convert yaml to json, since java protobuf impl supports only json
implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.11.3")

Expand Down
1 change: 1 addition & 0 deletions javaagent-core/src/main/java/org/hypertrace/agent/core/config/EnvironmentConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ private EnvironmentConfig() {}
private static final String HT_PREFIX = "ht.";

public static final String CONFIG_FILE_PROPERTY = HT_PREFIX + "config.file";
public static final String DYNAMIC_CONFIG_SERVICE_URL = HT_PREFIX + "dynamic.config.service.url";
static final String SERVICE_NAME = HT_PREFIX + "service.name";
static final String ENABLED = HT_PREFIX + "enabled";
static final String RESOURCE_ATTRIBUTES = HT_PREFIX + ".resource.attributes";
Expand Down
70 changes: 64 additions & 6 deletions javaagent-core/src/main/java/org/hypertrace/agent/core/config/HypertraceConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,20 @@
import com.google.common.annotations.VisibleForTesting;
import com.google.protobuf.BoolValue;
import com.google.protobuf.Int32Value;
import com.google.protobuf.Int64Value;
import com.google.protobuf.StringValue;
import com.google.protobuf.util.JsonFormat;
import com.google.protobuf.util.JsonFormat.Parser;
import io.grpc.Channel;
import io.grpc.ManagedChannelBuilder;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import org.hypertrace.agent.config.Config;
import org.hypertrace.agent.config.Config.AgentConfig;
import org.hypertrace.agent.config.Config.DataCapture;
Expand All @@ -38,6 +43,8 @@
import org.hypertrace.agent.config.Config.Opa.Builder;
import org.hypertrace.agent.config.Config.PropagationFormat;
import org.hypertrace.agent.config.Config.Reporting;
import org.hypertrace.agent.config.ConfigurationServiceGrpc;
import org.hypertrace.agent.config.Service;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

Expand All @@ -64,10 +71,18 @@ private HypertraceConfig() {}

public static AgentConfig get() {
if (agentConfig == null) {
String dynamicConfigServiceUrl =
EnvironmentConfig.getProperty(EnvironmentConfig.DYNAMIC_CONFIG_SERVICE_URL);
DynamicConfigFetcher dynamicConfigFetcher = null;
if (dynamicConfigServiceUrl != null) {
dynamicConfigFetcher = new DynamicConfigFetcher(dynamicConfigServiceUrl);
Executors.newScheduledThreadPool(1)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an executor service in the upstream that we could use? The upstream javaagent-tooling might have smth that we could reuse. We also have to set the thread to the daemon otherwise it will be blocking shutdown

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will check this

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upstream does not have any executor service that I could find. They have also used Executors directly https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation-api/src/main/java/io/opentelemetry/instrumentation/api/internal/SupportabilityMetrics.java#L81

.scheduleAtFixedRate(dynamicConfigFetcher, 60, 30, TimeUnit.SECONDS);
}
synchronized (HypertraceConfig.class) {
if (agentConfig == null) {
try {
agentConfig = load();
agentConfig = load(dynamicConfigFetcher);
log.info(
"Config loaded: {}",
JsonFormat.printer().omittingInsignificantWhitespace().print(agentConfig));
Expand Down Expand Up @@ -130,21 +145,28 @@ public static void reset() {
}
}

private static AgentConfig load() throws IOException {
private static AgentConfig load(DynamicConfigFetcher dynamicConfigFetcher) throws IOException {
String configFile = EnvironmentConfig.getProperty(EnvironmentConfig.CONFIG_FILE_PROPERTY);
if (configFile == null) {
return EnvironmentConfig.applyPropertiesAndEnvVars(applyDefaults(AgentConfig.newBuilder()))
.build();
AgentConfig.Builder configBuilder = AgentConfig.newBuilder();
if (dynamicConfigFetcher != null) {
configBuilder = dynamicConfigFetcher.initializeConfig().toBuilder();
}
return EnvironmentConfig.applyPropertiesAndEnvVars(applyDefaults(configBuilder)).build();
}
return load(configFile);
return load(configFile, dynamicConfigFetcher);
}

@VisibleForTesting
static AgentConfig load(String filename) throws IOException {
static AgentConfig load(String filename, DynamicConfigFetcher dynamicConfigFetcher)
throws IOException {
File configFile = new File(filename);
if (!configFile.exists() || configFile.isDirectory() || !configFile.canRead()) {
log.error("Config file {} does not exist", filename);
AgentConfig.Builder configBuilder = AgentConfig.newBuilder();
if (dynamicConfigFetcher != null) {
configBuilder = dynamicConfigFetcher.initializeConfig().toBuilder();
}
return EnvironmentConfig.applyPropertiesAndEnvVars(applyDefaults(configBuilder)).build();
}

Expand Down Expand Up @@ -236,4 +258,40 @@ private static String convertYamlToJson(InputStream yaml) throws IOException {
ObjectMapper jsonWriter = new ObjectMapper();
return jsonWriter.writeValueAsString(obj);
}

public static class DynamicConfigFetcher implements Runnable {

private final ConfigurationServiceGrpc.ConfigurationServiceBlockingStub blockingStub;

private static Int64Value configTimestamp;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

give it a better name.


private DynamicConfigFetcher(String dynamicConfigServiceUrl) {
Channel channel = ManagedChannelBuilder.forTarget(dynamicConfigServiceUrl).build();
blockingStub = ConfigurationServiceGrpc.newBlockingStub(channel);
configTimestamp = Int64Value.newBuilder().setValue(System.currentTimeMillis()).build();
}

@Override
public void run() {
Service.UpdateConfigurationResponse response =
blockingStub.updateConfiguration(
Service.UpdateConfigurationRequest.newBuilder()
.setTimestamp(configTimestamp)
.build());
configTimestamp = response.getTimestamp();
AgentConfig.Builder configBuilder = HypertraceConfig.get().toBuilder();
configBuilder.setEnabled(response.getEnabled());
configBuilder.setDataCapture(response.getDataCapture());
configBuilder.setJavaagent(response.getJavaAgent());
agentConfig = configBuilder.build();
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this the field from the HypertraceConfig class?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we worry about concurrent modifications (read/write) at the same time?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is the field from HypertraceConfig class. We do not need to worry about concurrent modifications as it is a volatile field.

}

private AgentConfig initializeConfig() {
Service.InitialConfigurationResponse response =
blockingStub.initialConfiguration(
Service.InitialConfigurationRequest.newBuilder().build());
configTimestamp = response.getTimestamp();
return response.getAgentConfig();
}
}
}