Skip to content

Commit 4ea0f3b

Browse files
feat(monitoring): add HTTP Basic auth for Prometheus scrape endpoint
Protect /actuator/prometheus on GMS, MAE, and MCE with optional HTTP Basic auth. Auth is off by default and auto-enables with safe default credentials when Micrometer Prometheus export is enabled. Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent d79eba4 commit 4ea0f3b

16 files changed

Lines changed: 444 additions & 3 deletions

File tree

docker/monitoring/prometheus.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,9 @@ scrape_configs:
1111
- job_name: 'micrometer'
1212
scrape_interval: 10s
1313
metrics_path: /actuator/prometheus
14+
basic_auth:
15+
username: prometheus
16+
password: datahub
1417
static_configs:
1518
- targets:
1619
- 'datahub-gms:4319'
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Example: enable Micrometer Prometheus export with HTTP Basic auth on GMS, MAE, and MCE.
2+
# Copy to prometheus-scrape-auth.env (gitignored) or pass via DATAHUB_LOCAL_COMMON_ENV.
3+
#
4+
# Auth is enabled automatically when export is enabled (defaults: prometheus / datahub).
5+
# Override credentials in production:
6+
MANAGEMENT_METRICS_EXPORT_PROMETHEUS_ENABLED=true
7+
MANAGEMENT_METRICS_EXPORT_PROMETHEUS_AUTH_PASSWORD=change-me

docs/advanced/monitoring.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1112,6 +1112,37 @@ scrape job for port **4319**. Outside Docker, leave `MANAGEMENT_SERVER_PORT` uns
11121112
application port; set it when you want a separate management listener (Spring maps the env var to
11131113
`management.server.port`).
11141114

1115+
#### Securing the Micrometer Prometheus endpoint
1116+
1117+
By default, Micrometer Prometheus export is **disabled** (`MANAGEMENT_METRICS_EXPORT_PROMETHEUS_ENABLED=false`).
1118+
When you enable export on GMS, MAE consumer, or MCE consumer, HTTP Basic auth is **enabled automatically** on
1119+
`/actuator/prometheus` with default credentials `prometheus` / `datahub`. Change the password in production.
1120+
1121+
| Variable | Default | Description |
1122+
| -------- | ------- | ----------- |
1123+
| `MANAGEMENT_METRICS_EXPORT_PROMETHEUS_ENABLED` | `false` | Expose `/actuator/prometheus` via Micrometer |
1124+
| `MANAGEMENT_METRICS_EXPORT_PROMETHEUS_AUTH_ENABLED` | _(auto)_ | When unset, follows export enabled. Set `false` to expose metrics without auth |
1125+
| `MANAGEMENT_METRICS_EXPORT_PROMETHEUS_AUTH_USERNAME` | `prometheus` | Scrape username |
1126+
| `MANAGEMENT_METRICS_EXPORT_PROMETHEUS_AUTH_PASSWORD` | `datahub` | Scrape password (startup warns if default is used) |
1127+
1128+
Example Prometheus scrape config:
1129+
1130+
```yaml
1131+
scrape_configs:
1132+
- job_name: datahub-micrometer
1133+
metrics_path: /actuator/prometheus
1134+
basic_auth:
1135+
username: prometheus
1136+
password: datahub
1137+
static_configs:
1138+
- targets:
1139+
- datahub-gms:4319
1140+
- datahub-mae-consumer:4319
1141+
- datahub-mce-consumer:4319
1142+
```
1143+
1144+
For local Docker development, see [prometheus-scrape-auth.env.example](../../docker/profiles/prometheus-scrape-auth.env.example).
1145+
11151146
In the JVM dashboard, you can find detailed charts based on JVM metrics like CPU/memory/disk usage. In the DataHub
11161147
dashboard, you can find charts to monitor each endpoint and the kafka topics. Using the example implementation, go
11171148
to http://localhost:3001 to find the grafana dashboards! (Username: admin, PW: admin)

docs/how/updating-datahub.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,6 +156,7 @@ Requirements:
156156
- **(Ingestion / Sigma — Action required) Null `DatasetUpstream.name` now tolerated.** Previously raised a Pydantic `ValidationError` warning; now counted under `chart_dataset_upstream_name_missing` with a `SourceReport.warning` titled `Sigma workbook dataset upstream dropped (name missing)`. Lineage semantics unchanged. **Action:** migrate any alert keyed on the old Pydantic warning text to the new counter or warning title.
157157
- **(Actions / Kafka)** The default Kafka offset commit strategy for Actions has changed from **synchronous** (`async_commit_enabled: false`) to **asynchronous** (`async_commit_enabled: true`). With asynchronous commits, offsets are stored locally after each event and committed periodically by a background thread (default interval: 10 seconds via `async_commit_interval`). This provides up to **25x throughput improvement** for high-volume action pipelines. The tradeoff: on consumer crash, up to `async_commit_interval` milliseconds of events may be redelivered. All built-in actions are idempotent or tolerate redelivery, so no behavior change is expected for standard deployments. To restore the previous synchronous default for any action, add `async_commit_enabled: false` to the `source.config` section of your action YAML.
158158
- **Docker / Prometheus (Micrometer):** When using the stock DataHub Docker `start.sh` entrypoints, GMS, MAE consumer, MCE consumer, and the frontend expose Micrometer Actuator (including `GET /actuator/prometheus` and `/actuator/health`) on container port **4319** by default (`MANAGEMENT_SERVER_PORT`), separate from the main API/UI port. Profile compose ([docker/profiles/docker-compose.gms.yml](../../docker/profiles/docker-compose.gms.yml)) **`expose`s 4319** for on-network scraping and maps **`${DATAHUB_MAPPED_GMS_MANAGEMENT_PORT:-4319}:4319`** on the host for GMS (same pattern as **`${DATAHUB_MAPPED_GMS_PORT:-8080}:8080`** for HTTP). Override `DATAHUB_GMS_MANAGEMENT_URL` in smoke tests if needed, or change the host port via `DATAHUB_MAPPED_GMS_MANAGEMENT_PORT`. Scrape from another container on the same compose network (e.g. `http://datahub-gms:4319/actuator/prometheus` as in [docker/monitoring/prometheus.yaml](../../docker/monitoring/prometheus.yaml)). MAE and MCE image **HEALTHCHECK** probes the management port first, then falls back to the main consumer port for older layouts. The JMX Prometheus Java agent remains on **4318**. If you previously scraped Micrometer from the GMS HTTP port (e.g. `8080`), update scrapes to the management listener. To keep Actuator on the main port, unset `MANAGEMENT_SERVER_PORT` in the container or set it equal to the main server port.
159+
- **(Operations / monitoring) Micrometer Prometheus export and scrape auth:** Micrometer Prometheus export now defaults to **off** (`MANAGEMENT_METRICS_EXPORT_PROMETHEUS_ENABLED=false`) on GMS, MAE consumer, and MCE consumer. When you enable export, HTTP Basic auth is **automatically required** on `/actuator/prometheus` unless you set `MANAGEMENT_METRICS_EXPORT_PROMETHEUS_AUTH_ENABLED=false`. Default scrape credentials are `prometheus` / `datahub`; set `MANAGEMENT_METRICS_EXPORT_PROMETHEUS_AUTH_PASSWORD` before enabling export in production. Update Prometheus, Grafana Agent, or managed scrape configs to use `basic_auth` when upgrading. See [monitoring.md](../advanced/monitoring.md#securing-the-micrometer-prometheus-endpoint).
159160
- (Operations / monitoring) Docker images that bundle the **JMX Prometheus Java agent** (`datahub-gms`, `datahub-frontend-react`, `datahub-mae-consumer`, `datahub-mce-consumer`, `datahub-upgrade`) now ship **`jmx_prometheus_javaagent` 1.0.1** (previously 0.20.0). The agent uses **Prometheus client_java 1.x**; upgrade scrapers and dashboards accordingly.
160161
- **HTTP scrape path:** Metrics are exposed at **`/metrics`**, not at **`/`**. If you scrape the JMX port directly (often **4318** when Prometheus export is enabled), set your Prometheus `metrics_path` (or Kubernetes `ServiceMonitor` / `PodMonitor` `path`) to **`/metrics`**. Anything still requesting **`/`** will receive the default HTML page, not the metrics exposition.
161162
- **JVM metrics:** Some built-in JVM metric **names** changed to align with OpenMetrics (for example, memory-related series). Update Grafana panels, recording rules, and alerts that referenced the old names. See the [client_java JVM migration notes](https://prometheus.github.io/client_java/migration/simpleclient/#jvm-metrics).

metadata-jobs/mae-consumer-job/src/main/resources/application.properties

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
server.port=9091
2-
management.endpoints.web.exposure.include=metrics, health, info
2+
management.endpoints.web.exposure.include=metrics, health, info, prometheus
33
spring.mvc.servlet.path=/
44
management.health.elasticsearch.enabled=false
55
management.health.neo4j.enabled=false

metadata-jobs/mce-consumer-job/src/main/resources/application.properties

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
server.port=9090
22
datahub.gms.uri=http://localhost:9090${DATAHUB_GMS_BASE_PATH}/gms
3-
management.endpoints.web.exposure.include=metrics, health, info
3+
management.endpoints.web.exposure.include=metrics, health, info, prometheus
44
spring.mvc.servlet.path=/
55
management.health.elasticsearch.enabled=false
66
management.health.neo4j.enabled=false

metadata-service/configuration/src/main/resources/application.yaml

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1009,7 +1009,12 @@ management:
10091009
jmx:
10101010
enabled: ${MANAGEMENT_METRICS_EXPORT_JMX_ENABLED:true} # Enable jmx metrics export
10111011
prometheus:
1012-
enabled: ${MANAGEMENT_METRICS_EXPORT_PROMETHEUS_ENABLED:true} # Enable prometheus metrics export
1012+
enabled: ${MANAGEMENT_METRICS_EXPORT_PROMETHEUS_ENABLED:false} # Enable prometheus metrics export
1013+
auth:
1014+
# When unset, auth is enabled automatically if prometheus export is enabled above.
1015+
# Set MANAGEMENT_METRICS_EXPORT_PROMETHEUS_AUTH_ENABLED=false to expose metrics without auth.
1016+
username: ${MANAGEMENT_METRICS_EXPORT_PROMETHEUS_AUTH_USERNAME:}
1017+
password: ${MANAGEMENT_METRICS_EXPORT_PROMETHEUS_AUTH_PASSWORD:}
10131018
tags:
10141019
application: ${spring.application.name}
10151020

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
package com.linkedin.gms.factory.system_telemetry;
2+
3+
import jakarta.annotation.PostConstruct;
4+
import jakarta.servlet.FilterChain;
5+
import jakarta.servlet.ServletException;
6+
import jakarta.servlet.http.HttpServletRequest;
7+
import jakarta.servlet.http.HttpServletResponse;
8+
import java.io.IOException;
9+
import java.nio.charset.StandardCharsets;
10+
import java.security.MessageDigest;
11+
import java.util.Base64;
12+
import lombok.extern.slf4j.Slf4j;
13+
import org.springframework.util.StringUtils;
14+
import org.springframework.web.filter.OncePerRequestFilter;
15+
16+
/**
17+
* Optional HTTP Basic authentication for {@code /actuator/prometheus}.
18+
*
19+
* <p>Independent of DataHub token auth. When disabled, requests pass through unchanged.
20+
*/
21+
@Slf4j
22+
public class PrometheusScrapeAuthFilter extends OncePerRequestFilter {
23+
24+
public static final String PROMETHEUS_ACTUATOR_PATH = "/actuator/prometheus";
25+
private static final String BASIC_PREFIX = "Basic ";
26+
27+
private final PrometheusScrapeAuthSettings settings;
28+
29+
public PrometheusScrapeAuthFilter(PrometheusScrapeAuthSettings settings) {
30+
this.settings = settings;
31+
}
32+
33+
@PostConstruct
34+
public void validateConfiguration() {
35+
if (!settings.isEnabled()) {
36+
return;
37+
}
38+
log.info("Prometheus scrape HTTP Basic auth enabled for {}", PROMETHEUS_ACTUATOR_PATH);
39+
if (settings.isUsingDefaultPassword()) {
40+
log.warn(
41+
"Prometheus scrape auth is using the default password ({}). Set "
42+
+ "MANAGEMENT_METRICS_EXPORT_PROMETHEUS_AUTH_PASSWORD to a strong secret.",
43+
PrometheusScrapeAuthSettings.DEFAULT_PASSWORD);
44+
}
45+
}
46+
47+
@Override
48+
protected boolean shouldNotFilter(HttpServletRequest request) {
49+
if (!settings.isEnabled()) {
50+
return true;
51+
}
52+
String path = request.getServletPath();
53+
return path == null || !PROMETHEUS_ACTUATOR_PATH.equals(path);
54+
}
55+
56+
@Override
57+
protected void doFilterInternal(
58+
HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
59+
throws ServletException, IOException {
60+
61+
if (isAuthorized(request)) {
62+
filterChain.doFilter(request, response);
63+
return;
64+
}
65+
66+
response.setHeader("WWW-Authenticate", "Basic realm=\"Prometheus\"");
67+
response.sendError(HttpServletResponse.SC_UNAUTHORIZED, "Unauthorized");
68+
}
69+
70+
private boolean isAuthorized(HttpServletRequest request) {
71+
String authorization = request.getHeader("Authorization");
72+
if (!StringUtils.hasText(authorization) || !authorization.startsWith(BASIC_PREFIX)) {
73+
return false;
74+
}
75+
76+
try {
77+
byte[] decoded =
78+
Base64.getDecoder().decode(authorization.substring(BASIC_PREFIX.length()).trim());
79+
String credentials = new String(decoded, StandardCharsets.UTF_8);
80+
int separator = credentials.indexOf(':');
81+
if (separator < 0) {
82+
return false;
83+
}
84+
String username = credentials.substring(0, separator);
85+
String password = credentials.substring(separator + 1);
86+
return constantTimeEquals(username, settings.getUsername())
87+
&& constantTimeEquals(password, settings.getPassword());
88+
} catch (IllegalArgumentException e) {
89+
log.debug("Invalid Basic authorization header on {}", PROMETHEUS_ACTUATOR_PATH);
90+
return false;
91+
}
92+
}
93+
94+
private static boolean constantTimeEquals(String actual, String expected) {
95+
if (actual == null || expected == null) {
96+
return false;
97+
}
98+
return MessageDigest.isEqual(
99+
actual.getBytes(StandardCharsets.UTF_8), expected.getBytes(StandardCharsets.UTF_8));
100+
}
101+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
package com.linkedin.gms.factory.system_telemetry;
2+
3+
import org.springframework.beans.factory.annotation.Value;
4+
import org.springframework.boot.context.properties.EnableConfigurationProperties;
5+
import org.springframework.boot.web.servlet.FilterRegistrationBean;
6+
import org.springframework.context.annotation.Bean;
7+
import org.springframework.core.Ordered;
8+
9+
/** Shared filter beans for Prometheus scrape auth. */
10+
@EnableConfigurationProperties(PrometheusScrapeAuthProperties.class)
11+
class PrometheusScrapeAuthFilterConfigurationSupport {
12+
13+
@Bean
14+
PrometheusScrapeAuthFilter prometheusScrapeAuthFilter(
15+
PrometheusScrapeAuthProperties authProperties,
16+
@Value("${management.metrics.export.prometheus.enabled:false}")
17+
boolean prometheusExportEnabled) {
18+
PrometheusScrapeAuthSettings settings =
19+
PrometheusScrapeAuthSettings.resolve(
20+
prometheusExportEnabled,
21+
authProperties.getEnabled(),
22+
authProperties.getUsername(),
23+
authProperties.getPassword());
24+
return new PrometheusScrapeAuthFilter(settings);
25+
}
26+
27+
@Bean
28+
FilterRegistrationBean<PrometheusScrapeAuthFilter> prometheusScrapeAuthFilterRegistration(
29+
PrometheusScrapeAuthFilter filter) {
30+
FilterRegistrationBean<PrometheusScrapeAuthFilter> registration = new FilterRegistrationBean<>();
31+
registration.setFilter(filter);
32+
registration.addUrlPatterns(PrometheusScrapeAuthFilter.PROMETHEUS_ACTUATOR_PATH);
33+
registration.setOrder(Ordered.HIGHEST_PRECEDENCE);
34+
registration.setAsyncSupported(true);
35+
return registration;
36+
}
37+
38+
private PrometheusScrapeAuthFilterConfigurationSupport() {}
39+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
package com.linkedin.gms.factory.system_telemetry;
2+
3+
import org.springframework.boot.actuate.autoconfigure.web.server.ConditionalOnManagementPort;
4+
import org.springframework.boot.actuate.autoconfigure.web.server.ManagementPortType;
5+
import org.springframework.context.annotation.Configuration;
6+
import org.springframework.context.annotation.Import;
7+
8+
/**
9+
* Registers scrape auth on the main web server when management shares the application port
10+
* (e.g. Aiven {@code MANAGEMENT_SERVER_PORT=8080}).
11+
*/
12+
@Configuration
13+
@ConditionalOnManagementPort(ManagementPortType.SAME)
14+
@Import(PrometheusScrapeAuthFilterConfigurationSupport.class)
15+
public class PrometheusScrapeAuthFilterFactory {}

0 commit comments

Comments
 (0)