feat(otel): add support for otel metrics #5021

mabdinur · 2025-11-04T21:20:23Z

What does this PR do?

Adds OpenTelemetry metrics SDK integration with OTLP exporter support for gRPC and HTTP/protobuf protocols.

Motivation:
Enables customers to use the OpenTelemetry Metrics API and export metrics via OTLP to any compatible backend, including the Datadog Agent.

Change log entry
Adds OpenTelemetry metrics SDK integration with OTLP exporter support. Configure via DD_METRICS_OTEL_ENABLED and standard OpenTelemetry environment variables.

Key Features:

OTLP exporter support (gRPC and HTTP/protobuf)
Automatic endpoint resolution based on protocol and agent hostname
Datadog service metadata mapped to OpenTelemetry resource attributes
General OTLP settings (OTEL_EXPORTER_OTLP_*) as defaults for metrics-specific settings

Required Gems for metrics support:

opentelemetry-metrics-sdk (~> 0.8)
opentelemetry-exporter-otlp-metrics (~> 0.4)

Testing:

BUNDLE_GEMFILE=gemfiles/ruby_3.2_opentelemetry.gemfile bundle exec rspec spec/datadog/opentelemetry/metrics_spec.rb

pr-commenter · 2025-11-04T21:56:46Z

Benchmarks

Benchmark execution time: 2025-11-27 18:28:06

Comparing candidate commit bee73bb in PR branch munir/add-otel-metrics-configs with baseline commit 2c84fc0 in branch master.

Found 1 performance improvements and 0 performance regressions! Performance is the same for 43 metrics, 2 unstable metrics.

scenario:profiling - intern mixed existing and new

🟩 throughput [+2.476op/s; +3.608op/s] or [+8.729%; +12.720%]

lib/datadog/core/configuration/option.rb

lib/datadog/opentelemetry/configuration/settings.rb

lib/datadog/opentelemetry/metrics.rb

lib/datadog/opentelemetry/configuration/settings.rb

…uns/19580472034

appraisal/ruby-3.2.rb

marcotc · 2025-11-21T19:27:01Z

appraisal/ruby-3.5.rb

  gem 'opentelemetry-sdk', '~> 1.1'
+  gem 'opentelemetry-metrics-sdk', '>= 0.8'
+  gem 'opentelemetry-exporter-otlp-metrics', '>= 0.4'
+  # opentelemetry-metrics-sdk 0.11+ requires opentelemetry-common >= 0.23.0 (for time_in_nanoseconds)


Where is time_in_nanoseconds needed? I can't find it in this PR.

It's defined in opentelemetry-common and used in the opentelemetry metrics sdk: https://github.com/search?q=repo%3Aopen-telemetry%2Fopentelemetry-ruby%20time_in_nanoseconds&type=code. It's not directly used in our library.

This is just note for us so we do not install incompatible opentelemetry libraries.

lib/datadog/opentelemetry.rb

Co-authored-by: Marco Costa <[email protected]>

lib/datadog/opentelemetry/configuration/settings.rb

…uns/19581796018

lib/datadog/opentelemetry/metrics.rb

lib/datadog/opentelemetry/configuration/settings.rb

y9v · 2025-11-24T12:50:47Z

lib/datadog/opentelemetry/sdk/metrics_exporter.rb

+        rescue => e
+          Datadog.logger.error("Failed to export OpenTelemetry Metrics: #{e.message}")
+          telemetry&.inc('tracers', METRIC_EXPORT_FAILURES, 1, tags: @telemetry_tags)
+          raise


do we want to raise this error again so that the user receives it?

This is a good question! @mabdinur, can you check if this API expects errors to propagate?

The metrics exporter does not have any error handling. If it raises an exception then the errors will bubble up. We're doing the same here: https://github.com/open-telemetry/opentelemetry-ruby/blob/8c02da8de62df1464a6e4c8cc35f838a0ccd0dd5/exporter/otlp-metrics/lib/opentelemetry/exporter/otlp/metrics/metrics_exporter.rb#L86.

lib/datadog/opentelemetry/metrics.rb

y9v · 2025-11-24T13:03:26Z

lib/datadog/opentelemetry/metrics.rb

+      def self.resolve_metrics_endpoint
+        metrics_config = @settings.opentelemetry.metrics
+        exporter_config = @settings.opentelemetry.exporter
+
+        return metrics_config.endpoint.to_s if metrics_config.endpoint
+        scheme = @agent_ssl ? 'https' : 'http'
+
+        if metrics_config.protocol
+          protocol = metrics_config.protocol
+          port = (protocol == 'http/protobuf') ? 4318 : 4317
+          path = (protocol == 'http/protobuf') ? '/v1/metrics' : ''
+          return "#{scheme}://#{@agent_host}:#{port}#{path}"
+        end
+
+        return exporter_config.endpoint.to_s if exporter_config.endpoint
+
+        protocol = exporter_config.protocol || 'http/protobuf'
+        port = (protocol == 'http/protobuf') ? 4318 : 4317
+        path = (protocol == 'http/protobuf') ? '/v1/metrics' : ''
+        "#{scheme}://#{@agent_host}:#{port}#{path}"
+      end


I think we can definitely extract a function here that builds the full url from either metrics config or exporter config, and simply use it here:

metrics_endpoint(@settings.opentelemetry.metrics) || metrics_endpoint(@settings.opentelemetry.exporter)

I removed most of this logic. Since grpc is not supported yet. We can really simplify this

lib/datadog/opentelemetry/metrics.rb

y9v · 2025-11-24T13:06:10Z

lib/datadog/opentelemetry/metrics.rb

+        timeout = metrics_config.timeout_millis || exporter_config.timeout_millis
+        headers = metrics_config.headers || exporter_config.headers || {}
+
+        protocol = metrics_config.protocol || exporter_config.protocol


I think it's better to check if all the needed settings are in metrics_config, if not - use exporter_config. With the way it is now, the user could configure timeout for metrics, headers for exporter and protocol for metrics - and it would somehow work.

This makes sense but it goes against the opentelemetry spec. Each configuration needs to be evaluated separately

lib/datadog/opentelemetry/metrics.rb

lib/datadog/opentelemetry/sdk/metrics_exporter.rb

ivoanjo

Gave it another pass! I think this is getting in quite good shape :)

ivoanjo · 2025-11-27T10:57:13Z

docs/OpenTelemetry.md

+| Metrics SDK   | https://rubygems.org/gems/opentelemetry-metrics-sdk  | 2.25.0+         |  >= 0.8             |
+| OTLP Metrics Exporter | https://rubygems.org/gems/opentelemetry-exporter-otlp-metrics  | 2.25.0+         |  >= 0.4            |


Why 2.25.0? Are we not planning to ship this with the next 2.23.0 release, or at least the 2.24.0 after that? 👀

I wasn't sure when this PR would get merged so I picked a release that would likely go out at the end of Q4. Since we are close to getting this merged we can set it to 2.23.0 😄

docs/OpenTelemetry.md

ivoanjo · 2025-11-27T11:00:10Z

docs/OpenTelemetry.md

+    # Set environment variable before initializing metrics support
+    ENV['DD_METRICS_OTEL_ENABLED'] = 'true'
+    require 'opentelemetry/sdk'
+    require 'opentelemetry-metrics-sdk'
+    require 'opentelemetry/exporter/otlp_metrics'
+    require 'datadog/opentelemetry'


Have you considered turning these 5 lines into a datadog/opentelemetry/metrics_autoload (or boot or some name) or something like that?

That way customers would only need to copy-paste the simpler require :)

Actually only opentelemetry/sdk and datadog/opentelemetry are required. opentelemetry/sdk is required to call OpenTelemetry::SDK.configure and 'datadog/opentelemetry' is required to enable ddtrace-opentelemetry support for both metrics and traces.

We do not need to import 'opentelemetry/exporter/otlp_metrics' and 'opentelemetry/exporter/otlp_metrics' here. This is handled by 'datadog/opentelemetry'. I added these require statements to signal to users that they NEED to install the metrics-sdk and the metrics-exporter. I'll remove these imports and make the description a bit clearer.

ivoanjo · 2025-11-27T11:08:36Z

lib/datadog/core/runtime/metrics.rb

        rescue => e
-          Datadog.logger.warn("Error while sending runtime metric. Cause: #{e.class.name} #{e.message}")
+          Datadog.logger.warn("Error while sending runtime metric. Cause: #{e.class.name}  #{e.class}: #{e}")
        end


Minor: Why add both e.class and e.class.name? 👀

This was a product of my find and replace to address another comment. I missed the #{e.class.name} component. I will remove it.

ivoanjo · 2025-11-27T11:49:10Z

spec/datadog/opentelemetry/metrics_spec.rb

+  describe 'Basic Functionality' do
+    it 'exports counter metrics' do
+      setup_metrics
+      provider = ::OpenTelemetry.meter_provider


Minor: Consider making this into a let? lets are lazy, so they will only be called when needed, which I think is exactly what we need here (e.g. setup_metrics will still correctly run before)

Yup a let here is a good idea. Also the code when through a few iterations. We can simplify the tests and consolidate some of the helpers

ivoanjo · 2025-11-27T11:50:40Z

spec/datadog/opentelemetry/metrics_spec.rb

+      flush_and_wait(provider)
+
+      metric = find_metric_in_json(get_testagent_metrics, 'requests_myapp')


Minor: Every call to get_testagent_metrics seems to be preceded by a flush_and_wait(provider) -- consider combining them to simplify the tests a bit (the less boilerplate they have, the clearer we can focus on what's actually being tested)

Yup, just did this

ivoanjo · 2025-11-27T11:51:53Z

spec/datadog/opentelemetry/metrics_spec.rb

+    it 'exports gauge metrics' do
+      setup_metrics('OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE' => 'cumulative')
+      provider = ::OpenTelemetry.meter_provider
+      gauge = provider.meter('app').create_gauge('temperature')
+
+      gauge.record(72)
+      flush_and_wait(provider)
+      gauge.record(72)
+      flush_and_wait(provider)
+
+      metric = find_metric_in_json(get_testagent_metrics, 'temperature')
+      value = metric['gauge']['data_points'].first['as_int']&.to_i
+      expect(value).to eq(72)
+    end


Consider turning some of ⬆️ into a comment in the test -- I think it's not obvious why the two records are there

ivoanjo · 2025-11-27T12:00:46Z

spec/datadog/opentelemetry/metrics_spec.rb

+    it 'metrics-specific configs take precedence over general OTLP configs' do
+      setup_metrics(
+        'DD_METRICS_OTEL_ENABLED' => 'true',
+        'OTEL_EXPORTER_OTLP_METRICS_ENDPOINT' => 'http://metrics:4318/v1/metrics',
+        'OTEL_EXPORTER_OTLP_METRICS_PROTOCOL' => 'http/protobuf',
+        'OTEL_EXPORTER_OTLP_METRICS_TIMEOUT' => '5000',
+        'OTEL_EXPORTER_OTLP_METRICS_HEADERS' => 'metrics=value',
+        'OTEL_METRIC_EXPORT_INTERVAL' => '4000',
+        'OTEL_METRIC_EXPORT_TIMEOUT' => '3000',
+        'OTEL_EXPORTER_OTLP_ENDPOINT' => 'http://general:4317',
+        'OTEL_EXPORTER_OTLP_PROTOCOL' => 'grpc',
+        'OTEL_EXPORTER_OTLP_TIMEOUT' => '2000',
+        'OTEL_EXPORTER_OTLP_HEADERS' => 'general=value'
+      )
+
+      provider = ::OpenTelemetry.meter_provider
+      reader = provider.metric_readers.first
+      exporter = reader.instance_variable_get(:@exporter)
+
+      expect(exporter.instance_variable_get(:@uri).to_s).to eq('http://metrics:4318/v1/metrics')
+      expect(exporter.instance_variable_get(:@timeout)).to eq(5.0)
+      expect(reader.instance_variable_get(:@export_interval)).to eq(4.0)
+      expect(reader.instance_variable_get(:@export_timeout)).to eq(3.0)
+      expect(exporter.instance_variable_get(:@headers)['metrics']).to eq('value')
+    end
+
+    it 'general OTLP configs work in isolation' do
+      setup_metrics(
+        'OTEL_EXPORTER_OTLP_ENDPOINT' => 'http://general:4317',
+        'OTEL_EXPORTER_OTLP_PROTOCOL' => 'http/protobuf',
+        'OTEL_EXPORTER_OTLP_TIMEOUT' => '8000',
+        'OTEL_EXPORTER_OTLP_HEADERS' => 'general=value'
+      )


Consider factoring out the OTLP configs per-test. That way it's even more clear that they get overridden! Something like

describe 'configuration priority' do let(:env_vars) do # general otlp configs end before { setup_metrics(env_vars) } it 'uses the general OTLP configs in isolation' do ...the current test... end context 'when metrics-specific configs are provided' do let(:env_vars) do super().merge( # metrics configs only -- general come from super() ) end it 'gives metrics-specific precendence' do ...etc... end end

Yup, this wouldve actually saved me a lot time. Debugging this test is hell

ivoanjo · 2025-11-27T12:01:33Z

spec/datadog/opentelemetry/metrics_spec.rb

+      reader = provider.metric_readers.first
+      exporter = reader.instance_variable_get(:@exporter)


Minor: These two also could become lets after provider also becomes one :)

Yup and we can also put attributes and resource in lets as well

mabdinur · 2025-11-27T17:59:47Z

Gave it another pass! I think this is getting in quite good shape :)

Thanks so much for the thorough review and feedback. I really appreciate your time and patience. I learned a lot about ruby along the way 😸

ivoanjo

👍 Looks great to me! I really like how clean the tests ended up :)

ivoanjo · 2025-11-28T08:56:13Z

lib/datadog/opentelemetry/configuration/settings.rb

+          proc do |value|
+            if value && value.to_s.downcase != 'http/protobuf'
+              Datadog.logger.warn("#{env_var_name}=#{value} is not supported. Using http/protobuf instead.")
+              'http/protobuf'
+            else
+              value
+            end
+          end


Minor: I suggest this small tweak for two reasons

It's clearer only protobuf is available

In the prior version we actually did not downcase the actual value so I think we could end up with 'HtTp/PrOtObUf' which is kinda confusing?

Suggested change

proc do |value|

if value && value.to_s.downcase != 'http/protobuf'

Datadog.logger.warn("#{env_var_name}=#{value} is not supported. Using http/protobuf instead.")

'http/protobuf'

else

value

end

end

proc do |value|

if value && value.to_s.downcase != 'http/protobuf'

Datadog.logger.warn("#{env_var_name}=#{value} is not supported. Using http/protobuf instead.")

end

'http/protobuf'

end

ivoanjo · 2025-11-28T09:01:38Z

lib/datadog/opentelemetry/metrics.rb

+        @logger.warn("Could not load OTLP metrics exporter:  #{e.class}: #{e}")
+      end
+
+      private :create_resource, :configure_metric_reader, :resolve_metrics_endpoint, :configure_otlp_exporter


Minor: Since this is now a regular class, you can leave a private in a single line at the beginning of the private section of the methods and not need to list them here ;)

ivoanjo · 2025-11-28T09:03:22Z

spec/datadog/opentelemetry/metrics_spec.rb

+  describe 'Basic Functionality' do
+    it 'exports counter metrics' do
+      setup_metrics
+      provider = ::OpenTelemetry.meter_provider


Minor: (Can be removed? Since it's a let now?)

Suggested change

provider = ::OpenTelemetry.meter_provider

ivoanjo · 2025-11-28T09:03:35Z

spec/datadog/opentelemetry/metrics_spec.rb

+
+    it 'exports histogram metrics' do
+      setup_metrics
+      provider = ::OpenTelemetry.meter_provider


Minor: (Can be removed? Since it's a let now?)

Suggested change

provider = ::OpenTelemetry.meter_provider

ivoanjo · 2025-11-28T09:05:50Z

spec/datadog/opentelemetry/metrics_spec.rb

+            'OTEL_METRIC_EXPORT_INTERVAL' => '4000',
+            'OTEL_METRIC_EXPORT_TIMEOUT' => '3000',
+            'OTEL_EXPORTER_OTLP_PROTOCOL' => 'grpc',
+            'OTEL_EXPORTER_OTLP_TIMEOUT' => '2000'


Minor: I think this one can be removed, since it's already defined in the outer block (to 8000)

Suggested change

'OTEL_EXPORTER_OTLP_TIMEOUT' => '2000'

chore(otel): add support for otel metrics [Part 1]

3571707

github-actions bot added core Involves Datadog core libraries otel OpenTelemetry-related changes labels Nov 4, 2025

consolidate tests

2d64146

add support for exporting otlp

5ad9754

mabdinur changed the title ~~chore(otel): add support for otel metrics [Part 1]~~ chore(otel): add support for otel metrics Nov 4, 2025

add docs and justify minimum supported versions

9eb0a24

mabdinur force-pushed the munir/add-otel-metrics-configs branch from a611845 to 9eb0a24 Compare November 5, 2025 00:29

ivoanjo reviewed Nov 5, 2025

View reviewed changes

lib/datadog/core/configuration/option.rb Show resolved Hide resolved

mabdinur added 9 commits November 5, 2025 11:52

use testagent for otlp tests

d9cd08d

use default instead of default_proc in otel configs

eb5c3a2

consolidate agent resolutation logic

d6bbff9

simplify settings

4883393

clean up testagent helpers

f1cb22c

clean up docs

1818637

Merge branch 'master' into munir/add-otel-metrics-configs

2e8cdc1

clean up gemfile

060757d

fix metric export timeout

270a3e9

mabdinur commented Nov 5, 2025

View reviewed changes

lib/datadog/opentelemetry/configuration/settings.rb Outdated Show resolved Hide resolved

mabdinur commented Nov 5, 2025

View reviewed changes

lib/datadog/opentelemetry/metrics.rb Outdated Show resolved Hide resolved

mabdinur commented Nov 5, 2025

View reviewed changes

lib/datadog/opentelemetry/metrics.rb Outdated Show resolved Hide resolved

mabdinur commented Nov 5, 2025

View reviewed changes

lib/datadog/opentelemetry/configuration/settings.rb Outdated Show resolved Hide resolved

mabdinur added 7 commits November 5, 2025 15:44

use datadog_env

e138f38

make http protobuf the default

2eefa81

use agent resolver

65bcf2c

use opentelemetry configure

a894798

fix spacing

c9abc8a

fix spacing

94b46d8

update config tests to validate the exporter that is set

9f3751f

mabdinur and others added 2 commits November 21, 2025 13:53

Merge bb8502a into 2c84fc0

0277207

[🤖] Lock Dependency: https://github.com/DataDog/dd-trace-rb/actions/r…

d282e0f

…uns/19580472034

marcotc reviewed Nov 21, 2025

View reviewed changes

appraisal/ruby-3.2.rb Outdated Show resolved Hide resolved

marcotc reviewed Nov 21, 2025

View reviewed changes

lib/datadog/opentelemetry.rb Outdated Show resolved Hide resolved

mabdinur added 2 commits November 21, 2025 14:32

fix steep check

ff85daf

simplify try_wait_until logic

3faa913

mabdinur force-pushed the munir/add-otel-metrics-configs branch from cfcb81a to 3faa913 Compare November 21, 2025 19:40

mabdinur and others added 3 commits November 21, 2025 14:43

revert change to ruby 3.2

4aa5529

Update lib/datadog/opentelemetry.rb

ca22f30

Co-authored-by: Marco Costa <[email protected]>

Merge ca22f30 into 2c84fc0

09059fd

marcotc reviewed Nov 21, 2025

View reviewed changes

lib/datadog/opentelemetry/configuration/settings.rb Outdated Show resolved Hide resolved

marcotc reviewed Nov 21, 2025

View reviewed changes

lib/datadog/opentelemetry/configuration/settings.rb Outdated Show resolved Hide resolved

dd-apm-ecosystems-autobot bot and others added 2 commits November 21, 2025 19:56

[🤖] Lock Dependency: https://github.com/DataDog/dd-trace-rb/actions/r…

7384a8b

…uns/19581796018

fix how otlp headers are parsed

447d925

marcotc reviewed Nov 21, 2025

View reviewed changes

lib/datadog/opentelemetry/metrics.rb Outdated Show resolved Hide resolved

clean up resource attribute creation

0feb1be

y9v reviewed Nov 24, 2025

View reviewed changes

p-datadog reviewed Nov 24, 2025

View reviewed changes

lib/datadog/opentelemetry/metrics.rb Outdated Show resolved Hide resolved

lib/datadog/opentelemetry/sdk/metrics_exporter.rb Outdated Show resolved Hide resolved

mabdinur added 3 commits November 25, 2025 00:08

address comments round 3

fe8d703

lint

59bb33b

update docs to include configuration issue

a13a5a7

mabdinur requested review from ivoanjo and marcotc November 26, 2025 16:58

ivoanjo reviewed Nov 27, 2025

View reviewed changes

more clean ups, address all open comments

bee73bb

mabdinur force-pushed the munir/add-otel-metrics-configs branch from a6ddfdc to bee73bb Compare November 27, 2025 17:57

mabdinur requested a review from ivoanjo November 27, 2025 19:42

ivoanjo approved these changes Nov 28, 2025

View reviewed changes

		\| Metrics SDK \| https://rubygems.org/gems/opentelemetry-metrics-sdk \| 2.25.0+ \| >= 0.8 \|
		\| OTLP Metrics Exporter \| https://rubygems.org/gems/opentelemetry-exporter-otlp-metrics \| 2.25.0+ \| >= 0.4 \|

		flush_and_wait(provider)

		metric = find_metric_in_json(get_testagent_metrics, 'requests_myapp')

		reader = provider.metric_readers.first
		exporter = reader.instance_variable_get(:@exporter)

feat(otel): add support for otel metrics #5021

Are you sure you want to change the base?

feat(otel): add support for otel metrics #5021

Conversation

mabdinur commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter bot commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

scenario:profiling - intern mixed existing and new

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ivoanjo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mabdinur commented Nov 4, 2025 •

edited

Loading

pr-commenter bot commented Nov 4, 2025 •

edited

Loading