test: test helpers for metrics

Author: zvkemp

test_helpers/README.md

@@ -22,6 +22,46 @@ gem install opentelemetry-test-helpers
 
 Or, if you use [bundler][bundler-home], include `opentelemetry-test-helpers` in your `Gemfile`.
 
+## Testing with metrics
+
+The `metrics` module facilitates testing instrumentation libraries with respect to the OpenTelemetry Metrics API and SDK. It is not required by default. It is designed to work with or without the metrics API and SDK defined, but you may experience issues if the API or SDK gem is in your Gemfile but not yet loaded when the test helpers are initialized.
+
+### Usage
+
+In a test_helper.rb, after the `configure` block,
+require this library:
+
+```ruby
+OpenTelemetry::SDK.configure do |c|
+  c.error_handler = ->(exception:, message:) { raise(exception || message) }
+  c.add_span_processor span_processor
+end
+require 'opentelemetry/test_helpers/metrics'
+```
+
+If the library uses Appraisals, it is recommended to appraise with and without the metrics api and sdk gems. Note that any metrics implementation in instrumentation libraries should be written against the API only, but for testing the SDK is required to collect metrics data - testing under all three scenarios (no metrics at all, api only, and with the sdk) helps ensure compliance with this requirement.
+
+In a test:
+
+```ruby
+with_metrics_sdk do
+  let(:metric_snapshots) do
+    metrics_exporter.tap(&:pull)
+      .metric_snapshots.select { |snapshot| snapshot.data_points.any? }
+      .group_by(&:name)
+  end
+
+  it "uses metrics", with_metrics_sdk: true do
+    # do something here ...
+    _(metric_snapshots).count.must_equal(4)
+  end
+end
+```
+
+- `metrics_exporter` is automatically reset before each test.
+- `#with_metrics_sdk` will only yield if the SDK is present.
+- `#with_metrics_api` will only yield if the API is present
+
 ## How can I get involved?
 
 The `opentelemetry-test-helpers` gem source is [on github][repo-github], along with related gems including `opentelemetry-api`.

test_helpers/lib/opentelemetry/test_helpers/metrics.rb

@@ -0,0 +1,90 @@
+# frozen_string_literal: true
+
+# Copyright The OpenTelemetry Authors
+#
+# SPDX-License-Identifier: Apache-2.0
+
+require 'minitest/spec'
+
+module OpenTelemetry
+  module TestHelpers
+    # Convenience features and Minitest extensions to support testing
+    # around the metrics-api or metrics-sdk libraries.
+    module Metrics
+      module LoadedMetricsFeatures
+        OTEL_METRICS_API_LOADED = !Gem.loaded_specs['opentelemetry-metrics-api'].nil?
+        OTEL_METRICS_SDK_LOADED = !Gem.loaded_specs['opentelemetry-metrics-sdk'].nil?
+
+        extend self
+
+        def api_loaded?
+          OTEL_METRICS_API_LOADED
+        end
+
+        def sdk_loaded?
+          OTEL_METRICS_SDK_LOADED
+        end
+      end
+
+      module MinitestExtensions
+        def self.prepended(base)
+          base.extend(self)
+        end
+
+        def self.included(base)
+          base.extend(self)
+        end
+
+        def before_setup
+          super
+          reset_metrics_exporter
+        end
+
+        def with_metrics_sdk
+          yield if LoadedMetricsFeatures.sdk_loaded?
+        end
+
+        def without_metrics_sdk
+          yield unless LoadedMetricsFeatures.sdk_loaded?
+        end
+
+        def metrics_exporter
+          with_metrics_sdk { METRICS_EXPORTER }
+        end
+
+        def reset_meter_provider
+          with_metrics_sdk do
+            resource = OpenTelemetry.meter_provider.resource
+            OpenTelemetry.meter_provider = OpenTelemetry::SDK::Metrics::MeterProvider.new(resource: resource)
+            OpenTelemetry.meter_provider.add_metric_reader(METRICS_EXPORTER)
+          end
+        end
+
+        def reset_metrics_exporter
+          with_metrics_sdk do
+            METRICS_EXPORTER.pull
+            METRICS_EXPORTER.reset
+          end
+        end
+
+        def it(desc = 'anonymous', with_metrics_sdk: false, without_metrics_sdk: false, &block)
+          return super(desc, &block) unless with_metrics_sdk || without_metrics_sdk
+
+          raise ArgumentError, 'without_metrics_sdk and with_metrics_sdk must be mutually exclusive' if without_metrics_sdk && with_metrics_sdk
+
+          return if with_metrics_sdk && !LoadedMetricsFeatures.sdk_loaded?
+          return if without_metrics_sdk && LoadedMetricsFeatures.sdk_loaded?
+
+          super(desc, &block)
+        end
+      end
+
+      if LoadedMetricsFeatures.sdk_loaded?
+        METRICS_EXPORTER = OpenTelemetry::SDK::Metrics::Export::InMemoryMetricPullExporter.new
+        OpenTelemetry.meter_provider.add_metric_reader(METRICS_EXPORTER)
+      end
+
+      Minitest::Spec.prepend(MinitestExtensions)
+    end
+  end
+end

test_helpers/opentelemetry-test-helpers.gemspec

@@ -25,8 +25,9 @@ Gem::Specification.new do |spec|
   spec.require_paths = ['lib']
   spec.required_ruby_version = '>= 3.0'
 
+  spec.add_dependency 'minitest', '~> 5.0'
+
   spec.add_development_dependency 'bundler', '>= 1.17'
-  spec.add_development_dependency 'minitest', '~> 5.0'
   spec.add_development_dependency 'opentelemetry-sdk'
   spec.add_development_dependency 'pry'
   spec.add_development_dependency 'pry-byebug' unless RUBY_ENGINE == 'jruby'

test_helpers/test/opentelemetry/test_helpers/metrics_test.rb

@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+# Copyright The OpenTelemetry Authors
+#
+# SPDX-License-Identifier: Apache-2.0
+
+require 'test_helper'
+
+require 'opentelemetry/test_helpers/metrics' # not loaded by default
+
+describe OpenTelemetry::TestHelpers::Metrics do
+  describe 'dependencies' do
+    let(:gemspec) { Gem.loaded_specs.fetch('opentelemetry-test-helpers') }
+    let(:dependencies) { gemspec.dependencies.map(&:name) }
+
+    # NOTE: The `metrics` module here is intended to facilitate testing
+    # for instrumentation libraries that should function with or without
+    # the metrics-api in the bundle. Including it in this test helper
+    # should be considered a mistake unless additional provisions are made to preserve
+    # this feature.
+    it 'does not include the api or sdk gems' do
+      _(dependencies).wont_include('opentelemetry-metrics-sdk')
+      _(dependencies).wont_include('opentelemetry-metrics-api')
+    end
+  end
+end

test_helpers/test/test_helper.rb

@@ -4,13 +4,12 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
-require 'opentelemetry-sdk'
-require 'opentelemetry-test-helpers'
-require 'minitest/autorun'
-require 'pry'
-
 if RUBY_ENGINE == 'ruby'
   require 'simplecov'
   SimpleCov.start
-  SimpleCov.minimum_coverage 85
+  SimpleCov.minimum_coverage 70
 end
+require 'opentelemetry-sdk'
+require 'opentelemetry-test-helpers'
+require 'minitest/autorun'
+require 'pry'