docs(experimental): reviewer-focused documentation for Metrics API/SDK (spec crosswalks, architecture, review guides)

apps/opentelemetry_api_experimental/ARCHITECTURE.md

@@ -0,0 +1,40 @@
+opentelemetry_api_experimental – Architecture (Metrics API)
+==========================================================
+
+Purpose: Describe the instrumentation surface and how it delegates to the SDK so non-Erlang reviewers can reason about behavior.
+
+API surface
+-----------
+- Erlang macros in `include/otel_meter.hrl` provide a minimal, ergonomic API:
+  - Instrument creation: `?create_counter`, `?create_updown_counter`, `?create_histogram`, and observable counterparts.
+  - Recording: `?counter_add`, `?updown_counter_add`, `?histogram_record`.
+  - Observables: `?register_callback` for one or many instruments.
+- Macros resolve the “current meter” automatically from the module’s application scope and call underlying modules (`otel_meter`, `otel_counter`, `otel_histogram`, etc.).
+
+Delegation to SDK
+-----------------
+- Instrument creation and recording calls are forwarded to the SDK’s MeterProvider (`otel_meter_server`).
+- The SDK applies views, aggregations, temporality, and export; the API does not implement these rules.
+
+Elixir wrapper modules
+----------------------
+- `lib/open_telemetry/*.ex` expose equivalent APIs for Elixir users, delegating to the same underlying functions.
+
+Conceptual flow
+---------------
+1) Application code includes `otel_meter.hrl` and uses macros to create instruments and record measurements.
+2) Macros capture the current context and meter, and call the appropriate API module.
+3) The API module calls into the SDK MeterProvider, which routes to streams and aggregations.
+
+Interoperability notes
+----------------------
+- Attributes are passed through as maps; filtering happens in the SDK via Views.
+- Units and descriptions are provided at creation and propagated to the SDK.
+- Callback lifecycles for observables are controlled by the SDK’s Readers during collection.
+
+See also
+--------
+- `SPEC_COMPLIANCE.md` for spec crosswalk and code anchors.
+- Package README for quickstart and macro cheat sheet.
+
+

apps/opentelemetry_api_experimental/GLOSSARY.md

@@ -0,0 +1,15 @@
+Glossary (API)
+==============
+
+- Instrument: A Counter, UpDownCounter, Histogram, or Observable variant used to capture measurements.
+- Measurement: A numeric value plus optional attributes recorded against an instrument.
+- Attributes: Key/value pairs attached to a measurement for dimensionality.
+- Meter: Factory for instruments within an instrumentation scope; resolved implicitly by macros.
+- Instrumentation scope: Name/version/schema of the library emitting telemetry.
+- View: SDK rule that selects instruments, filters attributes, and chooses an aggregation.
+- Aggregation: Sum, Last Value, Explicit Bucket Histogram, or Drop.
+- Temporality: Delta (per-interval) or Cumulative (running total), selected by the Reader.
+- Reader: SDK component that collects and exports metrics.
+- Exporter: Component that serializes and sends metrics (Console, OTLP).
+
+

apps/opentelemetry_api_experimental/README.md

@@ -164,3 +164,13 @@ AtomGauge = ?create_observable_gauge(AtomCountName, #{description => <<"Number o
 The callbacks are run when the Metric Reader collects metrics for export. See
 the Experimental SDK's `README.md` for more details on Metric Readers and their
 configuration.
+
+## For non-Erlang reviewers
+
+- **What to read first**: See `SPEC_COMPLIANCE.md` in this folder for a plain-language crosswalk of the Metrics API to the OpenTelemetry spec and where behaviors are implemented.
+- **How to validate quickly**:
+  - Ensure the experimental SDK app `opentelemetry_experimental` is included and configured with a console reader (see that app's README).
+  - Use the macros in `include/otel_meter.hrl` to create an instrument and record a value.
+  - Expect to see aggregated metric points printed by the SDK's console exporter.
+- **Where defaults live**: Aggregation and temporality are SDK concerns; see `../opentelemetry_experimental/SPEC_COMPLIANCE.md` for defaults and reader behavior.
+

apps/opentelemetry_api_experimental/REVIEW_GUIDE.md

@@ -0,0 +1,81 @@
+Review Guide (Metrics API)
+==========================
+
+Audience: Validate instrumentation behavior using macros, without reading Erlang.
+
+Prerequisites
+-------------
+- Ensure `opentelemetry_experimental` (SDK) is included and configured with a console reader (see SDK REVIEW_GUIDE/README).
+- Include the API header in your module: `-include_lib("opentelemetry_api_experimental/include/otel_meter.hrl").`
+
+1) Create instruments
+---------------------
+Goal: Instruments can be created with name, unit, and description.
+
+Example (Counter):
+```erlang
+?create_counter(app_counter, #{description => <<"Requests">>, unit => '1'}).
+```
+
+Expect: No error on creation; SDK will create streams on first record.
+
+2) Record synchronous measurements
+----------------------------------
+Goal: Recording calls forward context, meter, and attributes to SDK.
+
+Counter:
+```erlang
+?counter_add(app_counter, 5, #{host => <<"a">>}).
+```
+
+UpDownCounter and Histogram similarly:
+```erlang
+?updown_counter_add(app_load, -2, #{}).
+?histogram_record(latency_ms, 37, #{path => <<"/ping">>}).
+```
+
+Expect (with console reader): Sum/Histogram points printed periodically.
+
+3) Observable instruments
+-------------------------
+Goal: Callback-based measurements appear only on collection.
+
+Single instrument with callback at creation:
+```erlang
+?create_observable_gauge(proc_count,
+                         fun(_) -> [{erlang:system_info(process_count), #{}}] end,
+                         [],
+                         #{description => <<"Proc count">>, unit => '1'}).
+```
+
+Multiple instruments with one callback:
+```erlang
+G1 = ?create_observable_gauge(proc_count, #{}),
+G2 = ?create_observable_gauge(atom_count, #{}),
+?register_callback([G1, G2],
+                   fun(_) ->
+                       [{proc_count, [{erlang:system_info(process_count), #{}}]},
+                        {atom_count, [{erlang:system_info(atom_count), #{}}]}]
+                   end, []).
+```
+
+Expect: Values printed each collection tick; nothing in between.
+
+4) Attribute filtering via SDK Views
+------------------------------------
+Goal: Attributes are forwarded intact by API; filtering happens in SDK.
+- Configure a View in SDK with `attribute_keys` limited to `[host]`.
+- Record with extra attributes; expect only `host` to remain.
+
+5) Negative value handling
+--------------------------
+Goal: API forwards values; SDK enforces monotonicity.
+- Add negative values to a Counter; the SDK will discard (see SDK REVIEW_GUIDE step 1).
+
+Pointers
+--------
+- Architecture: `ARCHITECTURE.md`
+- Spec crosswalk: `SPEC_COMPLIANCE.md`
+- SDK validation: `../opentelemetry_experimental/REVIEW_GUIDE.md`
+
+

apps/opentelemetry_api_experimental/SPEC_COMPLIANCE.md

@@ -0,0 +1,108 @@
+opentelemetry_api_experimental – Spec Compliance Crosswalk
+==========================================================
+
+Audience: Reviewers validating OpenTelemetry Metrics API semantics without Erlang knowledge.
+
+Scope: Instrument types, recording semantics, observables, attributes, scope propagation, and how these map to SDK behavior.
+
+How to read code anchors: Blocks below cite file, start:end lines. They are illustrative; exact line numbers may drift slightly across commits.
+
+1) Instrument kinds supported
+-----------------------------
+- Counter, UpDownCounter, Histogram
+- ObservableCounter, ObservableUpDownCounter, ObservableGauge
+
+API surface (macros) maps to implementation functions. The macros resolve the current meter automatically and call the appropriate module.
+
+```41:52:apps/opentelemetry_api_experimental/include/otel_meter.hrl
+-define(counter_add(Name, Number, Attributes),
+        otel_counter:add(otel_ctx:get_current(), ?current_meter, Name, Number, Attributes)).
+
+-define(histogram_record(Name, Number, Attributes),
+        otel_histogram:record(otel_ctx:get_current(), ?current_meter, Name, Number, Attributes)).
+
+-define(register_callback(Instruments, Callback, CallbackArgs),
+        otel_meter:register_callback(?current_meter, Instruments, Callback, CallbackArgs)).
+```
+
+Where implemented (Elixir shims):
+- `lib/open_telemetry/*.ex` provide Elixir wrappers matching the same instrument set.
+
+2) Instrument creation
+----------------------
+Macros to create instruments delegate to `otel_meter` (API) which communicates with the SDK meter provider.
+
+```10:35:apps/opentelemetry_api_experimental/include/otel_meter.hrl
+-define(create_counter(Name, Opts),
+        otel_meter:create_counter(?current_meter, Name, Opts)).
+-define(create_histogram(Name, Opts),
+        otel_meter:create_histogram(?current_meter, Name, Opts)).
+-define(create_updown_counter(Name, Opts),
+        otel_meter:create_updown_counter(?current_meter, Name, Opts)).
+-define(create_observable_counter(Name, Callback, CallbackArgs, Opts),
+        otel_meter:create_observable_counter(?current_meter, Name, Callback, CallbackArgs, Opts)).
+```
+
+Behavioral expectations (from spec):
+- Instrument name uniqueness per meter.
+- Observable instruments tie to callbacks; callbacks are invoked on collection.
+
+Where enforced (SDK): see SDK crosswalk (Streams/View matching and reader callback execution).
+
+3) Recording semantics
+----------------------
+Synchronous instruments record via macros which capture the current context and meter.
+
+```41:49:apps/opentelemetry_api_experimental/include/otel_meter.hrl
+-define(counter_add(Name, Number, Attributes),
+        otel_counter:add(otel_ctx:get_current(), ?current_meter, Name, Number, Attributes)).
+-define(updown_counter_add(Name, Number, Attributes),
+        otel_updown_counter:add(otel_ctx:get_current(), ?current_meter, Name, Number, Attributes)).
+```
+
+Spec alignment notes:
+- Attributes: arbitrary key/value map is forwarded; filtering is performed by SDK views.
+- Units/descriptions are provided at creation time and carried to SDK.
+
+4) Observable semantics
+-----------------------
+Observable instruments can be created with a callback, or callbacks can later be registered for multiple instruments.
+
+```50:52:apps/opentelemetry_api_experimental/include/otel_meter.hrl
+-define(register_callback(Instruments, Callback, CallbackArgs),
+        otel_meter:register_callback(?current_meter, Instruments, Callback, CallbackArgs)).
+```
+
+Spec alignment notes:
+- Callback returns a list of measurements with attributes per instrument.
+- Callbacks are run when the configured Metric Reader collects (see SDK crosswalk).
+
+5) Scope and resource propagation
+---------------------------------
+The API resolves a “current meter” tied to the application/module scope. The SDK attaches instrumentation scope and resource to exported metrics.
+
+```7:9:apps/opentelemetry_api_experimental/include/otel_meter.hrl
+-define(current_meter, opentelemetry_experimental:get_meter(
+                         opentelemetry:get_application_scope(?MODULE))).
+```
+
+6) Defaults (aggregation and temporality)
+-----------------------------------------
+Defaults are an SDK concern. API defers to SDK’s per-instrument defaults and per-reader temporality. See SDK SPEC_COMPLIANCE.md for exact mapping.
+
+Verification steps (manual)
+---------------------------
+1. Configure SDK reader with console exporter and 1s interval (see SDK README).
+2. Create each instrument via macros.
+3. Record values; observe expected aggregation on console.
+4. Register an observable callback; confirm values appear only on collection cycles.
+5. Add a view in SDK that filters attributes; confirm only allowed keys appear.
+
+Related code/tests
+------------------
+- API macros: `apps/opentelemetry_api_experimental/include/otel_meter.hrl`
+- Erlang modules: `apps/opentelemetry_api_experimental/src/otel_*`
+- Elixir wrappers: `apps/opentelemetry_api_experimental/lib/open_telemetry/*.ex`
+- Tests: `apps/opentelemetry_api_experimental/test/otel_metrics_test.exs`
+
+

apps/opentelemetry_api_experimental/TEST_COVERAGE.md

@@ -0,0 +1,24 @@
+Test Coverage (API)
+===================
+
+Files
+-----
+- `test/otel_metrics_test.exs` – Elixir tests covering API instrumentation semantics.
+
+Spec areas exercised
+--------------------
+- Instrument creation via macros.
+- Recording of counters, histograms, and updown counters with attributes.
+- Observable callback behavior when paired with SDK readers.
+
+How to run
+----------
+- Using Mix (from umbrella root with Elixir available):
+  - `mix test apps/opentelemetry_api_experimental`
+
+Gaps to consider
+----------------
+- Negative value propagation relies on SDK enforcement; see SDK tests for monotonic handling.
+- Per-reader temporality behavior belongs to SDK tests.
+
+

apps/opentelemetry_experimental/ARCHITECTURE.md

@@ -0,0 +1,108 @@
+opentelemetry_experimental – Architecture (Metrics SDK)
+======================================================
+
+Purpose: Explain how the experimental Metrics SDK is structured so non-Erlang reviewers can validate behavior against the OpenTelemetry spec.
+
+Key components
+--------------
+- MeterProvider: `otel_meter_server`
+  - Creates meters, registers views, accepts instruments and callbacks, and orchestrates per-reader streams and aggregations.
+- View engine: `otel_view`
+  - Compiles selection criteria (instrument name/kind/unit, meter scope) to ETS match specifications; matches instruments to views; supplies attribute filtering and aggregation overrides.
+- Readers: `otel_metric_reader`
+  - Runs collection (periodic or pull), applies temporality, exports via configured exporter.
+- Aggregations: `otel_aggregation_sum`, `otel_aggregation_last_value`, `otel_aggregation_histogram_explicit`, `otel_aggregation_drop`
+  - Maintain and collect metric state per stream; enforce spec semantics.
+- Exporters: `otel_metric_exporter_console`, `otel_exporter_metrics_otlp`
+  - Serialize and send metrics to stdout or OTLP backends.
+- Observables: `otel_observables`
+  - Executes callbacks on collection; supports multi-instrument callbacks.
+- Exemplars: `otel_metric_exemplar_*`
+  - Select and store exemplars according to filter/reservoir configuration.
+
+Data model and storage
+----------------------
+ETS tables (per MeterProvider instance):
+- `instruments_tab`: Registry of created instruments (per meter, per name).
+- `callbacks_tab`: Registered observable callbacks keyed by reader id.
+- `streams_tab`: View matches per reader; each stream holds aggregation config, temporality, attribute filter, exemplar reservoir.
+- `metrics_tab`: Aggregation state bucketed by stream name and attribute set.
+- `exemplars_tab`: Exemplar storage per stream.
+
+Supervision and processes
+-------------------------
+- `otel_meter_server` is a `gen_server` registered as the global MeterProvider.
+- Each `otel_metric_reader` is a `gen_server` with an optional periodic timer.
+- Exporters are invoked synchronously from readers during `collect`.
+
+Recording flow (synchronous instruments)
+---------------------------------------
+1) Instrument creation inserts into `instruments_tab` and computes streams by matching views per reader.
+2) Application code records a measurement; MeterProvider finds streams for the instrument and updates aggregations.
+3) Negative values for monotonic instruments are discarded.
+
+```mermaid
+sequenceDiagram
+  participant App
+  participant MeterServer as otel_meter_server
+  participant Streams as streams_tab
+  participant Metrics as metrics_tab
+  App->>MeterServer: record(Name, Value, Attributes)
+  MeterServer->>Streams: lookup streams for Name
+  MeterServer->>Metrics: aggregation update per stream
+  Note right of MeterServer: Reject negative for Counter/Histogram
+```
+
+Collection flow (observables and export)
+----------------------------------------
+1) Reader wakes (timer) or is triggered (pull) and runs `collect`.
+2) Reader fetches callbacks for its id and executes them, populating metrics for observable instruments.
+3) Reader iterates streams for its id, collects aggregated data (respecting temporality), and calls exporter with resource and scope.
+
+```mermaid
+sequenceDiagram
+  participant Reader as otel_metric_reader
+  participant MeterServer as otel_meter_server
+  participant Callbacks as callbacks_tab
+  participant Streams as streams_tab
+  participant Metrics as metrics_tab
+  participant Exporter
+  Reader->>MeterServer: register_with_server (startup)
+  Reader->>Callbacks: get callbacks for ReaderId
+  Reader->>MeterServer: run_callbacks(...)
+  Reader->>Streams: iterate streams for ReaderId
+  Reader->>Metrics: collect per stream (temporality)
+  Reader->>Exporter: export(metrics, resource)
+```
+
+Views and streams
+-----------------
+- Adding a view triggers recomputation of streams for all existing instruments.
+- Streams are per reader; a single instrument may yield multiple streams (one per reader, plus per matching view), each with its own aggregation and temporality.
+- Default behavior when no view matches: create a stream using the instrument defaults.
+
+Temporality and forgetting
+--------------------------
+- Readers hold a mapping from instrument kind to temporality (delta or cumulative).
+- Streams with delta temporality reset aggregation state after collection ("forget").
+- Observable instruments use "forget" semantics on each collection.
+
+Configuration surface
+---------------------
+- Readers: `{module, config}` with keys:
+  - `export_interval_ms`: integer (periodic) or undefined (pull).
+  - `exporter`: `{Module, #{...}}` passed to exporter `init/1`.
+  - Optional: default aggregation and temporality mappings.
+- Views: list of maps with `selector`, optional `name`, `description`, `attribute_keys`, `aggregation_module`, `aggregation_options` (e.g., histogram bucket boundaries).
+
+Responsibility boundaries
+-------------------------
+- API (`opentelemetry_api_experimental`) provides instrumentation macros and delegates to the SDK.
+- SDK (`opentelemetry_experimental`) enforces aggregation rules, views, temporality, and export.
+
+See also
+--------
+- `SPEC_COMPLIANCE.md` for spec crosswalk and code anchors.
+- Package README for configuration examples.
+
+