Skip to content

chore(api): update benchmarks #2107

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kaylareopelle merged 2 commits into from
May 8, 2026
Merged

chore(api): update benchmarks #2107

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
eea25ae
chore(api): update benchmarks
xuan-cao-swi Apr 22, 2026
f867e89
add additional rake task group and add readme for that benchmark and …
xuan-cao-swi May 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions api/Gemfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,6 @@ gemspec
eval_gemfile '../contrib/Gemfile.shared'

group :test, :development do
gem 'opentelemetry-sdk', path: '../sdk', require: false
gem 'opentelemetry-test-helpers', path: '../test_helpers', require: false
end
15 changes: 15 additions & 0 deletions api/Rakefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,18 @@ default_tasks =
end

task default: default_tasks

namespace :bench do
bench_files = FileList['benchmarks/*_bench.rb']

bench_files.each do |file|
name = File.basename(file, '_bench.rb')
desc "Run #{name} benchmarks"
task name do
sh "bundle exec ruby #{file}"
end
end

desc 'Run all benchmarks'
task all: bench_files.map { |f| File.basename(f, '_bench.rb') }
end
133 changes: 133 additions & 0 deletions api/benchmarks/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# Trace API Benchmarks

This directory contains [benchmark-ips](https://github.com/evanphx/benchmark-ips) benchmarks for the OpenTelemetry Ruby API and SDK.

## Running the Benchmarks

Run from the `api/` directory:

```bash
# Run all benchmarks
rake bench:all

# Or run individual benchmarks:
bundle exec ruby benchmarks/context_bench.rb
bundle exec ruby benchmarks/id_generation_bench.rb
bundle exec ruby benchmarks/span_bench.rb
bundle exec ruby benchmarks/tracer_bench.rb
```

## Benchmark Files

| File | What it measures |
| ---- | --------------- |
| `context_bench.rb` | Context storage implementations — standard and fiber-local variants with both single and recursive value operations |
| `id_generation_bench.rb` | ID generation performance — trace IDs, span IDs, and random byte generation |
| `span_bench.rb` | Span operations — setting name, attributes, and adding events |
| `tracer_bench.rb` | Span creation performance — basic, with parent context, with attributes, and with links |

## Sample Run

### System Specifications

**OS Information:**

- Distribution: Ubuntu 24.04.3 LTS (Noble Numbat)
- Kernel: Linux 6.14.0-1018-aws
- Architecture: x86_64

**Memory:**

- Total: ~3.91 GB (4,006,000 kB)
- Available: ~3.40 GB (3,470,496 kB)
- Free: ~3.13 GB (3,195,972 kB)

**CPU:**

- Processor: Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
- Cores: 2
- Threads: 2
- Virtualization: Xen (Full)
- Cache: L1d 64 KiB × 2 | L1i 64 KiB × 2 | L2 512 KiB × 2 | L3 45 MiB

**Runtime:**

- Ruby: 3.4.0dev (2024-12-25 master f450108330) +PRISM [x86_64-linux]

### Benchmark Results

#### Context Operations

`bundle exec ruby benchmarks/context_bench.rb`

**Standard with_value Operations:**

| Context Type | Throughput | Time/Op | Relative Performance |
| --- | --- | --- | --- |
| FiberLocalArrayContext.with_value | 756,047.3 i/s | 1.32 μs/i | **Fastest** |
| FiberLocalLinkedListContext.with_value | 689,024.1 i/s | 1.45 μs/i | 1.10x slower |
| FiberAttributeContext.with_value | 651,214.4 i/s | 1.54 μs/i | 1.16x slower |
| ArrayContext.with_value | 633,636.0 i/s | 1.58 μs/i | 1.19x slower |
| LinkedListContext.with_value | 543,032.7 i/s | 1.84 μs/i | 1.39x slower |
| FiberLocalImmutableArrayContext.with_value | 488,004.8 i/s | 2.05 μs/i | 1.55x slower |
| FiberLocalVarContext.with_value | 477,231.8 i/s | 2.10 μs/i | 1.58x slower |
| ImmutableArrayContext.with_value | 473,185.9 i/s | 2.11 μs/i | 1.60x slower |

**Recursive with_value Operations:**

| Context Type | Throughput | Time/Op | Relative Performance |
| --- | --- | --- | --- |
| FiberLocalArrayContext.with_value | 79,682.3 i/s | 12.55 μs/i | **Fastest** |
| FiberLocalLinkedListContext.with_value | 70,142.6 i/s | 14.26 μs/i | 1.14x slower |
| FiberAttributeContext.with_value | 69,806.8 i/s | 14.33 μs/i | 1.14x slower |
| ArrayContext.with_value | 66,249.2 i/s | 15.09 μs/i | 1.20x slower |
| LinkedListContext.with_value | 57,670.2 i/s | 17.34 μs/i | 1.38x slower |
| FiberLocalVarContext.with_value | 49,221.0 i/s | 20.32 μs/i | 1.62x slower |
| FiberLocalImmutableArrayContext.with_value | 48,464.7 i/s | 20.63 μs/i | 1.64x slower |
| ImmutableArrayContext.with_value | 47,063.2 i/s | 21.25 μs/i | 1.69x slower |

#### ID Generation

`bundle exec ruby benchmarks/id_generation_bench.rb`

**Trace ID Generation:**

| Method | Throughput | Time/Op | Relative Performance |
| --- | --- | --- | --- |
| generate_trace_id_while | 3,620,066.2 i/s | 276.24 ns/i | **Fastest** |
| generate_trace_id | 2,052,539.5 i/s | 487.20 ns/i | 1.76x slower |

**Span ID Generation:**

| Method | Throughput | Time/Op | Relative Performance |
| --- | --- | --- | --- |
| generate_span_id_while | 3,797,911.0 i/s | 263.30 ns/i | **Fastest** |
| generate_span_id | 1,963,041.8 i/s | 509.41 ns/i | 1.93x slower |

**Random Bytes Generation:**

| Method | Throughput | Time/Op | Relative Performance |
| --- | --- | --- | --- |
| generate_r_in_place | 1,712,273.8 i/s | 584.02 ns/i | **Fastest** |
| generate_r | 1,348,496.2 i/s | 741.57 ns/i | 1.27x slower |

#### Span Operations

`bundle exec ruby benchmarks/span_bench.rb`

| Operation | Throughput | Time/Op | Relative Performance |
| --- | --- | --- | --- |
| name= | 3,642,110.6 i/s | 274.57 ns/i | **Fastest** |
| set_attribute | 1,996,785.3 i/s | 500.80 ns/i | 1.82x slower |
| add_event | 268,851.5 i/s | 3.72 μs/i | 13.55x slower |

#### Tracer Span Creation

`bundle exec ruby benchmarks/tracer_bench.rb`

| Operation | Throughput | Time/Op | Relative Performance |
| --- | --- | --- | --- |
| start span with parent context | 2,619,109.2 i/s | 381.81 ns/i | **Fastest** |
| start span | 2,269,594.3 i/s | 440.61 ns/i | 1.15x slower |
| start span with attributes | 2,007,011.5 i/s | 498.25 ns/i | 1.30x slower |
| start span with links | 1,991,015.3 i/s | 502.26 ns/i | 1.32x slower |
6 changes: 3 additions & 3 deletions api/benchmarks/context_bench.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
#
# SPDX-License-Identifier: Apache-2.0

require 'benchmark/ipsa'
require 'benchmark/ips'
require 'concurrent-ruby'
require 'opentelemetry'

Expand Down Expand Up @@ -784,7 +784,7 @@ def set_values(values) # rubocop:disable Naming/AccessorMethodName:
ROOT = empty.freeze
end

Benchmark.ipsa do |x|
Benchmark.ips do |x|

@thompson-tomo thompson-tomo Apr 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure that this isn't changing the measurements/data being captured/recorded?

@xuan-cao-swi xuan-cao-swi Apr 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ipsa only have additional allocations stats from MemoryProfiler.report()
e.g.

Allocations -------------------------------------
               name=       0/0  alloc/ret        0/0  strings/ret
       set_attribute       0/0  alloc/ret        0/0  strings/ret
           add_event       2/2  alloc/ret        0/0  strings/ret

I'm not sure this information is still valuable or relevant, as it doesn't provide meaningful comparisons like ips does (e.g., xxx is 1.45x faster than xxx). ipsa is also quite old and conflicts with using the latest version of ips. If allocation data is truly needed, we could simply inline/copy the relevant MemoryProfiler.report logic rather than depending on an unmaintained gem.

@thompson-tomo thompson-tomo Apr 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am All for switching from an unmaintained gem, my question steemed from reading the pr description.

x.report 'FiberAttributeContext.with_value' do
FiberAttributeContext.with_value('key', 'value') { |ctx, _| ctx }
end
Expand Down Expand Up @@ -820,7 +820,7 @@ def set_values(values) # rubocop:disable Naming/AccessorMethodName:
x.compare!
end

Benchmark.ipsa do |x| # rubocop:disable Metrics/BlockLength
Benchmark.ips do |x| # rubocop:disable Metrics/BlockLength
x.report 'LinkedListContext.with_value recursive' do
LinkedListContext.with_value('key', 'value') do
LinkedListContext.with_value('key', 'value') do
Expand Down
8 changes: 4 additions & 4 deletions api/benchmarks/id_generation_bench.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
#
# SPDX-License-Identifier: Apache-2.0

require 'benchmark/ipsa'
require 'benchmark/ips'

INVALID_SPAN_ID = ("\0" * 8).b
INVALID_TRACE_ID = ("\0" * 16).b
Expand Down Expand Up @@ -45,19 +45,19 @@ def generate_r_in_place(trace_id)
64 - x.bit_length
end

Benchmark.ipsa do |x|
Benchmark.ips do |x|
x.report('generate_trace_id') { generate_trace_id }
x.report('generate_trace_id_while') { generate_trace_id_while }
x.compare!
end

Benchmark.ipsa do |x|
Benchmark.ips do |x|
x.report('generate_span_id') { generate_span_id }
x.report('generate_span_id_while') { generate_span_id_while }
x.compare!
end

Benchmark.ipsa do |x|
Benchmark.ips do |x|
trace_id = generate_trace_id
x.report('generate_r') { generate_r(trace_id) }
x.report('generate_r_in_place') { generate_r_in_place(trace_id) }
Expand Down
4 changes: 2 additions & 2 deletions api/benchmarks/span_bench.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
#
# SPDX-License-Identifier: Apache-2.0

require 'benchmark/ipsa'
require 'benchmark/ips'
require 'opentelemetry/sdk'

OpenTelemetry::SDK.configure
Expand All @@ -18,7 +18,7 @@
'http.url' => 'blogs/index'
}

Benchmark.ipsa do |x|
Benchmark.ips do |x|
x.report 'name=' do
span.name = 'new_name'
end
Expand Down
4 changes: 2 additions & 2 deletions api/benchmarks/tracer_bench.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
#
# SPDX-License-Identifier: Apache-2.0

require 'benchmark/ipsa'
require 'benchmark/ips'
require 'opentelemetry'

tracer = OpenTelemetry::Trace::Tracer.new
Expand All @@ -26,7 +26,7 @@
)
end

Benchmark.ipsa do |x|
Benchmark.ips do |x|
x.report 'start span' do
span = tracer.start_span('test_span')
span.finish
Expand Down
2 changes: 1 addition & 1 deletion api/opentelemetry-api.gemspec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Gem::Specification.new do |spec|
spec.required_ruby_version = '>= 3.3'
spec.add_dependency 'logger'

spec.add_development_dependency 'benchmark-ipsa', '~> 0.2.0'
spec.add_development_dependency 'benchmark-ips', '~> 2.14.0'

@thompson-tomo thompson-tomo Apr 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because we are switching tools why don't we move this to the gemfile?

@xuan-cao-swi xuan-cao-swi Apr 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am ok with either put here or in gemfile

@thompson-tomo thompson-tomo Apr 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would rather gem so that we can move towards eliminating the disabling of the corresponding rubocop cop & the updating process is natively handled by renovate.

spec.add_development_dependency 'concurrent-ruby', '~> 1.3'
spec.add_development_dependency 'minitest', '~> 5.0'
spec.add_development_dependency 'opentelemetry-test-helpers'
Expand Down
Loading