Skip to content

Commit c3cd448

Browse files
quinna-hbouwkastsoulcutterivoanjoanmarchenko
authored
add supported versions workflow (#4210)
* add supported version script and table * update script * rubocop lint * modify script locations, add description to md table * improve table output * wip * refactor code * wip * add supported versions * add branch for testing * remove json to avoid merge conflict issues * update PR body * Update .github/scripts/generate_table_versions.rb Co-authored-by: Steven Bouwkamp <[email protected]> * switch to use gem declarations instead of hardcoded mappings * linting checks * cleanup comments * refactor code * cleanup code * Combine duplicate option table rows The documentation for instrumenting rake had two rows for the same option key. This consolidates those entries into a single row. * Enable type checking for AgentSettingsResolver/AgentSettings Steep doesn't seem to be a big fan of Structs so I just went ahead and turned the `AgentSettings` into a regular class that's equivalent to the struct we had before. (In particular, I decided to still keep every field as optional). Ideally this would be a `Data` class, but we're far from dropping support for Rubies that don't have it. * Move url building behavior from `AgentBaseUrl` to `AgentSettings` This is preparation to also share this behavior with profiling. * Refactor crashtracking to use `AgentSettings#url` The behavior from the old `AgentBaseUrl` is now contained in `AgentSettings` so we can clean up the extra logic. * [PROF-11078] Fix profiling exception when agent url is an ipv6 address **What does this PR do?** This PR builds atop #4237 and fixes a similar-ish issue in the profiler caused by the same mishandling of ipv6 addresses. In particular, when provided with an ipv6 address in the agent url, the profiler would fail with an exception: ``` $ env DD_AGENT_HOST=2001:db8:1::2 DD_PROFILING_ENABLED=true \ bundle exec ddprofrb exec ruby -e "sleep 2" dd-trace-rb/lib/datadog/profiling/http_transport.rb:27:in `initialize': Failed to initialize transport: invalid authority (ArgumentError) ``` **Motivation:** Luckily we didn't have any customers using this, as it fails immediately and loudly, but it's still a bug on a configuration that should be supported. **Additional Notes:** Since we had similar buggy logic copy-pasted in crashtracking and profiling (crashtracking had been fixed in #4237) I chose to extract out the relevant logic into the `AgentSettings` class, so that both can reuse it. **How to test the change?** I've added unit test coverage for this issue to profiling, and the snippet above can be used to end-to-end test it's working fine. Here's how it looks on my machine now: ``` E, [2025-01-02T17:32:32.398756 #359317] ERROR -- datadog: [datadog] (dd-trace-rb/lib/datadog/profiling/http_transport.rb:68:in `export') Failed to report profiling data (agent: http://[2001:db8:1::2]:8126/): failed ddog_prof_Exporter_send: error trying to connect: tcp connect error: Network is unreachable (os error 101): tcp connect error: Network is unreachable (os error 101): Network is unreachable (os error 101) ``` E.g. we correctly try to connect to the dummy address, and fail :) (Note: The error message is a bit ugly AND repeats itself a bit. That's being tracked separately in DataDog/libdatadog#283 ) * Implement `==` for new `AgentSettings` class Forgot this one, some of our tests relied on it! * use Ruby 3.4.1 for test-memcheck GHA * Update exceptions file with another variant of thread creation memory leak Since our exceptions match on the stack, they are affected by internal naming changes, and it looks like a new `ruby_xcalloc_body` function is now showing up in the stack. * Introduce Ruby 3.5 gemfile variant for testing with dev builds This is waaay incomplete in terms of adding support for Ruby 3.5 but should get us going for ASAN testing for now. * Update list of files used to compute cache checksum In practice this shouldn't make a difference, since the final lockfiles are supposed to be a superset of the root-level gemfile BUT the `Appraisals` file no longer exists anyway and "just in case" let's have it anyway as it seems more correct. * Bump Ruby 3.4 integration image to stable version * Remove workaround for strscan issue This is not expected to be an issue in 3.5 (and is probably fixed for 3.4 as well, but I'll leave that for a separate PR to not affect the appraisals). * Add unsafe api calls checker to track down issues such as #4195 This checker is used to detect accidental thread scheduling switching points happening during profiling sampling. See the bigger comment in unsafe_api_calls_check.h . I was able to check that this checker correctly triggers for the bug in #4195, and also the bug I'm going to fix next, which is the use of `rb_hash_lookup` in the otel context reading code. * Fix going into Ruby code when looking up otel context `rb_hash_lookup` calls `#hash` on the key being looked up so it's safe to use unless during sampling. This can cause the same issue as we saw in #4195 leading to ``` [BUG] unexpected situation - recordd:1 current:0 -- C level backtrace information ------------------------------------------- ruby(rb_print_backtrace+0x11) [0x55ba03ccf90f] vm_dump.c:820 ruby(rb_vm_bugreport) vm_dump.c:1151 ruby(bug_report_end+0x0) [0x55ba03e91607] error.c:1042 ruby(rb_bug_without_die) error.c:1042 ruby(die+0x0) [0x55ba03ac0998] error.c:1050 ruby(rb_bug) error.c:1052 ruby(disallow_reentry+0x0) [0x55ba03ab6dcc] vm_sync.c:226 ruby(rb_ec_vm_lock_rec_check+0x1a) [0x55ba03cb17aa] eval_intern.h:144 ruby(rb_ec_tag_state) eval_intern.h:155 ruby(rb_vm_exec) vm.c:2484 ruby(vm_invoke_proc+0x201) [0x55ba03cb62b1] vm.c:1509 ruby(rb_vm_invoke_proc+0x33) [0x55ba03cb65d3] vm.c:1728 ruby(thread_do_start_proc+0x176) [0x55ba03c63516] thread.c:598 ruby(thread_do_start+0x12) [0x55ba03c648a2] thread.c:615 ruby(thread_start_func_2) thread.c:672 ruby(nt_start+0x107) [0x55ba03c65137] thread_pthread.c:2187 /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xd9) [0x7ff360b66609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7ff360a70353] ``` * Avoid trying to sample allocations when VM is raising exception During my experiments to reproduce issues around allocation profiling, I've noted that the VM is in an especially delicate state during exception raising, so let's just decline to sample in this situation. * Update tests with new signatures for test methods * Check if symbol is static before calling SYM2ID on it It occurs to me that if a symbol is dynamic, we were causing it to become a static symbol (e.g. making it never be able to be garbage collected). This can be very bad! And also, we know the symbol we're looking for must be a static symbol because if nothing else, our initialization caused it to become a static symbol. Thus, if we see a dynamic symbol, we can stop there, since by definition it won't be the symbol we're looking after. This is... really awkward to add a specific unit test for, so I've just relied on our existing test coverage to show that this has not affected the correctness of our otel code. * Document that unsafe api calls checker is only for test code * Add 3.4 support * Update DevelopmentGuide * Remove `racc` gem from 3.3 and 3.4 appraisal files * [🤖] Lock Dependency: https://github.com/DataDog/dd-trace-rb/actions/runs/12595964519 * Remove strscan specification in 3.4 gemfile * [🤖] Lock Dependency: https://github.com/DataDog/dd-trace-rb/actions/runs/12595969993 * add hardcoded update workflow file --------- Co-authored-by: Steven Bouwkamp <[email protected]> Co-authored-by: Bradley Schaefer <[email protected]> Co-authored-by: Ivo Anjo <[email protected]> Co-authored-by: Andrey Marchenko <[email protected]> Co-authored-by: Sarah Chen <[email protected]> Co-authored-by: ivoanjo <[email protected]>
1 parent d5f88ea commit c3cd448

File tree

18 files changed

+266
-0
lines changed

18 files changed

+266
-0
lines changed

Diff for: ‎.github/scripts/find_gem_version_bounds.rb

+149
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
require 'pathname'
2+
require 'rubygems'
3+
require 'json'
4+
require 'bundler'
5+
6+
lib = File.expand_path('lib', __dir__)
7+
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
8+
require 'datadog'
9+
10+
class GemfileProcessor
11+
SPECIAL_CASES = {
12+
"opensearch" => "OpenSearch" # special case because opensearch = OpenSearch not Opensearch
13+
}.freeze
14+
EXCLUDED_INTEGRATIONS = ["configuration", "propagation", "utils"].freeze
15+
16+
def initialize(directory: 'gemfiles/', contrib_dir: 'lib/datadog/tracing/contrib/')
17+
@directory = directory
18+
@contrib_dir = contrib_dir
19+
@min_gems = { 'ruby' => {}, 'jruby' => {} }
20+
@max_gems = { 'ruby' => {}, 'jruby' => {} }
21+
@integration_json_mapping = {}
22+
end
23+
24+
def process
25+
parse_gemfiles
26+
process_integrations
27+
include_hardcoded_versions
28+
write_output
29+
end
30+
31+
private
32+
33+
34+
def parse_gemfiles(directory = 'gemfiles/')
35+
gemfiles = Dir.glob(File.join(@directory, '*'))
36+
gemfiles.each do |gemfile_name|
37+
runtime = File.basename(gemfile_name).split('_').first # ruby or jruby
38+
next unless %w[ruby jruby].include?(runtime)
39+
# parse the gemfile
40+
if gemfile_name.end_with?(".gemfile")
41+
process_gemfile(gemfile_name, runtime)
42+
elsif gemfile_name.end_with?('.gemfile.lock')
43+
process_lockfile(gemfile_name, runtime)
44+
end
45+
end
46+
47+
end
48+
49+
def process_gemfile(gemfile_name, runtime)
50+
begin
51+
definition = Bundler::Definition.build(gemfile_name, nil, nil)
52+
definition.dependencies.each do |dependency|
53+
gem_name = dependency.name
54+
version = dependency.requirement.to_s
55+
update_gem_versions(runtime, gem_name, version)
56+
end
57+
rescue Bundler::GemfileError => e
58+
puts "Error reading Gemfile: #{e.message}"
59+
end
60+
end
61+
62+
def process_lockfile(gemfile_name, runtime)
63+
lockfile_contents = File.read(gemfile_name)
64+
parser = Bundler::LockfileParser.new(lockfile_contents)
65+
parser.specs.each do |spec|
66+
gem_name = spec.name
67+
version = spec.version.to_s
68+
update_gem_versions(runtime, gem_name, version)
69+
end
70+
end
71+
72+
def update_gem_versions(runtime, gem_name, version)
73+
return unless version_valid?(version)
74+
75+
gem_version = Gem::Version.new(version)
76+
77+
# Update minimum gems
78+
if @min_gems[runtime][gem_name].nil? || gem_version < Gem::Version.new(@min_gems[runtime][gem_name])
79+
@min_gems[runtime][gem_name] = version
80+
end
81+
82+
# Update maximum gems
83+
if @max_gems[runtime][gem_name].nil? || gem_version > Gem::Version.new(@max_gems[runtime][gem_name])
84+
@max_gems[runtime][gem_name] = version
85+
end
86+
end
87+
88+
# Helper: Validate the version format
89+
def version_valid?(version)
90+
return false if version.nil? || version.strip.empty?
91+
92+
Gem::Version.new(version)
93+
true
94+
rescue ArgumentError
95+
false
96+
end
97+
98+
99+
def process_integrations
100+
integrations = Datadog::Tracing::Contrib::REGISTRY.map(&:name).map(&:to_s)
101+
integrations.each do |integration|
102+
next if EXCLUDED_INTEGRATIONS.include?(integration)
103+
104+
integration_name = resolve_integration_name(integration)
105+
106+
@integration_json_mapping[integration] = [
107+
@min_gems['ruby'][integration_name],
108+
@max_gems['ruby'][integration_name],
109+
@min_gems['jruby'][integration_name],
110+
@max_gems['jruby'][integration_name]
111+
]
112+
end
113+
end
114+
115+
def include_hardcoded_versions
116+
# `httpx` is maintained externally
117+
@integration_json_mapping['httpx'] = [
118+
'0.11', # Min version Ruby
119+
'0.11', # Max version Ruby
120+
nil, # Min version JRuby
121+
nil # Max version JRuby
122+
]
123+
124+
# `makara` is part of `activerecord`
125+
@integration_json_mapping['makara'] = [
126+
'0.3.5', # Min version Ruby
127+
'0.3.5', # Max version Ruby
128+
nil, # Min version JRuby
129+
nil # Max version JRuby
130+
]
131+
end
132+
133+
def resolve_integration_name(integration)
134+
mod_name = SPECIAL_CASES[integration] || integration.split('_').map(&:capitalize).join
135+
module_name = "Datadog::Tracing::Contrib::#{mod_name}"
136+
integration_module = Object.const_get(module_name)::Integration
137+
integration_module.respond_to?(:gem_name) ? integration_module.gem_name : integration
138+
rescue NameError, NoMethodError
139+
puts "Fallback for #{integration}: module or gem_name not found."
140+
integration
141+
end
142+
143+
def write_output
144+
@integration_json_mapping = @integration_json_mapping.sort.to_h
145+
File.write("gem_output.json", JSON.pretty_generate(@integration_json_mapping))
146+
end
147+
end
148+
149+
GemfileProcessor.new.process

Diff for: ‎.github/scripts/generate_table_versions.rb

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
require 'json'
2+
3+
input_file = 'gem_output.json'
4+
output_file = 'integration_versions.md'
5+
6+
data = JSON.parse(File.read(input_file))
7+
8+
comment = "# Integrations\n\n"
9+
header = "| Integration | Ruby Min | Ruby Max | JRuby Min | JRuby Max |\n"
10+
separator = "|-------------|----------|-----------|----------|----------|\n"
11+
rows = data.map do |integration_name, versions|
12+
ruby_min, ruby_max, jruby_min, jruby_max = versions.map { |v| v || "None" }
13+
"| #{integration_name} | #{ruby_min} | #{ruby_max} | #{jruby_min} | #{jruby_max} |"
14+
end
15+
16+
File.open(output_file, 'w') do |file|
17+
file.puts comment
18+
file.puts header
19+
file.puts separator
20+
rows.each { |row| file.puts row }
21+
end

Diff for: ‎.github/workflows/generate-supported-versions.yml

+49
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
name: "Generate Supported Versions"
2+
3+
on:
4+
workflow_dispatch:
5+
6+
7+
concurrency:
8+
group: ${{ github.workflow }}
9+
cancel-in-progress: true
10+
11+
jobs:
12+
build:
13+
runs-on: ubuntu-22.04
14+
permissions:
15+
contents: read
16+
steps:
17+
- name: Checkout code
18+
uses: actions/checkout@v4
19+
20+
- name: Set up Ruby
21+
uses: ruby/setup-ruby@v1
22+
with:
23+
bundler-cache: true # runs bundle install
24+
ruby-version: "3.3"
25+
26+
- name: Update latest
27+
run: bundle exec ruby .github/scripts/find_gem_version_bounds.rb
28+
29+
- name: Generate versions table
30+
run: ruby .github/scripts/generate_table_versions.rb
31+
32+
- run: git diff
33+
34+
- name: Create Pull Request
35+
id: cpr
36+
uses: peter-evans/create-pull-request@v7
37+
with:
38+
token: ${{ secrets.GHA_PAT }}
39+
branch: auto-generate/update-supported-versions
40+
title: '[🤖] Update Supported Versions'
41+
base: master
42+
labels: dev/internal, integrations
43+
commit-message: "Test creating supported versions"
44+
delete-branch: true
45+
body: |
46+
This is a PR to update the table for supported integration versions.
47+
Workflow run: [Generate Supported Versions](https://github.com/DataDog/dd-trace-rb/actions/workflows/generate-supported-versions.yml)
48+
This should be tied to tracer releases, or triggered manually.
49+

Diff for: ‎lib/datadog/tracing/contrib/action_cable/integration.rb

+3
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,9 @@ class Integration
1717

1818
# @public_api Changing the integration name or integration options can cause breaking changes
1919
register_as :action_cable, auto_patch: false
20+
def self.gem_name
21+
'actioncable'
22+
end
2023

2124
def self.version
2225
Gem.loaded_specs['actioncable'] && Gem.loaded_specs['actioncable'].version

Diff for: ‎lib/datadog/tracing/contrib/action_mailer/integration.rb

+4
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,10 @@ class Integration
1818
# @public_api Changing the integration name or integration options can cause breaking changes
1919
register_as :action_mailer, auto_patch: false
2020

21+
def self.gem_name
22+
'actionmailer'
23+
end
24+
2125
def self.version
2226
Gem.loaded_specs['actionmailer'] && Gem.loaded_specs['actionmailer'].version
2327
end

Diff for: ‎lib/datadog/tracing/contrib/action_pack/integration.rb

+3
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,9 @@ class Integration
1818

1919
# @public_api Changing the integration name or integration options can cause breaking changes
2020
register_as :action_pack, auto_patch: false
21+
def self.gem_name
22+
'actionpack'
23+
end
2124

2225
def self.version
2326
Gem.loaded_specs['actionpack'] && Gem.loaded_specs['actionpack'].version

Diff for: ‎lib/datadog/tracing/contrib/action_view/integration.rb

+3
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,9 @@ class Integration
1818

1919
# @public_api Changing the integration name or integration options can cause breaking changes
2020
register_as :action_view, auto_patch: false
21+
def self.gem_name
22+
'actionview'
23+
end
2124

2225
def self.version
2326
# ActionView is its own gem in Rails 4.1+

Diff for: ‎lib/datadog/tracing/contrib/active_job/integration.rb

+3
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,9 @@ class Integration
1717

1818
# @public_api Changing the integration name or integration options can cause breaking changes
1919
register_as :active_job, auto_patch: false
20+
def self.gem_name
21+
'activejob'
22+
end
2023

2124
def self.version
2225
Gem.loaded_specs['activejob'] && Gem.loaded_specs['activejob'].version

Diff for: ‎lib/datadog/tracing/contrib/active_record/integration.rb

+4
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,10 @@ class Integration
2222
# @public_api Changing the integration name or integration options can cause breaking changes
2323
register_as :active_record, auto_patch: false
2424

25+
def self.gem_name
26+
'activerecord'
27+
end
28+
2529
def self.version
2630
Gem.loaded_specs['activerecord'] && Gem.loaded_specs['activerecord'].version
2731
end

Diff for: ‎lib/datadog/tracing/contrib/active_support/integration.rb

+3
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,9 @@ class Integration
1919

2020
# @public_api Changing the integration name or integration options can cause breaking changes
2121
register_as :active_support, auto_patch: false
22+
def self.gem_name
23+
'activesupport'
24+
end
2225

2326
def self.version
2427
Gem.loaded_specs['activesupport'] && Gem.loaded_specs['activesupport'].version

Diff for: ‎lib/datadog/tracing/contrib/aws/integration.rb

+3
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@ class Integration
1616

1717
# @public_api Changing the integration name or integration options can cause breaking changes
1818
register_as :aws, auto_patch: true
19+
def self.gem_name
20+
'aws-sdk-core'
21+
end
1922

2023
def self.version
2124
if Gem.loaded_specs['aws-sdk']

Diff for: ‎lib/datadog/tracing/contrib/concurrent_ruby/integration.rb

+3
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@ class Integration
1616

1717
# @public_api Changing the integration name or integration options can cause breaking changes
1818
register_as :concurrent_ruby
19+
def self.gem_name
20+
'concurrent-ruby'
21+
end
1922

2023
def self.version
2124
Gem.loaded_specs['concurrent-ruby'] && Gem.loaded_specs['concurrent-ruby'].version

Diff for: ‎lib/datadog/tracing/contrib/httprb/integration.rb

+3
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,9 @@ class Integration
1717

1818
# @public_api Changing the integration name or integration options can cause breaking changes
1919
register_as :httprb
20+
def self.gem_name
21+
'http'
22+
end
2023

2124
def self.version
2225
Gem.loaded_specs['http'] && Gem.loaded_specs['http'].version

Diff for: ‎lib/datadog/tracing/contrib/kafka/integration.rb

+3
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@ class Integration
1616

1717
# @public_api Changing the integration name or integration options can cause breaking changes
1818
register_as :kafka, auto_patch: false
19+
def self.gem_name
20+
'ruby-kafka'
21+
end
1922

2023
def self.version
2124
Gem.loaded_specs['ruby-kafka'] && Gem.loaded_specs['ruby-kafka'].version

Diff for: ‎lib/datadog/tracing/contrib/mongodb/integration.rb

+3
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,9 @@ class Integration
1717

1818
# @public_api Changing the integration name or integration options can cause breaking changes
1919
register_as :mongo, auto_patch: true
20+
def self.gem_name
21+
'mongo'
22+
end
2023

2124
def self.version
2225
Gem.loaded_specs['mongo'] && Gem.loaded_specs['mongo'].version

Diff for: ‎lib/datadog/tracing/contrib/opensearch/integration.rb

+3
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@ class Integration
1616

1717
# @public_api Changing the integration name or integration options can cause breaking changes
1818
register_as :opensearch, auto_patch: true
19+
def self.gem_name
20+
'opensearch-ruby'
21+
end
1922

2023
def self.version
2124
Gem.loaded_specs['opensearch-ruby'] \

Diff for: ‎lib/datadog/tracing/contrib/presto/integration.rb

+3
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@ class Integration
1616

1717
# @public_api Changing the integration name or integration options can cause breaking changes
1818
register_as :presto
19+
def self.gem_name
20+
'presto-client'
21+
end
1922

2023
def self.version
2124
Gem.loaded_specs['presto-client'] && Gem.loaded_specs['presto-client'].version

Diff for: ‎lib/datadog/tracing/contrib/rest_client/integration.rb

+3
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@ class Integration
1616

1717
# @public_api Changing the integration name or integration options can cause breaking changes
1818
register_as :rest_client
19+
def self.gem_name
20+
'rest-client'
21+
end
1922

2023
def self.version
2124
Gem.loaded_specs['rest-client'] && Gem.loaded_specs['rest-client'].version

0 commit comments

Comments
 (0)