[PROF-12743] Runtime stack collection callback registration

[gyuheon0h]

What does this PR do?
We want to collect runtime frames for the crashtracker. This approach follows how we access the call stack for profiling.

Motivation:

Change log entry

Additional Notes:

How to test the change?

"Ruby and C method runtime stack capture" RSpec test

Run a ruby program with the crashtracker initialized. Runtime stacks should be visible in the experimental section of the crash report. Stacktrace emitted from an example script I wrote to test

{
  "format": "Datadog Runtime Callback 1.0",
  "frames": [
    {
      "file": "<Fiddle (C extension)>",
      "function": "free"
    },
    {
      "file": "debug_runtime_callback.rb",
      "function": "final_crash_point",
      "line": 244
    },
    {
      "file": "<Integer (C extension)>",
      "function": "times"
    },
    {
      "file": "debug_runtime_callback.rb",
      "function": "final_crash_point",
      "line": 243
    },
   .....
    {
      "file": "debug_runtime_callback.rb",
      "function": "main_crash_test",
      "line": 251
    },
    {
      "file": "debug_runtime_callback.rb",
      "function": "<main>",
      "line": 276
    },
    {
      "file": "<Kernel (C extension)>",
      "function": "fork"
    }
  ]
}

[github-actions]

👋 Hey @DataDog/ruby-guild, please fill "Change log entry" section in the pull request description.

If changes need to be present in CHANGELOG.md you can state it this way

**Change log entry**

Yes. A brief summary to be placed into the CHANGELOG.md

(possible answers Yes/Yep/Yeah)

Or you can opt out like that

**Change log entry**

None.

(possible answers No/Nope/None)

^{Visited at: 2025-11-18 21:06:17 UTC}

[github-actions]

Typing analysis

Ignored files

This PR introduces 1 ignored file. It decreases the percentage of typed files from 38.32% to 38.28% (-0.04%).

Ignored files (+1-0)

❌ Introduced:

lib/datadog/core/crashtracking/component.rb

Note: Ignored files are excluded from the next sections.

Untyped other declarations

This PR clears 1 untyped other declaration. It decreases the percentage of typed other declarations from 68.02% to 67.96% (-0.06%).

Untyped other declarations (+0-1)

✅ Cleared:

sig/datadog/core/crashtracking/component.rbs:33
└── attr_reader logger: untyped

[datadog-datadog-prod-us1]

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage
• Patch Coverage: 92.59%
• Total Coverage: 95.16% (-0.01%)

View detailed report

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 6e6e165 | Docs | Datadog PR Page | Was this helpful? Give us feedback!}

[gyuheon0h]

• [PROF-12743] Runtime stack collection callback registration #4984 👈 (View in Graphite)
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[gyuheon0h]

Paper trail

[ ] Checking unreasonable string sizes
[ ] Checking that string pointers point to valid strings
[ ] Checking that control frames are valid
[ ] Checking that iseq is valid, and the instruction size is not unreasonable
[ ] Validating that pointers are readable using mincore
[ ] Checking for recursive frames
[ ] Strings can have different representations, take that into account
[ ] Ruby apps commonly have very deep stacks. We default in the profiler to collecting 400 and we've seen GitHub go close to 600, handle this
[ ] Pay attention to structure keeping the bytecode-to-line mapping

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ivoanjo]

Gave it a big pass! I think in general this seems reasonable -- most of my thoughts/worries are around ruby_runtime_stack_callback and making sure we get it into shape.

[ivoanjo]

Is this AI...? This seems a redundant check...?

[gyuheon0h]

This might be due to my unfamiliarity with Ruby internals, but the first check is intended to ensure the VALUE actually points to a live struct RBasic. Without the first check, even touching RSTRING(str) would dereference garbage and crash.The second check validates the string-specific payload (struct RString) since Ruby string’s header extends RBasic with extra fields.

[ivoanjo]

Ah... In fact RSTRING(...) is just a cast so that's why I was confused:

/**
 * Convenient casting macro.
 *
 * @param   obj  An object, which is in fact an ::RString.
 * @return  The passed object casted to ::RString.
 */
#define RSTRING(obj)            RBIMPL_CAST((struct RString *)(obj))

but TBH it's probably best to avoid as much as possible Ruby macros and other calls here to avoid relying on too many assumptions of what they do.

That is -- this should probably be a is_pointer_readable((const void *)str, sizeof(struct RString)) and thus avoid the reliance on Ruby headers other than to know "what's the size of it".

[ivoanjo]

We're validating some pointers, but not others... Why?

[ivoanjo]

So this function is very complex, because it also does a very complex thing. My question is -- what happens if we get something wrong here? That is, we already have a bunch of safeguards, but what can actually happen if we miss a spot in pratice? Is there a safety net for us, or not?

[ivoanjo]

So this slightly makes me nervous, and comes back to my question of "do we have a safety net for this or not". We're asking the VM to get something for us and thus we have no control over valid pointers and the VM doing sane things if something is wrong...