[BUG]: Increase in Node heap size due to datdog-pprof after upgrading to v24

[wolt-kgarg]

Tracer Version(s)

5.73.0

Node.js Version(s)

24.11.0

Bug Report

Hi team,

We recently migrated our application from Node v22.20.0 to v24.11.0, and after the upgrade, we noticed a significant increase in the heap live size.

We have Datadog profiling enabled for our app, and as shown in the attached screenshot, the heap live size inside (Datadog pprof) increased by almost 26% post-upgrade.

We’d like to understand whether this behavior is expected with Node v24 with v8 now reserving more space or if there are any known issues or configuration changes that could help mitigate this increase.

@datadog/pprof version is 5.11.1

Reproduction Code

No response

Error Logs

No response

Tracer Config

No response

Operating System

No response

Bundling

Webpack

[szegedi]

Hi, Attila from Datadog profiling team here.

I looked into this and found that what you see is indeed a curious artifact of how we capture profiles, although some of the memory increase might actually be real, albeit temporary. Let me elaborate.

Every 65 seconds we create a profile and upload it to Datadog. The process is (for purposes of this discussion) two-step: In step one we stop and restart every profiler component (these are normally the CPU profiler and the Heap Live Size profiler) and we do this sequentially – this will be important. As part of the stop operation, the profiler asks V8 for a profile (either CPU or memory), then encodes it into an internal object representation – a tree of pprof-format objects. These objects are created in profile-serializer.js you see in Heap Live Size reports. Then when all the profiles are thus obtained, as step two they are asynchronously encoded into their wire representation and uploaded to the Datadog agent. Then they become garbage.

If you don't specify the order of profiler components using an explicit DD_PROFILING_PROFILERS env var (e.g. DD_PROFILING_PROFILERS=space,cpu) then the default ordering is cpu,space ("space" is the Heap Live Size component profiler.) And herein lies the crux of the matter: since "space" is stopped/restarted after "cpu" in step one, it will capture as part of the live set the object tree of "cpu" waiting to be encoded and uploaded in step two. That object tree will become garbage immediately afterwards, but it was live when the heap live size snapshot was taken so it shows up there.

So for me, the right way to fix this is to ensure that "space" is always the first profiler component to snapshot as then it won't be capturing the temporary memory use of the others. You can try yourself setting this env var and see how it changes your heap live size.

On the other hand, it is doubtless that what you see now with Node.js 24 is an increase in the memory used for temporarily holding the CPU profile. I'm admittedly not sure what to make of that. A CPU profile will always have roughly the same amount of samples. Its memory footprint depends on the depths of call stacks in samples as well as the variance in them across samples – samples with same stack traces share most of objects among themselves. Our profiler is built on top of V8's profiler so if for some reason the V8's CpuProfiler in Node.js 24 starts returning us samples that contain a higher variance of stack traces, either because of a change in V8, a change in Node.js, or a change in how your application executes on Node.js 24, then this can happen.

Again, if you want your heap live size to reflect actual long-lived objects and not contain temporary CPU profile artifacts, you can set DD_PROFILING_PROFILERS=space,cpu. I will make changes in dd-trace-js going forward to change the default ordering.