Skip to content

Heap profiler with leak tracking #5763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Heap profiler with leak tracking #5763

wants to merge 5 commits into from

Conversation

rdettai
Copy link
Collaborator

@rdettai rdettai commented Apr 29, 2025

Description
This PR adds a very lightweight, feature gated, heap profiling endpoint. It superseeds the previous attempt in #5724

Using the jemalloc native profiling is complicated, it's hard to asses its performance impact, and the resulting traces are hard to interpret.

This PR proposes an custom and simple heap profiler:

  • written so that the profiler has almost no performance impact when it's not being used actively. Every allocation/deallocation simply checks for an atomic ENABLED flag with relaxed ordering and does no extra work if that flag is false.
  • the profiler can be activated with a webservice. It builds allocation statistics by call site and prints a backtrace when they exceed their last highest by a given threshold.

When started, the profiling allocates ~30MB of symbols.

How was this PR tested?
Python scripts, see results below.

@rdettai rdettai self-assigned this Apr 29, 2025
@rdettai rdettai requested a review from PSeitz April 29, 2025 15:04
@rdettai rdettai changed the base branch from custom-heap-prof to main April 30, 2025 10:18
@rdettai
Copy link
Collaborator Author

rdettai commented May 6, 2025

I ran some tests both during indexing (ingest V2) and search (float aggregation). On a pretty system (10 indexes with 1 milion docs ingested to each index):

Without the feature flag:

  • indexing duration: 96.81
  • avg search duration: 0.0058

With the feature flag but without starting the profiling:

  • indexing duration: 96.80
  • avg search duration: 0.0060

With the profiling activated with min_alloc_size=64KiB

  • indexing duration: 96.77
  • avg search duration: 0.0059

With the profiling activated for search only and min_alloc_size=1KiB

  • indexing duration: 96.80
  • avg search duration: 0.014

With the profiling activeted for indexing and min_alloc_size=1KiB

  • heap profiling stopped, memory_locations full

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant