USDTs on important functions

[brancz]

Other popular allocators, such as jemalloc have built-in mechanisms for profiling; mimalloc does not. To allow some level of debugging, it would be great if mimalloc added USDTs in important places, such as at the end of malloc and free. USDTs have a very minimal impact when not being traced (inserts a single NOP instruction) and much lower impact than having to use uprobes/uretprobes.

Is this something that would be accepted into the mimalloc project?

[shikhar]

We would love to have more insight into mimalloc behaviour for our workloads and this seems like a great low-cost way to achieve that.

[brancz]

Daan (@daanx) with jemalloc being archived, it might start to be more important to make mimalloc more observable. Any chance this is something you'd be interested in accepting?

[daanx]

Apologies for the late response. Yikes, I didn't know jemalloc was getting archived -- that's unexpected :-(
Anyways, I am quite interested in USDT's but I would have to study it a bit more -- if the overhead is indeed this low that would be quite interesting. At the moment, there is already quite some support for tracing with tools like valgrind and asan (see mimalloc-trace.h) and there has been integration with Windows perf counters as well. If we add USDT it would be good if we could reuse the same mechanism? Or at least impact the code as little as possible. I'll study this a bit more soon.

[brancz]

At first sight I would say that should work. A cherry on top would be if mimalloc sampled allocations every X bytes allocated and only fired the trace probes then (or additionally to trace probes that always fire unconditionally). That way mimalloc doesn’t have to know how to unwind a stack and can offload that part to the profiler, but the profiler doesn’t have to trace every single allocation which would be cost prohibitive in a production environment (which is arguably exactly where you want to profile). That would give us everything we need to build external heap and allocation profilers with very low overhead.

[lucab]

I'm also coming from the jemalloc-profiling world and would love to have some way to observe mimalloc allocation patterns under real-world workloads in production environments.
This is a somehow common wishlist item when developing backend/infrastructure services (in my case, Rust binaries running on Linux servers).
For completeness, tracing through valgrind/heaptrack is nice for local development but is not well suited for investigating memory usage-patterns of a remotely deployed binary.

I'm quite excited if this ends up going in the direction suggested by Frederic Branczyk (@brancz), which wouldn't even require active instrumentation in the consuming application and can be directly driven by an external profiler!

[danielsn]

I'm really glad to see this issue, I'm also excited about having memory profiling in mimalloc. I came to it from the direction of supporting in-process memory profiling from python (which is using mimalloc for the new NOGIL mode), which it turns out we could do with a fairly small code change.

For allocation profiling, this provides a callback that could be used to trigger the probe for an external profiler, or to directly sample and store the values for in-process profiling.

Supporting live-heap profiling requires a cheap way of knowing whether a given pointer was tracked when free is called. Doing so externally turns out to be surprisingly expensive, as even a hash-table incurs cache misses, which are avoided if the metadata lives on the same cache line as an allocator data stucture that will already be read as part of the free operation. In this case, the on_free callback could either do in-process or out-of-process untracking of the object. In the common case where there is no tracked object on the page, this is a single branch on data that is already in cache.

I have a draft PR exploring the approach: #1266

I'm not wedded to this code, and would love to work with you on whatever approach best fits with the mimalloc goals. Does this fit with what you're looking for Frederic Branczyk (@brancz)? If you could give this a try and let me know if there are changes that would work better for your out-of-process goals, that would be very helpful.

[danielsn]

I've done some polish on the PR and I think it is ready for review Frederic Branczyk (@brancz) Daan (@daanx).

[lucab]

Sorry for not posting this earlier, but last year I did some exploration around mimalloc allocations profiling.
Compared to #1266, I went in a different direction and hooked USDT probes to relevant entrypoints, see the drafted code at lucab#1.
The code in that PR is roughly working as an experiment, but it is not polished nor complete, so I didn't try to push it upstream.

In particular, before proposing this, I still wanted to explore some way to support externally-configured sampling.
I had some plans around trying to apply consistent probabilistic sampling (similar to OpenTelemetry head sampling algorithm) to pointer arithmetic, in order to get consistent sample/no-sample decisions for both allocs and frees based on pointer values (without full state tracking).
An external tracer would use a dedicated USDT semaphore (which IIRC is basically just a volatile u16 at a known location) to set the sampling probability.

Unfortunately I didn't found the time/stamina to close the loop on this experiment in the past months. I'm not actively working on these topics, so it's unlikely that I'll manage to conclude this exploration anytime soon. But maybe the code draft and the braindump above could be useful to somebody else.

[danielsn]

I had some plans around trying to apply consistent probabilistic sampling (similar to OpenTelemetry head sampling algorithm) to pointer arithmetic, in order to get consistent sample/no-sample decisions for both allocs and frees based on pointer values (without full state tracking).

I did some experiments around doing this in python, and found that at least in python it easy to get into a situation where a set of objects are being allocated/freed repeatedly as python reusused objects, which started to mess with the statistics of which allocations to sample if they happened to all match or not match the determinstic algorithm.

[danielsn]

Luca Bruno (@lucab) I'd be interested in looking into pointer arithmetic based sampling: my experiment was a quick one, and its possible that with proper upscaling it might work better.

[danielsn]

Daan (@daanx) is the approach of adding the sampling decision inside mimalloc, with either direct probes or flexible callbacks when sampling occurs, one you'd be interested in exploring?

[brancz]

In an unrelated context we’ve been playing with usdt semaphores which might be another thing we can use to make sure we’re only paying for the instrumentation when something actually attaches to the USDT(s).

[lucab]

Frederic Branczyk (@brancz) I think stap-based probes can already handle this internally, see https://sourceware.org/systemtap/wiki/UserSpaceProbeImplementation#Semaphore_Handling (though I didn't explicitly verify that). But this is still a boolean full-on/full-off behavior when the profiler attaches.

Daniel Schwartz-Narbonne (@danielsn) I didn't consider the skewed bias in pointer values, that's indeed a bummer for an unlucky single-process/single-run case.
This is a bit outside of my area of expertise, but I still think in cases like a distributed profiler observing a N>1 number of executions, the statistical properties of the distribution of values may still make this work. That is, while a single observation can be highly biased, the aggregated set can still expose a useful signal on average.