Support perf jitdump format

[vchuravy]

Multiple languages support perf's jitdump format. This includes:

• Node.js/Chrome's V8
• Java through https://github.com/torvalds/linux/tree/master/tools/perf/jvmti
• dotnet/runtime
• LLVM based languages:
  • Julia
  • Wasmtime
  • https://github.com/llvm/llvm-project/blob/main/llvm/lib/ExecutionEngine/PerfJITEvents/PerfJITEventListener.cpp

[r1viollet]

Thanks highlighting this gap.

I have started adding some support to read symbols taking dotnet as an example. Would you have a language in particular I should test against ? Not all languages publish frame pointers (or dwarf) to unwind through stacks. I would need to test the quality of the unwinding.

Also I would need to understand where this is published. For now we check in
/tmp/perf-<PID>.map

Note that we also have in process profilers dedicated to languages (for Node, Java, .NET) which will offer better results and other features like memory profiling.

[aristotelhs]

This looks like a very useful feature. We are using Julia and it supports publishing jitdump format (https://docs.julialang.org/en/v1/manual/profile/#External-Profiling). Is there a patch for this support which we can help by testing?

[vchuravy]

Would you have a language in particular I should test against ? Not all languages publish frame pointers (or dwarf) to unwind through stacks. I would need to test the quality of the unwinding.

You could test against julia

ENABLE_JITPROFILING=1 julia test.jl

test.kl:

function fibonacci(n)
    if n == 1 || n == 2
        return 1
    else
        return fibonacci(n - 1) + fibonacci(n - 2)
    end
end


while true
    @show fibonacci(42)
    sleep(1)
end

The function fibonacci should symbolize in the profile.

[aristotelhs]

I did and I created 2 flamegraphs (that is what we want in the end to do using ddprof) to show what it captures. The first one is without the perf inject --jit command, the second is after running the command. Both files are attached.

What I did notice is that Julia did not create a /tmp/perf-$pid.map file but on the other hand it created some files in .debug/jit/llvm-IR-jit-$date-$random whose names were in the format: jit-$pid.dump

[vchuravy]

So the .map file is created during the execution of the process and perf --inject then creates these files IIRC

[vchuravy]

Ah right I remember now. There are two different formats:

The older map https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/jit-interface.txt and the newer jitdump https://elixir.bootlin.com/linux/v4.10/source/tools/perf/Documentation/jitdump-specification.txt

Julia and all other LLVM languages use the jitdump format. So yes the correct thing is to scan maps for jit-$pid.dump.

[aristotelhs]

So as for now ddprof does not support jitdump format?

[r1viollet]

Thanks for clarifying the request. We do not support Julia for now.

I need to check what makes the most sense between using a native profiler vs sending profiles from a profiler tuned for Julia.
There is prior art in uploading pprof data for Julia link

[aristotelhs]

The main difference using perf linux subsystem is to sample stack profiles that are able to also include the kernel part (which I doubt that can be achieved by a native profiler). ddprof would instrument starting the profiling and uploading the data to datadog where we could visualize them.

[vchuravy]

I need to check what makes the most sense between using a native profiler vs sending profiles from a profiler tuned for Julia.

Both have value, but having integration with the native profiler is a must to analyze IO an network issues. Parca is adding support for jitdump here parca-dev/parca-agent#1051

[r1viollet]

To address the topic of what is the best approach to get good profiling data. I agree with the kernel frames argument, though this is also something we can add to a Julia profiler.

There are a few issues with using ddprof as I see it:

• Julia does not like when we hook dlopen. Cf a link refering to the same issue for sanitizers.
  https://github.com/JuliaLang/julia/blob/master/src/dlload.c#L178
  We do this to ensure we are aware of when we should instrument new allocation functions.
• Julia has it's own allocator, so hooking the standard allocator is not interesting to the user.
• Julia creates it's own stack traces with Fibers, which makes it harder for us to sample from outside the runtime.

Overall having access to the Julia APIs makes it easier to build profiling data. Could you try using the profiler that was mentioned in the link and the Datadog uploader it was using ?

For ddprof, I do agree that having a general perf map support is valuable and that we could improve the visibility into the general CPU consumption.

[aristotelhs]

How would a Julia profiler be able to profile the kernel stack? Unless it also asks from the kernel to show that by using perf_open.
One of the issues we face is that at times there is a complete freeze of Julia process and gets stuck there. (One of the theories that we look into is that it might be a heavy GC that stops the world and/or something gets stuck). In that case I am not sure if the profiler would be able to continue running. ddprof using perf_open is an external tool (using the kernel for instrumentation) so it could be able to show something.

[NHDaly]

@r1viollet: Thanks for engaging with us on the topic! :)

Indeed, you make some great points about the benefits of an integrated profiler! Thanks for outlining them.
In particular, the point about dlopen is new to me. Other than that, we agree that an integrated profiler would be better able to collect allocation profiles, and better able to report on things like which Fiber a profile came from.

Until now, we have been using our own integrated profiling package, here: https://github.com/JuliaPerf/ProfileEndpoints.jl, to some degree of success. We have also had some success trying out Pete's uploader from DataDog/dd-trace-go#1534. 👍

However, as @aristotelhs has pointed out, we have been suffering general issues with DataDog metrics blackouts, due to unreliabilites in the Julia scheduler.

Currently, our metrics exporter spawns a fiber, or coroutine (called a Task in Julia), to sit in the background and write out the metrics to the datadog agent every second or so. But when the scheduler is overloaded, we can sometimes see that exporter task starved for tens of minutes or sometimes even an hour! During that whole hour, we get no metrics from our julia process! 😬 We currently have no way to tell what was happening during that blackout. And we have an observability goal this quarter to find out.

Finally, there are a couple unrelated issues that lower our confidence in the Julia thread profiler, such as: JuliaLang/julia#33490. This was partially resolved, but it still stops all the threads for every sample instead of just one thread at a time, which can still introduce artificial contention. (I need to file another issue for that, but haven't done it yet.) We were hoping that switching to a more mainstream external profiler can also buy us some robustness improvements.

We basically see two possible paths forward for continuous CPU profiling:

1. Keep the profiling in-process, so that we can take advantage of its built-in support for JIT functions, but somehow change our integration to shield it from the Julia scheduler.
   • For example, this could involve moving the background task that collects and reports the latest profiling samples out of Julia and into a C thread running in the shared process,
   • or possibly sharing the results to another Julia process over shared memory, or something like that.
2. Integrate with an existing, robust, external CPU profiler, and upload those results to DataDog via an external process.
   • This works around any issues with the Julia scheduler, or bugs in the profiler.
   • But it needs to support reading the JIT names somehow.
   • This could be done via either:
     1. ddprof (as described in this thread so far), or
     2. manually running perf and then reverse-engineering the DataDog api, as done in Help with Julia profiling support? dd-trace-go#1534.
        • perf already supports the jitdump format, so this works out of the box. But the continuous process to upload to DataDog of course would need to be rewritten from scratch.

Of all of the above options, it seems to us that adding JIT support to ddprof would be the quickest path to unlock our goals for continuous profiling of our Julia processes in prod. Does that reasoning sound right to you as well?

Thanks a bunch again! :)