Repository navigation
LTTNG-UST 2.13 changes soname breaking loading of libcoreclrtraceptprovider #57784
Description
Activity
using compilation tricks like the ones used in System.Security for openssl
Note, same (shim) tricks are used for libnuma.so, which is closer to where coreclrtraceptprovider lives. 🙂
Thanks. The shim approach is tempting. We sort of codegen the trace points, so this one will be interesting if we decide to support both.
After some digging, it turned out
lttng/tracepoint.hheader (the one we use) does things quite differently than libssl, libnuma ones. It differs in the following manner:- We use
TACEPOINT_*macros which (during the macro expansion) binds to__tracepoints__initmethod and the related code chunks.- We stay consistent and do not directly use any non-macro APIs.
- The
__tracepoints__inititself usesdlopen(3)and uses the ABI-versioned flavorliblttng-ust.so.0,liblttng-ust.so.1(For 2.13).- They even have explicit validation checks in 2.13 to ensure that the current process does not have
so.0loaded. e.g. https://github.com/lttng/lttng-ust/blob/d2a010d/src/lib/lttng-ust-tracepoint/tracepoint.c#L611-L642
- They even have explicit validation checks in 2.13 to ensure that the current process does not have
Here is the expansion of macros for our generated
tpdotnetruntimemonoprofiler.h:# on Ubuntu x64 $ ./build.sh clr $ clang-9 -E \ artifacts/obj/coreclr/Linux.x64.Debug/pal/src/eventprovider/lttngprovider/lttng/tpdotnetruntimemonoprofiler.h \ -Isrc/coreclr/pal/inc/rt -Isrc/coreclr/pal/inc > dump
the dump looks like this: http://sprunge.us/J5rsv3 (and file tpdotnetruntimemonoprofiler.h: http://sprunge.us/LPmIuJ). Search for
__tracepoints__initordlopen("liblttng. It is quite noisy, so I made a copy, removed#include "palrt.h"etc. and kept only one probe, here is the shorter version of this expansion: http://sprunge.us/BK0q1V, at the very end of this file is the code of interest.I will defer to @janvorli as to the best approach for lttngshim. One option is that we install both versions on the official build machine, and compile both v0 and v1 implementations in our code with our pseudo init; which will probe for .so.1, and if not found, .so.0, then select the implementation based on the result. This approach will prevent us from stamping out symbols of
liblttng-ust{-tracepoint}.soand we will still be using their init mechanism (which seem to take care of "shared access" scenarios).Reacted by John Salem- We use
Thanks. I actually found about dlopen/dlsym pattern in their code when I commented out these lines (to see what breaks)
just to find out it had no effect on the build result (still succeeds) other than we (unnecessarily) link toliblttng-ust{-tracepoint}.so.0. 🙂 All public APIs in their header are already shim-like macro (without multi-version support):It seems that we will need to create and distribute two flavors of the libcoreclrtraceptprovider.so, one compiled with one version of lttng and the other with the other one. Then we would try to load one and if we fail, try to load the other. It seems that's essentially what @am11 has suggested, right @am11?
Yes, and that also seems to be what needs to happen from my understanding.
@janvorli, yes, that's what I was thinking. It will be cleaner, as it will save us from using macro post-expansion calls. The quickest way to get there is to duplicate python code (as macro names are changed in 2.13).
Part II - python free implementation
However, in a separate project, we can replace the python scripts with lttngshim which will dynamically define those macros (at build time) and we can expand it without needing to build code. It is more work as python scripts read the XML (ETL manifest) file and generates multiple header and code files. If we think replacing python is a good idea, then I was imagining for:
- input: we define a static header with our events and groups and keep it in sync with
src/coreclr/vm/ClrEtwAll.man(manually at first, as it's only a few times a year we add/modify events; if we have more time, we can use XmlPeek msbuild task to update the header files when msbuild is present .. because cmake doesn't support xml parsing) - output, instead of creating too many header and code files (this is half of what we produce: https://gist.github.com/am11/404d99384eaae7259bcabdd06ff5e307), we can make one and ifdef out the flavors: coreclr, mono, rundom, private and stress.
- input: we define a static header with our events and groups and keep it in sync with
One thing to add here: Using
src\coreclr\vm\ClrEtwAll.manas the one source of truth means that all other tools that consume events only have to look at one source of truth. For example, TraceEvent-based tools use the manifest to generate C# code that gets used to parse the events.One thing to add here: Using
src\coreclr\vm\ClrEtwAll.manas the one source of truth means that all other tools that consume events only have to look at one source of truthI understand the rationale, and using
XmlPeek-based parsing to dynamically generate the header is an option. However, in varied parts of runtime code base, we do manually keep lists in-sync which infrequently change, a few examples:
27 remaining items
Source build has patching mechanism in place, you can disable lttng feature at build time as discussed above.
- added a commit that references this issue
on Mar 31, 2022 Is it still not resolved after 3 years?
This is getting a complete mess; the current latest .NET runtime 9.0.4 of arm and arm64 for linux each link to a different version of liblttng-ust.so:
henri@seven:~/Downloads/dotnet$ objdump -x dotnet-runtime-9.0.4-linux-arm/shared/Microsoft.NETCore.App/9.0.4/libcoreclrtraceptprovider.so | grep NEEDED NEEDED liblttng-ust.so.1 NEEDED libstdc++.so.6 NEEDED libm.so.6 NEEDED libgcc_s.so.1 NEEDED libc.so.6 NEEDED ld-linux-armhf.so.3 henri@seven:~/Downloads/dotnet$ objdump -x dotnet-runtime-9.0.4-linux-arm64/shared/Microsoft.NETCore.App/9.0.4/libcoreclrtraceptprovider.so | grep NEEDED NEEDED liblttng-ust.so.0 NEEDED libstdc++.so.6 NEEDED libm.so.6 NEEDED libgcc_s.so.1 NEEDED libc.so.6 NEEDED ld-linux-aarch64.so.1Please make a consistent decision which dependency is needed and make it consistent least within a certain DotNet release.
Reacted by Werner BreitwieserThe version we link to depends on what distro we build against. We try to build against the oldest distro we can to ensure as broad compatibility as possible. For arm64/x64, the target is Ubuntu 16.04, so that .NET can work on distros with glibc 2.23 or newer. But for arm32, we couldn't choose that due to some things not working properly on arm32 when built against that glibc. I am not sure if I remember it correctly, but I believe it was a libc move to 64 bit file time even on 32 bit systems. So we have to target Ubuntu 22.04 for arm32. We could move arm64/x64 to that one too, but that would mean loosing support for distros with glibc older than 2.35.
You can find details on the compatibility here: https://github.com/dotnet/core/blob/main/release-notes/9.0/supported-os.md
Reacted by Adeel MujahidThe version we link to depends on what distro we build against
Yup and that is by design, as-in https://github.com/dotnet/runtime/blob/main/docs/project/linux-build-methodology.md.
We could move arm64/x64 to that one too, but that would mean loosing support for distros with glibc older than 2.35.
Yes, and I don't think lttng-ust is a good reason to bump libc version by 12.
Distro packages maintain that kind of thing behind the scenes. For MSFT builds, we could add lttng-ust to
src/native/externalto make it seamless. This is how we are streamlining all other dependencies.we could add lttng-ust to src/native/external to make it seamless
We are not actively investing into making lttng support better. Our bet is that it will get replaced by https://docs.kernel.org/trace/user_events.html
Reacted by Brian RobbinsWe are not actively investing into making lttng support better. Our bet is that it will get replaced by docs.kernel.org/trace/user_events.html
Yup, I'm following
src/native/external/LinuxTracepointsprogress and related discussions. I was suggesting that compared to updating glibc baseline (by 12 minor versions) for lttng-ust, it is better to align it with rest of the external deps which solves the problem without changing the libc baseline.If we don't want to make any further changes in this regard, then maybe we should close this issue?
With .NET 10, the problem persists. I've built a Linux version of my application on Windows and when I pack it to RPM on Fedora 43 (x64), this RPM cannot be installed because the binaries (.NET application and its dependencies) still reference liblttng-ust.so.0, whereas this Fedora 43 has liblttng-ust.so.1 installed. Possibly, the toolchain on Linux has to resolve this kind of issue, but it does not, which means that my shiny new .NET application simply won't run on many modern systems. Am I missing something?
my shiny new .NET application simply won't run on many modern systems. Am I missing something?
We load the lttng library dynamically using
dlopen. Failure to load it is handled gracefully. It does not prevent application from running. It just prevents the application from being traced using lttng.What is the error that you see?
my shiny new .NET application simply won't run on many modern systems. Am I missing something?
We load the lttng library dynamically using
dlopen. Failure to load it is handled gracefully. It does not prevent application from running. It just prevents the application from being traced using lttng.What is the error that you see?
Thank you for the information. The problem was that RPM installer complained about the unsatisfied dependency: "nothing provides liblttng-ust.so.0()(64bit) needed by abcalc-1.1.0-1.fc43.x86_64 from @commandline" . I solved this problem by excluding the dependency - I added the following to the .spec file: "%global __requires_exclude liblttng-ust.so." . This allowed me to install the RPM, and after this, the application worked fine. So, indeed, it's just an installation issue.
Thank you for confirming the dynamic loading - this lets me be sure that the approach I use will work for users.
Reacted by Jan Kotas, Omair Majid and Alexandre H.T.R. Bonfitto- added a commit that references this issue
on Sep 28, 2026
There's an ABI breaking change between LTTNG-ust 2.12 and 2.13, so they've reved their soname from
liblttng-ust.so.0toliblttng-ust.so.1. We link against the former. Given some distros will start supporting newer versions, customers will likely end up upgrading packages and realize they can't loadlibcoreclrtraceptprovider.so. We should investigate avenues to support multiple versions as needed using compilation tricks like the ones used in System.Security for openssl. Other possibility is to only support 2.13 and that brings us come code sanity advantages - this would only happen in .NET 7+ realistically.@brianrob @dotnet/dotnet-diag