Bootstrap fails due to system LLVM libffi link error

[MagaTailor]

Trying to bootstrap rust on aarch64 Linux using --llvm-root fails:

error: linking with `cc` failed: exit code: 1
note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/bin/rustc.0.o" "-o" "aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/bin/rustc" "-Wl,--gc-sections" "-pie" "-Wl,-O1" "-nodefaultlibs" "-L" "/usr/lib/llvm-3.8/lib" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_driver-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_passes-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_borrowck-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_lint-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_resolve-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_typeck-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_plugin-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_privacy-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_trans-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_platform_intrinsics-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_incremental-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "syntax_ext-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_mir-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_const_eval-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_metadata-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_save_analysis-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_const_math-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "getopts-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "flate-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_back-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rbml-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_data_structures-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "syntax-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "serialize-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "log-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "term-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "fmt_macros-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "arena-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_llvm-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "graphviz-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "std-fe3cdf61" "-l" "rt" "-l" "dl" "-l" "tinfo" "-l" "pthread" "-l" "z" "-l" "m" "-l" "stdc++" "-l" "dl" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "util" "-Wl,-rpath,$ORIGIN/../lib" "-Wl,-rpath,/usr/local/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,--enable-new-dtags" "-s" "-l" "compiler-rt"

note: ../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_float'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_sint64'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_pointer'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_sint32'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_void'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_sint8'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_call'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_sint16'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_double'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_prep_cif'
collect2: error: ld returned 1 exit status

error: aborting due to previous error
../mk/target.mk:220: recipe for target 'aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/bin/rustc' failed
make: *** [aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/bin/rustc] Error 101

For some reason the correct llvm configuration is not being detected and -lffi has to be passed manually for the bootstrap to succeed.

[brson]

In-tree, we configure LLVM without -lffi so maybe there's a discrepancy when we encounter an LLVM with it compiled in.

[MagaTailor]

Yup, the LLVM I was bootsrapping against was clearly built with LLVM_ENABLE_FFI.

[jethrogb]

Running into the same isse. Shouldn't llvm-config output this requirement?

Btw for anyone reading this: a quick fix is to add:

#[link(name = "ffi")] extern {}

to src/librustc_llvm/lib.rs.

[japaric]

I'm also experiencing this on x86_64 Ubuntu 16.04 using the llvm-dev package and --llvm-root.

[MagaTailor]

The title should be edited then, and btw, the quickest non-permanent solution looks like this:
RUSTFLAGS='-C link-args=-lffi'

[japaric]

It seems this is a duplicate of #29694

[japaric]

Recently, I have been annoyed by this a lot. I've been boostraping rustc against system llvm a lot in different Docker containers and I have to remember to apply @jethrogb patch/fix to every rust branch/worktree I work on and then I have to be careful to not commit this fix to the branch I'm working on. 😞

Could we simply commit @jethrogb's fix into master? I tested bootstrapping rust against our llvm fork with that fix merged into master (5a5736d) and at least, on Linux, it bootstraped without problems.

P.S. @petevine's RUSTFLAGS solution didn't work for me.

[MagaTailor]

@japaric That's all there's to it, and it definitely works for the Makefiles bootstrap.

[alexcrichton]

This is of course not intentional, and fixes are always welcome! The trick here is detecting whether we're building against an LLVM that needs libffi. For example the in-tree LLVM doesn't and we shouldn't link to it as a result.

[cuviper]

FWIW, Fedora's LLVM is built with FFI, but this is usually fine because libLLVM.so links to libffi.so itself. However, with llvm-static.rpm installed, llvm-config starts giving static link options (but not -lffi), and ldd -r librustc_llvm-*.so indeed shows a few undefined symbol: ffi_....

[cuviper]

IOW I blame llvm-config, not that blame solves anything.
(I also don't like that it doesn't let me choose shared linkage even when static libraries are present...)

[userwithuid]

@cuviper Regarding 'choose shared over static llvm':
If I understand correctly, since 3.9 llvm-config will prefer the combo-dynlib by default over any static libs present. And even if you don't have the combo, but say shared and static component libs installed, there are now options to specifically choose --link-static or --link-shared.

Relevant llvm commit explains it: llvm-mirror/llvm@93083d4

Doesn't fix this bug here, external llvm + ffi enabled + intentionally linking llvm static libs still triggers it, just thought it would be nice to know.

(BTW, the ffi-issue seems to be sort of a known shortcoming of llvm-config --system-libs. It only statically lists the LLVMSupport sysdeps, but ffi is the (currently only?) sysdep from another component - LLVMExecutionEngine)