LLVM 5.0 Tracking Issue

[TimNN]

The time has come again, LLVM is releasing 5.0 soon and Rust wants to / needs to follow.

I have started work on the upgrade in these branches:

• rust-lang/rust: https://github.com/alexcrichton/rust/tree/llvm50
• rust-lang/llvm: https://github.com/rust-lang/llvm/tree/rust-llvm-2017-07-20
• rust-lang/compiler-rt: https://github.com/rust-lang/compiler-rt/tree/rust-2017-07-20
• rust-lang-nursery/compiler-builtins: https://github.com/rust-lang-nursery/compiler-builtins/tree/llvm50

Open tasks

• get asm.js working again - reenable JSBackend as a target
  • add asm.js target to our fork - rust-lang/llvm@c9a2123
  • update emscripten to LLVM 5.0

Completed tasks

Next steps:

• Get rustllvm to compile (error log)

Current issues:

• A lot changed around attributes (Update Rust LLVM bindings for LLVM 5.0 #43387)
• SynchronizationScope changed / was renamed? (Update Rust LLVM bindings for LLVM 5.0 #43387)
• LLVMMetadataRef was added, this can / should likely replace LLVMRustMetadataRef (Update Rust LLVM bindings for LLVM 5.0 #43387)
• createPointerType gained a new parameter (Update Rust LLVM bindings for LLVM 5.0 #43387)
• createNameSpace lost some parameters (Update Rust LLVM bindings for LLVM 5.0 #43387)
• DebugLoc was replaced by DiagnosticLocation in LLVM DiagnosticInfo (Update Rust LLVM bindings for LLVM 5.0 #43387)
• compiler segfault when testing in stage2 - using x86_64-gnu image
  • sample backtrace
  • upstream bug
  • upstream fix (temporary workaround) - commit1, commit2
• fix compilation of fuzzer on various platforms
  • i686 freebsd
  • x86_64 freebsd
  • x86_64 netbsd
  • upstream bug local fix/workaround
  • upstream fix
  • rustbuild: Tweak how we cross-compile LLVM #43527
• fix compilation of LLVM on i686-unknown-linux-gnu
  • logs
  • upstream bug local workaround
  • upstream fix
• codegen/issue-37945.rs - info below
  • upstream bug
  • upstream fix (ignore it)
• FreeBSD failed to compile std::to_string references
  • rustbuild: Update cross-compilers for FreeBSD #43530
• fix run-make/sanitizer-leak - apparently hit a fatal error when run locally?
  • --privileged needed when running the docker container Flag docker invocations as --privileged on CI #43536
• run-make/simd-ffi
  • upstream bug
  • upstream fix
• mips failed to compile cargo
  • mipsel logs
  • mips logs
  • upstream bug
  • upstream fix
• related debuginfo-gdb test failures - LLVM5: Update DW_OP_plus to DW_OP_plus_uconst #47610
  • sample fix
  • explanation
• rustc with LLVM 5 reports invalid DIExpression (rustc with LLVM 5 reports invalid DIExpression #47464) - fixed in LLVM5: Update DW_OP_plus to DW_OP_plus_uconst #47610
• LLVM 5 fails debuginfo/by-value-self-argument-in-trait-impl.rs (LLVM 5 fails debuginfo/by-value-self-argument-in-trait-impl.rs #47611)

All but the first issues seem straight forward to fix, so I'll try to send PRs for those soon.

[TimNN]

cc @nagisa, @michaelwoerister, @arielb1, @eddyb: You have been recommend as familiar with the rust llvm wrapper, would one of you mind looking into the attributes changes?

[alexcrichton]

With #43387 looks like rustc can bootstrap on x86_64-unknown-linux-gnu, but there's a lot of test failures. A full stage2 test run has tons of errors, lots of compiler segfaults. A stage1 test run has much fewer errors:

test [compile-fail] compile-fail/proc-macro/derive-bad.rs ... FAILED
test [compile-fail] compile-fail/proc-macro/expand-to-unstable-2.rs ... FAILED
test [compile-fail] compile-fail/proc-macro/expand-to-unstable.rs ... FAILED
test [compile-fail] compile-fail/proc-macro/issue-38586.rs ... FAILED
test [compile-fail] compile-fail/proc-macro/item-error.rs ... FAILED
test [compile-fail] compile-fail/proc-macro/lints_in_proc_macros.rs ... FAILED
test [compile-fail] compile-fail/proc-macro/proc-macro-attributes.rs ... FAILED
test [codegen] codegen/lifetime_start_end.rs ... FAILED
test [codegen] codegen/issue-37945.rs ... FAILED
test [debuginfo-gdb] debuginfo-gdb/var-captured-in-nested-closure.rs ... FAILED
test [debuginfo-gdb] debuginfo-gdb/var-captured-in-sendable-closure.rs ... FAILED
test [debuginfo-gdb] debuginfo-gdb/var-captured-in-stack-closure.rs ... FAILED
test [run-make] run-make/llvm-pass ... FAILED
test [run-make] run-make/rustc-macro-dep-files ... FAILED
test [run-make] run-make/simd-ffi ... FAILED
test [run-pass] run-pass/debuginfo-lto.rs ... FAILED
test [run-pass] run-pass/issue-34569.rs ... FAILED
test [run-pass] run-pass/backtrace.rs ... FAILED
test [run-pass] run-pass/backtrace-debuginfo.rs ... FAILED
test [run-pass] run-pass/proc-macro/add-impl.rs ... FAILED
test [run-pass] run-pass/proc-macro/append-impl.rs ... FAILED
test [run-pass] run-pass/proc-macro/attr-args.rs ... FAILED
test [run-pass] run-pass/proc-macro/bang-macro.rs ... FAILED
test [run-pass] run-pass/proc-macro/count_compound_ops.rs ... FAILED
test [run-pass] run-pass/proc-macro/crate-var.rs ... FAILED
test [run-pass] run-pass/proc-macro/derive-same-struct.rs ... FAILED
test [run-pass] run-pass/proc-macro/issue-40001.rs ... FAILED
test [run-pass] run-pass/proc-macro/issue-39889.rs ... FAILED
test [run-pass] run-pass/proc-macro/hygiene_example.rs ... FAILED
test [run-pass] run-pass/proc-macro/use-reexport.rs ... FAILED
test [run-pass] run-pass/proc-macro/load-two.rs ... FAILED
test [run-pass] run-pass/issue-40663.rs ... FAILED

[alexcrichton]

@eddyb indicated on IRC that this function call may be responsible for the debuginfo-gdb failures, and he was spot on!

[alexcrichton]

I've filed https://bugs.llvm.org/show_bug.cgi?id=33883 related to the run-make/simd-ffi failure

[eddyb]

So LLVM got even worse (the current implementation sometimes creates an alloca to avoid pretty much this kind of misoptimization ruining complex llvm.dbg.declare).

The proper solution here at this point is to produce a pointer to every single closure capture (which means loading the respective pointer from the environment, if the capture is already by-ref), and then use llvm.dbg.value to indicate that... oh, wait, that won't work, let me step back.

Using llvm.dbg.declare once means that even if the value is modified later, a debugger knows where to get the value during the scope the name is defined for.
OTOH, llvm.dbg.value gives the value at that time, and even if you say the value is behind a pointer, that won't let the debugger see updated values from later writes to the same location.

Which means we need one alloca for every single capture (holding the pointers mentioned above). This is because of idiosyncratic LLVM limitations - even though there is a system for indicating multiple pointer offsets and dereferences to get from the argument to llvm.dbg.declare (or llvm.dbg.value) (a subset of "DWARF expressions"), and there's enough information to start from an arbitrary pointer on stack or in a register (for llvm.dbg.value), analloca is still required.

But with one llvm.dbg.declare per alloca, they should be simple enough not to break.

[alexcrichton]

The codegen/issue-37945.rs failure looks related to rust-lang/llvm#90. @arielb1 would you be able to help out diagnosing this one? I've verified that the llvm branch we have for 5.0 does indeed include those commits:

• rust-lang/llvm@7df0651
• rust-lang/llvm@156cc49
• rust-lang/llvm@477bd75
• rust-lang/llvm@5a057dc
• rust-lang/llvm@2b66aee

Do you think that this may have regressed in upstream LLVM?

[alexcrichton]

I think I fixed codegen/lifetime_start_end.rs in 38e40ce

[TimNN]

Edit: Nevermind this comment, those tests are expected to fail on stage 1.

Original comment

For a stage 1 test run, I get the same failures as @alexcrichton, for compile-fail proc macro derive:

thread '<unnamed>' panicked at 'proc_macro::__internal::with_sess() called before set_parse_sess()!', /home/logic/build-tmp/rust-src/src/libproc_macro/lib.rs:678:8
note: Run with `RUST_BACKTRACE=1` for a backtrace.
fatal runtime error: failed to initiate panic, error 5

full logs

I'm not sure about the initial panic message, but the "failed to initiate panic" seems kind of bad (error 5 is apparently _URC_END_OF_STACK)

For run-pass proc-macro-derive:

index out of bounds: the len is 0 but the index is 4294967206

That also seems bad.

[alexcrichton]

@TimNN oh no need to worry about those, that's a benign failure. Procedural macros don't work in stage1 and not all tests are flagged as // ignore-stage1

[alexcrichton]

Filed an upstream bug for the segfaults in rustc I've been seeing. (segfaults were happening on all platforms)