Program cross compiled for mips-unknown-linux-musl crashes with SIGILL

[japaric]

cc @Bigomby @alexcrichton

Originally reported as japaric/rust-cross#27

Original report below:

---

Hi, I'm trying to cross compile a Rust app for LEDE (fork of OpenWRT) but I'm stuck.

The application is just an hello_world example:

fn main() {
    println!("Hello, world");
}

These are the steps I performed:

1. In first place I downloaded the LEDE SDK and compiled successfully the image and the toolchain for my device (mips-openwrt-linux-musl-gcc).

2. Install the Rust std using rustup add mips-unknown-linux-musl.

3. Create a .cargo/config file in my project with the following content:

   [target.mips-unknown-linux-musl]
   linker = "mips-openwrt-linux-musl-gcc"
   

4. Build the app using cargo build --release --target=mips-unknown-linux-musl.

5. Deploy the executable to the device using scp.

6. Got Illegal instruction

On target

$ ldd hello_world
/lib/ld-musl-mips-sf.so.1 (0x558c2000)                                                 
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x77cc4000)                                       
libc.so => /lib/ld-musl-mips-sf.so.1 (0x558c2000)  

$ ldd --version
musl libc (mips-sf)
Version 1.1.15
Dynamic Program Loader
Usage: ldd [options] [--] pathname

On host

$ mips-openwrt-linux-musl-gcc -v
Reading specs from /opt/lede/staging_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15/bin/../lib/gcc/mips-openwrt-linux-musl/5.4.0/specs
COLLECT_GCC=mips-openwrt-linux-musl-gcc
COLLECT_LTO_WRAPPER=/opt/lede/staging_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15/bin/../libexec/gcc/mips-openwrt-linux-musl/5.4.0/lto-wrapper
Target: mips-openwrt-linux-musl
Configured with: /home/diego/source/build_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15/gcc-5.4.0/configure --with-bugurl=http://www.lede-project.org/bugs/ --with-pkgversion='LEDE GCC 5.4.0 r2032' --prefix=/home/diego/source/staging_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15 --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=mips-openwrt-linux-musl --with-gnu-ld --enable-target-optspace --disable-libgomp --disable-libmudflap --disable-multilib --disable-libmpx --disable-nls --without-isl --without-cloog --with-host-libstdcxx=-lstdc++ --with-float=soft --with-gmp=/home/diego/source/staging_dir/host --with-mpfr=/home/diego/source/staging_dir/host --with-mpc=/home/diego/source/staging_dir/host --disable-decimal-float --with-mips-plt --with-diagnostics-color=auto-if-env --disable-libssp --enable-__cxa_atexit --with-headers=/home/diego/source/staging_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15/include --disable-libsanitizer --enable-languages=c,c++ --enable-shared --enable-threads --with-slibdir=/home/diego/source/staging_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15/lib --enable-lto --with-libelf=/home/diego/source/staging_dir/host
Thread model: posix
gcc version 5.4.0 (LEDE GCC 5.4.0 r2032)

Disabling jemalloc

If I use rust nightly and disable jemalloc as following:

#![feature(alloc_system)]
extern crate alloc_system;

fn main() {
    println!("Hello, world");
}

Then it works! But as I keep adding code the application eventually crashes with another Illegal instruction. For example:

Works

TcpStream::connect("130.206.193.115:80").unwrap();

Does not work

TcpStream::connect("google.com:80").unwrap();'

I don't know what I'm doing wrong. Any help will be appreciated.

[japaric]

@Bigomby

I don't know what I'm doing wrong.

The only thing that I think you could be doing wrong is compiling the binary for
MIPS (big endian) but running it on a MIPSel (little endian) device but then not
even the "Hello, world!" program that uses system_alloc would work so I'm
going to discard that possibility.

I'd like to check a few things:

• What device are you running the binaries on? I'd like to know known the exact
  architecture and CPU model but if you can at least tell me the brand and model
  of the router/board/device you are using, that would be a great start.
• Could you point me to the LEDE SDK, its URL, that you are using?
• Do you have access to GDB on the target device? Any chance you could get a
  backtrace/disassemble of a program at the moment that it crashes?

My guess, since C code is involved in all the crashes, is that we are building
our binary releases of std with C compiler flags that generate instructions
that are not available on your device. We'll hopefully only need to tweak those
and then a new nightly release of std should work for you.

If it's not the C side, then we may need to tweak our LLVM codegen options.

Or it could be a bad libc definition.

[Bigomby]

• The device is a Huawei EchoLife HG556a.

• I got the SDK from this repo https://github.com/lede-project/source (commit 76847c7)

• I got this when running with gdb

  #![feature(alloc_system)]
  extern crate alloc_system;
  
  use std::net::TcpStream;
  
  fn main() {
    TcpStream::connect("google.com:80").unwrap();
  }

  (gdb) r
  Starting program: /root/kinton-yeelight 
  warning: Unable to find dynamic linker breakpoint function.
  GDB will be unable to debug shared library initializers
  and track explicitly loaded dynamic code.
  
  Program received signal SIGILL, Illegal instruction.
  0x5556269c in ?? ()
  (gdb) bt
  #0  0x5556269c in ?? ()
  #1  0x55565fdc in _$LT$str$u20$as$u20$std..net..addr..ToSocketAddrs$GT$::to_socket_addrs::h7d1fe6bb64ce9c00 ()
  #2  0x55560cec in _ftext ()
  

[Bigomby]

This is what happends when using jemalloc

fn main() {
    println!("Hello world");
}

(gdb) r
Starting program: /root/hello_world 
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.

Program received signal SIGILL, Illegal instruction.
je_arena_mapbits_binind_get () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:801
801     /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h: No such file or directory.
(gdb) bt
#0  je_arena_mapbits_binind_get () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:801
#1  je_arena_salloc () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:1317
#2  je_isalloc () at include/jemalloc/internal/jemalloc_internal.h:924
#3  je_iallocztm () at include/jemalloc/internal/jemalloc_internal.h:937
#4  a0ialloc () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:321
#5  je_a0malloc () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:336
#6  je_arena_tdata_get_hard () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:519
#7  0x5558f7dc in je_arena_tdata_get () at include/jemalloc/internal/jemalloc_internal.h:818
#8  je_decay_ticker_get () at include/jemalloc/internal/jemalloc_internal.h:842
#9  je_arena_decay_ticks () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:1253
#10 je_arena_decay_tick () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:1264
#11 je_arena_malloc_large () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:2476
#12 0x55590ad4 in je_arena_malloc () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:1287
#13 je_arena_palloc () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:2601
#14 0x555b2420 in je_ipallocztm () at include/jemalloc/internal/jemalloc_internal.h:982
#15 je_tcache_create () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/tcache.c:328
#16 0x555849b0 in je_tcache_get () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/tcache.h:236
#17 je_imalloc () at include/jemalloc/internal/jemalloc_internal.h:954
#18 imallocx_no_prof () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:2135
#19 mallocx () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:2164
#20 0x5557bb30 in alloc_jemalloc::imp::__rust_allocate () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/liballoc_jemalloc/lib.rs:106
#21 0x5557aaa4 in alloc::heap::allocate () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/liballoc/heap.rs:59
#22 alloc::raw_vec::{{impl}}::with_capacity<u8> () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/liballoc/raw_vec.rs:96
#23 collections::vec::{{impl}}::with_capacity<u8> () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libcollections/vec.rs:356
#24 collections::slice::hack::to_vec<u8> () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libcollections/slice.rs:160
#25 collections::slice::{{impl}}::to_vec<u8> () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libcollections/slice.rs:1161
#26 collections::slice::{{impl}}::to_owned<u8> () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libcollections/slice.rs:1281
#27 collections::str::{{impl}}::to_owned () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libcollections/str.rs:167
#28 0x5556c934 in std::rt::lang_start () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libstd/rt.rs:50
#29 0x55561cd8 in main ()
warning: GDB can't find the start of the function at 0x77f78acf.

    GDB is unable to find the start of the function at 0x77f78acf
and thus can't determine the size of that function's stack frame.
This means that GDB may be unable to access that stack frame, or
the frames below it.
    This problem is most likely caused by an invalid program counter or
stack pointer.
    However, if you think GDB should simply search farther back
from 0x77f78acf for code which looks like the beginning of a
function, you can increase the range of the search using the `set
heuristic-fence-post' command.

[alexcrichton]

A SIGILL probably means that we're miscompiling jemalloc somehow (maybe some weird codegen option) but if it still faults with the system allocator then it's likely not exclusively a jemalloc problem.

[Bigomby]

I tried to run the program on an OM2P (MIPS device) and it's working fine, even with jemalloc. OM2P uses an Atheros AR7240 so the issue seems to be something more specific to Broadcom 6358.

UPDATE: It crashes also when using glibc instead musl on the Huawei EchoLife HG556a.

mips-openwrt-linux-gnu-gcc -v

Reading specs from /opt/l/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/bin/../lib/gcc/mips-openwrt-linux-gnu/6.2.0/specs
COLLECT_GCC=/opt/l/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/bin/mips-openwrt-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/opt/l/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/bin/../libexec/gcc/mips-openwrt-linux-gnu/6.2.0/lto-wrapper
Target: mips-openwrt-linux-gnu
Configured with: /home/diego/lede/build_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/gcc-6.2.0/configure --with-bugurl=http://www.lede-project.org/bugs/ --with-pkgversion='LEDE GCC 6.2.0 r2065' --prefix=/home/diego/lede/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24 --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=mips-openwrt-linux-gnu --with-gnu-ld --enable-target-optspace --disable-libgomp --disable-libmudflap --disable-multilib --disable-libmpx --disable-nls --without-isl --without-cloog --with-host-libstdcxx=-lstdc++ --with-float=soft --with-gmp=/home/diego/lede/staging_dir/host --with-mpfr=/home/diego/lede/staging_dir/host --with-mpc=/home/diego/lede/staging_dir/host --disable-decimal-float --with-mips-plt --with-diagnostics-color=auto-if-env --enable-libssp --enable-__cxa_atexit --with-headers=/home/diego/lede/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/include --disable-libsanitizer --enable-languages=c,c++ --enable-shared --enable-threads --with-slibdir=/home/diego/lede/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/lib --enable-lto --with-libelf=/home/diego/lede/staging_dir/host
Thread model: posix
gcc version 6.2.0 (LEDE GCC 6.2.0 r2065) 

[japaric]

Interesting. Then this could be due to codegen options related to "architecture level" (-march) or CPU optimizations (-mcpu) in the sense that the current rustc options generate instructions that are not available on some of these devices. Unfortunately, I don't know the MIPS ISA well enough to confirm this.

[jcowgill]

This is probably due to the ISA level. The HG556a seems to use MIPS32r1 and the OM2P uses MIPS32r2. I think LLVM defaults to MIPS32r2 unless told otherwise. It would be interesting to get a disassembly at the SIGILL, then I could tell you for certain that this was the problem.

[matan129]

I'm experiencing this issue too (on a Broadcom chip, though).
The chip is BCM63168. I have a working OpenWRT toolchain for it. Using mips-unknown-linux-musl as a target with rustc\Cargo works for a simple hello world application, but for anything more complicated I get an illegal instruction.

[Jedzia]

I have suffered similar losses in the battle for Illegal Instructions on mips. My research so far and i hope this helps:

Minimal build for VU+ DUO2 mipsel-unknown-linux-gnu with Rust on Windows

To save you some hours i describe the process how i build a hello world without
ending up with Illegal instruction errors on the BCM7425B2 STB platform.

Target Specs

root@vuduo2:~# cat /proc/cpuinfo
system type             : BCM7425B2 STB platform
machine                 : Unknown
processor               : 0
cpu model               : Broadcom BMIPS5000 V1.1  FPU V0.1
BogoMIPS                : 864.25
cpu MHz                 : 1305.097
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 64
extra interrupt vector  : yes
hardware watchpoint     : no
isa                     : mips1 mips2 mips32r1
ASEs implemented        :
shadow register sets    : 1
kscratch registers      : 0
core                    : 0
VCED exceptions         : not available
VCEI exceptions         : not available

processor               : 1
cpu model               : Broadcom BMIPS5000 V1.1  FPU V0.1
BogoMIPS                : 655.36
cpu MHz                 : 1305.097
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 64
extra interrupt vector  : yes
hardware watchpoint     : no
isa                     : mips1 mips2 mips32r1
ASEs implemented        :
shadow register sets    : 1
kscratch registers      : 0
core                    : 0
VCED exceptions         : not available
VCEI exceptions         : not available

What you need

To compile for the target i used Codescape GNU Tools 2018.09-03 Binaries
https://codescape.mips.com/components/toolchain/2018.09-03/downloads.html
installed in C:\Toolchain\MIPS\mips-2016.05.

Rust components used:

E:\Projects\Rust\vuduo2\hello-vuduo>rustup show
Default host: x86_64-pc-windows-gnu

installed targets for active tool chain
--------------------------------------

mipsel-unknown-linux-gnu
x86_64-pc-windows-gnu

active tool chain
----------------

nightly-x86_64-pc-windows-gnu (default)
rustc 1.35.0-nightly (719b0d984 2019-03-13)

Install "The sysroot manager that lets you build and customize std"
xargo.

cargo install xargo

Setup

Environment

Paths are specific to my setup and locations. This can vary in your case and you
have to adjust them.

Add your cross-compiler to the search path

set PATH=C:\Toolchain\MIPS\mips-2016.05\bin;%PATH%

Specify the compiler setup to use:

set CC=mips-linux-gnu-gcc
set CFLAGS_mipsel-unknown-linux-gnu=-EL
set AR_mipsel-unknown-linux-gnu=mips-linux-gnu-ar

• where CC specifies the compiler you use to build the target and std crate binaries
• CFLAGS_mipsel-unknown-linux-gnu the specific compiler switches to use. In this case
  -EL tells the compiler to build code for little endian processors (the el in mipsel).
  Other compilers may come for a specific architecture and are named mipsel-linux-gnu-gcc,
  but in my case the compiler is mips-linux-gnu-gcc and can output in both endianess. You
  have to consider this when using a different tool chain.
  To learn more about the capabilities of the compiler use mips-linux-gnu-gcc --target-help.
• The AR_mipsel-unknown-linux-gnu binary util is used to create, modify, and extract from
  archives (libraries).

Build files

Any minimal Cargo.toml. Nothing special:

[package]
name = "hello-vuduo"
version = "0.1.0"
authors = ["Jedzia <jed69@gmx.de>"]
edition = "2018"

Build file for Xargo Xargo.toml:

[dependencies]
std = {features = ["panic_unwind"]}

[build]
target = "mipsel-linux-gnu-gcc"

• std: The std dependency is what you want to build on your own.
  This is the cause of Illegal instruction errors on mips32r1 targets as the
  upstream target from the Rust repository is built for mips32r2 with fpxx and
  nooddspreg enabled. This change was done on Mar 15, 2018:
  bump mipsel isa leval and enable fpxx #48874

• The [build] target specifies the default architecture to build.
  Can also be specified at the command-line with

    cargo build --target=mipsel-unknown-linux-gnu
    xargo build --target=my-unknown-system-dos
  

Cargo configuration

Local cargo configuration .cargo/config:

[build]
target = "mipsel-unknown-linux-gnu"

[target.mipsel-unknown-linux-gnu]
rustflags = [
  "-C", "link-arg=-EL",
  "-C", "target-cpu=mips32",
  "-C", "target-feature=+mips32,-mips32r2,-fpxx,-nooddspreg",
]

ar = "mips-linux-gnu-ar"
linker = "mips-linux-gnu-gcc"
runner = "run.bat"  

• The [build] target specifies the default architecture to build (see above).
• Under [target.mipsel-unknown-linux-gnu] we specify the flags used for creating the binary
  and linking it. -C is forwarded to the rustc-compiler and indicates a code-generation option.
  rustc -C help will list all options available. The default target specs for the current rustc
  compiler can be listed with:
  rustc -Z unstable-options --print target-spec-json --target mipsel-unknown-linux-gnu.
  The used options in detail:
  • link-arg=-EL is daisy-chained to the linker(mips-linux-gnu-gcc): little endian output is generated.
  • target-cpu=mips32 for use with our good old mips32r1. A list can be retrieved with
    rustc --print target-cpus. We are building for mips so with our target:
    rustc --target mipsel-unknown-linux-gnu --print target-cpus.
  • target-feature: For a list of available options use rustc --print target-features or in our case:
    rustc --target mipsel-unknown-linux-gnu -C target-cpu=mips32 -C target-feature=help.
    • +mips32: Enable Mips32 ISA Support.
    • -mips32r2: Disable Mips32r2 ISA Support ... illegal instruction.
    • -fpxx: Disable support for FPXX. The FPXX extension mandates that all code must execute correctly
      when run using 32-bit or 64-bit registers. The code can be interlinked with either FP32 or FP64,
      but not both.
    • -nooddspreg: Disable odd numbered single-precision registers.
• ar: binary util used to create, modify, and extract from archives (libraries). Use our target
  specific one.
• linker: mips-linux-gnu-gcc is used as linker in conjunction with the specified
  link-arg=-EL flag to produce little endian binaries.
• runner: What (x/c)argo uses to execute the run command. The first command-line parameter is
  the relative path of your binary. I use a combination of scp to the target-box or copy to a network
  share and ssh to run it remotely.
  Something like:
  run.bat:
  cp %1 \VUDUO2/Harddisk/jedzia/rust
  ssh jedzia@vuduo2 /media/hdd/jedzia/rust/hello-vuduo

I was able to successfully compile and run with the fpxx and nooddspreg options enabled
("-C", "target-feature=+mips32,-mips32r2,+fpxx,+nooddspreg").
Maybe they are not necessary for
the VU+ Duo2. But i can't prove this as i only compile a little hello world with floating point
arithmetic in it. When it fails you have to tweak it for your target. I have disabled these options
as they are part of the #48874 change set (see above).

Compiling

run

xargo build

or if you have set up a runner in .cargo/config for remote execution

xargo run 

to see it work.

and

cargo build

to see it fail again with illegal instruction as it is compiled against the Rust
mipsel-unknown-linux-gnu target and not our custom one built by xargo.

[Jedzia]

I cant withstand not to add an example: https://github.com/Jedzia/RokerDepp