CI is failing due to "/usr/bin/ld: final link failed: No space left on device"

Describe the bug

The CI on main is failing due to the runners running out of disk space

To Reproduce

Example: https://github.com/apache/arrow-datafusion/actions/runs/5277391664/jobs/9545549038

"-Wl,-Bdynamic" "-llzma" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/__w/arrow-datafusion/arrow-datafusion/target/debug/deps/roundtrip_physical_plan-8dff8073998bbd8f" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
  = note: /usr/bin/ld: final link failed: No space left on device
          collect2: error: ld returned 1 exit status
          

error: could not compile `datafusion-substrait` (test "roundtrip_physical_plan") due to previous error
Error: Process completed with exit code 101.

Expected behavior

Should pass

Additional context

No response

If someone else could look at this that would be great, otherwise I plan to do so in a few hours

Based off of this experiment:

Before any builds start the builder has 11G/35GB if its total disk, and has 33MB in the datafusion checkout:

11G	/
33M	/__w/arrow-datafusion/arrow-datafusion

After the build ends there is 35GB on the disk, all of it in the datafusion checkout:

35G	/
24G	/__w/arrow-datafusion/arrow-datafusion

Of that, almost all of it is in target:

4.0K	/__w/arrow-datafusion/arrow-datafusion/CHANGELOG.md
4.0K	/__w/arrow-datafusion/arrow-datafusion/CODE_OF_CONDUCT.md
4.0K	/__w/arrow-datafusion/arrow-datafusion/CONTRIBUTING.md
108K	/__w/arrow-datafusion/arrow-datafusion/Cargo.lock
4.0K	/__w/arrow-datafusion/arrow-datafusion/Cargo.toml
12K	/__w/arrow-datafusion/arrow-datafusion/LICENSE.txt
4.0K	/__w/arrow-datafusion/arrow-datafusion/NOTICE.txt
4.0K	/__w/arrow-datafusion/arrow-datafusion/README.md
224K	/__w/arrow-datafusion/arrow-datafusion/benchmarks
24K	/__w/arrow-datafusion/arrow-datafusion/ci
44K	/__w/arrow-datafusion/arrow-datafusion/conbench
13M	/__w/arrow-datafusion/arrow-datafusion/datafusion
196K	/__w/arrow-datafusion/arrow-datafusion/datafusion-cli
160K	/__w/arrow-datafusion/arrow-datafusion/datafusion-examples
792K	/__w/arrow-datafusion/arrow-datafusion/dev
444K	/__w/arrow-datafusion/arrow-datafusion/docs
4.0K	/__w/arrow-datafusion/arrow-datafusion/header
2.7M	/__w/arrow-datafusion/arrow-datafusion/parquet-testing
4.0K	/__w/arrow-datafusion/arrow-datafusion/pre-commit.sh
8.0K	/__w/arrow-datafusion/arrow-datafusion/python
4.0K	/__w/arrow-datafusion/arrow-datafusion/rustfmt.toml
24G	/__w/arrow-datafusion/arrow-datafusion/target.   <-- here is the problem
28K	/__w/arrow-datafusion/arrow-datafusion/test-utils
8.8M	/__w/arrow-datafusion/arrow-datafusion/testing

I will next sort out how to reduce the number and size of binaries

At least locally, for me, incremental compilation intermediate files seem to consume a large amount of space:

(arrow_dev) alamb@MacBook-Pro-8:~/Software/arrow-datafusion$ du -s -h target/debug/*
463M	target/debug/build
1.6G	target/debug/deps
780K	target/debug/examples
4.2G	target/debug/incremental

I will experiment with disabling that:
https://doc.rust-lang.org/cargo/reference/profiles.html?highlight=incremental#incremental

incremental seems to take 2GB of space on the builder -- will try disabling it and see what happens

https://github.com/apache/arrow-datafusion/actions/runs/5281745040/jobs/9555736433?pr=6678

Run du -s -h $GITHUB_WORKSPACE/target/debug/* || true
429M	/__w/arrow-datafusion/arrow-datafusion/target/debug/build
21G	/__w/arrow-datafusion/arrow-datafusion/target/debug/deps
4.0K	/__w/arrow-datafusion/arrow-datafusion/target/debug/examples
2.3G	/__w/arrow-datafusion/arrow-datafusion/target/debug/incremental

LOL well I have some fixes that I think will give us some breathing room for a while:

Disable incremental compilation on CI #6688 (Save 2GB)
Minor: combine dataframe and dataframe_functions tests to save build time and builder disk space #6682 (save 125MB)
Minor: combine statistics, filter_pushdown and custom_sources provider tests together to reduce CI disk space #6683 (save 250MB)
Minor: consolidate repartition test into sql_integration to save builder space and build time #6685 (save 125MB)
Minor: combine fuzz tests into a single binary to save builder space and build time #6684 (save 1GB)
Minor: consolidate datafusion_substrait tests into substrait_integration to save builder space and build time #6685 #6686 (save 250MB)

For a total of 22% ( 3.75GB out of 22GB currently made by this job)

After applying all our the updates from yesterday, I measured disk usage again: https://github.com/apache/arrow-datafusion/actions/runs/5291555219/jobs/9577253624?pr=6678

Before the work the disk had 35GB at the end of the test run

35G	/

After all the changes it has 8GB to spare

27G	/

🎉

Navigation Menu

Repository navigation

CI is failing due to "/usr/bin/ld: final link failed: No space left on device" #6676

Describe the bug

To Reproduce

Expected behavior

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CI is failing due to "/usr/bin/ld: final link failed: No space left on device" #6676

Description

Describe the bug

To Reproduce

Expected behavior

Additional context

Activity

alamb commented on Jun 15, 2023

alamb commented on Jun 15, 2023

alamb commented on Jun 15, 2023

alamb commented on Jun 15, 2023

alamb commented on Jun 15, 2023

alamb commented on Jun 16, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions