Skip to content

use half of the available cores for rocBLAS - #281

Open
bedroge wants to merge 2 commits into
EESSI:mainfrom
bedroge:rocblas_parallel
Open

use half of the available cores for rocBLAS#281
bedroge wants to merge 2 commits into
EESSI:mainfrom
bedroge:rocblas_parallel

Conversation

@bedroge

@bedroge bedroge commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

This should hopefully prevent it from running out of memory.

@bedroge

bedroge commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Let's give it a try

bot: build repo:eessi.io-2025.06-software instance:eessi-bot-aws-eu-south on:arch=x86_64/amd/zen5 for:arch=x86_64/amd/zen5,accel=amd/gfx908+amd/gfx90a+amd/gfx942+amd/gfx1030+amd/gfx1100+amd/gfx1101+amd/gfx1200+amd/gfx1201

@eessi-bot-aws-eu-south

eessi-bot-aws-eu-south Bot commented Aug 6, 2026

Copy link
Copy Markdown

New job on instance eessi-bot-aws-eu-south for repository eessi.io-2025.06-software
Building on: amd-zen5
Building for: x86_64/amd/zen5 and accelerator amd/gfx908+amd/gfx90a+amd/gfx942+amd/gfx1030+amd/gfx1100+amd/gfx1101+amd/gfx1200+amd/gfx1201
Job dir: /project/def-users/SHARED/jobs/2026.08/pr_281/241

date job status comment
Aug 06 09:56:20 UTC 2026 submitted job id 241 awaits release by job manager
Aug 06 09:56:52 UTC 2026 released job awaits launch by Slurm scheduler
Aug 06 10:01:56 UTC 2026 running job 241 is running

@bedroge

bedroge commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Not sure what's going on with these jobs. It doesn't look like they run out of memory, but it's getting stuck at some point, and ultimately Slurm aborts and requeues the job on the same node? On the node itself I only see these messages in the syslog:

[Thu Aug  6 11:11:51 2026] INFO: task bash:1680114 blocked for more than 122 seconds.
[Thu Aug  6 11:11:51 2026]       Not tainted 5.14.0-611.49.1.el9_7.x86_64 #1
[Thu Aug  6 11:11:51 2026] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Aug  6 11:11:51 2026] task:bash            state:D stack:0     pid:1680114 tgid:1680114 ppid:1680110 flags:0x00004002
[Thu Aug  6 11:11:51 2026] Call Trace:
[Thu Aug  6 11:11:51 2026]  <TASK>
[Thu Aug  6 11:11:51 2026]  __schedule+0x231/0x540
[Thu Aug  6 11:11:51 2026]  schedule+0x2c/0xb0
[Thu Aug  6 11:11:51 2026]  io_schedule+0x41/0x70
[Thu Aug  6 11:11:51 2026]  folio_wait_bit_common+0x11d/0x330
[Thu Aug  6 11:11:51 2026]  ? xas_init_marks+0x1f/0x50
[Thu Aug  6 11:11:51 2026]  ? __pfx_wake_page_function+0x10/0x10
[Thu Aug  6 11:11:51 2026]  invalidate_inode_pages2_range+0x179/0x3f0
[Thu Aug  6 11:11:51 2026]  fuse_open_common+0x1bf/0x220 [fuse]
[Thu Aug  6 11:11:51 2026]  ? __pfx_fuse_open+0x10/0x10 [fuse]
[Thu Aug  6 11:11:51 2026]  do_dentry_open+0x14c/0x440
[Thu Aug  6 11:11:51 2026]  do_open+0x219/0x400
[Thu Aug  6 11:11:51 2026]  path_openat+0x110/0x280
[Thu Aug  6 11:11:51 2026]  do_filp_open+0xb0/0x160
[Thu Aug  6 11:11:51 2026]  do_open_execat+0x5c/0xf0
[Thu Aug  6 11:11:51 2026]  open_exec+0x27/0x50
[Thu Aug  6 11:11:51 2026]  load_elf_binary+0x20c/0xe30
[Thu Aug  6 11:11:51 2026]  ? __kernel_read+0x18a/0x320
[Thu Aug  6 11:11:51 2026]  search_binary_handler+0xd2/0x250
[Thu Aug  6 11:11:51 2026]  exec_binprm+0x4f/0x1a0
[Thu Aug  6 11:11:51 2026]  bprm_execve.part.0+0x179/0x210
[Thu Aug  6 11:11:51 2026]  do_execveat_common.isra.0+0x1a0/0x220
[Thu Aug  6 11:11:51 2026]  __x64_sys_execve+0x33/0x40
[Thu Aug  6 11:11:51 2026]  do_syscall_64+0x5c/0xe0
[Thu Aug  6 11:11:51 2026]  ? xas_start+0x5f/0xd0
[Thu Aug  6 11:11:51 2026]  ? _raw_spin_unlock_irqrestore+0xa/0x30
[Thu Aug  6 11:11:51 2026]  ? next_uptodate_folio+0x85/0x260
[Thu Aug  6 11:11:51 2026]  ? _raw_spin_unlock+0xa/0x30
[Thu Aug  6 11:11:51 2026]  ? filemap_map_pages+0x45e/0x540
[Thu Aug  6 11:11:51 2026]  ? do_read_fault+0x10b/0x1d0
[Thu Aug  6 11:11:51 2026]  ? do_pte_missing+0x157/0x200
[Thu Aug  6 11:11:51 2026]  ? __handle_mm_fault+0x2fe/0x650
[Thu Aug  6 11:11:51 2026]  ? set_next_entity+0xda/0x150
[Thu Aug  6 11:11:51 2026]  ? __count_memcg_events+0x4f/0xb0
[Thu Aug  6 11:11:51 2026]  ? mm_account_fault+0x6c/0x100
[Thu Aug  6 11:11:51 2026]  ? handle_mm_fault+0x138/0x270
[Thu Aug  6 11:11:51 2026]  ? do_user_addr_fault+0x35d/0x620
[Thu Aug  6 11:11:51 2026]  ? exc_page_fault+0x61/0x150
[Thu Aug  6 11:11:51 2026]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[Thu Aug  6 11:11:51 2026] RIP: 0033:0x14f133880ce7
[Thu Aug  6 11:11:51 2026] RSP: 002b:00007ffe20441e68 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
[Thu Aug  6 11:11:51 2026] RAX: ffffffffffffffda RBX: 00005562b7abfdb0 RCX: 000014f133880ce7
[Thu Aug  6 11:11:51 2026] RDX: 00005562b7ac3ac0 RSI: 00005562b7b052f0 RDI: 00005562b7abfdb0
[Thu Aug  6 11:11:51 2026] RBP: 0000000000000000 R08: 00007ffe20441e40 R09: 0000000000000000
[Thu Aug  6 11:11:51 2026] R10: 000014f1337bd9f8 R11: 0000000000000202 R12: 00000000ffffffff
[Thu Aug  6 11:11:51 2026] R13: 00005562b7b052f0 R14: 00005562b7ac3ac0 R15: 00005562b7b025d0
[Thu Aug  6 11:11:51 2026]  </TASK>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant