Skip to content

Reduce 32-bit zstdmt maximum job size to 256 MB - #4946

Merged
kevinjzhang merged 1 commit into
meta:devfrom
kevinjzhang:fix/zstdmt-32bit-jobsize-max
Nov 13, 2026
Merged

kevinjzhang merged 1 commit into
meta:devfrom
kevinjzhang:fix/zstdmt-32bit-jobsize-max

Conversation

@kevinjzhang

@kevinjzhang kevinjzhang commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

On 32-bit builds, a level 22 stream with nbWorkers=1 needs a 2 GiB round buffer (4 x 512 MB), which malloc cannot provide in a 32-bit address space, so compression fails with memory_allocation (#4628).

Lower ZSTDMT_JOBLOG_MAX from 29 to 28 and ZSTDMT_JOBSIZE_MAX from 512 MB to 256 MB on 32-bit builds, as such systems should not be using such resource intensive jobs anyway.

Notes:

  • On 32-bit, inputs larger than 256 MB are split into jobs at different points, so compressed bytes can differ for those configs. Output is still a valid zstd frame.
  • Related: Fix 32-bit zstdmt max job size allocation failure #4675 proposes the same cap change, but the tests are not necessary which just check the bounds match the architecture.

@meta-cla meta-cla Bot added the CLA Signed label Nov 12, 2026
@kevinjzhang
kevinjzhang requested a review from Cyan4973 November 12, 2026 23:04
@kevinjzhang
kevinjzhang marked this pull request as ready for review November 12, 2026 23:05

@Cyan4973 Cyan4973 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

A rebase should be enough to fix the remaining CI test jobs.

#define ZSTDMT_JOBLOG_MAX (MEM_32bits() ? 29 : 30)
#define ZSTDMT_JOBSIZE_MAX (MEM_32bits() ? (512 MB) : (1024 MB))
#define ZSTDMT_JOBLOG_MAX (MEM_32bits() ? 28 : 30)
#define ZSTDMT_JOBSIZE_MAX (MEM_32bits() ? (256 MB) : (1024 MB))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: I would have made ZSTDMT_JOBSIZE_MAX derived from ZSTDMT_JOBLOG_MAX,
like:

#define ZSTDMT_JOBSIZE_MAX  (1 << (ZSTDMT_JOBLOG_MAX))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think although this is cleaner, in this case it made sense to keep the MB values so we know the actual quantity of ZSTDMT_JOBSIZE_MAX without calculating it so I will keep it as is.

On 32-bit builds, a level 22 stream with nbWorkers=1 needs a 2 GiB
round buffer (4 x 512 MB), which malloc cannot provide in a 32-bit
address space, so compression fails with memory_allocation.

Lower ZSTDMT_JOBLOG_MAX from 29 to 28 and ZSTDMT_JOBSIZE_MAX from
512 MB to 256 MB on 32-bit builds, so the same case needs 1 GiB.
64-bit builds are unchanged.

Addresses meta#4628.
@kevinjzhang
kevinjzhang force-pushed the fix/zstdmt-32bit-jobsize-max branch from 0d6b4aa to 37a900a Compare November 13, 2026 18:54
@kevinjzhang
kevinjzhang merged commit 07490bd into meta:dev Nov 13, 2026
109 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants