Skip to content

dictBuilder: validate ZDICT_addEntropyTablesFromBuffer capacities - #4869

Open
professor-moody wants to merge 1 commit into
meta:devfrom
professor-moody:fix-zdict-entropy-capacity-bounds
Open

professor-moody wants to merge 1 commit into
meta:devfrom
professor-moody:fix-zdict-entropy-capacity-bounds

Conversation

@professor-moody

@professor-moody professor-moody commented Oct 29, 2026 •

Copy link
Copy Markdown

Summary

This is defensive hardening of the deprecated ZDICT_addEntropyTablesFromBuffer() entry point and its private helper ZDICT_addEntropyTablesFromBuffer_advanced() in lib/dictBuilder/zdict.c. The normal legacy trainer already bounds dictionary content to destination capacity.

ZDICT_addEntropyTablesFromBuffer_advanced() reserves an 8-byte dictionary header (size_t hSize = 8;) and computes:

  • (char*)dictBuffer + hSize with remaining capacity dictBufferCapacity - hSize, and
  • (char*)dictBuffer + dictBufferCapacity - dictContentSize for the pre-populated dictionary content at the end of dictBuffer.

Before this change, ZDICT_addEntropyTablesFromBuffer_advanced() did not check these destination preconditions before performing unsigned subtraction and pointer arithmetic:

  1. When dictBufferCapacity < hSize (for example 0 through 7), dictBufferCapacity - hSize wraps unsigned size_t and (char*)dictBuffer + hSize points past the destination allocation, allowing ZDICT_analyzeEntropy() to write Huffman and FSE tables out of bounds on the heap.
  2. When dictContentSize > dictBufferCapacity, ((char*)dictBuffer + dictBufferCapacity) - dictContentSize attempts to form a pointer before dictBuffer, which is undefined behavior and causes out-of-bounds reads in ZDICT_analyzeEntropy() and XXH64().
  3. When dictBuffer == NULL, (char*)dictBuffer + hSize performs non-zero offset arithmetic on a null pointer before writing header fields.

Fix

Add an upfront precondition check in ZDICT_addEntropyTablesFromBuffer_advanced(), consistent with the capacity checks in ZDICT_finalizeDictionary():

if ((dictBuffer == NULL) || (dictBufferCapacity < dictContentSize) || (dictBufferCapacity < hSize))
    return ERROR(dstSize_tooSmall);

This preserves existing truncation behavior when dictBufferCapacity >= hSize and dictContentSize <= dictBufferCapacity (including ZDICT_trainFromBuffer_unsafe_legacy() calls where dictContentSize == dictBufferCapacity). This update rebases the branch onto dev (9d728bb0) to resolve an adjacent unit-test context conflict in tests/fuzzer.c.

Testing

  • Added unit regression coverage in tests/fuzzer.c (basicUnitTests) verifying:
    • ZSTD_error_dstSize_tooSmall rejection for dictBufferCapacity < 8 (0 and 7), dictContentSize > dictBufferCapacity (17 > 16), and dictBuffer == NULL;
    • successful dictionary header and entropy-table generation on a valid pre-populated buffer (ZDICT_getDictID() != 0).
  • Verified the full unfiltered basicUnitTests suite in tests/fuzzer.c (./tests/fuzzer -i0 -s0 -v) under AddressSanitizer and UndefinedBehaviorSanitizer as well as the default unit-test build.

Disclosure: Automated static analysis and AI coding assistance were used to draft and verify this defensive bounds check and regression test. AI assistance was used in drafting and validating this change; the patch and tests were reviewed and verified by the author.

Validate destination buffer preconditions in
`ZDICT_addEntropyTablesFromBuffer_advanced` (`lib/dictBuilder/zdict.c`)
before subtracting `hSize` (`8`) or `dictContentSize` from
`dictBufferCapacity` and offsetting `dictBuffer`:

- Return `ERROR(dstSize_tooSmall)` when `dictBuffer == NULL`,
  `dictBufferCapacity < dictContentSize`, or
  `dictBufferCapacity < hSize`.
- Add unit tests in `tests/fuzzer.c` (`basicUnitTests`) covering too
  small capacities (`7` and `0`), `dictContentSize > dictBufferCapacity`
  (`17 > 16`), `dictBuffer == NULL`, and a valid pre-populated buffer.

Automated static analysis and AI coding assistance were used to draft
and verify this defensive bounds check and regression test. AI
assistance was used in drafting and validating this change; the patch
and tests were reviewed and verified by the author.
@professor-moody professor-moody changed the title dictBuilder: validate buffer capacities in ZDICT_addEntropyTablesFromBuffer_advanced dictBuilder: validate ZDICT_addEntropyTablesFromBuffer capacities Nov 14, 2026
@professor-moody
professor-moody force-pushed the fix-zdict-entropy-capacity-bounds branch from e084949 to 72c83e1 Compare November 14, 2026 19:44

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant