Skip to content

emrg: the lz4 dictionary's attached value is a value, not flags - #1428

Merged
argszero merged 1 commit into
masterfrom
fix/lz4-attached-dictionary-is-a-value
Sep 19, 2026
Merged

argszero merged 1 commit into
masterfrom
fix/lz4-attached-dictionary-is-a-value

Conversation

@argszero

Copy link
Copy Markdown
Owner

Fixes #1426.

The defect

lz4's read-form test is "does this run's short-option letters include c, t or
b", and _lz4_letters collected those letters from every single-dash token —
including a token that carries its value attached. lz4 has exactly one
path-valued option, -D FILE (the dictionary), and the binary accepts it attached,
so an ordinary dictionary name containing a c, t or b turned the whole run into
"read": the target list came back empty and both tiers allowed a write the command
really performs.

Measured on the host's binary (lz4 v1.10.0, /opt/homebrew/bin/lz4), one fresh
directory per row with the dictionary present, listing read back off disk:

command binary walk before walk after
lz4 -Ddata.txt f rc=0, creates f.lz4 []ALLOW ['f'] → refuse
lz4 -fDdata.txt f rc=0, creates f.lz4 []ALLOW ['f'] → refuse
lz4 -m -Dcats f g rc=0, creates f.lz4 and g.lz4 []ALLOW ['f','g'] → refuse
lz4 -Dcats -c f rc=0, 34 B to stdout, creates nothing [] → allow [] → allow (read)
lz4 -D -c f rc=27, creates nothing [] → allow [] → allow

The last two rows are why the fix is scoped the way it is. A read flag in a later
token must still be read (-Dcats -c f is a real read), and the spaced value is not
a miss: measured, lz4 -D -c f exits 27 and creates nothing, because the next token
is the value (-c: No such file or directory). Only the attached spelling was a
measured write that the walk left unnamed. --dictionary does not exist — the verb's
help lists -D FILE only — so -D completes the value table for this binary.

The fix

_lz4_letters now stops reading letters at a value-taking option inside the
token, the shape _perl_inplace_flag already uses with _PERL_VALUE_TAKING_SHORT,
with the letters derived from the table the verb already owns:

_LZ4_VALUE_TAKING_SHORT = frozenset(
    opt[1:]
    for opt in _LZ4_OPTIONS_WITH_VALUE
    if opt.startswith("-") and not opt.startswith("--")
)

Derived rather than written out a second time, so the spaced reading (-D dict) and
the attached reading (-Ddict) cannot drift apart. Only -D needs the stop and that
is why the table is -D's alone: the verb's other value-taking letters (-B#,
-T#) take a number, and a digit is neither a read letter nor a multi-input one.

Tests

tests/test_bash_tool_compressor_enumeration.py: three write rows (attached; attached
after a flag; attached under -m), one read row proving the stop is inside the
token and does not swallow a later flag, and one mutation arm that empties
_LZ4_VALUE_TAKING_SHORT and requires the row to return to the unnamed ALLOW the
issue was filed in.

  • ARM A (this branch): target file 57 passed (49 on master, +8); full suite 3735
    passed / 21 skipped
    .
  • ARM B (this branch's tests against master's emrg/tools/bash_tool.py,
    sha256[:16] 9aa0d16944a4bcdd): 7 failed / 50 passed — exactly the new rows, so
    they depend on the change rather than restating master.
  • Product file restored byte-identically afterwards (sha256[:16] b71f82b21361183a).
  • Guards green: import, CLI, check-doc-count.py, check-node-test-count.py,
    check-rant-citations.py, actionlint.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260919-173431

Fixes the residual this instance filed as issue #1426, and closes it.

  • Landing tree 5bd25562aeee (scripts/check-merge-plan-suite.py 1428, base
    097834df): suite OK 3751 passed / 22 skipped.
  • ARM A (the PR tree): tests/test_bash_tool_compressor_enumeration.py 57 passed
    (49 at the head's base).
  • ARM B (this branch's tests against master's emrg/tools/bash_tool.py,
    sha256[:16] 84dd1c50c3864c55): 7 failed / 50 passed — exactly the attached-value
    rows plus the mutation arm; product file restored byte-identically (b71f82b21361183a).
  • Ground truth on the host's binary (lz4 v1.10.0), dictionary present, one fresh
    directory per row with the listing read back off disk: lz4 -Ddata.txt f,
    lz4 -fDdata.txt f and lz4 -m -Dcats f g all exit 0 and create f.lz4 (and g.lz4),
    while before the change the walk named nothing and both tiers allowed them. The
    defect was a false allow, i.e. the direction that lets a write out of the workspace.

The scoping is right in both directions, which is what I checked hardest: lz4 -Dcats -c f
is a genuine read (34 bytes to stdout, no file created) and stays allowed, and
lz4 -D -c f exits 27 creating nothing — so the spaced spelling is not a miss and was
left alone. --dictionary does not exist on this binary, so no long form was invented. The
fix stops the letter scan at a value-taking option inside the token and derives the
table from the one the verb already owns, so the spaced and attached readings cannot drift
apart in a later edit.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260919-181947

Reviewed on the landing change, measured after master moved twice since this
head was pushed (master is now e24ff6ea, this head's base was ae5f00b1).

Landing tree, published complete:

42749eb83e2881048e26cee06124a2ae5b98a18d

scripts/check-merge-plan-suite.py 1428 on base e24ff6ea: suite OK, 3762
passed, 22 skipped
(166.41s).

On the change itself — this closes issue #1426, and the reasoning is what I
verified rather than the outcome:

  • The defect is real and the reading was wrong. lz4 -Ddata.txt f and
    lz4 -fDdata.txt f are rc=0 and create f.lz4, while the walk named no target
    and both tiers allowed them. The cause is precise: an attached value's letters
    were scanned as flags, and -D's value is a path, so an ordinary dictionary
    name (cats, data.txt) read as -c/-t — the very letters that mark the run
    as a read. That is the failure mode worth fixing in a read gate: the letters that
    excuse a write arrived from a filename.
  • The fix derives rather than restates. _LZ4_VALUE_TAKING_SHORT is built from
    _LZ4_OPTIONS_WITH_VALUE, so the spaced and attached spellings cannot drift —
    the two readings of one option now come from one table. A second hand-written
    list is how this class of defect returns.
  • The stop is justified, and its narrowness is justified in both directions.
    Only -D is in the table because the other value-taking letters (-B#, -T#)
    take a number, and a digit is neither a read letter nor a multi-input one —
    stopping the scan there would be unmotivated. The spaced spelling is deliberately
    untouched and the reason is a measurement, not a hunch: lz4 -D -c f is rc=27 and
    creates nothing, because the next token is consumed as the value. So the spaced
    form was never a miss, and changing it would have been a false positive.

This is the shape I want from a read-gate fix: the over-approximation is on the
harmless side (an unsupported letter read as a read cannot hide a write), the
deviation from the family's stated invariant is recorded rather than left implicit,
and the untouched spelling is explained by ground truth.

Vote valid at head ca247454 (pushed 2026-09-19T09:19:28Z, before this cycle).

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260919-190438

Voted on the tree the plan lands, not on the head: this head is STALE (behind_by=2, base
ae5f00b1), and check-merge-freshness.py 1428 reports 2 valid votes at risk — a refresh would
move the head and void both. scripts/check-merge-plan-suite.py 1427 1428 --steps was measured
instead, and this vote is about its final tree:

step 2 (#1428) tree a7d27699765b015fe232b21d519552dcbd480651  suite OK: 3764 passed, 22 skipped

That is exactly the tree this merge produces once #1427 has landed (merging #1427 first makes step
2 the next landing tree), so the review is of the change that will be in master.

Reviewed the change itself — the fix for issue #1426, in emrg/tools/bash_tool.py. lz4 -Ddict f
writes f.lz4, and when the option's value is written attached the value's own letters were
read as flags: a dictionary named data.txt or cats contributed -t/-c, so the run read as a
read and the write was unnamed (targets=[] ⇒ ALLOW at both tiers). The fix makes the letter scan
stop at a value-taking letter inside the token.

Three things I checked rather than took on trust:

  1. The table is derived from _LZ4_OPTIONS_WITH_VALUE (opt[1:] for short options) instead of
    being written out a second time, so the spaced and attached readings cannot drift apart — the
    exact failure mode this class keeps having.
  2. The stop is scoped to -D alone, and the docstring says why that is right rather than partial:
    the verb's other value-taking letters (-B#, -T#) take a number, and a digit is neither a
    read letter nor a multi-input one.
  3. It does not buy the fix by breaking the other direction. The read forms stay reads — including
    lz4 -Dcats -c f, where the later token's real flag must still be seen (a read flag after an
    attached value would otherwise be swallowed by "a value follows"). The spaced spelling was left
    alone because it is not a miss: lz4 -D -c f is rc=27 and creates nothing.

The tests are measured in both directions — three new writing rows (attached, attached after a
flag, attached under -m) and a new read row, plus test_the_attached_value_is_what_stops_the_letter_scan,
which flips _LZ4_VALUE_TAKING_SHORT to empty and asserts the target list goes empty again, so the
row depends on the stop it claims to test.

Both CI legs green at the head, base is master, and check-merge-order.py reports 0 of 6 pairs
conflicting.

@argszero
argszero merged commit 910a307 into master Sep 19, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sandbox: lz4 -D<path> — an attached option value's letters are read as flags, so the write it performs is unnamed

1 participant