add st_fcheck and st_fclose, for output whose write can fail - #27
Open
glennhickey wants to merge 2 commits into
Open
add st_fcheck and st_fclose, for output whose write can fail#27glennhickey wants to merge 2 commits into
glennhickey wants to merge 2 commits into
Conversation
stdio reports a failed write -- ENOSPC, a quota, a read-only mount -- by setting the stream's error indicator, and nothing is obliged to look at it. A program that writes an output file and never checks therefore cannot tell a complete file from a truncated one, and still exits successfully. When the format is line oriented, as most of ours are, the short file also still parses: a truncated fasta is a valid fasta with less sequence in it. This is not hypothetical. Red hit exactly this in a VGP 577-way cactus alignment, silently losing up to 473 Mb from a 2.1 Gb genome across 9 genomes, all preprocessed on one node inside a 42 minute window. st_fcheck flushes and then tests the indicator; the flush is part of the check rather than an optimisation, because the buffer is not necessarily handed to the operating system until it happens, so a stream that has already lost data can still look clean beforehand. st_fclose adds a close whose return value is also checked, closing being the last point at which buffered data reaches the operating system and so the only place a failure there is ever reported. Both die via the existing st_errAbort/st_errnoAbort path, matching st_fopen, which is the neighbouring wrapper they are meant to pair with. Purely additive: no existing caller changes behaviour. Callers get converted separately. Tested with a child process writing to /dev/full, where every write fails with ENOSPC. Before, the write returned and the stream looked fine; now the child dies with status 1 naming the file. The happy path is asserted to be untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test distinguished only "no /dev/full" from "the real /dev/full". A third case exists: a rootfs unpacked without device nodes, or a sandbox that stubs /dev, leaves a plain writable file at that path. There the write succeeds, st_fcheck correctly returns, and the test reports a failure that says nothing about st_fcheck. Prove the device is the one we think it is, with a raw write that has to fail with ENOSPC, and treat anything else as a skip. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add some explicit checks to detect situations when writing to a full disk. The idea is to fail rather than silently truncating output.