Skip to content

fix(bed): read BED files through htslib so gzip and bgzip work (#230) - #240

Merged
TimD1 merged 2 commits into
devfrom
230_td_bed-htslib
Aug 8, 2026
Merged

fix(bed): read BED files through htslib so gzip and bgzip work (#230)#240
TimD1 merged 2 commits into
devfrom
230_td_bed-htslib

Conversation

@TimD1-bot

@TimD1-bot TimD1-bot commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

Note

Authorship: the content below was drafted by Claude Opus 5 (an AI coding agent) and
filed via gh under @TimD1-bot, a bot account operated by @TimD1. It reflects the
agent's analysis, not a statement authored by @TimD1.

Summary

bedData::bedData() read with an std::ifstream, so a compressed BED could not be parsed — and
it failed in the worst possible way. The constructor's fopen existence check succeeded on a
.bed.gz, the first line then decoded to binary garbage, and parse_coord() reported:

[ERROR vcfdist] Invalid coordinate '<garbage>' on line 1 of BED file 'regions.bed.gz'

which names the wrong cause: the coordinate is not malformed, the file is compressed. ERROR()
calls std::exit, so the try/catch around the -b constructor never ran and there was no
second chance to improve the message.

This blocks #47 outright, since the GIAB stratification sets ship every region file as .bed.gz.

Change

hts_open() + hts_getline() on a kstring_t replace the ifstream. htslib decodes plain,
gzip, and bgzip transparently and detects the encoding from the file's leading bytes rather than
its extension, so a compressed BED misnamed .bed is read correctly too. htslib was already a
link dependency — bcf_open() reads the VCFs.

The malformed-coordinate ERROR paths added under #200 are untouched: the field, the 1-based
line, and the filename are still reported, and now identically whether the file was compressed.

Two incidental consequences of the reader change, both tested:

  • A read failure is no longer mistaken for end of file. hts_getline returns -1 at EOF and
    <= -2 on error; only -1 ends the loop. A truncated or corrupt compressed BED previously
    would have parsed as a partial region set and scored against it — it now errors naming the line.
  • A CRLF line terminator is accepted. hts_getline strips \r\n, where std::getline left
    the \r on the stop coordinate and rejected the file.

The -b usage line now reads BED file containing regions to evaluate (plain, gzip, or bgzip).

Testing

Nine BedCtor cases added alongside the existing ones, none replaced:

test covers
ParsesUncompressed / ParsesGzip / ParsesBgzip the same three regions on two contigs parse identically from all three encodings
DetectsEncodingFromContents a bgzip file named .bed still parses
MalformedCoordInGzipNamesLine a bad coordinate inside a gzip BED names the same field and decoded line
TruncatedGzipErrors a half-truncated gzip BED errors rather than parsing short
AcceptsCarriageReturns chr1\t2\t8\r yields stop 8, not "8\r"
AcceptsEmptyFile an empty BED remains vacuously valid
OpenFailErrors (existing) an unopenable path still errors naming the file

write_tmp_bed() gained a bedzip_t encoding argument (defaulting to plain, so existing callers
are unchanged) that writes the compressed variants via bgzf_open's wg/w modes.

One integration case, test_bed_gzipped-regions-match-plain, runs the swallowed_snps callsets
against a gzip copy of synthetic.bed compressed into the working directory — no compressed
fixture is committed — and pins the same counts as the uncompressed run.

Build fix the tests forced

The second commit adds -I$(SRC) to TEST_CXXFLAGS in tests/unit/build/Makefile.

Writing the compressed fixtures needs "htslib/bgzf.h" from tests/unit/src/test_helpers.cpp. A
quoted include is resolved against the including file's own directory before any -I, so that
reached CI's headers only from a file in src/ — where CI copies them, rather than installing
them. It compiled locally regardless, because Homebrew's htslib is on the default search path, so
the first CI run failed with fatal error: htslib/bgzf.h: No such file or directory even
though every local check was green.

Both resolution paths were then verified explicitly: with the headers copied to src/htslib (the
CI layout) -H shows bgzf.h resolving from ../../../src/htslib/bgzf.h, and with that
directory absent it falls back to the system path. CXXFLAGS is deliberately untouched, since it
must stay identical to src/Makefile.

Verification

  • Unit suite: 756 tests, all passing — in both header layouts.
  • pytest (integration + unit workflow): 115 passed, up from 111.
  • Both builds compile clean under -Wall -Wextra -Werror=missing-field-initializers; doxygen src/Doxyfile emits no warnings.
  • CI passes on 862db48.
  • CI pins htslib 1.17; KS_INITIALIZE, ks_free, hts_getline, bgzf_open's g mode, and
    hts_hopen's handling of empty_format were each confirmed present in the 1.17 sources, since
    local development is against 1.20.

Part of #47; first of nine. Resolves #230 — note this will not auto-close, since the PR targets
dev while the default branch is master.

bedData::bedData() read with an ifstream, so a compressed BED could not be
parsed. The fopen existence check succeeded on a .bed.gz, the first line then
decoded to binary garbage, and parse_coord() reported an invalid coordinate --
naming the wrong cause, since the coordinate was not malformed. ERROR() exits,
so the try/catch around the -b constructor never ran.

Replace the ifstream with hts_open() plus hts_getline() on a kstring_t, which
decodes plain, gzip, and bgzip transparently and detects the encoding from the
file's leading bytes rather than its extension. A read failure below EOF is now
reported rather than being mistaken for a short file, so a truncated compressed
BED no longer parses as a partial region set. The malformed-coordinate ERROR
paths are unchanged.

htslib was already a link dependency; bcf_open() reads the VCFs.
A quoted include is resolved against the including file's own directory before
any -I path, so "htslib/bgzf.h" in tests/unit/src/test_helpers.cpp searched
tests/unit/src/htslib/ and stopped. It built locally only because Homebrew's
htslib is on the default search path; CI copies the headers into src/ instead
of installing them, so the test build failed to find bgzf.h.

Add -I$(SRC) to TEST_CXXFLAGS, giving the test objects the same htslib
resolution the src objects get for free by living in src/. Verified both ways:
with the headers copied to src/htslib (the CI layout) they resolve from there,
and with that directory absent they fall back to the system path.

CXXFLAGS is untouched, since it must stay identical to src/Makefile.
@TimD1
TimD1 merged commit 62b7859 into dev Aug 8, 2026
1 check passed
@TimD1
TimD1 deleted the 230_td_bed-htslib branch August 8, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants