feat: add block-level Brotli/GZip compression for SingleFile storage - #344
Merged
MPCoreDeveloper merged 1 commit intoAug 30, 2026
Merged
Conversation
|
This was referenced Aug 30, 2026
MPCoreDeveloper
pushed a commit
that referenced
this pull request
Aug 30, 2026
MPCoreDeveloper
pushed a commit
that referenced
this pull request
Aug 30, 2026
MPCoreDeveloper
pushed a commit
that referenced
this pull request
Aug 30, 2026
MPCoreDeveloper
pushed a commit
that referenced
this pull request
Aug 30, 2026
… block flag on rewrites + read row-cache via ReadBlockAsync A table row-cache block that was rewritten while it already existed (auto-flush as JSON grows past the compression threshold, or grow/realloc) was stored compressed WITHOUT the per-block Compressed flag, so on reopen the raw Brotli/GZip bytes were handed to the JSON parser (JsonException '0x0B is an invalid start of a value'). WriteBlockAsync now recomputes the flag for every write, and SingleFileTable.EnsureCacheLoaded reads through ReadBlockAsync (transparent decrypt+decompress). Regression tests added for compression and compression+encryption reopen+SELECT.
MPCoreDeveloper
pushed a commit
that referenced
this pull request
Aug 30, 2026
Updates documentation and code comments that still described block compression as 'not implemented' or 'reserved' despite being shipped in v1.9.8 (PR #344). Co-authored-by: scdb-dev <scdb-dev@localhost>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Block-level Brotli/GZip compression for SingleFile (.scdb) storage
Summary
Adds transparent, per-block compression to the SingleFile storage mode. Compression is applied before encryption on write and removed after decryption on read, so the two layers compose cleanly without interfering with each other. A per-block
Compressedflag in the block registry tracks state, allowing mixed compressed and uncompressed blocks within the same file.Motivation
SingleFile mode pre-allocates metadata pages (registry, FSM, WAL, table directory) and stores every block uncompressed. For workloads with repetitive or text-heavy data, this wastes significant disk space. Block-level compression lets the engine shrink on-disk payloads without changing the file format's structural layout or breaking compatibility with existing encrypted databases.
Implementation
BlockCompressionModeenum:None,Brotli,GZip— selectable viaDatabaseOptions.BlockCompressionBlockCompressorservice: applies compression only to blocks at or aboveDatabaseOptions.CompressionThreshold(default 64 bytes); smaller blocks pass through uncompressed to avoid negative compression ratiosSingleFileStorageProvider: compression happens between the serialization layer and the encryption layer on write; decompression happens between decryption and deserialization on readVacuumFullAsyncpropagatesBlockCompressionandCompressionThresholdto the temp file options, so vacuumed files retain their compression settingsPOC statistics
Statistics below are from the Brotli benchmark. GZip is supported and tested but not benchmarked in the POC.
A 10 million record proof-of-concept was run before implementing the production path:
Test coverage
16 regression tests in
SingleFileCompressionTests.cscovering:InvalidOperationExceptionGetBlockMetadataDatabaseFactoryintegration with compression-enabled optionsDatabaseFactorywrong compression mode on reopen throwsAll 16 tests pass.
Compatibility note
If you have a different compression or storage strategy planned for v2.0 that I can't see from the public repo, no concerns at all — happy to rework, rebase, or withdraw this if it conflicts with your roadmap. This was built against the v1.9.8 codebase and is intended to be fully backward compatible (compression defaults to
None, so existing files open unchanged).Files changed
src/SharpCoreDB/Services/BlockCompressor.cs(new)src/SharpCoreDB/Storage/BlockCompressionMode.cs(new)src/SharpCoreDB/Storage/SingleFileStorageProvider.cs(modified — compression integration + vacuum fix)tests/SharpCoreDB.Tests/Storage/SingleFileCompressionTests.cs(new)