[Bug] SingleFile (.scdb) mode ignores DatabaseOptions.EncryptionKey, writing data in plaintext

[saltus7]

Description
When using StorageMode.SingleFile (.scdb), the database accepts DatabaseOptions.EnableEncryption = true and a 32-byte EncryptionKey, but completely bypasses the encryption pipeline. Data is written to disk in plaintext, and reopening with a completely different (wrong) 32-byte key succeeds without throwing a cryptographic exception.

Minimal Reproduction
The following snippet demonstrates the bug. It creates an encrypted database, inserts a secret, and then successfully reopens it with a different key, proving the key is ignored and data is unencrypted.

using System;
using System.IO;
using System.Security.Cryptography;
using Microsoft.Extensions.DependencyInjection;
using SharpCoreDB;

// 1. Setup DI
var services = new ServiceCollection();
services.AddSharpCoreDB();
using var sp = services.BuildServiceProvider();
var factory = sp.GetRequiredService<DatabaseFactory>();

var path = "test.scdb";
var correctKey = RandomNumberGenerator.GetBytes(32);
var wrongKey = RandomNumberGenerator.GetBytes(32);

var options = new DatabaseOptions
{
    StorageMode = StorageMode.SingleFile,
    EnableEncryption = true,
    EncryptionKey = correctKey,
    CreateImmediately = true
};

// 2. Create and write with correct key
using (var db = factory.CreateWithOptions(path, "unused", options))
{
    db.ExecuteSQL("CREATE TABLE Secrets (Id INT, Data TEXT)");
    db.ExecuteSQL("INSERT INTO Secrets VALUES (1, 'classified-payload')");
    db.ForceSave();
}

// 3. Reopen with WRONG key (BUG: This should fail, but it succeeds)
using (var dbWrong = factory.CreateWithOptions(path, "unused", options with { EncryptionKey = wrongKey }))
{
    var rows = dbWrong.ExecuteQuery("SELECT * FROM Secrets");
    Console.WriteLine($"Wrong key opened DB. Rows found: {rows.Count}"); // Outputs: 1
}

// 4. Verify plaintext on disk (BUG: Secret is visible in hex editor)
var fileBytes = File.ReadAllBytes(path);
var secretBytes = System.Text.Encoding.UTF8.GetBytes("classified-payload");
bool isPlaintext = fileBytes.AsSpan().IndexOf(secretBytes.AsSpan()) >= 0;
Console.WriteLine($"Plaintext found on disk: {isPlaintext}"); // Outputs: True

Expected Behavior

• The .scdb file should contain AES-256-GCM encrypted ciphertext, not plaintext.
• Attempting to open the database with an incorrect 32-byte key should fail (e.g., throw a cryptographic authentication exception, or fail to decrypt the table directory, resulting in an empty/missing schema).

Actual Behavior

• The SingleFileStorageProvider accepts the EncryptionKey in DatabaseOptions but never instantiates AesGcmEncryption or passes the key to the I/O layer.
• WriteBlockAsync queues raw plaintext bytes directly to the write-behind cache.
• ReadBlockAsync, GetReadSpan, and GetReadStream return raw plaintext bytes without attempting decryption.
• Wrong keys are silently accepted because no cryptographic validation occurs during the read path.

Environment

• OS: Windows 11 / Linux x64 (Reproducible on both)
• .NET Version: .NET 10.0
• SharpCoreDB Version: 1.9.6
• Storage Mode: StorageMode.SingleFile (.scdb)

Root Cause Analysis
Forensic analysis of src/SharpCoreDB/Storage/SingleFileStorageProvider.cs shows:

1. The Open method reads options.EnableEncryption but never resolves ICryptoService or instantiates AesGcmEncryption.
2. The constructor does not accept or store an encryption instance.
3. I/O paths (WriteBlockAsync, ReadBlockAsync, GetReadSpan, GetReadStream) operate directly on the FileStream without any encryption/decryption interception.

(Note: Directory mode correctly uses ICryptoService and PageEncryption via Database.Core.cs, so this bug is isolated to the SingleFile storage provider.)

Proposed Fix
I have already implemented a working local patch that:

1. Updates DatabaseFactory.CreateSingleFileDatabase to resolve ICryptoService from DI when EnableEncryption is true.
2. Passes the resolved ICryptoService to SingleFileStorageProvider.Open.
3. Instantiates AesGcmEncryption in the provider and intercepts all read/write paths (WriteBlockAsync, ReadBlockAsync, GetReadSpan, GetReadStream) to encrypt/decrypt block data.
4. Includes a comprehensive regression test suite (SingleFileEncryptionTests.cs) covering plaintext-at-rest, correct-key roundtrip, wrong-key rejection, and file integrity.

I have opened a corresponding Pull Request (fix/singlefile-encryption-bypass) that implements this fix.

[MPCoreDeveloper]

Thank you — your analysis in this report (and in PR #342) was exactly right, including the note that directory mode already encrypts. The single-file gap is now closed on master and ships in v1.9.7.

Root cause confirmed

SingleFileStorageProvider accepted EnableEncryption + EncryptionKey but never instantiated a cipher — WriteBlockAsync queued raw plaintext to the write-behind queue and the WAL, and every read path returned it unmodified. Wrong keys were silently accepted because there was nothing to validate against. Your "Directory mode correctly uses ICryptoService/PageEncryption" note was also correct — the bug was isolated to the single-file provider, and the test suite never caught it because the shared test helper disabled encryption for speed and there were no wrong-key/plaintext-at-rest invariant tests for any storage mode.

What changed (v1.9.7)

• SingleFileStorageProvider now creates AesGcmEncryption from the caller-supplied DatabaseOptions.EncryptionKey and encrypts every block written through WriteBlockAsync (table rows, table directory, column/index definitions) with AES-256-GCM at rest; ReadBlockAsync, GetReadStream and GetReadSpan decrypt. The embedded WAL region receives ciphertext too.
• Wrong key now fails — GCM authentication error or empty schema, so no data is accessible (exactly the behavior your report specified).
• ValidateHeader enforces encryption-mode consistency on reopen, and UpdateBlockAsync/GetWriteStream (positional delta writes, incompatible with GCM) throw NotSupportedException when encrypted.
• Regression coverage (SingleFileEncryptionTests): no plaintext on disk, correct-key round-trip, wrong-key rejection, open-without-encryption rejected. Validation: 1,521/1,521 core + 114/114 EF Core tests.

We implemented this via our own commit rather than merging PR #342 directly, but your patch is a solid reference implementation and the integration points you identified match ours — we'd welcome your review of the merged change.

What remains plaintext — and SharpCoreDB v2.0

Two honest caveats:

1. The file header, block registry and free-space map stay plaintext by design — the registry must be readable to locate ciphertext blocks after a crash. That still leaks table/index block names (table:customers, sys:indexdefs:customers) and layout offsets.
2. Directory mode already encrypted (as you noted) — so this fix brings single-file mode to parity.

Both are addressed on the v2.0 roadmap: a new work package adds block-registry / on-disk metadata encryption (bootstrap super-block pointing at an encrypted registry, HKDF + AES-GCM per-file key wrap, EncryptionFormatVersion for a backward-compatible migration via Vacuum) so only a ~256-byte super-block and the header magic remain plaintext.

Thank you again for the report and the fix proposal — the wrong-key and plaintext-at-rest checks are now permanent regression tests.

— MPCoreDeveloper