feat(realtime): unify Arrow storage and support PK deletion vectors

[lxy-9602]

Purpose

Linked issue: #158

This change unifies real-time storage and schema handling for append-only and primary-key tables, adds PK + deletion-vector real-time reads, and supports querying an explicit snapshot.
The main changes are:

• Introduce RealtimeSchemaLayout for user input, store write, commit, and query schemas.
• Use ArrowRealtimeStore for both append-only and primary-key tables.
• Support real-time scans with an explicit scan.snapshot-id. Snapshots older than the context’s durable progress are rejected because the required memory range may already have been reclaimed.
• Support PK + deletion-vector reads by:
  • pushing value predicates into level > 0 files;
  • reading level-0 files and memory without predicate pushdown;
  • merging high-level, level-0, and memory readers in the final merge-on-read.

Writer lifecycle and recovery

• PrepareCommitWithProgress() seals current real-time data and returns its commit messages together with offset ranges.
• Before writer handoff or shutdown, all accepted data must be covered by a successful prepare. Closing with building or sealed-but-unprepared data returns an error and invalidates the RealtimeContext.
• Recovery recreates both RealtimeContext and FileStoreWrite from the durable offset stored in the committed snapshot.
• A Write, Seal, or PrepareCommitWithProgress failure must be treated as terminal by upstream: discard the writer and context, then replay from the previous durable checkpoint. This follows the Java-style fail-and-recreate recovery contract instead of retrying partially mutated writer state.

Note: The previous implementation allowed a writer containing only an unsealed building segment to be closed while keeping its context reusable. This can be unsafe because a replacement writer cannot enumerate or prepare that segment if there is no more data write to that bucket. This change intentionally tightens the lifecycle validation to avoid silently losing commit progress.

Tests

Coverage includes explicit snapshot scans, PK + deletion-vector merge-on-read, level-aware predicate pushdown, multiple level-0 runs, writer close validation, context recovery, and pinned read-view lifetime.

API and Format

API impact:

• Adds FileStoreWrite::Seal().
• Real-time scans now support the existing scan.snapshot-id option; no new scan API is introduced.
• RealtimeCommitProgress may represent one or more sealed segments.
• FileStoreWrite::Close() now returns an error and invalidates its RealtimeContext when building or sealed data has not been covered by a successful PrepareCommitWithProgress().
• After a write, seal, or prepare failure, upstream must discard the writer and context and recover from the durable snapshot offset.
• Normalizes the primary-key RealtimeStore write schema to:
  [_SEQUENCE_NUMBER, _VALUE_KIND, _REALTIME_OFFSET, user fields].

Documentation

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)