Skip to content

fix: handle empty batch from deletion in add_columns_from_stream - #7233

Merged
wjones127 merged 2 commits into
lance-format:mainfrom
zhangyang0418:fix/add-columns-empty-batch-deletion
Jun 16, 2026
Merged

wjones127 merged 2 commits into
lance-format:mainfrom
zhangyang0418:fix/add-columns-empty-batch-deletion

Conversation

@zhangyang0418

Copy link
Copy Markdown
Contributor

What

Fixes a panic in add_columns when a fragment's deleted rows fill an entire read batch.

Closes #7232.

Root cause

In add_columns_from_stream, the updater reads each fragment in physical batches and applies the deletion vector via filter_record_batch. When every row in a physical batch is deleted, the filter yields a 0-row batch. The outer loop then sets rows_remaining = 0, the inner while rows_remaining > 0 loop never runs, batches stays empty, and concat_batches(&batches[0].schema(), ..) panics with index out of bounds: the len is 0 but the index is 0.

This was hit in production using a merge-columns workflow over a dataset whose fragment had a deletion file, where the deleted rows happened to align with a read-batch boundary.

Fix

When the updater yields a 0-row batch, feed an empty batch back to the updater (to keep it in sync) and continue:

if rows_remaining == 0 {
    updater.update(RecordBatch::new_empty(stream.schema())).await?;
    continue;
}

Test

Adds test_add_columns_with_fully_deleted_batch: writes a single fragment with 105 rows, deletes the trailing 5 rows so that — read with batch_size=50 — the last batch [100..105) is fully deleted, then verifies add_columns succeeds and the new column has the correct values. The test panics on the unpatched code and passes with the fix.

When an entire read batch has been deleted (e.g., all rows in a batch
fall within the deletion vector), the updater yields a 0-row batch.
The old code would then skip the inner loop, leaving `batches` empty,
causing `concat_batches(&batches[0]..)` to panic with 'index out of
bounds: the len is 0 but the index is 0'.

This fix detects the empty batch case and feeds an empty batch back to
the updater to keep it in sync, then continues to the next batch.

Adds a regression test that creates a dataset with 105 rows, deletes
the trailing 5 rows (forming a fully-deleted trailing batch when read
with batch_size=50), and verifies add_columns succeeds.

Fixes the panic reported in production when using merge_columns on a
dataset with deletion files where deleted rows cluster at batch
boundaries.
@github-actions github-actions Bot added the bug Something isn't working label Jun 11, 2026

@wjones127 wjones127 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a clean fix, with a nice test. Thank you!

Good to merge once CI is passing.

@wjones127
wjones127 merged commit 1b78086 into lance-format:main Jun 16, 2026
30 checks passed
@codecov

codecov Bot commented Jun 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.81818% with 8 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance/src/dataset/schema_evolution.rs 81.81% 2 Missing and 6 partials ⚠️

📢 Thoughts on this report? Let us know!

Xuanwo added a commit that referenced this pull request Sep 1, 2026
…8612)

## What

Adding or merging a column fails with `NotSupported: Missing too many
rows in merge, run compaction to materialize deletions first` when a
fragment's deleted rows cover its first read batch.

This supersedes #7318, which goes after the same failure. Parameterizing
#7233's test by where the fully deleted batch sits shows the middle and
trailing cases already pass on `main`, for both nullable and
non-nullable new columns. So `add_blanks` rejecting an empty batch is
not a middle-of-fragment problem, and the oversized-batch slicing in
that PR does not change any observable behavior. Only a run starting at
physical row 0 fails.

## Root cause

Every data file in a fragment has to hold the same physical row count,
so the updater restores the deleted rows before writing. `add_blanks`
materializes a placeholder by copying the batch's first row, which means
the batch needs at least one live row.

A deleted run that trails live rows gets greedily appended to the
preceding batch, so it always has one. A run starting at physical row 0
arrives as an empty batch carrying every one of its offsets, and the
copy has nothing to copy from.

## Fix

Defer those blanks rather than invent values for an arbitrary schema:
`DeletionRestorer` remembers how many rows it owes and prepends them to
the next batch that does have a live row. Deleted rows sort before the
live rows that follow them, so the physical row order is unchanged.
Copying a real row also keeps the placeholder valid for a non-nullable
column, which a null placeholder would not.

Legacy files have to reproduce their original row group size, which
deferring would break, so they keep reporting the existing error and
`add_blanks` keeps rejecting an empty batch.

Two hardening changes in `add_blanks` came with it. Offsets that are not
strictly increasing used to underflow `u32` silently, and an offset past
the batch's live rows used to reach `arrow::compute::take` with bounds
checking off. Both now return `Internal` naming the offending offset.

## Tests

- `updater.rs`: six `DeletionRestorer` cases covering the deferral, a
second empty batch carrying the debt through, the offset shift,
`is_exhausted` while blanks are owed, and no double counting of deferred
rows. Plus `add_blanks` rejection cases for non-increasing, equal, and
out-of-range offsets, and a blank landing exactly at the end of a batch.
- `schema_evolution.rs`: `test_add_columns_with_fully_deleted_batch`
parameterized over leading/middle/trailing by nullable/non-nullable, and
a legacy case asserting the error still comes from `add_blanks`.
- `fragment.rs`: a deletion vector naming a row past the end of the
fragment must fail the stream at its end instead of writing a short data
file.
- `test_dataset.py`: `merge_columns` plus `LanceOperation::Merge` over a
dataset with a deletion file, parameterized the same three ways, with
the merged column declared non-nullable.

Co-authored-by: Xuanwo <github@xuanwo.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

panic in add_columns_from_stream: index out of bounds when entire read batch is deleted

2 participants