feat(index): integrate bulk planning and indexing - #166
Merged
Merged
Conversation
`vidxp index create` handled one media item per invocation, so indexing a repository meant driving it once per video and tracking the results by hand. Add a transport-neutral planning operation and a thin CLI adapter over it: - `Application.plan_bulk_index` resolves a selection to per-media targets and decides which ones the active snapshot already covers. The plan is read-only, so callers can show it before committing to any work. - `vidxp index bulk` indexes every registered media item, or a selection passed with repeated `--media-id`. `--plan-only` shows the decision without indexing, and `--reindex` plans covered media anyway. Media is skipped when the active snapshot holds a generation for it, that generation covers every requested modality, and its recorded input checksum still matches the registered media. Replacing a video's content or asking for a modality the generation lacks therefore plans it again. Media that is not in the ready state is reported as skipped rather than silently dropped. No new indexing behavior. Each pending target is submitted through the existing `submit_index` durable job, one job per media item, matching how `IngestionCoordinator` already sequences ingestion. That is what gives the batch its guarantees: a failure isolates to its own media, earlier successes stay committed, and rerunning the command retries only what is still missing because completed media is then skipped. The command exits non-zero when any media failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tulayha
marked this pull request as ready for review
September 14, 2026 11:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issue
Closes #68 when ready to merge. Integrates #157 and #158; both were retargeted and merged into this branch with their original commits preserved.
Summary
Add one
vidxp index bulkcommand for explicit media IDs or--all, with--plan-only, indexing options, per-video and overall progress, JSON summaries, and detached submission. This is user-visible behavior; no storage schema change or automatic index migration.The integration keeps @Hasnain2430's typed planning, preview, checksum checks and skip reasons, and @aimerscrypto's reusable runner, job submission, option forwarding and progress callbacks. Planning is available on the model-free control plane. Single indexing and bulk planning use the same validated configuration construction. A saved generation is skipped only when its content checksum and configuration fingerprint match; failed submitted jobs retain their IDs for retry. Each video still uses an ordinary durable indexing job.
The existing rule that an active repository uses one indexing profile remains in force. This PR does not silently clear old results or migrate a multi-video repository to different settings. Incompatible jobs report the existing error.
Validation
PYTHONPATH=src /Users/grayhat/Developer/vidxp/.venv/bin/python -m pytest -q tests/test_bulk_index.py tests/test_bulk_indexing.py— 32 passed, 3 subtests passed.PYTHONPATH=src /Users/grayhat/Developer/vidxp/.venv/bin/python -m pytest -q tests/test_bulk_index.py tests/test_bulk_indexing.py tests/test_application.py tests/test_control_plane.py tests/test_job_contracts.py tests/test_cli.py tests/test_local_snapshots.py— 148 passed, 7 subtests passed, 2 failed. Both failures are UI tests that require the absent Streamlit dependency; reproduced both failures on the unchanged checkout. No bulk/application/planning failures./Users/grayhat/Developer/vidxp/.venv/bin/ruff check .— passed.PYTHONPATH=src /Users/grayhat/Developer/vidxp/.venv/bin/python -m vidxp.cli index bulk --help— passed./opt/homebrew/bin/git diff --check— passed./private/tmp/bulk-real-smoke.py: generated two H.264 videos with FFmpeg, imported them into an isolated repository, and indexed both using the cached SigLIP2 model through real durable worker jobs. Repeating the plan skipped both; changing sampling scheduled both again. Passed./private/tmp/bulk-real-retry.py: temporarily withheld one imported video's stored content in the isolated repository. Its submitted job failed and retained its ID, while the next video indexed successfully. Restored the content, verified the plan selected only the failed item, retried its job independently, and searched all four indexed videos. Passed. No model downloads or changes to the user's media/indexes.npx --yes markdownlint-cli2@0.23.2 INSTALLATION_GUIDE.md— passed. Lychee is unavailable locally; local-link checking and the full validation suite remain for CI. No application builds or full suite run locally.Real indexing, skip behavior, failure isolation, independent retry and search have been exercised. Required CI must pass before merging. Transformers emitted existing model-config warnings about BOS/EOS token IDs while the real search completed successfully.
Contributor credit
Both original contributor commits are retained. If squash merging, retain both coauthor trailers in the final commit:
Co-authored-by: umair i250604@isb.nu.edu.pk
Co-authored-by: Hasnain Ibrar Hasnain2430@gmail.com