feat!: align media capabilities and add grounded-query tooling - #135
Merged
Merged
Conversation
Replace the dialogue and videoprism capability identifiers with speech and action across shared contracts, packages, extras, adapters, generated catalogs, benchmarks, and documentation. Bump the index schema so repositories rebuild under the new collection names. BREAKING CHANGE: dialogue and videoprism are removed as capability names and package extras. Use speech and action, then rebuild existing indexes.
tulayha
force-pushed
the
benchmark/codex-mcp-ablation
branch
from
August 29, 2026 19:56
34544b5 to
c34373d
Compare
Mahnoor-Zaffar
added a commit
to Mahnoor-Zaffar/vidxp
that referenced
this pull request
Sep 8, 2026
Upstream renamed the dialogue capability to speech (grayhatdevelopers#135), so the latency benchmark now targets the speech modality and its speech_indexing stage. The speech capability still reports the dialogue_phrases metric, so rate aggregation is unchanged.
tulayha
added a commit
that referenced
this pull request
Sep 14, 2026
…al DiDeMo corpus (#128) * feat(benchmarks): add reproducible indexing-latency benchmark Adds a command that generates synthetic media via FFmpeg testsrc2 and measures per-stage indexing throughput, per-stage wall time, and peak memory across configurable modalities. Supports regression detection against a prior baseline report. - : corpus generation, run orchestrator (drives real run_index/ModelRuntime), per-stage aggregation, baseline comparison - CLI command with --modalities, --videos, --duration-seconds, --resolution, --repetitions, --input-mode (transcript/transcribe), --audio-mode, --baseline, --baseline-tolerance - documents the protocol, output schema, and limitations - 23 unit tests for validation, aggregation, clip command building, baseline comparison, and corpus spec * fix(benchmarks): address CodeRabbit review issues - Reject resolutions with extra components (e.g. 320x180x1) - Accumulate record_counts across repetitions instead of overwriting - Move corpus generation inside try block for proper failure handling - Pass reset parameter through instead of hardcoded True - Validate baseline configuration compatibility before comparison * feat(benchmarks): run indexing-latency benchmark on real DiDeMo media Add a --corpus mode to vidxp benchmark index-latency so the throughput benchmark can index the real DiDeMo videos prepared by vidxp benchmark prepare, instead of only synthetic testsrc2 clips. Named corpora resolve under the application data directory and honor media-overrides.json; arbitrary media directories are supported too. Real corpora have no released transcripts, so dialogue requires --input-mode transcribe and is transcribed by Whisper. Reports record the corpus as kind 'real' with name, clip count, bytes, duration range, and containers, and baseline comparison rejects a baseline whose corpus signature differs so synthetic and real results never mix. * fix(benchmarks): address latency benchmark review feedback Ensure every repetition runs a fresh indexing cycle by forcing reset for repetitions after the first, so --repetitions N measures N full indexing runs instead of one run followed by resume/skip passes. Apply the --audio-mode flite requirement only to generated synthetic media; real corpora take audio from the video and document the flag as ignored, so the DiDeMo dialogue/transcribe command now works without it. Recognize non-named --corpus strings as media directory paths and validate they exist, making the documented --corpus <directory> mode usable from the command line. Add orchestration, real-corpus validation, and directory-resolution tests covering the changed behavior. * fix(benchmarks): align latency benchmark with speech capability rename Upstream renamed the dialogue capability to speech (#135), so the latency benchmark now targets the speech modality and its speech_indexing stage. The speech capability still reports the dialogue_phrases metric, so rate aggregation is unchanged. * fix(benchmarks): reset every measured latency repetition Always reset the benchmark index, including the first repetition when a completed run ID is reused. Keep the reset option accepted for compatibility and clarify its help. Update the existing orchestration assertion. User-visible benchmark correctness fix; not internal-only. Validation: 60 targeted latency and runner tests passed, targeted Ruff passed, CLI help and git diff --check passed. * fix(benchmarks): require explicit reset for existing latency runs Reject an existing run before generating media or opening storage unless reset is requested. New runs index normally and reset between repetitions. Restore the reset option meaning and document the overwrite behavior. User-visible benchmark correction. Validation: 61 targeted latency and runner tests, targeted Ruff, CLI help, and git diff --check passed. Markdown lint was unavailable in the offline npm cache; no links changed. Full suite and model-backed smoke were not run for this focused guard change. --------- Co-authored-by: Talha Amjad <tamjad91@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
speech,scene,action, andsoundcapability names across the shared CLI, HTTP, MCP, Desktop, package, and generated contractsqwen3.5:4b-q4_K_M, while preserving deterministic evidence retrieval when the model is unavailableBreaking change
dialogueandvideoprismare removed as public capability names and package extras. Usespeechandaction, then rebuild existing indexes. No compatibility aliases are retained.Cost and ownership boundary
Promptfoo Community and the local VidXP/Ollama processes have no per-run fee. The approximately 3.4 GB Qwen model uses local disk, memory, compute time, and electricity. Codex benchmark runs consume the user's applicable Codex allowance or API usage; hosted judge calls would be additional paid model usage. Desktop asks before installing Ollama or downloading the model and does not bundle model weights.
Validation
uv run --no-sync ruff check .uv run --no-sync python -m pytest -q: 732 passed, 5 skipped, 106 subtests passeduv run --no-sync pytest -q tests/test_mcp.py tests/test_codex_plugin.py tests/test_cli.py: 99 passed, 7 subtests passednpm --prefix desktop run check: typecheck, lint, production build, and 47 tests passedcargo test --manifest-path desktop/src-tauri/Cargo.toml: 89 passedcargo test --release --locked --manifest-path desktop/src-tauri/Cargo.toml: 89 passednpm run model-catalog:checkfromdesktop/npm run notices:checkfromdesktop/npx --yes markdownlint-cli2@0.23.2 INSTALLATION_GUIDE.md docs/local-api.md docs/architecture/platform.md docs/integrations/openai-plugin.mdgit diff --checknpm --prefix premiere run check: typecheck, lint, 14 tests across 6 files, and both UXP and CEP production builds passeduv run --no-sync python -m pytest -q tests/test_api.py tests/test_query_service.py: 50 passedNo Codex inference runs, Ollama/Qwen model download, paid API call, dataset media download, or real Premiere host run was performed. The Ollama executable is not installed on this workstation, so the real Qwen schema/resource gate remains explicit follow-up validation rather than a claimed end-to-end result. Lychee was unavailable and the Docker daemon was stopped, so the pinned link check could not run. The direct
uv run --no-sync pytest -qWindows shim omitted the checkout root when collecting fiveutils.*modules; the complete suite passed throughuv run --no-sync python -m pytest -q.