Skip to content

feat!: align media capabilities and add grounded-query tooling - #135

Merged
tulayha merged 6 commits into
mainfrom
benchmark/codex-mcp-ablation
Aug 31, 2026
Merged

tulayha merged 6 commits into
mainfrom
benchmark/codex-mcp-ablation

Conversation

@tulayha

@tulayha tulayha commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add FineLAP-backed sound search and use semantic speech, scene, action, and sound capability names across the shared CLI, HTTP, MCP, Desktop, package, and generated contracts
  • add grounded query planning and synthesis with official Ollama qwen3.5:4b-q4_K_M, while preserving deterministic evidence retrieval when the model is unavailable
  • add a Desktop Local grounded answers option that reuses a healthy loopback Ollama service or, with explicit approval, installs Ollama through Winget/Homebrew, starts only a VidXP-owned service when necessary, streams the approved model download, and never stops an externally owned service
  • propagate the private model endpoint and selected model into browser, worker, API, Premiere, copied stdio MCP, and Codex registration without exposing a user-entered URL or forwarding unrelated environment variables
  • wire the Premiere panel to the shared durable grounded-query job, render locally generated claims with numbered evidence, and retain ranked evidence-only results when generation is unavailable
  • add a pinned Promptfoo/Codex SDK harness for matched MCP-on and MCP-off runs, deterministic temporal/tool-boundary scoring, and a ten-task LongVALE part-nine pilot spanning scenes, actions, environmental sound, speech, and joint evidence
  • document the dataset subset and media acquisition boundary, setup/indexing procedure, cost boundary, OVSD scope, model-selection evidence, managed local-answer lifecycle, and current evidence limitations
  • bump the index schema so repositories using removed capability names rebuild against the new collection contract

Breaking change

dialogue and videoprism are removed as public capability names and package extras. Use speech and action, then rebuild existing indexes. No compatibility aliases are retained.

Cost and ownership boundary

Promptfoo Community and the local VidXP/Ollama processes have no per-run fee. The approximately 3.4 GB Qwen model uses local disk, memory, compute time, and electricity. Codex benchmark runs consume the user's applicable Codex allowance or API usage; hosted judge calls would be additional paid model usage. Desktop asks before installing Ollama or downloading the model and does not bundle model weights.

Validation

  • uv run --no-sync ruff check .
  • uv run --no-sync python -m pytest -q: 732 passed, 5 skipped, 106 subtests passed
  • uv run --no-sync pytest -q tests/test_mcp.py tests/test_codex_plugin.py tests/test_cli.py: 99 passed, 7 subtests passed
  • npm --prefix desktop run check: typecheck, lint, production build, and 47 tests passed
  • cargo test --manifest-path desktop/src-tauri/Cargo.toml: 89 passed
  • cargo test --release --locked --manifest-path desktop/src-tauri/Cargo.toml: 89 passed
  • npm run model-catalog:check from desktop/
  • npm run notices:check from desktop/
  • Promptfoo 0.122.2 configuration validation under Node 24.19.0
  • npx --yes markdownlint-cli2@0.23.2 INSTALLATION_GUIDE.md docs/local-api.md docs/architecture/platform.md docs/integrations/openai-plugin.md
  • git diff --check
  • all ten pilot intervals checked against the pinned local LongVALE evaluation annotations
  • npm --prefix premiere run check: typecheck, lint, 14 tests across 6 files, and both UXP and CEP production builds passed
  • uv run --no-sync python -m pytest -q tests/test_api.py tests/test_query_service.py: 50 passed

No Codex inference runs, Ollama/Qwen model download, paid API call, dataset media download, or real Premiere host run was performed. The Ollama executable is not installed on this workstation, so the real Qwen schema/resource gate remains explicit follow-up validation rather than a claimed end-to-end result. Lychee was unavailable and the Docker daemon was stopped, so the pinned link check could not run. The direct uv run --no-sync pytest -q Windows shim omitted the checkout root when collecting five utils.* modules; the complete suite passed through uv run --no-sync python -m pytest -q.

Replace the dialogue and videoprism capability identifiers with speech and action across shared contracts, packages, extras, adapters, generated catalogs, benchmarks, and documentation. Bump the index schema so repositories rebuild under the new collection names.

BREAKING CHANGE: dialogue and videoprism are removed as capability names and package extras. Use speech and action, then rebuild existing indexes.
@tulayha
tulayha force-pushed the benchmark/codex-mcp-ablation branch from 34544b5 to c34373d Compare August 29, 2026 19:56
@tulayha tulayha changed the title feat(benchmarks): add Codex MCP ablation harness feat!: rename media capabilities and add MCP ablation harness Aug 29, 2026
@tulayha tulayha changed the title feat!: rename media capabilities and add MCP ablation harness feat!: align media capabilities and add grounded-query tooling Aug 30, 2026
@tulayha
tulayha merged commit 10237cc into main Aug 31, 2026
14 checks passed
Mahnoor-Zaffar added a commit to Mahnoor-Zaffar/vidxp that referenced this pull request Sep 8, 2026
Upstream renamed the dialogue capability to speech (grayhatdevelopers#135), so the
latency benchmark now targets the speech modality and its speech_indexing
stage. The speech capability still reports the dialogue_phrases metric,
so rate aggregation is unchanged.
tulayha added a commit that referenced this pull request Sep 14, 2026
…al DiDeMo corpus (#128)

* feat(benchmarks): add reproducible indexing-latency benchmark

Adds a  command that generates synthetic
media via FFmpeg testsrc2 and measures per-stage indexing throughput,
per-stage wall time, and peak memory across configurable modalities.
Supports regression detection against a prior baseline report.

- : corpus generation, run orchestrator (drives real
  run_index/ModelRuntime), per-stage aggregation, baseline comparison
- CLI command  with --modalities, --videos,
  --duration-seconds, --resolution, --repetitions, --input-mode
  (transcript/transcribe), --audio-mode, --baseline, --baseline-tolerance
-  documents the protocol, output schema, and limitations
- 23 unit tests for validation, aggregation, clip command building,
  baseline comparison, and corpus spec

* fix(benchmarks): address CodeRabbit review issues

- Reject resolutions with extra components (e.g. 320x180x1)
- Accumulate record_counts across repetitions instead of overwriting
- Move corpus generation inside try block for proper failure handling
- Pass reset parameter through instead of hardcoded True
- Validate baseline configuration compatibility before comparison

* feat(benchmarks): run indexing-latency benchmark on real DiDeMo media

Add a --corpus mode to vidxp benchmark index-latency so the throughput
benchmark can index the real DiDeMo videos prepared by vidxp benchmark
prepare, instead of only synthetic testsrc2 clips. Named corpora resolve
under the application data directory and honor media-overrides.json;
arbitrary media directories are supported too. Real corpora have no
released transcripts, so dialogue requires --input-mode transcribe and is
transcribed by Whisper. Reports record the corpus as kind 'real' with name,
clip count, bytes, duration range, and containers, and baseline comparison
rejects a baseline whose corpus signature differs so synthetic and real
results never mix.

* fix(benchmarks): address latency benchmark review feedback

Ensure every repetition runs a fresh indexing cycle by forcing reset
for repetitions after the first, so --repetitions N measures N full
indexing runs instead of one run followed by resume/skip passes.

Apply the --audio-mode flite requirement only to generated synthetic
media; real corpora take audio from the video and document the flag as
ignored, so the DiDeMo dialogue/transcribe command now works without it.

Recognize non-named --corpus strings as media directory paths and
validate they exist, making the documented --corpus <directory> mode
usable from the command line.

Add orchestration, real-corpus validation, and directory-resolution
tests covering the changed behavior.

* fix(benchmarks): align latency benchmark with speech capability rename

Upstream renamed the dialogue capability to speech (#135), so the
latency benchmark now targets the speech modality and its speech_indexing
stage. The speech capability still reports the dialogue_phrases metric,
so rate aggregation is unchanged.

* fix(benchmarks): reset every measured latency repetition

Always reset the benchmark index, including the first repetition when a completed run ID is reused. Keep the reset option accepted for compatibility and clarify its help. Update the existing orchestration assertion.

User-visible benchmark correctness fix; not internal-only. Validation: 60 targeted latency and runner tests passed, targeted Ruff passed, CLI help and git diff --check passed.

* fix(benchmarks): require explicit reset for existing latency runs

Reject an existing run before generating media or opening storage unless reset is requested. New runs index normally and reset between repetitions. Restore the reset option meaning and document the overwrite behavior.

User-visible benchmark correction. Validation: 61 targeted latency and runner tests, targeted Ruff, CLI help, and git diff --check passed. Markdown lint was unavailable in the offline npm cache; no links changed. Full suite and model-backed smoke were not run for this focused guard change.

---------

Co-authored-by: Talha Amjad <tamjad91@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant