Skip to content

Default embedding: embeddinggemma via ollama (0.3.2) - #13

Merged
dominikpeter merged 1 commit into
mainfrom
embed-codebase-bench
Oct 7, 2026
Merged

dominikpeter merged 1 commit into
mainfrom
embed-codebase-bench

Conversation

@dominikpeter

Copy link
Copy Markdown
Member

Closes #12.

What changes

  • New preset gemma (embeddinggemma through a local ollama, loopback only). A bare --embed now means gemma; the other presets are written --embed=minilm|qwen3|potion|openai|cohere.
  • README: recommends ollama pull embeddinggemma && duplicatecode scan . --embed; MiniLM is no longer described as best.
  • Benchmark tooling for Recommend a default light embedding model: quality and speed across codebases #12: eval/make_codebase_task.py, eval/embed_codebases.py (new ollama:<model> backend), eval/analyze_codebases.py, eval/static_rank/.
  • Version 0.3.2 (merging publishes it).

Why gemma

Re-implementation retrieval over 8 codebases, 240 functions (details in #12): R@1 0.942 vs 0.892 for the static detector alone, and still 0.942 with function names masked. MiniLM-L6 0.854, bge-m3 0.912, Qwen3-0.6B 0.917. About 5 ms per text on a GPU.

Tested

  • cargo test --workspace --locked (52 pass), cargo fmt --check.
  • eval-groups on the holdout, default profile: static 0.7607 -> 0.9071 with --embed (gemma). Per set: fn-python 0.706 -> 0.801, fn-typescript 0.728 -> 0.920, javascript 0.497 -> 0.788, python 0.660 -> 0.948, react 0.980 -> 0.987, sql 0.993 -> 0.9995.

Not covered

Default is unchanged without --embed. Thresholds with embeddings were set for MiniLM; false-positive rate of gemma on a real scan is not measured. The 11 HF-only models were not run.

🤖 Generated with Claude Code

@dominikpeter
dominikpeter merged commit f4ae599 into main Oct 7, 2026
5 checks passed
@dominikpeter
dominikpeter deleted the embed-codebase-bench branch October 7, 2026 12:16
…ed; codebase benchmark; 0.3.2

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Recommend a default light embedding model: quality and speed across codebases

1 participant