Skip to content

feat(ai): adaptive embedding batch sizing circuit breaker and timeout fallback #117

Description

@dan-petty

Problem Statement

Distributed OpenTelemetry traces indicate ai.rag.ollama_embed_batch repeatedly encounters 15.0s ReadTimeout: timed out exceptions (75s+ total latency) during dense repository indexing, stalling RAG operations on overloaded local Ollama nodes.

Proposed Changes

  • Dynamic batch partitioning in devops_cli.ai.rag.embeddings adapting batch size dynamically (halving from 32 down to 16, 8, 4 on latency degradation or timeout).
  • Exponential backoff with jitter on transient network timeouts.
  • Pre-flight Valkey SHA-256 chunk cache checks before remote embedding dispatch.
  • Automatic fallback to single-chunk requests when batches fail.

Acceptance Criteria

  • Zero uncaught ReadTimeout exceptions during full workspace re-indexing.
  • Average batch latency reduced below 2.0s.
  • 100% unit test coverage of batch adaptation logic in tests/test_rag_embeddings.py.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority/p1-highHigh impact work required for upcoming release milestonescope/aiAI review pipeline, personas, prompts, LLM client, or RAGstatus/triageNew issue or PR awaiting technical triage and scopingtype/featureNew user-facing functionality or capability addition

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions