v0.1.10-4: Add Unified v1 LMCache Gateway, Tier, Adapter, and Queue Observability

Background

The v1 architecture keeps the high-frequency KV data path between vLLM, LMCacheMPConnector, and LMCache MP. KDN observes and controls LMCache through a versioned Gateway rather than inferring cache state from Redis keys or reimplementing storage maintenance.

Later Proxy queueing, knowledge-aware policy, and network-compute overlap require a trace that separates:

KDN knowledge resolution and policy;
KDN Cache Service Facade operations;
LMCache Gateway capability discovery and requests;
LMCache L1/L2, adapter, tier, prefetch, pin, clear, and operation observations;
Instance-side actual hit tokens and remote reads;
Proxy queueing and vLLM execution;
Legacy compatibility projections.

Objective

Introduce unified request and cache-operation traces for the v1 LMCache-backed path while preserving the existing Legacy path and current routing/forwarding behavior.

Required Stages

When applicable, the schema should represent:

Runtime Profile resolution and freeze;
KDN knowledge lookup and semantic resolution;
Artifact compatibility evaluation;
LMCache Endpoint and Capability Snapshot discovery;
token/artifact lookup request and result;
cache-object/tier/adapter observation;
logical Prefetch/Pin/Unpin/Clear/Rebuild task queue and execution;
LMCache MP HTTP/Coordinator/SDK request timing;
LMCache asynchronous operation status/completion;
L1/L2 or adapter-specific observations where publicly available;
network/remote-read measurements reported by LMCache;
Instance-side LMCache load and actual hit-token confirmation;
CachePlan/FusionPlan construction;
Prefill queue and execution;
Decode execution/completion;
fallback selection and execution;
Legacy scan/dump/restore/inject stages, explicitly marked as Legacy.

Provenance Fields

Add explicit fields for:

resolved runtime_profile;
lmcache_profile_id;
lmcache_endpoint_id and endpoint generation;
Gateway adapter/profile;
storage adapter and tier where available;
predicted_*, observed_*, actual_*, and inferred values;
source and source endpoint;
observation timestamp, freshness/age, and expiry;
request start/end/duration;
queue and execution duration;
token lookup range and matched coverage where available;
actual hit tokens and remote-read counters where available;
stage outcome;
unsupported/incompatible/stale/partial status;
error and fallback reason;
Legacy projection marker.

Implementation Scope

Define stable JSON-serializable request and CacheOperationTask trace models.
Link reusable cache-operation tasks to all waiting requests through IDs.
Distinguish KDN decisions from Gateway invocation, LMCache runtime observation, Proxy inference, and vLLM measurement.
Distinguish L1/L2/tier/adapter observations without copying LMCache's physical chunk index.
Prefer LMCache-native metrics, events, status, and hit-token observations over latency inference or Redis-key existence.
Use monotonic clocks for durations and wall-clock timestamps only for external correlation.
Preserve trace context across current Scheduler, Proxy, KDN, Instance, and Gateway boundaries.
Allow tracing to be disabled or sampled without affecting correctness.
Keep v1 fields optional for text-only and Legacy paths.
New observability features target v1; Legacy receives only the fields needed to preserve regression visibility.

Acceptance Criteria

A request exports a deterministic machine-readable stage timeline.
A shared LMCache-backed operation can be correlated with all waiting requests.
Runtime Profile is explicit and immutable for the trace lifetime.
Predicted, observed, measured, inferred, stale, unsupported, partial, skipped, and Legacy-projected values cannot be confused.
Every physical cache observation identifies source, profile, endpoint generation, and freshness where available.
Token/artifact lookup timing is separate from prefetch/load timing and vLLM compute.
LMCache Gateway request time is separate from LMCache asynchronous operation time.
Actual cache reuse can be confirmed by hit-token or remote-read metrics where the active Profile exposes them.
Missing, unsupported, incompatible, stale, failed, cancelled, and skipped stages are explicit.
Stage durations are non-negative and use consistent units.
Trace collection does not alter routing, cache reuse, injection, forwarding, or fallback decisions.
A v1 trace cannot silently contain a Legacy write stage.
Text-only, Legacy KV, v1 LMCache-backed KV, unknown-future Profile, fallback, and endpoint-restart paths are representable.
Unit tests cover serialization, ordering, shared-task correlation, endpoint generation, stale observations, unsupported capability, failed/skipped stages, and provenance separation.
At least one CPU-only integration test produces an end-to-end timeline using Mock Gateway/Profile data.

Non-Goals

Implementing the production LMCache Gateway I/O.
Implementing ExecutionGraph scheduling.
Maintaining an authoritative Proxy or KDN physical block index.
Reimplementing LMCache metrics, events, tier accounting, or eviction.
Claiming queueing or cache-performance improvement in v0.1.10.
Adding new Legacy functionality.
Requiring an external tracing backend.

Parent Epic

v0.1.10 Epic: KDN Distribution Infrastructure Contracts and Observability Baseline #137

Dependencies

v0.1.10-2: Define v1 Runtime, Artifact, LMCache Observation, Operation Task, and Queue State Models #139 for state/object naming.
v0.1.10-3: Define Versioned KDN Cache Service Facade and LMCache Gateway Contracts #140 for Cache Service, Gateway, Profile, and capability vocabulary.

Research kickoff after #140

PR #154 has merged and the versioned KDN Cache Service / LMCache Gateway vocabulary is now available. This comment records the initial codebase audit for #141 before implementation.

Existing assets we should reuse

kdn_server.contracts.VersionedMessage already provides request_id, optional correlation_id, resolved runtime_profile, and UTC timestamps.
Gateway-targeted contracts already provide compatibility profile, endpoint ID, and endpoint generation.
CacheOperationTask already provides logical task identity, operation type, endpoint/profile provenance, state, attempt count, and lifecycle timestamps.
ProxyTask and QueueManager already collect many useful queue, preparation, forwarding, first-token, decode, prediction, and actual-duration markers.
scripts/validate_v1_kdn_roundtrip.py already contains the validated vLLM/LMCache metric vocabulary for external-prefix queries/hits, requested/hit tokens, L1 reads/writes, L2 prefetch hits, completed loads, and failures.
Proxy already emits a final cacheroute_meta SSE event, so there is an existing compatibility output path for trace data.

Current gaps and correctness risks

ProxyTask.trace is a free-form dictionary whose annotation does not match its actual string/float/bool contents. Field names mix pred_, predict_, actual_, timestamps, durations, state, and path labels without a versioned schema.
Several queue paths use wall-clock time.time() values to derive durations. v0.1.10-4: Add Unified v1 LMCache Gateway, Tier, Adapter, and Queue Observability #141 requires monotonic clocks for duration measurement and wall-clock timestamps only for external correlation.
Some prediction fields are overwritten with measured values after execution. This makes predicted and actual values impossible to distinguish reliably.
Scheduler Request currently carries only integer Request_ID plus Prompt/Service/Task dataclasses. It has no stable trace context, correlation ID, Runtime Profile, LMCache profile, endpoint generation, or typed stage timeline.
Proxy reconstructs only the existing dataclass fields, so additional trace context will be dropped unless propagation is explicitly designed.
CacheOperationTask does not yet link one shared logical operation to all waiting request IDs.
LMCache/vLLM metrics are currently sampled only by a standalone validation script. Aggregate Prometheus deltas cannot automatically be called per-request actual values under concurrency; they must be marked observed/aggregate/inferred unless the active LMCache profile provides request-correlated events or labels.
Current failure/fallback paths are encoded through free-form strings such as kv_inject_failed_fallback_text, rather than stable stage outcomes and fallback reasons.

Proposed model boundary for research

Prefer a new dependency-light, JSON-serializable observability package rather than extending the free-form dictionary as the canonical v1 schema. A candidate location is kdn_server/observability/, reusing the domain and contract vocabulary without importing FastAPI, Redis, vLLM, or LMCache.

Candidate public models:

TraceContext: trace ID, request ID, correlation ID, resolved Runtime Profile, sampling state, creation timestamp.
TraceStageName: stable stage vocabulary spanning runtime resolution, knowledge lookup, artifact compatibility, Gateway discovery/invocation, cache observations, operation queue/execution, Proxy prepare/ready, vLLM prefill/decode, fallback, and explicit Legacy stages.
TraceValueKind: predicted, observed, actual, inferred, and legacy-projected.
TraceStageOutcome: success, unsupported, incompatible, stale, partial, failed, cancelled, skipped.
TraceProvenance: component/source, compatibility profile, endpoint/generation, Gateway adapter, storage adapter/tier, freshness, and Legacy marker.
TraceStage: ordered immutable stage record with wall-clock correlation timestamps plus monotonic duration, outcome, provenance, typed measurements, error, and fallback reason.
RequestTrace: deterministic ordered request timeline with linked cache-operation task IDs.
CacheOperationTrace: logical operation lifecycle with all waiting request IDs and separate Gateway request time versus asynchronous operation time.

Compatibility-first migration direction

Keep current routing, forwarding, queueing, injection, and fallback decisions unchanged.
Preserve the current trace/cacheroute_meta compatibility output initially.
Add an optional typed v1 trace alongside it, then provide an adapter from existing Proxy markers to typed stages.
Do not overwrite predicted values with actual values.
Use time.perf_counter_ns() (or an injected monotonic clock) for durations; use UTC timestamps for cross-process correlation only.
Propagate trace context Scheduler -> Proxy -> KDN/Gateway -> Instance without forwarding private trace fields blindly into the OpenAI/vLLM request body.
Keep tracing disable/sampling behavior orthogonal to correctness.
Treat Legacy as an explicit projection and never emit a v1 Legacy write stage.
Keep physical KV keys, bytes, chunk indexes, credentials, and private LMCache objects out of traces.

Research questions to settle before implementation

Canonical ownership: should a RequestTrace live beside core.Request, or should components append through a separate collector referenced by trace ID?
Cross-process propagation: internal payload field, dedicated internal envelope, or headers? The design must not leak internal trace data into the public vLLM payload.
Append semantics: immutable copy-on-write stages versus a mutable collector with immutable exported snapshots.
Ordering: explicit monotonically increasing sequence plus stable stage IDs, not timestamp sorting alone.
Shared operations: how waiting request IDs attach/detach from one CacheOperationTask without changing scheduling behavior.
Metrics attribution: which LMCache/vLLM observations are truly request-correlated, which are aggregate observations, and which are inferred in isolated tests.
Sampling/disable behavior: minimum context that must still propagate when detailed collection is disabled.
Compatibility output: how long to retain the current free-form trace object and how to version any serialized replacement.

Suggested first implementation slice after research

A contract-only, CPU-only slice should define the observability models, deterministic ordering/serialization, shared-task correlation, monotonic-duration validation, provenance/value-kind separation, Legacy representation, and a Mock Gateway end-to-end timeline. It should not yet instrument production HTTP/LMCache I/O or change Proxy scheduling behavior.

This research is scoped to #141 and builds directly on the merged #139/#140 vocabulary.

The first implementation slice for this umbrella is now tracked by #178.

Scope: establish canonical observability v1 contracts, immutable snapshots, a process-local monotonic collector, and a strict Legacy Proxy projection under src/cacheroute/observability, while reusing canonical RuntimeProfile, OutcomeCode, endpoint/profile, and CacheOperationTask vocabulary.

This slice intentionally excludes production instrumentation, cross-process propagation, Gateway/LMCache I/O, public response-shape changes, and routing/queue behavior changes. Existing PR #156 should be rebased and refactored in place to implement #178.

Refs #137
Refs #157
Refs #159

Phase 4A of this observability umbrella is complete through merged PR #179 (merge commit 5562c223ad8900a5ced473ce10982ca7f827f117) and completed Issue #178.

The merged foundation now provides canonical immutable observability v1 contracts, monotonic clocks, a process-local collector, safe Legacy projection, request/cache-operation correlation, and dependency/wheel governance under src/cacheroute/observability.

This Issue remains open. PR #179 intentionally did not implement cross-process TraceContext propagation, production Scheduler/Proxy/KDN/Gateway/Instance instrumentation, LMCache/vLLM runtime observations, exporters, or public wire-format changes. Those capabilities require separate reviewed implementation slices rather than being inferred as completed by Phase 4A.

Phase 4B is now tracked by #182.

#182 covers the next narrow observability slice only: Scheduler-created internal TraceContext, compatibility-preserving Scheduler-to-Proxy propagation, Proxy-local queue/first-response/decode/completion instrumentation, and immutable RequestTrace finalization.

It intentionally stops at the Proxy-to-Instance boundary and does not close #141. KDN, Gateway, Instance, LMCache, vLLM, external export, persistence, and client-visible trace output remain future work.

docs: establish the CacheRoute developer and maintenance handbook

Phase 4B completion and Phase 4C tracking

Phase 4B is complete through closed Issue #182 and merged PR #183.

Merge commit: 3ff12ce323f882b7ed3aba751aac252cf04b0029

The merged Phase 4B implementation now provides:

Scheduler-created internal TraceContext;
canonical reserved-header overwrite and deterministic sampling;
Scheduler-to-Proxy propagation;
Proxy validation and safe local fallback;
Proxy-local prepare queue, ready queue, first-response, stream/decode, and completion stages;
immutable process-local Proxy RequestTrace finalization;
unchanged client-visible request, response, SSE, metadata, routing, injection, fallback, retry, and timeout behavior.

PR #184 has also merged and established the developer handbook and same-PR documentation governance.

The next reviewed slice is now tracked by #185:

Proxy-to-Instance TraceContext propagation and Instance-observed downstream transport stages.

#185 intentionally remains limited to Instance-observed HTTP/mock boundaries. It must not fabricate authoritative vLLM prefill/decode timing, forward trace headers to vLLM, instrument LMCache, expose traces to clients, or close this umbrella Issue.

After #185, KDN/Gateway/LMCache/vLLM-native observations and any export/correlation surface still require separate reviewed slices. Issue #141 remains open.

Navigation Menu

Repository navigation

v0.1.10-4: Add Unified v1 LMCache Gateway, Tier, Adapter, and Queue Observability #141

Background

Objective

Required Stages

Provenance Fields

Implementation Scope

Acceptance Criteria

Non-Goals

Parent Epic

Dependencies

Research kickoff after #140

Existing assets we should reuse

Current gaps and correctness risks

Proposed model boundary for research

Compatibility-first migration direction

Research questions to settle before implementation

Suggested first implementation slice after research

Phase 4B completion and Phase 4C tracking

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

v0.1.10-4: Add Unified v1 LMCache Gateway, Tier, Adapter, and Queue Observability #141

Description

Background

Objective

Required Stages

Provenance Fields

Implementation Scope

Acceptance Criteria

Non-Goals

Parent Epic

Dependencies

Activity

rickisba commented on Jul 29, 2026

Research kickoff after #140

Existing assets we should reuse

Current gaps and correctness risks

Proposed model boundary for research

Compatibility-first migration direction

Research questions to settle before implementation

Suggested first implementation slice after research

rickisba commented on Aug 3, 2026

rickisba commented on Aug 4, 2026

rickisba commented on Aug 4, 2026

rickisba commented on Aug 5, 2026

Phase 4B completion and Phase 4C tracking

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions