Repository navigation
feat: add Application Insights instrumentation and worker observability metrics - #1226
Cedric Vidal (cedricvidal) wants to merge 19 commits into
Conversation
…ty metrics Add telemetry module to packages/shared with initTelemetry(), trackMetric(), trackTrace(), and trackEvent() helpers using applicationinsights v3. The module gracefully no-ops when APPLICATIONINSIGHTS_CONNECTION_STRING is not set. Worker instrumentation (coder-acp-copilot and coder-acp-copilot-windows): - worker.run_duration_ms: total run duration - worker.first_ai_call_ms: time from run start to first AI turn - worker.subprocess_idle_s: seconds since last ACP protocol event - worker.cold_start_ms: time from process start to first run - worker.run_started: custom event emitted at run start - Subprocess logs forwarded to trackTrace() with runId correlation API instrumentation: - initTelemetry() called at startup for auto HTTP/dependency tracking Deployment: - ExternalSecret for appinsights-connection-string from Key Vault - worker-secrets updated to include APPLICATIONINSIGHTS_CONNECTION_STRING - API envFrom includes appinsights-secrets (optional) Closes #1223 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Fixes unbounded memory allocation vulnerability in W3C Baggage propagation (affects versions < 2.8.0, pulled in transitively by applicationinsights). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…sue-1223-appinsights-instrumentation
…/monitor-opentelemetry - Replace applicationinsights v3 (maintenance-mode) with modern OTel distro - Use native OpenTelemetry Meter API for custom metrics - Add TELEMETRY_SAMPLING_RATIO and TELEMETRY_LOG_LEVEL env vars - Fix cold start measurement (use process.uptime) - Fix first AI call detection (use structured protocol event check) - Add periodic subprocess idle monitoring during active runs - Add unit tests for telemetry module - Add resetTelemetry() for test isolation - Remove @opentelemetry/core override (no longer needed) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- New docs/architecture/observability.md covering telemetry data flow, transport protocols (HTTPS/Breeze, not OTLP), metrics reference, configuration, and secret delivery - Link doc in AGENTS.md documentation table Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Move telemetry module from packages/shared/src/telemetry/ into its own packages/telemetry/ workspace package. This decouples telemetry from the shared package, keeping shared focused on core types and service clients. - New packages/telemetry/ with @azure/monitor-opentelemetry + @opentelemetry/api - Remove telemetry deps from packages/shared/package.json - Update imports in api, coder-acp-copilot, coder-acp-copilot-windows - Update Dockerfiles (api, copilot, windows) to include telemetry package - Update pnpm-workspace.yaml and observability docs Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Extend telemetry to three more services: coder-acp-claude-code: - worker.run_duration_ms, first_ai_call_ms, cold_start_ms, subprocess_idle_s (30s interval), run_started event - Subprocess log forwarding via trackTrace (Verbose, level-gated) coder-vscode-electron-driver-ext: - worker.run_duration_ms, cold_start_ms, run_started event - No subprocess idle monitor (Playwright interaction model) judge: - judge.evaluation_duration_ms, blob_download_ms, criteria_count - judge.evaluation_started/completed events with requestId All three: telemetry package dep, Dockerfile updates, initTelemetry() at startup. Graceful no-op when connection string is absent. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…essor, report-generator, and model scanners Extend Application Insights telemetry to all remaining containerized services using the same packages/telemetry module and graceful no-op pattern. Services instrumented: - scheduler: dispatch cycle timing, requests dispatched counts, reaper sweep metrics, cold start - token-manager: validation cycle timing, tokens validated/invalidated counts, cold start (Express auto-instrumentation for routes) - post-processor: processing duration per handler type, cold start, processing started/completed events - report-generator: generation duration, LLM session duration, cold start, generation started/completed events - model-scanners (copilot + anthropic): scan duration, token acquisition time, models found, scan completed events. Short-lived jobs call shutdownTelemetry() to flush before exit. Each service gets: - telemetry workspace dependency in package.json - initTelemetry() call at entry point - Custom metrics at key timing points - Dockerfile updated to include packages/telemetry in builder + prod Updated observability architecture doc with all new service metrics. All 2516 tests pass, all services build cleanly. Refs: #1223 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Kept main's updated DEVPROXY_HAR_OUTPUT_DIR description (gateway backend clarification) alongside our Observability section. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Resolve conflicts in AGENTS.md (keep both observability and kubedock docs) - Resolve conflicts in coder-acp-copilot/src/index.ts (add KubedockClient + telemetry imports) - Resolve conflicts in coder-acp-claude-code/src/index.ts (add KubedockClient + telemetry + coldStartTracked) - Add ssr.resolve.conditions to vitest.integration.config.ts so integration tests can resolve the telemetry workspace package via the 'source' condition Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Refactor the telemetry package to support two export modes: - Collector mode: OTLP HTTP to an in-cluster OTel Collector when OTEL_COLLECTOR_ENDPOINT is set (collector forwards to App Insights) - Direct mode: useAzureMonitor() to App Insights when only APPLICATIONINSIGHTS_CONNECTION_STRING is set (current behavior) Collector mode takes priority when both env vars are set. Add OTel Collector Kustomize Component (deploy/components/otel-collector/): - otel-collector-contrib:0.115.0 Deployment + Service - OTLP HTTP receiver → memory_limiter → batch → azuremonitor exporter - JSON patch injects OTEL_COLLECTOR_ENDPOINT into all Deployments - Enabled in integration overlay Update observability docs and ENV_VARIABLES.md with dual-mode architecture, collector pipeline details, and new env var. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8cafaad3-0a65-4b98-9966-9e1509ba6b0e
Add @opentelemetry/auto-instrumentations-node to the NodeSDK config so HTTP, Express, MongoDB, and other modules are automatically instrumented for trace generation in collector mode. Key changes: - Add getNodeAutoInstrumentations() to initWithCollector() with noisy modules (fs, dns, net) disabled to keep span volume manageable - Create telemetry/register entry point for use with node --import, ensuring instrumentation hooks register before app modules load - Update all production Dockerfiles: node --import telemetry/register - Add OTEL_SERVICE_NAME env var to non-worker K8s Deployments (workers already have WORKER_NAME which register.ts falls back to) - Add unit test verifying auto-instrumentations are configured The --import flag ensures OpenTelemetry's import-in-the-middle hooks patch ESM modules (express, mongodb, http) at load time. Without this, the NodeSDK starts too late and no auto-spans are produced. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8cafaad3-0a65-4b98-9966-9e1509ba6b0e
1. Fix test TS2556/TS2352 — add variadic args type to mock function so .mock.calls[0][0] is properly typed. 2. Fix OTel Collector image: upstream otel-collector-contrib images (both ghcr and Docker Hub) are dynamically linked without glibc in their base, causing 'exec: no such file or directory' on AKS. Solution: add a Dockerfile that copies the binary from the upstream contrib image onto gcr.io/distroless/cc-debian12:nonroot (has glibc). CI builds and pushes this as scoped/otel-collector to our ACR. 3. Add otel-collector to CI build matrix (detect-changes + image job). 4. Update integration overlay images.yaml with collector image reference. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8cafaad3-0a65-4b98-9966-9e1509ba6b0e
The 0.116.0 build was a one-off upstream regression (dynamically linked binary on scratch base). All adjacent versions (0.114, 0.115, 0.117+) are statically linked and work on plain scratch. Pin to 0.117.0 and remove: - Custom wrapper Dockerfile (no longer needed) - CI matrix entry for building scoped/otel-collector - Integration overlay image setter override The official Docker Hub image pulls directly — validated in dev with 607/607 spans and 745/745 metrics exported successfully. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8cafaad3-0a65-4b98-9966-9e1509ba6b0e
…sue-1223-appinsights-instrumentation # Conflicts: # apps/api/src/index.ts # pnpm-lock.yaml
Adds a 'telemetry' feature flag that allows team members to disable
all telemetry emission via the Portal Admin UI or API without requiring
infrastructure changes or code deploys.
Implementation:
- Seed 'telemetry' flag (default: enabled) in API startup
- Add checkTelemetryFlag() helper that queries the feature flags API
with a 2s timeout, defaulting to enabled on any failure
- register.ts checks the flag before calling initTelemetry()
- Inject SCOPE_API_URL into all Deployments via the otel-collector
Kustomize Component patch (same pattern as OTEL_COLLECTOR_ENDPOINT)
Toggle via API:
curl -X PUT <api>/api/v1/feature-flags/telemetry -d '{"enabled": false}'
Then rolling-restart pods to pick up the change.
Safety: defaults to enabled on network error, timeout, or if the flag
doesn't exist yet (first deploy). Telemetry never blocks service startup.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8cafaad3-0a65-4b98-9966-9e1509ba6b0e
Align with existing feature gate pattern — telemetry is disabled by
default and must be explicitly enabled via:
PUT /api/v1/feature-flags/telemetry {"enabled": true}
On any failure (no API URL, network error, timeout, flag not found),
the check now returns false (no telemetry) instead of true.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8cafaad3-0a65-4b98-9966-9e1509ba6b0e
Replace startup-only flag check with a runtime-polled approach: - Always initialize OTel (auto-instrumentation hooks must register before app modules load — this is an ESM requirement) - GatingSpanProcessor wraps BatchSpanProcessor, forwarding spans only when the feature flag is enabled - TelemetryFlagPoller checks the API every 60s (unref'd timer, won't keep the process alive) - Toggling the flag takes effect within 60 seconds, no pod restart This follows OTel best practice: control emission/export at the processor level rather than starting/stopping the SDK lifecycle. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8cafaad3-0a65-4b98-9966-9e1509ba6b0e
Replace the runtime feature flag (GatingSpanProcessor + TelemetryFlagPoller) with a simple OTEL_ENABLED env var, matching the SCOPE_AUTH_ENABLED pattern. Changes: - register.ts checks OTEL_ENABLED env var (default: unset/false) - otel-collector Kustomize Component injects OTEL_ENABLED=true into all Deployments (same patch as OTEL_COLLECTOR_ENDPOINT) - Removed: feature-flag.ts, gating-span-processor.ts, TelemetryFlagPoller, GatingSpanProcessor, SCOPE_API_URL injection, telemetry feature flag seed - telemetry-client.ts reverted to plain BatchSpanProcessor Toggle mechanism: - Enable: include otel-collector component in overlay (injects both OTEL_COLLECTOR_ENDPOINT and OTEL_ENABLED=true) - Disable: remove component from overlay, rollout restart - Requires pod restart (same as auth toggle), no API dependency Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8cafaad3-0a65-4b98-9966-9e1509ba6b0e
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2113
🎉 All tests passed! |
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2114
🎉 All tests passed! |
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2165
🎉 All tests passed! |
|
Original author: Josh Duffney (@duffney) Observability Review — Improvement SuggestionsGreat foundation here — clean abstraction, graceful no-op, and meaningful metrics. A few things to tighten up before merging: Issues to Address
Important 🏗️ Migrate to the Modern OpenTelemetry DistroMicrosoft recommends The modern pattern for Node.js + AKS + App Insights: // Top of entrypoint — before all other imports
import { useAzureMonitor, shutdownAzureMonitor } from "@azure/monitor-opentelemetry";
useAzureMonitor({
azureMonitorExporterOptions: {
connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING,
},
samplingRatio: 0.5, // adjust per environment
enableLiveMetrics: true,
});
// Graceful shutdown
process.on("SIGTERM", async () => {
await shutdownAzureMonitor();
process.exit(0);
});This gives you auto-instrumented HTTP, dependencies, exceptions, and native OpenTelemetry spans/metrics APIs for custom instrumentation — without the maintenance-mode SDK. For ESM projects, add |
|
Original author: Josh Duffney (@duffney) Does this support opentelemitry for AI? |
|
Original author: Josh Duffney (@duffney) Add a diagram for the flow of data. |
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2167
🎉 All tests passed! |
|
Original author: Josh Duffney (@duffney) OpenTelemetry for AI — Research SummaryYes, What's Available
Relevant gen_ai Attributes for Scope Workers
How It Would WorkSince our workers don't call OpenAI directly (they spawn a subprocess that does), we can't use auto-instrumentation packages. Instead:
import { trace } from "@opentelemetry/api";
const tracer = trace.getTracer("scope-workers");
const span = tracer.startSpan("gen_ai.chat", {
attributes: {
"gen_ai.system": "openai",
"gen_ai.agent.name": workerType,
"gen_ai.request.model": model,
"gen_ai.usage.input_tokens": tokenUsage.input,
"gen_ai.usage.output_tokens": tokenUsage.output,
"gen_ai.conversation.id": runId,
}
});
span.end();
RecommendationNot for this PR — but a natural follow-up after the HAR parser already extracts References |
|
Original author: Josh Duffney (@duffney) What are the transport protocols of the arrows in the diagram. |
|
Original author: Josh Duffney (@duffney) move the telem modual from the shared into it's own package. |
|
Original author: Josh Duffney (@duffney) Cosider having a node-set collector? it's a way to locally and then send to the exporter or centreal log storage. |
|
Original author: Josh Duffney (@duffney) Consider insumenting the other micro services, gateway, schdular, token manager model scanners. Each microservice should have a toggle to turn off telem. |
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2190
🎉 All tests passed! |
|
Original author: Josh Duffney (@duffney) @cedricvidal_microsoft I dug into the OTel collector which adds a side-car for collecting and shipping the logs to free up the worker. The trade off is slower metric gathering. Is implementing the collection something you'd like to have implemented before this PR merges? IMO we won't need it until Prod is getting several hundred concurrent runs at all times. Lmk what you think. |
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2219
🎉 All tests passed! |
|
Original author: Josh Duffney (@duffney) Requested to implement otel collector side-car before merging. |
|
Original author: Cedric Vidal (@cedricvidal) Let's add a node level collector |
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2233
🎉 All tests passed! |
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2239
🎉 All tests passed! |
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2240
🎉 All tests passed! |
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2242
🎉 All tests passed! |
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2265
🎉 All tests passed! |
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2267
🎉 All tests passed! |
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2268
🎉 All tests passed! |
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2269
🎉 All tests passed! |
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2291
🎉 All tests passed! |
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2292
🎉 All tests passed! |
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2293
🎉 All tests passed! |
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #2294
🎉 All tests passed! |
|
Original author: Cedric Vidal (@cedricvidal) Collector runs as a single replica — worth avoiding a SPOF for the telemetry tierThe OTel Collector is deployed as a I don't think this is a throughput concern at current volume — it's more an availability one. A single replica has no failover story for the usual pod-lifecycle events: a node drain / cluster upgrade / eviction, a bad rollout, or an OOM (the The good news: the collector looks stateless, so HA should be cheapIf I'm reading the pipeline correctly, this collector is effectively stateless:
If that holds, HA is close to free: the ClusterIP Service already load-balances across pods, and because it's OTLP over HTTP ( Roughly: spec:
replicas: 3 # was 1
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: otel-collector, namespace: scoped }
spec:
minAvailable: 1
selector: { matchLabels: { app: otel-collector } }
# + optional topologySpreadConstraints across kubernetes.io/hostnameTwo ways to resolve itOption A — HA at the gateway tier (my lean, pending your check): the diff above, contained to the collector component. Service, endpoint injection, and app wiring all stay identical. Given the statelessness, this looks like the lowest-risk path. Option B — node-level tier (DaemonSet agent): also valid, but it's a larger change for no benefit we're currently using. It would mean swapping the static endpoint for a downward-API node-IP construction (or Mapping to this PRFor quick reference, how this setup lines up against the usual guidance:
Suggestion: I'd lean toward at least 2–3 replicas + a PDB before this ships, but since the "HA is safe here" claim rests on the collector being stateless, it's worth you confirming there's no trace-affinity processor in the pipeline I've overlooked. Happy to be corrected if the topology was a deliberate choice. |
|
Original author: Josh Duffney (@duffney) Thanks for the thorough analysis Cedric — you're spot on across the board. Confirming: no trace-affinity processors. The pipeline is purely Implemented Option A in commit d508e2d4:
No app code or endpoint changes needed — ClusterIP Service already load-balances, and OTLP over HTTP (discrete requests per export) avoids the gRPC long-connection pinning issue you mentioned. Agreed that DaemonSet (Option B) is over-engineering for our current relay-only use case + the Windows node pool complication you identified. Worth revisiting only if we add node-level collection needs later. |
|
Original author: Cedric Vidal (@cedricvidal) LGTM |
Original author: Josh Duffney (@duffney)
Summary
Adds application-level telemetry via Azure Application Insights using the modern
@azure/monitor-opentelemetryOpenTelemetry distro with an optional in-cluster OTel Collector gateway. Enables alerting on worker issues like stale runs and subprocess hangs (motivated by #1179).Closes #1223
Architecture
The telemetry package (
packages/telemetry/) supports dual-mode export:OTEL_COLLECTOR_ENDPOINTis set, sends OTLP HTTP to an in-cluster OTel Collector which forwards to App Insights via theazuremonitorexporter.APPLICATIONINSIGHTS_CONNECTION_STRINGis set, exports directly to App Insights viauseAzureMonitor().graph LR subgraph "App Pods (11 services)" S[Services] end subgraph "OTel Collector Gateway" R["OTLP HTTP :4318"] P["memory_limiter → batch"] E["azuremonitor exporter"] end subgraph Azure AI[App Insights] LA[Log Analytics] G[Grafana] end S -->|"OTLP HTTP"| R R --> P --> E E -->|"HTTPS POST<br/>Breeze /v2.1/track"| AI AI --> LA --> G S -.->|"Direct mode<br/>(fallback)"| AIWhat's Instrumented
scope-apicoder-acp-copilotcoder-acp-copilot-windowscoder-acp-claude-codecoder-vscode-electronscope-judgescope-schedulerscope-token-managerscope-post-processorscope-report-generatormodel-scanners(copilot + anthropic)OTel Collector Component
Deployed as a Kustomize Component at
deploy/components/otel-collector/:otel-collector-contrib:0.156.0with OTLP HTTP receiver → memory_limiter → batch → azuremonitor exporterOTEL_COLLECTOR_ENDPOINTandOTEL_ENABLEDenv varsKey Design Decisions
packages/telemetry/package — decoupled fromshared, minimal depskubedockpattern, overlays opt inotel-collector-contrib— required for theazuremonitorexporter--import telemetry/register— ensures OTel hooks register before app modules load (ESM requirement)OTEL_ENABLEDenv var toggle — same pattern asSCOPE_AUTH_ENABLED, controlled per overlayOperational Toggle
Telemetry is controlled by the
OTEL_ENABLEDenvironment variable, following the same pattern asSCOPE_AUTH_ENABLED. The otel-collector Kustomize Component injects it automatically — including the component enables telemetry, removing it disables it.Enable telemetry
Include the otel-collector component in the overlay's
kustomization.yaml:This injects two env vars into all Deployments:
OTEL_COLLECTOR_ENDPOINT— OTLP HTTP endpoint for the in-cluster collectorOTEL_ENABLED=true— activates telemetry in thetelemetry/registerentry pointThen deploy and rollout restart:
Disable telemetry
Remove the
../../components/otel-collectorline from the overlay, deploy, and restart. WithoutOTEL_ENABLED,register.tsskips initialization entirely (no-op).Override per overlay
To keep the collector running but disable telemetry in a specific overlay (e.g., for debugging the collector itself), add a patch:
Decision Matrix
OTEL_ENABLEDtrue(auto)false(override)Configuration
OTEL_ENABLED"true"to enable telemetry (injected by collector component)OTEL_COLLECTOR_ENDPOINTAPPLICATIONINSIGHTS_CONNECTION_STRINGOTEL_SERVICE_NAMETELEMETRY_SAMPLING_RATIO1.0TELEMETRY_LOG_LEVELWarningFiles Changed
New
packages/telemetry/— standalone telemetry package (init, helpers, types, register, tests)deploy/components/otel-collector/— Kustomize Component (Deployment, PDB, Service, ConfigMap)docs/architecture/observability.md— full architecture referenceModified
deploy/base/external-secret.yaml—appinsights-secretsdeploy/base/workers/worker-secrets.yaml— connection string env vardeploy/base/api.yaml— secretRef + OTEL_SERVICE_NAMEdeploy/base/judge.yaml,scheduler.yaml,token-manager.yaml— OTEL_SERVICE_NAMEdeploy/base/workers/model-scanner-*.yaml— OTEL_SERVICE_NAMEdeploy/overlays/integration/kustomization.yaml— enable otel-collector componentENV_VARIABLES.md— new env var docspnpm-workspace.yaml— add telemetry package