Skip to content

feat: add Application Insights instrumentation and worker observability metrics - #1226

Closed
Cedric Vidal (cedricvidal) wants to merge 19 commits into
mainfrom
jduffney-microsoft-issue-1223-appinsights-instrumentation
Closed

Cedric Vidal (cedricvidal) wants to merge 19 commits into
mainfrom
jduffney-microsoft-issue-1223-appinsights-instrumentation

Conversation

@cedricvidal

Copy link
Copy Markdown
Collaborator

Original author: Josh Duffney (@duffney)

Summary

Adds application-level telemetry via Azure Application Insights using the modern @azure/monitor-opentelemetry OpenTelemetry distro with an optional in-cluster OTel Collector gateway. Enables alerting on worker issues like stale runs and subprocess hangs (motivated by #1179).

Closes #1223

Architecture

The telemetry package (packages/telemetry/) supports dual-mode export:

  1. Collector mode — when OTEL_COLLECTOR_ENDPOINT is set, sends OTLP HTTP to an in-cluster OTel Collector which forwards to App Insights via the azuremonitor exporter.
  2. Direct mode — when only APPLICATIONINSIGHTS_CONNECTION_STRING is set, exports directly to App Insights via useAzureMonitor().
  3. No-op — when neither is set, all helpers are zero-cost no-ops.
graph LR
    subgraph "App Pods (11 services)"
        S[Services]
    end
    subgraph "OTel Collector Gateway"
        R["OTLP HTTP :4318"]
        P["memory_limiter → batch"]
        E["azuremonitor exporter"]
    end
    subgraph Azure
        AI[App Insights]
        LA[Log Analytics]
        G[Grafana]
    end

    S -->|"OTLP HTTP"| R
    R --> P --> E
    E -->|"HTTPS POST<br/>Breeze /v2.1/track"| AI
    AI --> LA --> G

    S -.->|"Direct mode<br/>(fallback)"| AI
Loading

What's Instrumented

Service Custom Metrics
scope-api Auto-instrumentation only (HTTP, MongoDB, Redis)
coder-acp-copilot run_duration, first_ai_call, cold_start, subprocess_idle, run_started
coder-acp-copilot-windows Same as copilot
coder-acp-claude-code Same as copilot
coder-vscode-electron run_duration, cold_start, run_started
scope-judge evaluation_duration, blob_download, criteria_count, evaluation events
scope-scheduler dispatch_cycle, requests_dispatched, pp_dispatch, reaper_sweep, cold_start
scope-token-manager validation_cycle, tokens_validated/invalidated, cold_start
scope-post-processor processing_duration, cold_start, processing events
scope-report-generator generation_duration, llm_session_duration, cold_start, generation events
model-scanners (copilot + anthropic) scan_duration, token_acquisition, models_found, scan_completed

OTel Collector Component

Deployed as a Kustomize Component at deploy/components/otel-collector/:

  • otel-collector-contrib:0.156.0 with OTLP HTTP receiver → memory_limiter → batch → azuremonitor exporter
  • 2 replicas + PodDisruptionBudget (minAvailable: 1) + topology spread across nodes
  • Patches all Deployments with OTEL_COLLECTOR_ENDPOINT and OTEL_ENABLED env vars
  • Enabled in integration overlay; can be disabled by removing the component reference

Key Design Decisions

  1. Standalone packages/telemetry/ package — decoupled from shared, minimal deps
  2. Collector takes priority — when both env vars are set, OTLP export to collector wins
  3. Kustomize Component pattern — matches kubedock pattern, overlays opt in
  4. otel-collector-contrib — required for the azuremonitor exporter
  5. OTLP HTTP over gRPC — simpler, no proto deps, sufficient for workload
  6. Graceful no-op — zero cost when env vars are unset, safe for local dev
  7. --import telemetry/register — ensures OTel hooks register before app modules load (ESM requirement)
  8. OTEL_ENABLED env var toggle — same pattern as SCOPE_AUTH_ENABLED, controlled per overlay

Operational Toggle

Telemetry is controlled by the OTEL_ENABLED environment variable, following the same pattern as SCOPE_AUTH_ENABLED. The otel-collector Kustomize Component injects it automatically — including the component enables telemetry, removing it disables it.

Enable telemetry

Include the otel-collector component in the overlay's kustomization.yaml:

components:
  - ../../components/otel-collector

This injects two env vars into all Deployments:

  • OTEL_COLLECTOR_ENDPOINT — OTLP HTTP endpoint for the in-cluster collector
  • OTEL_ENABLED=true — activates telemetry in the telemetry/register entry point

Then deploy and rollout restart:

kubectl rollout restart deployment -n scoped

Disable telemetry

Remove the ../../components/otel-collector line from the overlay, deploy, and restart. Without OTEL_ENABLED, register.ts skips initialization entirely (no-op).

Override per overlay

To keep the collector running but disable telemetry in a specific overlay (e.g., for debugging the collector itself), add a patch:

patches:
  - target:
      kind: Deployment
    patch: |
      - op: replace
        path: /spec/template/spec/containers/0/env
        value:
          - name: OTEL_ENABLED
            value: "false"

Decision Matrix

Collector Component OTEL_ENABLED Result
✅ included true (auto) Full pipeline: Apps → OTLP → Collector → App Insights
❌ removed unset No telemetry — no collector, no export, zero overhead
✅ included false (override) Collector running but idle (apps skip init)

Configuration

Env Var Default Description
OTEL_ENABLED (unset) Set to "true" to enable telemetry (injected by collector component)
OTEL_COLLECTOR_ENDPOINT (none) OTLP HTTP endpoint for in-cluster collector
APPLICATIONINSIGHTS_CONNECTION_STRING (none) Direct export connection string
OTEL_SERVICE_NAME (none) Service name for trace attribution (set in K8s manifests)
TELEMETRY_SAMPLING_RATIO 1.0 Sampling fraction (direct mode only)
TELEMETRY_LOG_LEVEL Warning Min severity for trace forwarding

Files Changed

New

  • packages/telemetry/ — standalone telemetry package (init, helpers, types, register, tests)
  • deploy/components/otel-collector/ — Kustomize Component (Deployment, PDB, Service, ConfigMap)
  • docs/architecture/observability.md — full architecture reference

Modified

  • All 11 service entry points + Dockerfiles + package.json files
  • deploy/base/external-secret.yaml — appinsights-secrets
  • deploy/base/workers/worker-secrets.yaml — connection string env var
  • deploy/base/api.yaml — secretRef + OTEL_SERVICE_NAME
  • deploy/base/judge.yaml, scheduler.yaml, token-manager.yaml — OTEL_SERVICE_NAME
  • deploy/base/workers/model-scanner-*.yaml — OTEL_SERVICE_NAME
  • deploy/overlays/integration/kustomization.yaml — enable otel-collector component
  • ENV_VARIABLES.md — new env var docs
  • pnpm-workspace.yaml — add telemetry package

…ty metrics

Add telemetry module to packages/shared with initTelemetry(), trackMetric(),
trackTrace(), and trackEvent() helpers using applicationinsights v3. The module
gracefully no-ops when APPLICATIONINSIGHTS_CONNECTION_STRING is not set.

Worker instrumentation (coder-acp-copilot and coder-acp-copilot-windows):
- worker.run_duration_ms: total run duration
- worker.first_ai_call_ms: time from run start to first AI turn
- worker.subprocess_idle_s: seconds since last ACP protocol event
- worker.cold_start_ms: time from process start to first run
- worker.run_started: custom event emitted at run start
- Subprocess logs forwarded to trackTrace() with runId correlation

API instrumentation:
- initTelemetry() called at startup for auto HTTP/dependency tracking

Deployment:
- ExternalSecret for appinsights-connection-string from Key Vault
- worker-secrets updated to include APPLICATIONINSIGHTS_CONNECTION_STRING
- API envFrom includes appinsights-secrets (optional)

Closes #1223

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Fixes unbounded memory allocation vulnerability in W3C Baggage propagation
(affects versions < 2.8.0, pulled in transitively by applicationinsights).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…/monitor-opentelemetry

- Replace applicationinsights v3 (maintenance-mode) with modern OTel distro
- Use native OpenTelemetry Meter API for custom metrics
- Add TELEMETRY_SAMPLING_RATIO and TELEMETRY_LOG_LEVEL env vars
- Fix cold start measurement (use process.uptime)
- Fix first AI call detection (use structured protocol event check)
- Add periodic subprocess idle monitoring during active runs
- Add unit tests for telemetry module
- Add resetTelemetry() for test isolation
- Remove @opentelemetry/core override (no longer needed)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- New docs/architecture/observability.md covering telemetry data flow,
  transport protocols (HTTPS/Breeze, not OTLP), metrics reference,
  configuration, and secret delivery
- Link doc in AGENTS.md documentation table

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Move telemetry module from packages/shared/src/telemetry/ into its own
packages/telemetry/ workspace package. This decouples telemetry from the
shared package, keeping shared focused on core types and service clients.

- New packages/telemetry/ with @azure/monitor-opentelemetry + @opentelemetry/api
- Remove telemetry deps from packages/shared/package.json
- Update imports in api, coder-acp-copilot, coder-acp-copilot-windows
- Update Dockerfiles (api, copilot, windows) to include telemetry package
- Update pnpm-workspace.yaml and observability docs

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Extend telemetry to three more services:

coder-acp-claude-code:
- worker.run_duration_ms, first_ai_call_ms, cold_start_ms,
  subprocess_idle_s (30s interval), run_started event
- Subprocess log forwarding via trackTrace (Verbose, level-gated)

coder-vscode-electron-driver-ext:
- worker.run_duration_ms, cold_start_ms, run_started event
- No subprocess idle monitor (Playwright interaction model)

judge:
- judge.evaluation_duration_ms, blob_download_ms, criteria_count
- judge.evaluation_started/completed events with requestId

All three: telemetry package dep, Dockerfile updates, initTelemetry()
at startup. Graceful no-op when connection string is absent.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…essor, report-generator, and model scanners

Extend Application Insights telemetry to all remaining containerized
services using the same packages/telemetry module and graceful no-op
pattern.

Services instrumented:
- scheduler: dispatch cycle timing, requests dispatched counts, reaper
  sweep metrics, cold start
- token-manager: validation cycle timing, tokens validated/invalidated
  counts, cold start (Express auto-instrumentation for routes)
- post-processor: processing duration per handler type, cold start,
  processing started/completed events
- report-generator: generation duration, LLM session duration, cold
  start, generation started/completed events
- model-scanners (copilot + anthropic): scan duration, token
  acquisition time, models found, scan completed events. Short-lived
  jobs call shutdownTelemetry() to flush before exit.

Each service gets:
- telemetry workspace dependency in package.json
- initTelemetry() call at entry point
- Custom metrics at key timing points
- Dockerfile updated to include packages/telemetry in builder + prod

Updated observability architecture doc with all new service metrics.

All 2516 tests pass, all services build cleanly.

Refs: #1223

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Kept main's updated DEVPROXY_HAR_OUTPUT_DIR description (gateway
backend clarification) alongside our Observability section.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Resolve conflicts in AGENTS.md (keep both observability and kubedock docs)
- Resolve conflicts in coder-acp-copilot/src/index.ts (add KubedockClient + telemetry imports)
- Resolve conflicts in coder-acp-claude-code/src/index.ts (add KubedockClient + telemetry + coldStartTracked)
- Add ssr.resolve.conditions to vitest.integration.config.ts so integration tests
  can resolve the telemetry workspace package via the 'source' condition

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Refactor the telemetry package to support two export modes:
- Collector mode: OTLP HTTP to an in-cluster OTel Collector when
  OTEL_COLLECTOR_ENDPOINT is set (collector forwards to App Insights)
- Direct mode: useAzureMonitor() to App Insights when only
  APPLICATIONINSIGHTS_CONNECTION_STRING is set (current behavior)

Collector mode takes priority when both env vars are set.

Add OTel Collector Kustomize Component (deploy/components/otel-collector/):
- otel-collector-contrib:0.115.0 Deployment + Service
- OTLP HTTP receiver → memory_limiter → batch → azuremonitor exporter
- JSON patch injects OTEL_COLLECTOR_ENDPOINT into all Deployments
- Enabled in integration overlay

Update observability docs and ENV_VARIABLES.md with dual-mode
architecture, collector pipeline details, and new env var.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8cafaad3-0a65-4b98-9966-9e1509ba6b0e
Add @opentelemetry/auto-instrumentations-node to the NodeSDK config
so HTTP, Express, MongoDB, and other modules are automatically
instrumented for trace generation in collector mode.

Key changes:
- Add getNodeAutoInstrumentations() to initWithCollector() with noisy
  modules (fs, dns, net) disabled to keep span volume manageable
- Create telemetry/register entry point for use with node --import,
  ensuring instrumentation hooks register before app modules load
- Update all production Dockerfiles: node --import telemetry/register
- Add OTEL_SERVICE_NAME env var to non-worker K8s Deployments
  (workers already have WORKER_NAME which register.ts falls back to)
- Add unit test verifying auto-instrumentations are configured

The --import flag ensures OpenTelemetry's import-in-the-middle hooks
patch ESM modules (express, mongodb, http) at load time. Without this,
the NodeSDK starts too late and no auto-spans are produced.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8cafaad3-0a65-4b98-9966-9e1509ba6b0e
1. Fix test TS2556/TS2352 — add variadic args type to mock function
   so .mock.calls[0][0] is properly typed.

2. Fix OTel Collector image: upstream otel-collector-contrib images
   (both ghcr and Docker Hub) are dynamically linked without glibc
   in their base, causing 'exec: no such file or directory' on AKS.
   Solution: add a Dockerfile that copies the binary from the upstream
   contrib image onto gcr.io/distroless/cc-debian12:nonroot (has glibc).
   CI builds and pushes this as scoped/otel-collector to our ACR.

3. Add otel-collector to CI build matrix (detect-changes + image job).

4. Update integration overlay images.yaml with collector image reference.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8cafaad3-0a65-4b98-9966-9e1509ba6b0e
The 0.116.0 build was a one-off upstream regression (dynamically linked
binary on scratch base). All adjacent versions (0.114, 0.115, 0.117+)
are statically linked and work on plain scratch.

Pin to 0.117.0 and remove:
- Custom wrapper Dockerfile (no longer needed)
- CI matrix entry for building scoped/otel-collector
- Integration overlay image setter override

The official Docker Hub image pulls directly — validated in dev with
607/607 spans and 745/745 metrics exported successfully.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8cafaad3-0a65-4b98-9966-9e1509ba6b0e
…sue-1223-appinsights-instrumentation

# Conflicts:
#	apps/api/src/index.ts
#	pnpm-lock.yaml
Adds a 'telemetry' feature flag that allows team members to disable
all telemetry emission via the Portal Admin UI or API without requiring
infrastructure changes or code deploys.

Implementation:
- Seed 'telemetry' flag (default: enabled) in API startup
- Add checkTelemetryFlag() helper that queries the feature flags API
  with a 2s timeout, defaulting to enabled on any failure
- register.ts checks the flag before calling initTelemetry()
- Inject SCOPE_API_URL into all Deployments via the otel-collector
  Kustomize Component patch (same pattern as OTEL_COLLECTOR_ENDPOINT)

Toggle via API:
  curl -X PUT <api>/api/v1/feature-flags/telemetry -d '{"enabled": false}'
  Then rolling-restart pods to pick up the change.

Safety: defaults to enabled on network error, timeout, or if the flag
doesn't exist yet (first deploy). Telemetry never blocks service startup.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8cafaad3-0a65-4b98-9966-9e1509ba6b0e
Align with existing feature gate pattern — telemetry is disabled by
default and must be explicitly enabled via:
  PUT /api/v1/feature-flags/telemetry {"enabled": true}

On any failure (no API URL, network error, timeout, flag not found),
the check now returns false (no telemetry) instead of true.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8cafaad3-0a65-4b98-9966-9e1509ba6b0e
Replace startup-only flag check with a runtime-polled approach:
- Always initialize OTel (auto-instrumentation hooks must register
  before app modules load — this is an ESM requirement)
- GatingSpanProcessor wraps BatchSpanProcessor, forwarding spans
  only when the feature flag is enabled
- TelemetryFlagPoller checks the API every 60s (unref'd timer,
  won't keep the process alive)
- Toggling the flag takes effect within 60 seconds, no pod restart

This follows OTel best practice: control emission/export at the
processor level rather than starting/stopping the SDK lifecycle.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8cafaad3-0a65-4b98-9966-9e1509ba6b0e
Replace the runtime feature flag (GatingSpanProcessor + TelemetryFlagPoller)
with a simple OTEL_ENABLED env var, matching the SCOPE_AUTH_ENABLED pattern.

Changes:
- register.ts checks OTEL_ENABLED env var (default: unset/false)
- otel-collector Kustomize Component injects OTEL_ENABLED=true into all
  Deployments (same patch as OTEL_COLLECTOR_ENDPOINT)
- Removed: feature-flag.ts, gating-span-processor.ts, TelemetryFlagPoller,
  GatingSpanProcessor, SCOPE_API_URL injection, telemetry feature flag seed
- telemetry-client.ts reverted to plain BatchSpanProcessor

Toggle mechanism:
- Enable: include otel-collector component in overlay (injects both
  OTEL_COLLECTOR_ENDPOINT and OTEL_ENABLED=true)
- Disable: remove component from overlay, rollout restart
- Requires pod restart (same as auth toggle), no API dependency

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8cafaad3-0a65-4b98-9966-9e1509ba6b0e
@cedricvidal Cedric Vidal (cedricvidal) added status: in-progress Work is actively being investigated or implemented. author: duffney labels Jul 29, 2026
@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2113

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2406 2406 0 0 0 0 0 2m52s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2114

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2406 2406 0 0 0 0 0 2m27s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2165

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2500 2500 0 0 0 0 0 2m51s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: Josh Duffney (@duffney)

Observability Review — Improvement Suggestions

Great foundation here — clean abstraction, graceful no-op, and meaningful metrics. A few things to tighten up before merging:

Issues to Address

Area Issue Recommendation
SDK choice Uses legacy applicationinsights v3 (Classic SDK, now maintenance-mode). Switch to @azure/monitor-opentelemetry — see architecture note below.
Shutdown shutdownTelemetry() calls client.flush() without awaiting the callback. Pending telemetry may be lost on exit. Wrap in a Promise or use shutdownAzureMonitor() with the OTel distro.
Sampling No sampling configured. Every subprocess log line forwarded via trackTrace() at Verbose level will be expensive at scale. Configure samplingRatio and/or filter traces to Warning+ in production.
First AI call heuristic msg.toLowerCase().includes("turn") is fragile — any log containing "turn" triggers it. Use a structured protocol event type check instead of substring matching.
Cold start metric Date.now() at module level doesn't capture actual container start. Consider process.uptime() * 1000 for more accurate cold-start measurement.
Subprocess idle metric lastProtocolEventTime is tracked but it's unclear where worker.subprocess_idle_s is emitted. Emit on a periodic interval or at run-end to catch hangs while they're happening.
Trace volume Every subprocess log line → trackTrace(Verbose) will be extremely noisy/costly. Add a log-level env var to control verbosity; batch or sample verbose traces.
No unit tests Zero test files for the new telemetry module. Add tests for the no-op path, init guard, and helpers (mock the SDK client).
Singleton & test isolation Module-global let initialized = false can cause issues in tests/hot-reload. Add a resetTelemetry() export for test harnesses.

Important

🏗️ Migrate to the Modern OpenTelemetry Distro

Microsoft recommends @azure/monitor-opentelemetry over the classic applicationinsights package (migration guide). The classic SDK is in maintenance mode and won't receive new features.

The modern pattern for Node.js + AKS + App Insights:

// Top of entrypoint — before all other imports
import { useAzureMonitor, shutdownAzureMonitor } from "@azure/monitor-opentelemetry";

useAzureMonitor({
  azureMonitorExporterOptions: {
    connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING,
  },
  samplingRatio: 0.5, // adjust per environment
  enableLiveMetrics: true,
});

// Graceful shutdown
process.on("SIGTERM", async () => {
  await shutdownAzureMonitor();
  process.exit(0);
});

This gives you auto-instrumented HTTP, dependencies, exceptions, and native OpenTelemetry spans/metrics APIs for custom instrumentation — without the maintenance-mode SDK. For ESM projects, add --import @azure/monitor-opentelemetry/loader to Node.js startup flags.

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: Josh Duffney (@duffney)

Does this support opentelemitry for AI?

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: Josh Duffney (@duffney)

Add a diagram for the flow of data.

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2167

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2516 2516 0 0 0 0 0 2m50s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: Josh Duffney (@duffney)

OpenTelemetry for AI — Research Summary

Yes, @azure/monitor-opentelemetry supports the gen_ai semantic conventions — any OTel-compliant spans/metrics carrying gen_ai.* attributes will be exported to App Insights and surfaced in the new "Agents (Preview)" blade.

What's Available

Layer Package Status What It Does
Auto-instrumentation @opentelemetry/instrumentation-openai Experimental (v0.0.1) Auto-traces OpenAI SDK calls (chat completions, embeddings) with gen_ai.* attributes
Manual spans @opentelemetry/api (already installed) Stable Create custom spans with gen_ai.* attributes on any AI call
Community @traceloop/node-server-sdk (OpenLLMetry) Stable (v0.27) Auto-instruments OpenAI, Azure OpenAI, Anthropic, etc. Emits gen_ai conventions.
Azure distro @azure/monitor-opentelemetry (already installed) Stable Exports all OTel telemetry (including gen_ai spans) to App Insights

Relevant gen_ai Attributes for Scope Workers

Attribute Maps To (in our system)
gen_ai.agent.name Worker name / Copilot CLI
gen_ai.request.model options.model (gpt-5.4, etc.)
gen_ai.usage.input_tokens From HAR parsing (already extracted)
gen_ai.usage.output_tokens From HAR parsing
gen_ai.operation.name "chat" / ACP session type
gen_ai.conversation.id runId

How It Would Work

Since our workers don't call OpenAI directly (they spawn a subprocess that does), we can't use auto-instrumentation packages. Instead:

  1. After HAR parsing (which already extracts token usage + AI call count), create a manual span:
import { trace } from "@opentelemetry/api";
const tracer = trace.getTracer("scope-workers");
const span = tracer.startSpan("gen_ai.chat", {
  attributes: {
    "gen_ai.system": "openai",
    "gen_ai.agent.name": workerType,
    "gen_ai.request.model": model,
    "gen_ai.usage.input_tokens": tokenUsage.input,
    "gen_ai.usage.output_tokens": tokenUsage.output,
    "gen_ai.conversation.id": runId,
  }
});
span.end();
  1. This span flows through the Azure Monitor exporter → App Insights → shows up in the Agents (Preview) view with per-agent token consumption, latency, and model performance.

Recommendation

Not for this PR — but a natural follow-up after the HAR parser already extracts tokenUsage and aiCallCount. The plumbing (@opentelemetry/api + Azure exporter) is already in place from this PR. Just need to create spans with the right attributes post-HAR-parsing.

References

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: Josh Duffney (@duffney)

What are the transport protocols of the arrows in the diagram.

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: Josh Duffney (@duffney)

move the telem modual from the shared into it's own package.

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: Josh Duffney (@duffney)

Cosider having a node-set collector? it's a way to locally and then send to the exporter or centreal log storage.

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: Josh Duffney (@duffney)

Consider insumenting the other micro services, gateway, schdular, token manager model scanners. Each microservice should have a toggle to turn off telem.

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2190

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2575 2575 0 0 0 0 0 2m49s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: Josh Duffney (@duffney)

@cedricvidal_microsoft I dug into the OTel collector which adds a side-car for collecting and shipping the logs to free up the worker. The trade off is slower metric gathering. Is implementing the collection something you'd like to have implemented before this PR merges? IMO we won't need it until Prod is getting several hundred concurrent runs at all times. Lmk what you think.

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2219

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2575 2575 0 0 0 0 0 2m59s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: Josh Duffney (@duffney)

Requested to implement otel collector side-car before merging.

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: Cedric Vidal (@cedricvidal)

Let's add a node level collector

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2233

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2580 2580 0 0 0 0 0 2m53s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2239

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2580 2580 0 0 0 0 0 2m48s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2240

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2581 2581 0 0 0 0 0 3m10s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2242

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2581 2581 0 0 0 0 0 2m57s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2265

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2762 2762 0 0 0 0 0 3m9s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2267

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2769 2769 0 0 0 0 0 3m5s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2268

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2781 2781 0 0 0 0 0 3m11s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2269

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2762 2762 0 0 0 0 0 3m4s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2291

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2762 2762 0 0 0 0 0 2m54s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2292

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2762 2762 0 0 0 0 0 3m3s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2293

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2762 2762 0 0 0 0 0 3m9s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2294

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
2762 2762 0 0 0 0 0 3m8s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: Cedric Vidal (@cedricvidal)

Collector runs as a single replica — worth avoiding a SPOF for the telemetry tier

The OTel Collector is deployed as a Deployment with replicas: 1 (deploy/components/otel-collector/otel-collector.yaml:9) behind one ClusterIP Service, and every instrumented service routes its OTLP through it via OTEL_COLLECTOR_ENDPOINT=http://otel-collector.scoped.svc.cluster.local:4318. So that single pod sits in the path of traces/metrics/logs for all of API, judge, scheduler, token-manager, workers, and the post/report processors.

I don't think this is a throughput concern at current volume — it's more an availability one. A single replica has no failover story for the usual pod-lifecycle events: a node drain / cluster upgrade / eviction, a bad rollout, or an OOM (the memory_limiter is limit_mib: 400, sized for one shared instance) would take the collector to zero and, as far as I can see from the config, there's no sending_queue/file_storage on the exporter — so in-flight telemetry during that window would be dropped rather than buffered. It's also the kind of outage that's easy to miss, since the thing that would surface it is the thing that's down. Please sanity-check that reading, but if it's right, even at low load replicas: 1 means a single pod event blinds the whole platform.

The good news: the collector looks stateless, so HA should be cheap

If I'm reading the pipeline correctly, this collector is effectively stateless:

receivers: [otlp] → processors: [memory_limiter, batch] → exporters: [azuremonitor]
  • memory_limiter and the otlp receiver hold no cross-request state.
  • batch only holds an ephemeral in-memory buffer (≤send_batch_size: 1024 / ≤timeout: 5s) — not persisted, not shared across pods.
  • Importantly, there's no trace-affinity processor here (no tail_sampling, groupbytrace, spanmetrics, servicegraph) — those are what would require all spans of a trace to hit the same instance and force sticky routing. Worth Josh double-checking I haven't missed one, since the HA argument hinges on it.

If that holds, HA is close to free: the ClusterIP Service already load-balances across pods, and because it's OTLP over HTTP (:4318, discrete request per export) you avoid the gRPC long-lived-connection pinning caveat too. So bumping replicas should spread load with no coordination — no leader election, no loadbalancing exporter, no app or endpoint changes.

Roughly:

spec:
  replicas: 3            # was 1
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: otel-collector, namespace: scoped }
spec:
  minAvailable: 1
  selector: { matchLabels: { app: otel-collector } }
# + optional topologySpreadConstraints across kubernetes.io/hostname

Two ways to resolve it

Option A — HA at the gateway tier (my lean, pending your check): the diff above, contained to the collector component. Service, endpoint injection, and app wiring all stay identical. Given the statelessness, this looks like the lowest-risk path.

Option B — node-level tier (DaemonSet agent): also valid, but it's a larger change for no benefit we're currently using. It would mean swapping the static endpoint for a downward-API node-IP construction (or internalTrafficPolicy: Local) across every workload's injection, plus a per-node-pool scheduling story — note the repo already has heterogeneous nodes (coder-acp-copilot-windows uses kubernetes.io/os: windows + a NoSchedule taint), so a Linux DaemonSet would need the right nodeSelector/tolerations or some pods' nodes silently get no collector. And node-local's real payoff (host/node metrics, on-node volume reduction) isn't something this relay-only collector does today. Even the tidy internalTrafficPolicy: Local variant trades the central SPOF for a per-node one (no cross-node fallback if a node lacks a ready agent), which seems like a worse trade for a pure relay. Reasonable to revisit if/when node-local collection is actually wanted.

Mapping to this PR

For quick reference, how this setup lines up against the usual guidance:

Aspect Recommended This PR
Tier Agent (DaemonSet) → Gateway, or scaled gateway Single gateway
Replicas / HA ≥2 (+ HPA if load warrants) replicas: 1 (SPOF)
Scaling trace sampling loadbalancing exporter if >1 N/A — no tail sampling, only batch/memory_limiter
App → collector node-local loopback (agent) or Service (gateway) Service DNS (fine for a gateway)

Suggestion: I'd lean toward at least 2–3 replicas + a PDB before this ships, but since the "HA is safe here" claim rests on the collector being stateless, it's worth you confirming there's no trace-affinity processor in the pipeline I've overlooked. Happy to be corrected if the topology was a deliberate choice.

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: Josh Duffney (@duffney)

Thanks for the thorough analysis Cedric — you're spot on across the board.

Confirming: no trace-affinity processors. The pipeline is purely otlp → memory_limiter → batch → azuremonitor. No tail_sampling, groupbytrace, spanmetrics, or servicegraph — the collector is fully stateless. Each request is independent, and the batch processor holds only an ephemeral in-memory buffer (≤1024 items / ≤5s) with no cross-pod coordination.

Implemented Option A in commit d508e2d4:

  • replicas: 1 → replicas: 2
  • Added PodDisruptionBudget (minAvailable: 1) — guarantees at least one pod stays up during voluntary disruptions (node drains, cluster upgrades)
  • Added topologySpreadConstraints (maxSkew: 1, kubernetes.io/hostname) — prevents both replicas from landing on the same node

No app code or endpoint changes needed — ClusterIP Service already load-balances, and OTLP over HTTP (discrete requests per export) avoids the gRPC long-connection pinning issue you mentioned.

Agreed that DaemonSet (Option B) is over-engineering for our current relay-only use case + the Windows node pool complication you identified. Worth revisiting only if we add node-level collection needs later.

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: Cedric Vidal (@cedricvidal)

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status: in-progress Work is actively being investigated or implemented.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Application Insights instrumentation and worker observability metrics

2 participants