Skip to content

feat(vm): export driver traces over OTLP #2564

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
krishicks merged 1 commit into from
Aug 3, 2026
Merged

feat(vm): export driver traces over OTLP #2564

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

139 changes: 71 additions & 68 deletions MODULE.bazel.lock
View file
Open in desktop

Large diffs are not rendered by default.

26 changes: 15 additions & 11 deletions architecture/gateway.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -637,17 +637,21 @@ export: the table's presence is the on-switch, and `OTEL_EXPORTER_OTLP_ENDPOINT`
is ignored so enablement has a single source. TOML decides whether and where
to export; the SDK's `OTEL_*` variables tune how. Transport is OTLP over gRPC
only. Shared provider, resource, and tracing-layer construction lives in
`openshell-otel`.

Span emission requires no per-handler instrumentation. The `tower_http`
`TraceLayer` in `multiplex.rs` opens a span per inbound request, and that span
continues incoming W3C trace context when present or starts a new trace
otherwise. It is named for the RPC and carries the request ID that also appears
in the gateway's logs — the identifier that lets an operator pivot between a
trace and its log lines. Store and compute-driver spans become children of the
request span. Reconciliation, provider refresh, and driver-watch loops create
their own operation spans because they have no inbound request to provide a
parent. gRPC status is recorded when response trailers arrive.
`openshell-otel`, along with shared HTTP/tonic trace-context propagation and
gRPC failure recording.

The `tower_http` `TraceLayer` in `multiplex.rs` opens a span per inbound request,
and that span continues incoming W3C trace context when present or starts a new
trace otherwise. It is named for the RPC and carries the request ID that also
appears in the gateway's logs — the identifier that lets an operator pivot
between a trace and its log lines. Store and compute-driver spans become
children of the request span. Reconciliation, provider refresh, and
driver-watch loops create their own operation spans because they have no
inbound request to provide a parent. gRPC status is recorded when response
trailers arrive.

The gateway forwards OTLP configuration and W3C trace context to managed
external drivers. Each driver exports under its own service name.

Two invariants shape the failure behavior. Telemetry is diagnostic, so no OTLP
failure stops the gateway from serving: a malformed endpoint is logged at
Expand Down
9 changes: 9 additions & 0 deletions crates/openshell-driver-vm/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,15 @@ path = "src/main.rs"

[dependencies]
openshell-core = { path = "../openshell-core", default-features = false }
openshell-otel = { path = "../openshell-otel" }
openshell-policy = { path = "../openshell-policy" }
openshell-vfio = { path = "../openshell-vfio" }

bollard = { version = "0.20", features = ["ssh"] }
tokio = { workspace = true }
tonic = { workspace = true, features = ["transport"] }
tower-http = { workspace = true }
http = { workspace = true }
prost = { workspace = true }
prost-types = { workspace = true }
futures = { workspace = true }
Expand All @@ -34,6 +37,9 @@ nix = { workspace = true }
clap = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
opentelemetry = { workspace = true }
opentelemetry_sdk = { workspace = true }
tracing-opentelemetry = { workspace = true }
miette = { workspace = true }
url = { workspace = true }
serde = { workspace = true }
Expand All @@ -56,6 +62,9 @@ telemetry = ["openshell-core/telemetry"]

[dev-dependencies]
temp-env = "0.3"
tempfile = "3"
opentelemetry_sdk = { workspace = true, features = ["testing"] }
opentelemetry-proto = { version = "0.32", default-features = false, features = ["gen-tonic", "trace"] }

# smol-rs/polling drives the BSD/macOS parent-death detection in
# procguard via kqueue's EVFILT_PROC / NOTE_EXIT filter. We could use
Expand Down
2 changes: 1 addition & 1 deletion crates/openshell-driver-vm/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ RUST_LOG=openshell_server=debug,openshell_driver_vm=debug \
mise run gateway:vm
```

The VM guest's serial console is appended to `<state-dir>/<sandbox-id>/console.log`. Sandbox IDs must match `[A-Za-z0-9._-]{1,128}` before the driver uses them in host paths. The gateway-owned compute-driver socket lives at `<state-dir>/run/compute-driver.sock`; OpenShell creates `run/` with owner-only permissions, removes same-owner stale sockets, and the gateway removes the socket on clean shutdown via `ManagedDriverProcess::drop`. UDS clients must match the driver UID and provide the expected gateway process PID by default. Standalone same-UID UDS mode requires the explicit `--allow-same-uid-peer` development flag. TCP mode is disabled by default because it is unauthenticated; use `--allow-unauthenticated-tcp --bind-address 127.0.0.1:50061` only for local development.
The VM guest's serial console is appended to `<state-dir>/<sandbox-id>/console.log`. Sandbox IDs must match `[A-Za-z0-9._-]{1,128}` before the driver uses them in host paths. The gateway-owned compute-driver socket lives at `<state-dir>/run/compute-driver.sock`; OpenShell creates `run/` with owner-only permissions and removes same-owner stale sockets. On clean shutdown, the gateway sends the managed driver `SIGTERM`, waits up to five seconds for it to flush telemetry and exit, then force-kills it if necessary and removes the socket. UDS clients must match the driver UID and provide the expected gateway process PID by default. Standalone same-UID UDS mode requires the explicit `--allow-same-uid-peer` development flag. TCP mode is disabled by default because it is unauthenticated; use `--allow-unauthenticated-tcp --bind-address 127.0.0.1:50061` only for local development.

## Host-side nftables rules

Expand Down
Loading
Loading