Skip to content

Track processes outside units and containers, grouped by executable (opt-in) #382

Open

Description

@mayankpande88

Problem

On hosts without Kubernetes, a process that runs outside a systemd unit or a container is dropped. Examples are a process started with nohup or from a login shell, or one spawned by cron into the session's scope. calcId returns "" for cgroup type StandaloneProcess (user.slice/..., *.scope under system.slice, the root cgroup), so its connections, L7 requests and resources aren't reported at all.

Decision

Track these processes grouped by executable, behind an opt-in flag (--track-standalone-processes, default off), so the number of series stays bounded.

Design

  1. Identity. Container id /standalone/<exe>, where <exe> is the basename of /proc/<pid>/exe (sanitized). Two binaries with the same basename are merged; that will be documented.
  2. Created lazily, on the first listen or TCP connect, not on process start. Interactive shells spawn many short commands (ls, grep, ...). Groups created at process start would churn, and what's missing today is network and L7 visibility. Processes are registered against the group as usual.
  3. Registry mapping. A standalone group isn't keyed in containersByCgroupId: one session scope (or the root cgroup) holds many executables. The pid maps to the group through containersByPid, and the group is found by id.
  4. Resources per process, not per cgroup. A standalone group's cgroup is shared (the session scope, or / for the whole host), so cgroup-based CPU, memory, I/O and PSI would be wrong.
    • Summed per process: CPU from /proc/<pid>/stat (utime + stime), RSS from /proc/<pid>/statm, and disk I/O from /proc/<pid>/io.
    • Already per process: delays (taskstats) and OOM kills (exit events).
    • Not reported: PSI and limits.
  5. Lifecycle. The group goes zombie when its last process exits and is removed by the usual GC. A new process with the same executable reuses the id.
  6. Off by default. With the flag off, behaviour is unchanged.

Done when

  • A nohup'd HTTP server and client, started from a login shell on a VM, show up as /standalone/<exe>, with the same container_net_tcp_* and container_http_requests_total the equivalent systemd unit gets.
  • Its container_resources_cpu_usage_seconds_total and memory match /proc/<pid>/stat and statm, summed over the group's processes, within a scrape interval. They must not equal the session scope's totals.
  • Running 1000 short shell commands (no network) creates no /standalone/* series.
  • With the flag off, the series and the registry state are unchanged.
  • Tests cover the id derivation, the lazy creation and the per-process resource sum.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions