plugpool

A cluster-autoscaler cloud provider for bare-metal machines on smart plugs. It treats a group of machines like a cloud node pool with a minimum of zero: when pods are pending that would fit, it switches a machine's plug on and the machine boots and joins the cluster; when cluster-autoscaler finds a machine unneeded, it shuts the machine down and switches the plug off.

plugpool speaks cluster-autoscaler's externalgrpc protocol and drives ESPHome plugs through their web server REST API. It runs as a sidecar in the cluster-autoscaler pod, so the two talk over localhost and nothing is exposed on the network.

Requirements

• Each machine boots by itself when power returns (a BIOS/firmware "power on after AC loss" setting) and joins the cluster at boot.
• Each machine is on its own ESPHome plug with the web_server component enabled; the plug's relay switch is an entity of the web server.
• Node spec.providerID is <prefix><node name> (k3s sets k3s://<name>; set providerIDPrefix for other distributions).
• The pod runs on a node that can reach the plugs and is never scaled down.

How a machine is powered

• Scale-up: the plug of the first machine that is off goes on (a plug that fails is skipped for the next machine). The machine is "creating" until its Node registers. If it does not register within cluster-autoscaler's --max-node-provision-time, cluster-autoscaler lowers the target size again (or deletes the instance), and plugpool switches that plug off — after checking that the machine has not registered in the meantime.

• Scale-down: after cluster-autoscaler has drained the node, plugpool

  • starts a one-shot privileged pod on the node that runs systemctl poweroff in the host namespaces;
  • waits until the node is NotReady (at most powerOff.notReadyTimeout);
  • waits powerOff.grace more, because some machines hang at the very end of an ACPI power-off after everything is synced;
  • switches the plug off and deletes the Node object.

  A node that is already dead goes through the same steps without the poweroff pod.

Safety rules:

• plugpool only switches the plugs in its configuration.
• Each plug powers one machine; the configuration is rejected if two nodes share a plug.
• A plug goes off only for a node cluster-autoscaler asked to delete or to drop from the target size. A registered node without cluster-autoscaler's ToBeDeletedByClusterAutoscaler taint is never shut down or switched off, whatever the request: this is checked before the shutdown and before every plug-off attempt (a machine that registers meanwhile keeps its power).
• The Node is deleted only after its poweroff pod is removed; until then the machine is not used for a scale-up.
• A node matches its instance by spec.providerID; by name only when the node has no providerID. A providerID of another form is logged as a configuration error.

State comes from the plugs and the Node objects on every refresh, so a restart of plugpool loses nothing except a scale-down in progress. Plugs are read in parallel with a short timeout; one that cannot be read keeps its last known state and never fails the refresh (a failed refresh stops cluster-autoscaler entirely). A plug never read yet is not used for scale-up.

Configuration

listen: 127.0.0.1:8086         # gRPC address for cluster-autoscaler
providerIDPrefix: "k3s://"     # node spec.providerID = prefix + name
powerOff:
  namespace: plugpool          # where poweroff pods run (privileged)
  image: busybox:1.37.0        # must provide nsenter
  notReadyTimeout: 5m          # default
  grace: 2m                    # default
nodeGroups:
  - id: workers
    minSize: 0
    maxSize: 2                 # at most the number of nodes listed
    template:                  # a node of this group, for scale-up from zero
      cpu: "24"
      memory: 64Gi
      pods: "110"              # default
      labels: { example.com/pool: workers }
      taints: []
    nodes:
      - name: worker-1
        plug: { url: "http://plug-worker-1.example", switch: "Relay" }

Plug credentials come from the environment: PLUGPOOL_PLUG_USERNAME and PLUGPOOL_PLUG_PASSWORD.

The template node is what cluster-autoscaler assumes a new machine offers; give the group's pool label in template.labels and select it in the workloads that should scale the pool.

Deploy

deploy/examples has a complete example: namespace, RBAC (cluster-autoscaler's rules plus Node delete and the poweroff pods), configuration and the Deployment with both containers. Set a nodeSelector that keeps the pod off the machines it powers. Raise the externalgrpc grpc_timeout (default 5s) as in the example.

DRA devices (GPUs) in the template

To scale up from zero for pods with ResourceClaims, cluster-autoscaler must know which devices a machine that is off would publish. plugpool records the DRA ResourceSlices of each machine while it is up (the last non-empty set, kept across power-offs in ConfigMap plugpool-template-slices in the poweroff namespace) and returns them, renamed to the template node, with the group's template. Nothing to configure; a machine must have been up once with its DRA driver running.

• Needs cluster-autoscaler with ResourceSlices in externalgrpc templates (1.37+); older versions ignore them.
• cluster-autoscaler assumes the machines of a group are alike: the template uses the first machine (in config order) with a snapshot, and plugpool logs a warning when the machines publish different devices.

Limits

• One replica. A scale-down in progress when the pod stops is not resumed; cluster-autoscaler then sees an unready node and deletes it again later.

Development

make test     # go vet + go test -race
make lint     # golangci-lint
make proto    # regenerate internal/protos (generated code is committed)
make image

License

Apache-2.0. internal/protos/externalgrpc.proto is from kubernetes/autoscaler (cluster-autoscaler-1.36.1), Copyright The Kubernetes Authors.