The API server's egress rules name DNS, the database and the Bots' computers. They never name 443. So in computers.mode: shared — the chart's default, and what ci/self-hosted-values.yaml installs with networkPolicy.enabled: true — the server can reach neither CopilotKit Intelligence, nor an identity provider, nor a Bot endpoint. The pod comes up green regardless, because /health answers from a literal.
What it looks like
On a cluster whose CNI enforces policy, curl from a pod carrying the labels the chart gives the API server:
shared mode, ci/self-hosted-values.yaml
api.cloud.copilotkit.ai TIMEOUT accounts.google.com TIMEOUT example.com TIMEOUT
a control pod in the same namespace, selected by no policy
api.cloud.copilotkit.ai 404 accounts.google.com 302 example.com 200
sandbox mode, same chart, same labels
api.cloud.copilotkit.ai 404 accounts.google.com 302 example.com 200
The control pod is what shows the cluster enforces policy at all, so the timeouts are the rule rather than the network.
Why it happens
charts/openbot/templates/networkpolicy.yaml:33-72 is the server's whole egress list: DNS, the bundled database, the computers on 4100, and then one rule that mentions 443 at all:
{{- if eq .Values.computers.mode "sandbox" }}
{{- /* The API server, which is where a per-Bot computer is asked for. */}}
- ports:
- port: 443
It is gated on sandbox and it has no to:. One omission, two opposite results. In shared and external the rule is absent and every outbound HTTPS call is dropped. In sandbox it is present and unscoped, so a rule whose comment says it is about the Kubernetes API server is what silently carries Intelligence, sign-in and every Bot endpoint — to every host on the internet. kubectl describe reads it back as To: <any> (traffic not restricted by destination).
Reproduction
helm template ci charts/openbot --values charts/openbot/ci/self-hosted-values.yaml --set-string secrets.keyEncryptionKey="$(openssl rand -base64 32)"- On a cluster that enforces NetworkPolicy (
kind create cluster is enough), apply the rendered ci-openbot-server NetworkPolicy into a namespace.
- Run a pod there carrying the labels from the rendered server Deployment's pod template, with
curl.
curl --max-time 8 https://api.cloud.copilotkit.ai — times out.- Repeat from a pod with no such labels — answers 404.
Against ci/eks-sandbox-values.yaml, step 4 succeeds instead, which is the second half.
Why nothing catches it
validation.yaml:77-78 refuses to install without config.intelligence.apiUrl, and :130-131 already refuses networkPolicy.enabled with an external database and nothing letting the API reach it. The same check was never written for the other outbound the server cannot work without.
Nor does anything report it afterwards. server/src/app.ts:173 answers /health with a literal { status: "ok" }, and server/deployment.yaml:117-138 points the startup, readiness and liveness probes at it, so the pod goes Ready, stays Live, the HPA scales it and the PDB protects it, while nobody can sign in.
What a fix probably has to do
The server's real outbound set is hostnames the deployment configures — Intelligence, the identity provider, managedAgent.url — plus Bot and MCP addresses not knowable at install time. NetworkPolicy cannot match a hostname, so none of it becomes a to:. That leaves a decision rather than a repair:
- Give the server the shape the computer already has at
networkpolicy.yaml:122-136 — 80 and 443 to 0.0.0.0/0 with the private ranges cut out — in every mode rather than only sandbox. It says what is true, that this pod talks to the internet, instead of saying it once by accident.
- Or refuse the install, the way the database check already does, and make whoever turns the policy on supply CIDRs.
Either way the sandbox-only 443/6443 rule should stop being the thing quietly carrying everything else.
Severity
networkPolicy.enabled is false by default and a deployment that leaves it there is unaffected. That is the limit on this.
Past it, the failure is total and differs by cloud from one values file. On GKE, Dataplane V2 is the default network plugin, always on for Autopilot and not disableable, so it is certain. On EKS the VPC CNI needs --enable-network-policy=true, which validation.yaml:127-128 notes is off by default, so the same file is fine there. Two of the five shipped targets set the flag, and nothing distinguishes the outcomes.
The API server's egress rules name DNS, the database and the Bots' computers. They never name 443. So in
computers.mode: shared— the chart's default, and whatci/self-hosted-values.yamlinstalls withnetworkPolicy.enabled: true— the server can reach neither CopilotKit Intelligence, nor an identity provider, nor a Bot endpoint. The pod comes up green regardless, because/healthanswers from a literal.What it looks like
On a cluster whose CNI enforces policy,
curlfrom a pod carrying the labels the chart gives the API server:The control pod is what shows the cluster enforces policy at all, so the timeouts are the rule rather than the network.
Why it happens
charts/openbot/templates/networkpolicy.yaml:33-72is the server's whole egress list: DNS, the bundled database, the computers on 4100, and then one rule that mentions 443 at all:It is gated on
sandboxand it has noto:. One omission, two opposite results. Insharedandexternalthe rule is absent and every outbound HTTPS call is dropped. Insandboxit is present and unscoped, so a rule whose comment says it is about the Kubernetes API server is what silently carries Intelligence, sign-in and every Bot endpoint — to every host on the internet.kubectl describereads it back asTo: <any> (traffic not restricted by destination).Reproduction
helm template ci charts/openbot --values charts/openbot/ci/self-hosted-values.yaml --set-string secrets.keyEncryptionKey="$(openssl rand -base64 32)"kind create clusteris enough), apply the renderedci-openbot-serverNetworkPolicy into a namespace.curl.curl --max-time 8 https://api.cloud.copilotkit.ai— times out.Against
ci/eks-sandbox-values.yaml, step 4 succeeds instead, which is the second half.Why nothing catches it
validation.yaml:77-78refuses to install withoutconfig.intelligence.apiUrl, and:130-131already refusesnetworkPolicy.enabledwith an external database and nothing letting the API reach it. The same check was never written for the other outbound the server cannot work without.Nor does anything report it afterwards.
server/src/app.ts:173answers/healthwith a literal{ status: "ok" }, andserver/deployment.yaml:117-138points the startup, readiness and liveness probes at it, so the pod goes Ready, stays Live, the HPA scales it and the PDB protects it, while nobody can sign in.What a fix probably has to do
The server's real outbound set is hostnames the deployment configures — Intelligence, the identity provider,
managedAgent.url— plus Bot and MCP addresses not knowable at install time. NetworkPolicy cannot match a hostname, so none of it becomes ato:. That leaves a decision rather than a repair:networkpolicy.yaml:122-136— 80 and 443 to0.0.0.0/0with the private ranges cut out — in every mode rather than onlysandbox. It says what is true, that this pod talks to the internet, instead of saying it once by accident.Either way the
sandbox-only 443/6443 rule should stop being the thing quietly carrying everything else.Severity
networkPolicy.enabledis false by default and a deployment that leaves it there is unaffected. That is the limit on this.Past it, the failure is total and differs by cloud from one values file. On GKE, Dataplane V2 is the default network plugin, always on for Autopilot and not disableable, so it is certain. On EKS the VPC CNI needs
--enable-network-policy=true, whichvalidation.yaml:127-128notes is off by default, so the same file is fine there. Two of the five shipped targets set the flag, and nothing distinguishes the outcomes.