[improve][functions] Allow customizing Kubernetes service domain suffix in Function Worker

[iantowey]

Motivation

Currently, the KubernetesRuntime hardcodes the Kubernetes service domain suffix to .svc.cluster.local when constructing the gRPC address used by the Function Worker to connect to function
instances. If a Function Worker is deployed outside of the Kubernetes cluster, it cannot communicate with the function pods because it cannot resolve the internal cluster DNS.

This change solves the problem by allowing users to customize the domain suffix, enabling external Function Workers to route traffic to function instances via an external Gateway or Ingress.

Modifications

• Added a new kubernetesServiceDomainSuffix configuration parameter to functions_worker.yml and KubernetesRuntimeFactoryConfig.
• Passed the new parameter through KubernetesRuntimeFactory down into KubernetesRuntime.
• Updated KubernetesRuntime.getServiceUrl() to use the configured suffix if provided, safely falling back to .svc.cluster.local if left blank to preserve backwards compatibility.
• Added a unit test testGetServiceUrl in KubernetesRuntimeTest to verify both the default behavior and the customized suffix behavior.

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

• Added unit test testGetServiceUrl in KubernetesRuntimeTest to ensure that getServiceUrl constructs the correct FQDN using either the default suffix or the custom
  kubernetesServiceDomainSuffix.

Does this pull request potentially affect one of the following parts:

 - [ ] Dependencies (add or upgrade a dependency)                                                                                                                                                     
 - [ ] The public API                                                                                                                                                                                 
 - [ ] The schema                                                                                                                                                                                     
 - [x] The default values of configurations                                                                                                                                                           
 - [ ] The threading model                                                                                                                                                                            
 - [ ] The binary protocol                                                                                                                                                                            
 - [ ] The REST endpoints                                                                                                                                                                             
 - [ ] The admin CLI options                                                                                                                                                                          
 - [ ] The metrics                                                                                                                                                                                    
 - [x] Anything that affects deployment  

[lhotari]

LGTM

[lhotari]

@iantowey Please don't commit "force ci re-run" commits. CI needs to be approved to run. If you want to work on early changes and run CI, please setup "Personal CI".
If the PR build fails due to flaky tests in apache/pulsar, you can add a comment /pulsarbot rerun to trigger a rerun.

[iantowey]

@lhotari apologies for this, i thought job would run in my forked repo CI https://github.com/iantowey/pulsar/actions. I'll

@iantowey Please don't commit "force ci re-run" commits. CI needs to be approved to run. If you want to work on early changes and run CI, please setup "Personal CI". If the PR build fails due to flaky tests in apache/pulsar, you can add a comment /pulsarbot rerun to trigger a rerun.

[lhotari]

@lhotari apologies for this, i thought job would run in my forked repo CI https://github.com/iantowey/pulsar/actions. I'll

@iantowey Please don't commit "force ci re-run" commits. CI needs to be approved to run. If you want to work on early changes and run CI, please setup "Personal CI". If the PR build fails due to flaky tests in apache/pulsar, you can add a comment /pulsarbot rerun to trigger a rerun.

@iantowey Sure that's fine, the commits don't cause problems itself. I got the impression that you weren't aware of "Personal CI". When using "Personal CI", you can trigger a rerun in the UI or with the gh command line tool. It requires that all build jobs have completed. You need to cancel the current workflow run to before rerun can be done. The UI handles this and provides the cancel option.
It's fine to have the same PR branch in both your fork PR and apache/pulsar PR so no problem caused.

[lhotari]

@iantowey Thanks for the contribution. This has been merged.

Some questions about the context of this PR:

What higher level goal are you planning to achieve with this PR?

Regarding the use case "This change solves the problem by allowing users to customize the domain suffix, enabling external Function Workers to route traffic to function instances via an external Gateway or Ingress.".
Just wondering what other parts of configuration are needed to achieve this. The default Pulsar Functions KubernetesRuntime doesn't directly expose ways to configure the Function Worker's Kubernetes Client authentication.
How would you configure the Function Worker to create the function instances in another Kubernetes cluster?
It's possible to set k8Uri in the config, but that's not helpful without having the ability to configure authentication. There seems to be a way to workaround it by using KUBECONFIG env variable so that the configuration of the target k8s cluster for the functions could be configured, at least in theory.

One notable detail of the GRPC connection from the Function Worker to the Function Instances is that it doesn't use authentication or TLS (it uses plaintext GRPC). The same applies to the exposed Prometheus metrics.

Regarding security of Pulsar Functions, the provided solution in Apache Pulsar and Apache Pulsar Helm chart assume that the deployed solution is used by trusted users running trusted code, isolated with network perimeter security. Some notes of this were recently added to SECURITY.md. There are hooks to harden the security, but this is not covered currently.

[iantowey]

@lhotari

Happy to share our context. We are still in the testing phase for this architecture, so some specifics might evolve, but this is the general direction we are heading.

Higher Level Goal
We run the Pulsar core components (Proxy, Broker, BookKeeper, Zookeeper, and the Functions Worker) on bare metal as Podman containers. Our Pulsar deployment relies heavily on connectors—approximately 1,000 individual connectors, some with a high degree of parallelism. Currently, the Function Worker hosts the functions via the process runtime setup. We want to move away from this setup for a number of reasons (e.g., changing Function Worker configurations requires us to cycle through all the FWs, bringing down all the functions on that FW). Our goal is to deploy the connectors on Kubernetes while continuing to maintain the FW on bare metal.

By default, the worker attempts to reach function instances using internal Kubernetes DNS (e.g., [function-name].[namespace].svc.cluster.local). Because our worker runs outside the cluster, it cannot resolve these internal addresses. This PR adds flexibility to customize the domain suffix, routing the worker's gRPC traffic through an external Kubernetes Gateway that we have configured to map to the internal function services.

Automating Gateway Routing with Kyverno
We deploy a Kyverno ClusterPolicy that watches for the creation of these function K8s Services (e.g., matching the pf-* naming convention). As soon as a Service is created, Kyverno automatically generates a corresponding GRPCRoute resource. This auto-generated route binds the function's specific hostname (utilizing the custom domain suffix introduced in this PR) to the Gateway, establishing the path from the external worker to the internal pod.

Kubernetes Authentication Configuration
You are exactly right regarding the KUBECONFIG workaround. We generate a kubeconfig file with the appropriate cluster details and client certificates for a pulsar namespace, mount that file into the Function Worker container, and set the KUBECONFIG environment variable to point to it. This allows the worker to authenticate successfully without needing explicit auth fields added to functions_worker.yml.

Security Context
Regarding the plaintext gRPC connections and metrics, we acknowledge this design and are deploying in alignment with the assumption of a trusted perimeter. The traffic between the external Function Worker and the Kubernetes Gateway occurs strictly within a secured, private network boundary (VPC/LAN), isolating it from untrusted traffic. To enforce this trusted perimeter at the ingress point, our design utilizes IP restrictions on the Gateway to explicitly drop any gRPC or metrics traffic that does not originate from the known, static IP addresses of our external Function Worker nodes.

Thanks again for the review and merge! Let me know if you need any more details.

[lhotari]

@iantowey Thanks for sharing the context of this PR and explaining how you have addressed the different concerns.