From 2218d658fb9cf4284c80cda3d346533e6c7c9670 Mon Sep 17 00:00:00 2001 From: Josh Duffney Date: Thu, 8 Oct 2026 14:05:41 -0500 Subject: [PATCH 01/18] feat(deploy): add Helm chart skeleton for Scope app MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Umbrella chart at deploy/helm/scope/ with Chart.yaml, values.yaml, shared helpers, workload-identity ServiceAccount, Secrets Store CSI SecretProviderClass, common-env ConfigMap, and the api service (deployment, service, ingress) as the reference implementation for remaining services. Pairs with the Azure infrastructure provisioned by deploy/azure/ (PR #1477): reads its outputs (acrLoginServer, keyVaultName, workloadIdentityClientId, workloadIdentityServiceAccountName, storageAccountName) as values, uses CSI-synced secrets for static connection strings/API keys, and workload identity for direct Blob/Queue/Key Vault SDK access. Remaining services (portal, judge, token-manager, scheduler, workers, jobs) are configured in values.yaml but not yet templated — tracked as follow-up work. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff --- deploy/helm/scope/.helmignore | 6 + deploy/helm/scope/Chart.yaml | 19 ++ deploy/helm/scope/README.md | 96 +++++++++ deploy/helm/scope/templates/NOTES.txt | 21 ++ deploy/helm/scope/templates/_helpers.tpl | 61 ++++++ .../helm/scope/templates/api/deployment.yaml | 62 ++++++ deploy/helm/scope/templates/api/ingress.yaml | 29 +++ deploy/helm/scope/templates/api/service.yaml | 17 ++ .../scope/templates/configmap-common-env.yaml | 18 ++ .../scope/templates/secretproviderclass.yaml | 52 +++++ .../helm/scope/templates/serviceaccount.yaml | 10 + deploy/helm/scope/values.yaml | 183 ++++++++++++++++++ 12 files changed, 574 insertions(+) create mode 100644 deploy/helm/scope/.helmignore create mode 100644 deploy/helm/scope/Chart.yaml create mode 100644 deploy/helm/scope/README.md create mode 100644 deploy/helm/scope/templates/NOTES.txt create mode 100644 deploy/helm/scope/templates/_helpers.tpl create mode 100644 deploy/helm/scope/templates/api/deployment.yaml create mode 100644 deploy/helm/scope/templates/api/ingress.yaml create mode 100644 deploy/helm/scope/templates/api/service.yaml create mode 100644 deploy/helm/scope/templates/configmap-common-env.yaml create mode 100644 deploy/helm/scope/templates/secretproviderclass.yaml create mode 100644 deploy/helm/scope/templates/serviceaccount.yaml create mode 100644 deploy/helm/scope/values.yaml diff --git a/deploy/helm/scope/.helmignore b/deploy/helm/scope/.helmignore new file mode 100644 index 000000000..0b4ffec36 --- /dev/null +++ b/deploy/helm/scope/.helmignore @@ -0,0 +1,6 @@ +.git/ +.gitignore +*.swp +*.bak +*.tmp +.DS_Store diff --git a/deploy/helm/scope/Chart.yaml b/deploy/helm/scope/Chart.yaml new file mode 100644 index 000000000..fa43fbf7d --- /dev/null +++ b/deploy/helm/scope/Chart.yaml @@ -0,0 +1,19 @@ +apiVersion: v2 +name: scope +description: >- + Helm chart for deploying the Scope application (API, Portal, Judge, Token + Manager, Scheduler, and coding-agent workers) onto an AKS cluster — pairs + with the infrastructure provisioned by deploy/azure/. +type: application +version: 0.1.0 +appVersion: "0.1.0" +home: https://microsoft.github.io/scope/ +sources: + - https://github.com/microsoft/scope +keywords: + - scope + - ai-benchmarking + - coding-agents +maintainers: + - name: Scope maintainers + url: https://github.com/microsoft/scope diff --git a/deploy/helm/scope/README.md b/deploy/helm/scope/README.md new file mode 100644 index 000000000..f998d2ee8 --- /dev/null +++ b/deploy/helm/scope/README.md @@ -0,0 +1,96 @@ +# Scope Helm chart + +Installs the Scope application onto the AKS cluster provisioned by +[`deploy/azure/`](../azure/README.md) (the "Deploy to Azure" Bicep template). + +> **Status**: skeleton. Only the `api` service is fully templated so far +> (deployment, service, ingress) along with the shared infrastructure +> (ServiceAccount, SecretProviderClass, common-env ConfigMap). `portal`, +> `judge`, `token-manager`, `scheduler`, and the workers are configured in +> `values.yaml` but their templates are follow-up work. + +## What this does **NOT** automate + +- Provisioning Azure infrastructure — that's `deploy/azure/`. +- Populating bring-your-own secrets in Key Vault (GitHub token, Anthropic API + key, etc.) — add those yourself before or after installing this chart. +- Building/pushing container images to a registry — point `global.imageRegistry` + at your own registry (e.g. the Bicep-provisioned ACR) and build/push images + from this repo's Dockerfiles. +- TLS/cert-manager or DNS — bring your own ingress controller and certificates. + +## Prerequisites + +- An AKS cluster with: + - The **Azure Key Vault Secrets Provider** CSI driver add-on enabled + (`--enable-addons azure-keyvault-secrets-provider`) + - **OIDC issuer** + **workload identity** enabled + - A user-assigned managed identity federated to + `system:serviceaccount::`, granted + `Key Vault Secrets User` on the Key Vault and + `Storage Blob Data Contributor` + `Storage Queue Data Contributor` on the + Storage account + + All of the above is exactly what `deploy/azure/main.bicep` provisions. + +- `helm` and `kubectl` configured against the target cluster. + +## Install + +Read the Bicep deployment's outputs and populate a values override file: + +```bash +az deployment group show \ + --resource-group \ + --name \ + --query properties.outputs -o json +``` + +```yaml +# my-values.yaml +global: + namespace: scope + imageRegistry: + +azure: + tenantId: + keyVaultName: + workloadIdentityClientId: + serviceAccountName: + storageAccountName: + +api: + ingress: + enabled: true + host: api.example.com +``` + +```bash +kubectl create namespace scope +helm install scope deploy/helm/scope -n scope -f my-values.yaml +``` + +## Configuration + +See [`values.yaml`](./values.yaml) for the full set of configurable values — +every service section supports `enabled`, `image.{repository,tag}`, +`replicas`, and `resources`. + +### Secrets + +Two Azure auth patterns are used side by side: + +1. **CSI-synced secrets** (static values like connection strings and API + keys): listed in `secrets.keyVaultSecretNames`, synced from Key Vault into + a single Kubernetes `Secret` (`secrets.k8sSecretName`) by the + `SecretProviderClass`, and consumed via `envFrom.secretRef`. + `mongoConnectionString` and `redisConnectionString` are pre-populated in + Key Vault by `deploy/azure/`; everything else is bring-your-own. +2. **Direct workload identity** (in-process `DefaultAzureCredential` calls): + used for Blob/Queue Storage access (RBAC-only, no secret needed) and by + `token-manager`, which calls the Key Vault SDK directly for live token + round-robin management. + +All Azure-dependent Pods run under the shared ServiceAccount named by +`azure.serviceAccountName`, which must match the Bicep template's federated +subject. diff --git a/deploy/helm/scope/templates/NOTES.txt b/deploy/helm/scope/templates/NOTES.txt new file mode 100644 index 000000000..55d45806f --- /dev/null +++ b/deploy/helm/scope/templates/NOTES.txt @@ -0,0 +1,21 @@ +Scope has been deployed as release "{{ .Release.Name }}" in namespace +"{{ .Values.global.namespace }}". + +{{- if .Values.api.ingress.enabled }} +API: https://{{ .Values.api.ingress.host }} +{{- else }} +API: kubectl -n {{ .Values.global.namespace }} port-forward svc/{{ include "scope.componentName" (dict "component" "api" "context" .) }} 3100:{{ .Values.api.port }} +{{- end }} +{{- if .Values.portal.ingress.enabled }} +Portal: https://{{ .Values.portal.ingress.host }} +{{- else }} +Portal: kubectl -n {{ .Values.global.namespace }} port-forward svc/{{ include "scope.componentName" (dict "component" "portal" "context" .) }} 5100:{{ .Values.portal.port }} +{{- end }} + +Before this chart can fully start: + 1. Populate any bring-your-own secrets in Key Vault "{{ .Values.azure.keyVaultName }}" + (github-token, anthropic-api-key, etc. — see values.yaml secrets.keyVaultSecretNames). + 2. Confirm azure.workloadIdentityClientId matches the user-assigned identity + federated by deploy/azure/ to system:serviceaccount:{{ .Values.global.namespace }}:{{ .Values.azure.serviceAccountName }}. + +See deploy/helm/scope/README.md for full install instructions. diff --git a/deploy/helm/scope/templates/_helpers.tpl b/deploy/helm/scope/templates/_helpers.tpl new file mode 100644 index 000000000..151c07a84 --- /dev/null +++ b/deploy/helm/scope/templates/_helpers.tpl @@ -0,0 +1,61 @@ +{{/* +Chart base name. +*/}} +{{- define "scope.name" -}} +{{- .Chart.Name -}} +{{- end -}} + +{{/* +Fully qualified app name, used as a prefix for resource names. +*/}} +{{- define "scope.fullname" -}} +{{- .Release.Name -}} +{{- end -}} + +{{/* +Common labels applied to every resource. +*/}} +{{- define "scope.labels" -}} +app.kubernetes.io/name: {{ include "scope.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end -}} + +{{/* +Selector labels for a given service (pass `component` in the calling scope's +dict, e.g. (dict "component" "api" "context" $)). +*/}} +{{- define "scope.selectorLabels" -}} +app.kubernetes.io/name: {{ include "scope.name" .context }} +app.kubernetes.io/instance: {{ .context.Release.Name }} +app.kubernetes.io/component: {{ .component }} +{{- end -}} + +{{/* +Resource name for a given service component, e.g. "scope-api". +*/}} +{{- define "scope.componentName" -}} +{{- printf "%s-%s" (include "scope.fullname" .context) .component -}} +{{- end -}} + +{{/* +Fully qualified image reference, combining global.imageRegistry with a +service's image.repository/tag (falling back to global.imageTag). +*/}} +{{- define "scope.image" -}} +{{- $tag := .image.tag | default .context.Values.global.imageTag -}} +{{- if .context.Values.global.imageRegistry -}} +{{- printf "%s/%s:%s" .context.Values.global.imageRegistry .image.repository $tag -}} +{{- else -}} +{{- printf "%s:%s" .image.repository $tag -}} +{{- end -}} +{{- end -}} + +{{/* +Name of the shared workload-identity ServiceAccount. Must match the Bicep +template's federated subject. +*/}} +{{- define "scope.serviceAccountName" -}} +{{- .Values.azure.serviceAccountName -}} +{{- end -}} diff --git a/deploy/helm/scope/templates/api/deployment.yaml b/deploy/helm/scope/templates/api/deployment.yaml new file mode 100644 index 000000000..2ab7e43e3 --- /dev/null +++ b/deploy/helm/scope/templates/api/deployment.yaml @@ -0,0 +1,62 @@ +{{- if .Values.api.enabled -}} +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "scope.componentName" (dict "component" "api" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + app.kubernetes.io/component: api +spec: + replicas: {{ .Values.api.replicas }} + selector: + matchLabels: + {{- include "scope.selectorLabels" (dict "component" "api" "context" .) | nindent 6 }} + template: + metadata: + labels: + {{- include "scope.selectorLabels" (dict "component" "api" "context" .) | nindent 8 }} + azure.workload.identity/use: "true" + spec: + serviceAccountName: {{ include "scope.serviceAccountName" . }} + {{- with .Values.global.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + containers: + - name: api + image: {{ include "scope.image" (dict "image" .Values.api.image "context" .) }} + imagePullPolicy: {{ .Values.global.imagePullPolicy }} + ports: + - name: http + containerPort: {{ .Values.api.port }} + envFrom: + - configMapRef: + name: {{ include "scope.componentName" (dict "component" "common-env" "context" .) }} + - secretRef: + name: {{ .Values.secrets.k8sSecretName }} + env: + - name: PORT + value: {{ .Values.api.port | quote }} + volumeMounts: + - name: kv-secrets + mountPath: /mnt/secrets + readOnly: true + readinessProbe: + httpGet: { path: /health, port: http } + initialDelaySeconds: 5 + periodSeconds: 10 + livenessProbe: + httpGet: { path: /health, port: http } + initialDelaySeconds: 15 + periodSeconds: 20 + resources: + {{- toYaml .Values.api.resources | nindent 12 }} + volumes: + - name: kv-secrets + csi: + driver: secrets-store.csi.k8s.io + readOnly: true + volumeAttributes: + secretProviderClass: {{ include "scope.componentName" (dict "component" "azure-kv" "context" .) }} +{{- end -}} diff --git a/deploy/helm/scope/templates/api/ingress.yaml b/deploy/helm/scope/templates/api/ingress.yaml new file mode 100644 index 000000000..07bf95397 --- /dev/null +++ b/deploy/helm/scope/templates/api/ingress.yaml @@ -0,0 +1,29 @@ +{{- if and .Values.api.enabled .Values.api.ingress.enabled -}} +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: {{ include "scope.componentName" (dict "component" "api" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + app.kubernetes.io/component: api +spec: + {{- with .Values.api.ingress.className }} + ingressClassName: {{ . }} + {{- end }} + {{- with .Values.api.ingress.tls }} + tls: + {{- toYaml . | nindent 4 }} + {{- end }} + rules: + - host: {{ required "api.ingress.host is required when api.ingress.enabled is true" .Values.api.ingress.host | quote }} + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: {{ include "scope.componentName" (dict "component" "api" "context" .) }} + port: + number: {{ .Values.api.port }} +{{- end -}} diff --git a/deploy/helm/scope/templates/api/service.yaml b/deploy/helm/scope/templates/api/service.yaml new file mode 100644 index 000000000..271ca0d5c --- /dev/null +++ b/deploy/helm/scope/templates/api/service.yaml @@ -0,0 +1,17 @@ +{{- if .Values.api.enabled -}} +apiVersion: v1 +kind: Service +metadata: + name: {{ include "scope.componentName" (dict "component" "api" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + app.kubernetes.io/component: api +spec: + selector: + {{- include "scope.selectorLabels" (dict "component" "api" "context" .) | nindent 4 }} + ports: + - name: http + port: {{ .Values.api.port }} + targetPort: http +{{- end -}} diff --git a/deploy/helm/scope/templates/configmap-common-env.yaml b/deploy/helm/scope/templates/configmap-common-env.yaml new file mode 100644 index 000000000..6d94009ce --- /dev/null +++ b/deploy/helm/scope/templates/configmap-common-env.yaml @@ -0,0 +1,18 @@ +{{/* +Non-secret environment shared by most services: internal service hostnames, +storage account name (RBAC-only, no key needed), and queue names — mirrors +the `x-common-env`/`x-worker-env` YAML anchors in docker-compose.yml. +*/}} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ include "scope.componentName" (dict "component" "common-env" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} +data: + AZURE_STORAGE_ACCOUNT_NAME: {{ .Values.azure.storageAccountName | quote }} + TOKEN_MANAGER_URL: "http://{{ include "scope.componentName" (dict "component" "token-manager" "context" .) }}:{{ .Values.tokenManager.port }}" + JUDGE_SERVICE_URL: "http://{{ include "scope.componentName" (dict "component" "judge" "context" .) }}:{{ .Values.judge.port }}" + SCOPE_MT_API_URL: "http://{{ include "scope.componentName" (dict "component" "api" "context" .) }}:{{ .Values.api.port }}" + AZURE_STORAGE_QUEUE_REPORT: report-queue diff --git a/deploy/helm/scope/templates/secretproviderclass.yaml b/deploy/helm/scope/templates/secretproviderclass.yaml new file mode 100644 index 000000000..de20a4280 --- /dev/null +++ b/deploy/helm/scope/templates/secretproviderclass.yaml @@ -0,0 +1,52 @@ +{{/* +Secrets Store CSI driver SecretProviderClass: syncs the Key Vault secrets +listed in Values.secrets.keyVaultSecretNames into a single K8s Secret +(Values.secrets.k8sSecretName) via secretObjects, so application Deployments +can consume them with a plain `envFrom.secretRef` instead of mounting the CSI +volume individually. At least one Pod must still mount this SecretProviderClass +as a volume for the sync to occur (see the `kv-secrets` volume on the `api` +Deployment) — the Secrets Store CSI driver syncs on volume mount, not +continuously in the background. + +Uses workload identity (clientID) rather than pod identity or the VM's +system-assigned identity, matching the Bicep template's AKS configuration. +*/}} +{{- $logicalToEnvKey := dict + "mongoConnectionString" "MONGO_CONNECTION_STRING" + "redisConnectionString" "REDIS_CONNECTION_STRING" + "githubToken" "GITHUB_TOKEN" + "anthropicApiKey" "ANTHROPIC_API_KEY" +-}} +apiVersion: secrets-store.csi.x-k8s.io/v1 +kind: SecretProviderClass +metadata: + name: {{ include "scope.componentName" (dict "component" "azure-kv" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} +spec: + provider: azure + secretObjects: + - secretName: {{ .Values.secrets.k8sSecretName }} + type: Opaque + data: + {{- range $logical, $kvObjectName := .Values.secrets.keyVaultSecretNames }} + {{- $envKey := get $logicalToEnvKey $logical }} + {{- if $envKey }} + - objectName: {{ $kvObjectName }} + key: {{ $envKey }} + {{- end }} + {{- end }} + parameters: + usePodIdentity: "false" + useVMManagedIdentity: "false" + clientID: {{ required "azure.workloadIdentityClientId is required" .Values.azure.workloadIdentityClientId | quote }} + keyvaultName: {{ required "azure.keyVaultName is required (see deploy/azure output keyVaultName)" .Values.azure.keyVaultName | quote }} + tenantId: {{ required "azure.tenantId is required" .Values.azure.tenantId | quote }} + objects: | + array: + {{- range $logical, $kvObjectName := .Values.secrets.keyVaultSecretNames }} + - | + objectName: {{ $kvObjectName }} + objectType: secret + {{- end }} diff --git a/deploy/helm/scope/templates/serviceaccount.yaml b/deploy/helm/scope/templates/serviceaccount.yaml new file mode 100644 index 000000000..de7fd15f1 --- /dev/null +++ b/deploy/helm/scope/templates/serviceaccount.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "scope.serviceAccountName" . }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + azure.workload.identity/use: "true" + annotations: + azure.workload.identity/client-id: {{ required "azure.workloadIdentityClientId is required (see deploy/azure output workloadIdentityClientId)" .Values.azure.workloadIdentityClientId | quote }} diff --git a/deploy/helm/scope/values.yaml b/deploy/helm/scope/values.yaml new file mode 100644 index 000000000..1985ee764 --- /dev/null +++ b/deploy/helm/scope/values.yaml @@ -0,0 +1,183 @@ +# ----------------------------------------------------------------------------- +# Default values for the Scope umbrella chart. +# +# This chart is designed to be deployed against the AKS cluster provisioned by +# deploy/azure/ (the "Deploy to Azure" Bicep template). Values under `azure:` +# correspond directly to that template's outputs: +# +# az deployment group show -g -n --query properties.outputs +# +# azure.acrLoginServer <- output acrLoginServer +# azure.keyVaultName <- output keyVaultName +# azure.keyVaultUri <- output keyVaultUri +# azure.workloadIdentityClientId <- output workloadIdentityClientId +# azure.serviceAccountName <- output workloadIdentityServiceAccountName +# azure.tenantId <- your Azure AD tenant ID +# azure.storageAccountName <- output storageAccountName +# +# This repo ships ONE values.yaml with sane open-source defaults. There are no +# per-environment overlay files in-repo — layer your own `-f my-values.yaml` +# (or `--set`) on top for environment-specific overrides. +# ----------------------------------------------------------------------------- + +global: + # Namespace all resources are deployed into. Must match the namespace the + # Bicep template federated the workload identity to + # (output workloadIdentityServiceAccountNamespace, default "scope"). + namespace: scope + + # Container registry the service images are pulled from. Point this at the + # Bicep-provisioned ACR's login server (output acrLoginServer) in + # production. AKS node kubelet identity already has AcrPull on that + # registry, so no imagePullSecrets are required. Leave empty to use images + # exactly as named in each service's `image.repository` (e.g. for a local + # registry or GHCR during development). + imageRegistry: "" + imagePullPolicy: IfNotPresent + imagePullSecrets: [] + + imageTag: "latest" + +# ----------------------------------------------------------------------------- +# Azure integration: Key Vault (via Secrets Store CSI driver) + workload +# identity (direct SDK access for Blob/Queue storage and token-manager's +# live Key Vault calls). +# ----------------------------------------------------------------------------- +azure: + tenantId: "" + keyVaultName: "" + # Client ID of the user-assigned managed identity the Bicep template + # created and federated to the Kubernetes ServiceAccount below. + workloadIdentityClientId: "" + # Must exactly match the Bicep's federated subject + # (system:serviceaccount::). + serviceAccountName: scope-workload-identity + storageAccountName: "" + +# Secret object names as they exist in Key Vault, synced into a single K8s +# Secret (named by `secrets.k8sSecretName`) via the Secrets Store CSI +# driver's secretObjects feature. `mongoConnectionString` and +# `redisConnectionString` are pre-populated by the Bicep template; the rest +# are bring-your-own (add them to the Key Vault before or after install). +secrets: + k8sSecretName: scope-secrets + keyVaultSecretNames: + mongoConnectionString: cosmos-connection-string + redisConnectionString: redis-connection-string + githubToken: github-token + anthropicApiKey: anthropic-api-key + +# ----------------------------------------------------------------------------- +# Per-service configuration. Each service supports `enabled`, `image`, +# `replicas`, and `resources`; resource defaults mirror the "mirrors prod +# K8s" limits already documented in docker-compose.yml. +# ----------------------------------------------------------------------------- + +api: + enabled: true + image: + repository: scope-api + tag: "" + port: 80 + replicas: 1 + resources: + requests: { cpu: 250m, memory: 256Mi } + limits: { cpu: "500m", memory: 512Mi } + ingress: + enabled: false + className: "" + host: "" + tls: [] + +portal: + enabled: true + image: + repository: scope-portal + tag: "" + port: 80 + replicas: 1 + resources: + requests: { cpu: 100m, memory: 64Mi } + limits: { cpu: 200m, memory: 128Mi } + ingress: + enabled: false + className: "" + host: "" + tls: [] + +judge: + enabled: true + image: + repository: scope-judge + tag: "" + port: 80 + replicas: 1 + resources: + requests: { cpu: 500m, memory: 512Mi } + limits: { cpu: "1", memory: 1Gi } + judgeModel: gpt-5.4-mini + feedbackModel: gpt-5.4-mini + judgeStrategy: bundled + +tokenManager: + enabled: true + image: + repository: scope-token-manager + tag: "" + port: 80 + replicas: 1 + resources: + requests: { cpu: 100m, memory: 128Mi } + limits: { cpu: 200m, memory: 256Mi } + +scheduler: + enabled: true + image: + repository: scope-scheduler + tag: "" + port: 8080 + replicas: 1 + resources: + requests: { cpu: 100m, memory: 128Mi } + limits: { cpu: 200m, memory: 256Mi } + +workers: + reportGenerator: + enabled: true + image: + repository: scope-report-generator + tag: "" + replicas: 1 + resources: + requests: { cpu: 250m, memory: 256Mi } + limits: { cpu: "500m", memory: 512Mi } + coderAcpCopilot: + enabled: false + image: + repository: scope-coder-acp-copilot + tag: "" + replicas: 1 + resources: + requests: { cpu: 500m, memory: 512Mi } + limits: { cpu: "1", memory: 1Gi } + coderAcpClaudeCode: + enabled: false + image: + repository: scope-coder-acp-claude-code + tag: "" + replicas: 1 + resources: + requests: { cpu: 500m, memory: 512Mi } + limits: { cpu: "1", memory: 1Gi } + +jobs: + dbMigrate: + enabled: true + image: + repository: scope-api + tag: "" + registerAgents: + enabled: true + image: + repository: alpine + tag: "3.21" From a0ff4090d73ff55421b7f03b8faa6637d16631da Mon Sep 17 00:00:00 2001 From: Josh Duffney Date: Fri, 9 Oct 2026 09:10:56 -0500 Subject: [PATCH 02/18] feat(deploy): split Helm core app onto GHCR, defer ACP workers to ACR phase - Add per-image registry override to scope.image helper so core services (api, portal, judge, token-manager, scheduler, report-generator) default to GHCR while the ACP coding-agent workers and db-migrate (builder stage) point at the Bicep-provisioned ACR instead. - Template the remaining core services: judge service, token-manager, scheduler, portal (deployment/service/ingress), and the report-generator background worker. - Add the db-migrate pre-install/pre-upgrade Job, running against a dedicated :builder-tagged api image since the production image only ships compiled db-migrations output (no tsx/src). - Fix Redis secret wiring: the Bicep template's KV secret is redis-primary-key (a bare access key), not a connection string; wire discrete REDIS_HOST/PORT/PASSWORD/TLS env vars to match what every service actually reads. - Add AZURE_KEYVAULT_URI to the common-env ConfigMap for token-manager's direct Key Vault SDK client. - Add .github/workflows/publish-images.yml to publish the 6 core images (+ an api :builder tag) to GHCR on push to main. - Document the 3-phase flow (Azure infra -> helm install core app from GHCR -> build/push ACP workers to ACR + register secrets via Key Vault or the token-manager admin API) in the chart README and NOTES.txt. - Defer ACP worker Deployment manifests (coder-acp-copilot, coder-acp-claude-code) entirely; they need the Kubedock/MCP-gateway sidecar pattern from docs/architecture/kubedock.md, not yet templated. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff --- .github/workflows/publish-images.yml | 103 +++++++++++++ deploy/helm/scope/README.md | 138 +++++++++++++++--- deploy/helm/scope/templates/NOTES.txt | 4 + deploy/helm/scope/templates/_helpers.tpl | 14 +- .../scope/templates/configmap-common-env.yaml | 10 ++ .../helm/scope/templates/jobs/db-migrate.yaml | 45 ++++++ .../scope/templates/judge/deployment.yaml | 73 +++++++++ .../helm/scope/templates/judge/service.yaml | 17 +++ .../scope/templates/portal/deployment.yaml | 52 +++++++ .../helm/scope/templates/portal/ingress.yaml | 29 ++++ .../helm/scope/templates/portal/service.yaml | 17 +++ .../scope/templates/scheduler/deployment.yaml | 68 +++++++++ .../scope/templates/scheduler/service.yaml | 17 +++ .../scope/templates/secretproviderclass.yaml | 2 +- .../templates/token-manager/deployment.yaml | 66 +++++++++ .../templates/token-manager/service.yaml | 17 +++ .../workers/report-generator/deployment.yaml | 54 +++++++ deploy/helm/scope/values.yaml | 82 +++++++++-- 18 files changed, 769 insertions(+), 39 deletions(-) create mode 100644 .github/workflows/publish-images.yml create mode 100644 deploy/helm/scope/templates/jobs/db-migrate.yaml create mode 100644 deploy/helm/scope/templates/judge/deployment.yaml create mode 100644 deploy/helm/scope/templates/judge/service.yaml create mode 100644 deploy/helm/scope/templates/portal/deployment.yaml create mode 100644 deploy/helm/scope/templates/portal/ingress.yaml create mode 100644 deploy/helm/scope/templates/portal/service.yaml create mode 100644 deploy/helm/scope/templates/scheduler/deployment.yaml create mode 100644 deploy/helm/scope/templates/scheduler/service.yaml create mode 100644 deploy/helm/scope/templates/token-manager/deployment.yaml create mode 100644 deploy/helm/scope/templates/token-manager/service.yaml create mode 100644 deploy/helm/scope/templates/workers/report-generator/deployment.yaml diff --git a/.github/workflows/publish-images.yml b/.github/workflows/publish-images.yml new file mode 100644 index 000000000..079e286cd --- /dev/null +++ b/.github/workflows/publish-images.yml @@ -0,0 +1,103 @@ +name: Publish core images to GHCR + +# Builds and publishes the 6 core Scope app images (NOT the two ACP +# coding-agent workers, which are built/pushed to the Bicep-provisioned ACR +# as a separate step — see deploy/helm/scope/README.md "Phase 3") to GHCR, +# so the Helm chart under deploy/helm/scope/ can install the app straight +# from ghcr.io//scope- with no private registry required. +on: + push: + branches: [main] + paths: + - "apps/api/**" + - "apps/judge/**" + - "apps/token-manager/**" + - "apps/scheduler/**" + - "apps/portal/**" + - "apps/workers/report-generator/**" + - "packages/shared/**" + - "packages/db-migrations/**" + - "pnpm-workspace.yaml" + - "pnpm-lock.yaml" + - ".github/workflows/publish-images.yml" + workflow_dispatch: {} + +permissions: + contents: read + packages: write + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: false + +env: + REGISTRY: ghcr.io + +jobs: + publish: + # Fork PRs never trigger this workflow (push to main / manual dispatch + # only), so there's no need for the ci.yml TRUSTED_CODE gate here. + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + image: + - name: scope-api + dockerfile: apps/api/Dockerfile + - name: scope-judge + dockerfile: apps/judge/Dockerfile + - name: scope-token-manager + dockerfile: apps/token-manager/Dockerfile + - name: scope-scheduler + dockerfile: apps/scheduler/Dockerfile + - name: scope-portal + dockerfile: apps/portal/Dockerfile + - name: scope-report-generator + dockerfile: apps/workers/report-generator/Dockerfile + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v4.0.0 + + - name: Log in to GHCR + uses: docker/login-action@v3 + with: + registry: ${{ env.REGISTRY }} + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Build and push (production image) + uses: docker/build-push-action@v6 + with: + context: . + file: ${{ matrix.image.dockerfile }} + push: true + tags: | + ${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ matrix.image.name }}:latest + ${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ matrix.image.name }}:${{ github.sha }} + build-args: | + GIT_COMMIT=${{ github.sha }} + BUILD_TIME=${{ github.event.head_commit.timestamp || '' }} + GIT_BRANCH=${{ github.ref_name }} + cache-from: type=gha,scope=${{ matrix.image.name }} + cache-to: type=gha,mode=min,scope=${{ matrix.image.name }} + + # The db-migrate Helm Job runs migrations via the api Dockerfile's + # `builder` stage (full TS source + tsx) — the slim production image + # above only ships packages/db-migrations/dist, which mongo-migrate-ts's + # CLI can't use (see deploy/helm/scope/templates/jobs/db-migrate.yaml). + - name: Build and push (api builder stage, for db-migrate job) + if: matrix.image.name == 'scope-api' + uses: docker/build-push-action@v6 + with: + context: . + file: apps/api/Dockerfile + target: builder + push: true + tags: | + ${{ env.REGISTRY }}/${{ github.repository_owner }}/scope-api:builder + ${{ env.REGISTRY }}/${{ github.repository_owner }}/scope-api:builder-${{ github.sha }} + cache-from: type=gha,scope=scope-api-builder + cache-to: type=gha,mode=min,scope=scope-api-builder diff --git a/deploy/helm/scope/README.md b/deploy/helm/scope/README.md index f998d2ee8..a063713a3 100644 --- a/deploy/helm/scope/README.md +++ b/deploy/helm/scope/README.md @@ -3,20 +3,45 @@ Installs the Scope application onto the AKS cluster provisioned by [`deploy/azure/`](../azure/README.md) (the "Deploy to Azure" Bicep template). -> **Status**: skeleton. Only the `api` service is fully templated so far -> (deployment, service, ingress) along with the shared infrastructure -> (ServiceAccount, SecretProviderClass, common-env ConfigMap). `portal`, -> `judge`, `token-manager`, `scheduler`, and the workers are configured in -> `values.yaml` but their templates are follow-up work. +## Three-phase deployment + +Deploying Scope end to end is three separate steps: + +1. **Deploy Azure infrastructure** — `deploy/azure/` (the "Deploy to Azure" + Bicep template) provisions AKS, Cosmos DB, Key Vault, Redis, Storage, ACR, + and networking. It does **not** install the app. +2. **`helm install` the core app** (this chart) — api, portal, judge, + token-manager, scheduler, and report-generator. Images are pulled from + **GHCR** (`ghcr.io//scope-`, published by + [`.github/workflows/publish-images.yml`](../../../.github/workflows/publish-images.yml)) + by default, so no registry build step is required for this phase — AKS + pulls these public images directly. +3. **Build/push the ACP coding-agent workers separately** — + `coder-acp-copilot` and `coder-acp-claude-code` are *not* installed by + this chart (they need the Kubedock/MCP-gateway sidecar pattern described + in [`docs/architecture/kubedock.md`](../../docs/architecture/kubedock.md), + which isn't templated here yet). Build and push those two images to the + Bicep-provisioned **ACR** (`az acr build`), then register the + GitHub/Anthropic credentials they need — either directly in Key Vault or + via the running token-manager's admin API — and enable them with + `helm upgrade` once their manifests exist. See + ["Phase 3: ACP workers"](#phase-3-acp-coding-agent-workers-not-automated) + below. + +This chart only automates phase 2. ## What this does **NOT** automate - Provisioning Azure infrastructure — that's `deploy/azure/`. - Populating bring-your-own secrets in Key Vault (GitHub token, Anthropic API - key, etc.) — add those yourself before or after installing this chart. -- Building/pushing container images to a registry — point `global.imageRegistry` - at your own registry (e.g. the Bicep-provisioned ACR) and build/push images - from this repo's Dockerfiles. + key, etc.) — add those yourself before or after installing this chart (see + [Secrets](#secrets) below). +- Building/pushing the two ACP coding-agent worker images, or deploying them + — that's the separate "Phase 3" step above; this chart disables both by + default (`workers.coderAcpCopilot.enabled` / `workers.coderAcpClaudeCode.enabled`) + and ships no Deployment templates for them yet. +- Registering agent types with the API (`scripts/register-agent.sh`) — a + manual Phase 3 step alongside enabling the workers. - TLS/cert-manager or DNS — bring your own ingress controller and certificates. ## Prerequisites @@ -35,7 +60,7 @@ Installs the Scope application onto the AKS cluster provisioned by - `helm` and `kubectl` configured against the target cluster. -## Install +## Install (Phase 2) Read the Bicep deployment's outputs and populate a values override file: @@ -48,21 +73,24 @@ az deployment group show \ ```yaml # my-values.yaml -global: - namespace: scope - imageRegistry: - azure: tenantId: keyVaultName: + keyVaultUri: workloadIdentityClientId: serviceAccountName: storageAccountName: + redisHostName: api: ingress: enabled: true host: api.example.com + +portal: + ingress: + enabled: true + host: app.example.com ``` ```bash @@ -70,11 +98,20 @@ kubectl create namespace scope helm install scope deploy/helm/scope -n scope -f my-values.yaml ``` +This pulls all 6 core images from `global.imageRegistry` (defaults to +`ghcr.io/microsoft`, i.e. `ghcr.io/microsoft/scope-api`, etc. — override with +`--set global.imageRegistry=ghcr.io/` if you publish your +own fork's images). The `db-migrate` Job (a `pre-install,pre-upgrade` Helm +hook) runs automatically before the Deployments start. + ## Configuration See [`values.yaml`](./values.yaml) for the full set of configurable values — -every service section supports `enabled`, `image.{repository,tag}`, -`replicas`, and `resources`. +every service section supports `enabled`, `image.{repository,tag,registry}`, +`replicas`, and `resources`. `image.registry` overrides `global.imageRegistry` +per-image — used by the workers (which point at ACR, not GHCR) and +`jobs.dbMigrate` (which needs a distinct `:builder`-tagged image; see +`templates/jobs/db-migrate.yaml`). ### Secrets @@ -84,13 +121,70 @@ Two Azure auth patterns are used side by side: keys): listed in `secrets.keyVaultSecretNames`, synced from Key Vault into a single Kubernetes `Secret` (`secrets.k8sSecretName`) by the `SecretProviderClass`, and consumed via `envFrom.secretRef`. - `mongoConnectionString` and `redisConnectionString` are pre-populated in - Key Vault by `deploy/azure/`; everything else is bring-your-own. -2. **Direct workload identity** (in-process `DefaultAzureCredential` calls): - used for Blob/Queue Storage access (RBAC-only, no secret needed) and by - `token-manager`, which calls the Key Vault SDK directly for live token - round-robin management. + `mongoConnectionString` (`cosmos-connection-string`, a full connection + string) and `redisPassword` (`redis-primary-key`, a bare access key — + pairs with the non-secret `azure.redisHostName`/`redisPort`/`redisTls` + values to form `REDIS_HOST`/`REDIS_PORT`/`REDIS_PASSWORD`/`REDIS_TLS`) are + pre-populated in Key Vault by `deploy/azure/`. `githubToken` + (`github-token`) and `anthropicApiKey` (`anthropic-api-key`) are + bring-your-own and are what the ACP workers (Phase 3) and the portal's + GitHub Models fallback (`apps/api/src/llm-token.ts`) read directly as + plain `GITHUB_TOKEN`/`ANTHROPIC_API_KEY` env vars — this is the simplest + path and bypasses token-manager entirely: + + ```bash + az keyvault secret set --vault-name --name github-token --value + az keyvault secret set --vault-name --name anthropic-api-key --value + ``` + + Pods pick these up on their next restart (`kubectl rollout restart` after + setting a secret, since the CSI driver only re-syncs on pod start/the + configured polling interval). + +2. **Via the token-manager API** (more capable: multiple keys per + capability, round-robin, rotation, OAuth tokens) — register credentials + directly against the running `token-manager` service instead of writing + fixed-name Key Vault secrets: + + ```bash + kubectl port-forward -n scope svc/scope-token-manager 3300:80 + curl -X POST http://localhost:3300/api/v1/keys \ + -H 'Content-Type: application/json' \ + -d '{"type":"github-oauth","value":"","capabilities":["github-models","copilot-cli"]}' + ``` + + `token-manager` stores these itself (its own `DefaultAzureCredential` + Key Vault client — see `azure.keyVaultUri`), independent of the + `secrets.keyVaultSecretNames` list above. See `apps/token-manager/src/routes.ts` + for the full `type`/`capabilities` enum. + +3. **Direct workload identity** (in-process `DefaultAzureCredential` calls, + no secret at all): used for Blob/Queue Storage access (RBAC-only) and by + `token-manager` itself for its live Key Vault calls (option 2 above). All Azure-dependent Pods run under the shared ServiceAccount named by `azure.serviceAccountName`, which must match the Bicep template's federated subject. + +## Phase 3: ACP coding-agent workers (not automated) + +Once the core app is up, bring the two ACP workers online separately: + +```bash +az acr build --registry --image scope-coder-acp-copilot:latest \ + -f apps/workers/coder-acp-copilot/Dockerfile \ + --build-arg COPILOT_CLI_VERSION=$(grep COPILOT_CLI_VERSION apps/workers/coder-acp-copilot/versions.env | cut -d= -f2) . + +az acr build --registry --image scope-coder-acp-claude-code:latest \ + -f apps/workers/coder-acp-claude-code/Dockerfile \ + --build-arg CLAUDE_CODE_ACP_VERSION=$(grep CLAUDE_CODE_ACP_VERSION apps/workers/coder-acp-claude-code/versions.env | cut -d= -f2) \ + --build-arg CLAUDE_AGENT_SDK_VERSION=$(grep CLAUDE_AGENT_SDK_VERSION apps/workers/coder-acp-claude-code/versions.env | cut -d= -f2) . +``` + +Then register the GitHub/Anthropic credentials the workers need (see +[Secrets](#secrets) above — either Key Vault directly or the token-manager +API) and register each agent type with the API +(`scripts/register-agent.sh`, pointed at `API_URL`/`TOKEN_MANAGER_URL`). +Deploying the worker Pods themselves (with their Kubedock/MCP-gateway +sidecars) is not yet templated in this chart — see +`docs/architecture/kubedock.md` for the intended production pattern. diff --git a/deploy/helm/scope/templates/NOTES.txt b/deploy/helm/scope/templates/NOTES.txt index 55d45806f..95532acda 100644 --- a/deploy/helm/scope/templates/NOTES.txt +++ b/deploy/helm/scope/templates/NOTES.txt @@ -18,4 +18,8 @@ Before this chart can fully start: 2. Confirm azure.workloadIdentityClientId matches the user-assigned identity federated by deploy/azure/ to system:serviceaccount:{{ .Values.global.namespace }}:{{ .Values.azure.serviceAccountName }}. +NOTE: The ACP coding-agent workers (coder-acp-copilot, coder-acp-claude-code) +are NOT deployed by this chart. Build/push their images to your ACR and +bring them online as a separate step — see "Phase 3" in README.md. + See deploy/helm/scope/README.md for full install instructions. diff --git a/deploy/helm/scope/templates/_helpers.tpl b/deploy/helm/scope/templates/_helpers.tpl index 151c07a84..3037b5cf0 100644 --- a/deploy/helm/scope/templates/_helpers.tpl +++ b/deploy/helm/scope/templates/_helpers.tpl @@ -40,13 +40,19 @@ Resource name for a given service component, e.g. "scope-api". {{- end -}} {{/* -Fully qualified image reference, combining global.imageRegistry with a -service's image.repository/tag (falling back to global.imageTag). +Fully qualified image reference, combining a registry with a service's +image.repository/tag. Per-image `.image.registry` (used by the workers and +the db-migrate job, which live on the Bicep-provisioned ACR rather than +GHCR) wins; otherwise falls back to `global.imageRegistry` (the default +distribution point for the core app services, conventionally +ghcr.io//scope). Falls back to `global.imageTag` when `.image.tag` +is unset, and to a bare (unregistered) name when no registry applies at all. */}} {{- define "scope.image" -}} {{- $tag := .image.tag | default .context.Values.global.imageTag -}} -{{- if .context.Values.global.imageRegistry -}} -{{- printf "%s/%s:%s" .context.Values.global.imageRegistry .image.repository $tag -}} +{{- $registry := .image.registry | default .context.Values.global.imageRegistry -}} +{{- if $registry -}} +{{- printf "%s/%s:%s" $registry .image.repository $tag -}} {{- else -}} {{- printf "%s:%s" .image.repository $tag -}} {{- end -}} diff --git a/deploy/helm/scope/templates/configmap-common-env.yaml b/deploy/helm/scope/templates/configmap-common-env.yaml index 6d94009ce..f61a76a3f 100644 --- a/deploy/helm/scope/templates/configmap-common-env.yaml +++ b/deploy/helm/scope/templates/configmap-common-env.yaml @@ -12,7 +12,17 @@ metadata: {{- include "scope.labels" . | nindent 4 }} data: AZURE_STORAGE_ACCOUNT_NAME: {{ .Values.azure.storageAccountName | quote }} + # Only token-manager calls the Key Vault SDK directly (DefaultAzureCredential + # via the workload-identity-injected AZURE_CLIENT_ID/AZURE_FEDERATED_TOKEN_FILE + # env vars); harmless no-op for the other services. + AZURE_KEYVAULT_URI: {{ required "azure.keyVaultUri is required (see deploy/azure output keyVaultUri)" .Values.azure.keyVaultUri | quote }} TOKEN_MANAGER_URL: "http://{{ include "scope.componentName" (dict "component" "token-manager" "context" .) }}:{{ .Values.tokenManager.port }}" JUDGE_SERVICE_URL: "http://{{ include "scope.componentName" (dict "component" "judge" "context" .) }}:{{ .Values.judge.port }}" SCOPE_MT_API_URL: "http://{{ include "scope.componentName" (dict "component" "api" "context" .) }}:{{ .Values.api.port }}" AZURE_STORAGE_QUEUE_REPORT: report-queue + # Discrete Redis connection settings (Azure Cache for Redis: TLS-only on + # 6380). REDIS_PASSWORD (the access key) comes from the KV-synced + # `scope-secrets` Secret, not this ConfigMap. + REDIS_HOST: {{ required "azure.redisHostName is required (see deploy/azure output redisHostName)" .Values.azure.redisHostName | quote }} + REDIS_PORT: {{ .Values.azure.redisPort | quote }} + REDIS_TLS: {{ .Values.azure.redisTls | quote }} diff --git a/deploy/helm/scope/templates/jobs/db-migrate.yaml b/deploy/helm/scope/templates/jobs/db-migrate.yaml new file mode 100644 index 000000000..e5ba99d77 --- /dev/null +++ b/deploy/helm/scope/templates/jobs/db-migrate.yaml @@ -0,0 +1,45 @@ +{{- if .Values.jobs.dbMigrate.enabled -}} +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ include "scope.componentName" (dict "component" "db-migrate" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + app.kubernetes.io/component: db-migrate + annotations: + # Runs before api/judge/scheduler/token-manager come up, mirroring + # docker-compose's `depends_on: db-migrate: condition: + # service_completed_successfully`. Re-run on every `helm upgrade` too, + # since new migrations can ship with any release. + "helm.sh/hook": pre-install,pre-upgrade + "helm.sh/hook-weight": "0" + "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded +spec: + backoffLimit: 3 + template: + metadata: + labels: + {{- include "scope.selectorLabels" (dict "component" "db-migrate" "context" .) | nindent 8 }} + spec: + restartPolicy: Never + {{- with .Values.global.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + containers: + # packages/db-migrations' migrate:up script runs its .ts migrations + # directly via tsx, so this must use the Dockerfile's `builder` + # stage (full TS source + dev deps) rather than the slim production + # scope-api image, which only ships compiled dist/ (no src/, no + # tsx) — see jobs.dbMigrate.image.tag default ("builder") in + # values.yaml. + - name: db-migrate + image: {{ include "scope.image" (dict "image" .Values.jobs.dbMigrate.image "context" .) }} + imagePullPolicy: {{ .Values.global.imagePullPolicy }} + workingDir: /app + command: ["pnpm", "--filter", "db-migrations", "migrate:up"] + envFrom: + - secretRef: + name: {{ .Values.secrets.k8sSecretName }} +{{- end -}} diff --git a/deploy/helm/scope/templates/judge/deployment.yaml b/deploy/helm/scope/templates/judge/deployment.yaml new file mode 100644 index 000000000..9ccda568f --- /dev/null +++ b/deploy/helm/scope/templates/judge/deployment.yaml @@ -0,0 +1,73 @@ +{{- if .Values.judge.enabled -}} +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "scope.componentName" (dict "component" "judge" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + app.kubernetes.io/component: judge +spec: + replicas: {{ .Values.judge.replicas }} + selector: + matchLabels: + {{- include "scope.selectorLabels" (dict "component" "judge" "context" .) | nindent 6 }} + template: + metadata: + labels: + {{- include "scope.selectorLabels" (dict "component" "judge" "context" .) | nindent 8 }} + azure.workload.identity/use: "true" + spec: + serviceAccountName: {{ include "scope.serviceAccountName" . }} + {{- with .Values.global.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + containers: + - name: judge + image: {{ include "scope.image" (dict "image" .Values.judge.image "context" .) }} + imagePullPolicy: {{ .Values.global.imagePullPolicy }} + ports: + - name: http + containerPort: {{ .Values.judge.port }} + envFrom: + - configMapRef: + name: {{ include "scope.componentName" (dict "component" "common-env" "context" .) }} + - secretRef: + name: {{ .Values.secrets.k8sSecretName }} + env: + - name: PORT + value: {{ .Values.judge.port | quote }} + # Criteria loaded from the API (primary); filesystem fallback + # (CRITERIA_DIR) is unused in prod since this image ships no + # /app/config/criteria — kept only for parity with dev. + - name: CRITERIA_API_URL + value: "http://{{ include "scope.componentName" (dict "component" "api" "context" .) }}:{{ .Values.api.port }}" + - name: JUDGE_MODEL + value: {{ .Values.judge.judgeModel | quote }} + - name: FEEDBACK_MODEL + value: {{ .Values.judge.feedbackModel | quote }} + - name: JUDGE_STRATEGY + value: {{ .Values.judge.judgeStrategy | quote }} + volumeMounts: + - name: kv-secrets + mountPath: /mnt/secrets + readOnly: true + readinessProbe: + httpGet: { path: /health, port: http } + initialDelaySeconds: 5 + periodSeconds: 10 + livenessProbe: + httpGet: { path: /health, port: http } + initialDelaySeconds: 15 + periodSeconds: 20 + resources: + {{- toYaml .Values.judge.resources | nindent 12 }} + volumes: + - name: kv-secrets + csi: + driver: secrets-store.csi.k8s.io + readOnly: true + volumeAttributes: + secretProviderClass: {{ include "scope.componentName" (dict "component" "azure-kv" "context" .) }} +{{- end -}} diff --git a/deploy/helm/scope/templates/judge/service.yaml b/deploy/helm/scope/templates/judge/service.yaml new file mode 100644 index 000000000..801441eea --- /dev/null +++ b/deploy/helm/scope/templates/judge/service.yaml @@ -0,0 +1,17 @@ +{{- if .Values.judge.enabled -}} +apiVersion: v1 +kind: Service +metadata: + name: {{ include "scope.componentName" (dict "component" "judge" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + app.kubernetes.io/component: judge +spec: + selector: + {{- include "scope.selectorLabels" (dict "component" "judge" "context" .) | nindent 4 }} + ports: + - name: http + port: {{ .Values.judge.port }} + targetPort: http +{{- end -}} diff --git a/deploy/helm/scope/templates/portal/deployment.yaml b/deploy/helm/scope/templates/portal/deployment.yaml new file mode 100644 index 000000000..8995aef73 --- /dev/null +++ b/deploy/helm/scope/templates/portal/deployment.yaml @@ -0,0 +1,52 @@ +{{- if .Values.portal.enabled -}} +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "scope.componentName" (dict "component" "portal" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + app.kubernetes.io/component: portal +spec: + replicas: {{ .Values.portal.replicas }} + selector: + matchLabels: + {{- include "scope.selectorLabels" (dict "component" "portal" "context" .) | nindent 6 }} + template: + metadata: + labels: + {{- include "scope.selectorLabels" (dict "component" "portal" "context" .) | nindent 8 }} + spec: + {{- with .Values.global.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + containers: + - name: portal + image: {{ include "scope.image" (dict "image" .Values.portal.image "context" .) }} + imagePullPolicy: {{ .Values.global.imagePullPolicy }} + ports: + - name: http + containerPort: {{ .Values.portal.port }} + env: + # Nginx serves purely static, pre-built assets — no Azure + # identity, Key Vault secrets, or backend env vars are needed + # here. Only these two toggles are read at container start by + # docker-entrypoint.sh (see apps/portal/docker-entrypoint.sh). + - name: SCOPE_AUTH_ENABLED + value: {{ .Values.portal.authEnabled | quote }} + {{- if .Values.portal.docsBaseUrl }} + - name: SCOPE_DOCS_BASE_URL + value: {{ .Values.portal.docsBaseUrl | quote }} + {{- end }} + readinessProbe: + httpGet: { path: /health, port: http } + initialDelaySeconds: 5 + periodSeconds: 10 + livenessProbe: + httpGet: { path: /health, port: http } + initialDelaySeconds: 15 + periodSeconds: 20 + resources: + {{- toYaml .Values.portal.resources | nindent 12 }} +{{- end -}} diff --git a/deploy/helm/scope/templates/portal/ingress.yaml b/deploy/helm/scope/templates/portal/ingress.yaml new file mode 100644 index 000000000..aba290408 --- /dev/null +++ b/deploy/helm/scope/templates/portal/ingress.yaml @@ -0,0 +1,29 @@ +{{- if and .Values.portal.enabled .Values.portal.ingress.enabled -}} +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: {{ include "scope.componentName" (dict "component" "portal" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + app.kubernetes.io/component: portal +spec: + {{- with .Values.portal.ingress.className }} + ingressClassName: {{ . }} + {{- end }} + {{- with .Values.portal.ingress.tls }} + tls: + {{- toYaml . | nindent 4 }} + {{- end }} + rules: + - host: {{ required "portal.ingress.host is required when portal.ingress.enabled is true" .Values.portal.ingress.host | quote }} + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: {{ include "scope.componentName" (dict "component" "portal" "context" .) }} + port: + number: {{ .Values.portal.port }} +{{- end -}} diff --git a/deploy/helm/scope/templates/portal/service.yaml b/deploy/helm/scope/templates/portal/service.yaml new file mode 100644 index 000000000..b661025e2 --- /dev/null +++ b/deploy/helm/scope/templates/portal/service.yaml @@ -0,0 +1,17 @@ +{{- if .Values.portal.enabled -}} +apiVersion: v1 +kind: Service +metadata: + name: {{ include "scope.componentName" (dict "component" "portal" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + app.kubernetes.io/component: portal +spec: + selector: + {{- include "scope.selectorLabels" (dict "component" "portal" "context" .) | nindent 4 }} + ports: + - name: http + port: {{ .Values.portal.port }} + targetPort: http +{{- end -}} diff --git a/deploy/helm/scope/templates/scheduler/deployment.yaml b/deploy/helm/scope/templates/scheduler/deployment.yaml new file mode 100644 index 000000000..b98bf174a --- /dev/null +++ b/deploy/helm/scope/templates/scheduler/deployment.yaml @@ -0,0 +1,68 @@ +{{- if .Values.scheduler.enabled -}} +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "scope.componentName" (dict "component" "scheduler" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + app.kubernetes.io/component: scheduler +spec: + replicas: {{ .Values.scheduler.replicas }} + selector: + matchLabels: + {{- include "scope.selectorLabels" (dict "component" "scheduler" "context" .) | nindent 6 }} + template: + metadata: + labels: + {{- include "scope.selectorLabels" (dict "component" "scheduler" "context" .) | nindent 8 }} + azure.workload.identity/use: "true" + spec: + serviceAccountName: {{ include "scope.serviceAccountName" . }} + {{- with .Values.global.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + containers: + - name: scheduler + image: {{ include "scope.image" (dict "image" .Values.scheduler.image "context" .) }} + imagePullPolicy: {{ .Values.global.imagePullPolicy }} + ports: + - name: http + containerPort: {{ .Values.scheduler.port }} + envFrom: + - configMapRef: + name: {{ include "scope.componentName" (dict "component" "common-env" "context" .) }} + - secretRef: + name: {{ .Values.secrets.k8sSecretName }} + env: + - name: PORT + value: {{ .Values.scheduler.port | quote }} + - name: SCHEDULER_TARGET_QUEUE_DEPTH + value: "2" + - name: SCHEDULER_POLL_INTERVAL_MS + value: "2000" + volumeMounts: + - name: kv-secrets + mountPath: /mnt/secrets + readOnly: true + # The scheduler's health server returns 200 on every path (no + # dedicated /health route) — see apps/scheduler/src/index.ts. + readinessProbe: + httpGet: { path: /, port: http } + initialDelaySeconds: 5 + periodSeconds: 10 + livenessProbe: + httpGet: { path: /, port: http } + initialDelaySeconds: 15 + periodSeconds: 20 + resources: + {{- toYaml .Values.scheduler.resources | nindent 12 }} + volumes: + - name: kv-secrets + csi: + driver: secrets-store.csi.k8s.io + readOnly: true + volumeAttributes: + secretProviderClass: {{ include "scope.componentName" (dict "component" "azure-kv" "context" .) }} +{{- end -}} diff --git a/deploy/helm/scope/templates/scheduler/service.yaml b/deploy/helm/scope/templates/scheduler/service.yaml new file mode 100644 index 000000000..ca0b91178 --- /dev/null +++ b/deploy/helm/scope/templates/scheduler/service.yaml @@ -0,0 +1,17 @@ +{{- if .Values.scheduler.enabled -}} +apiVersion: v1 +kind: Service +metadata: + name: {{ include "scope.componentName" (dict "component" "scheduler" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + app.kubernetes.io/component: scheduler +spec: + selector: + {{- include "scope.selectorLabels" (dict "component" "scheduler" "context" .) | nindent 4 }} + ports: + - name: http + port: {{ .Values.scheduler.port }} + targetPort: http +{{- end -}} diff --git a/deploy/helm/scope/templates/secretproviderclass.yaml b/deploy/helm/scope/templates/secretproviderclass.yaml index de20a4280..e2b42bc91 100644 --- a/deploy/helm/scope/templates/secretproviderclass.yaml +++ b/deploy/helm/scope/templates/secretproviderclass.yaml @@ -13,7 +13,7 @@ system-assigned identity, matching the Bicep template's AKS configuration. */}} {{- $logicalToEnvKey := dict "mongoConnectionString" "MONGO_CONNECTION_STRING" - "redisConnectionString" "REDIS_CONNECTION_STRING" + "redisPassword" "REDIS_PASSWORD" "githubToken" "GITHUB_TOKEN" "anthropicApiKey" "ANTHROPIC_API_KEY" -}} diff --git a/deploy/helm/scope/templates/token-manager/deployment.yaml b/deploy/helm/scope/templates/token-manager/deployment.yaml new file mode 100644 index 000000000..741f7b525 --- /dev/null +++ b/deploy/helm/scope/templates/token-manager/deployment.yaml @@ -0,0 +1,66 @@ +{{- if .Values.tokenManager.enabled -}} +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "scope.componentName" (dict "component" "token-manager" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + app.kubernetes.io/component: token-manager +spec: + replicas: {{ .Values.tokenManager.replicas }} + selector: + matchLabels: + {{- include "scope.selectorLabels" (dict "component" "token-manager" "context" .) | nindent 6 }} + template: + metadata: + labels: + {{- include "scope.selectorLabels" (dict "component" "token-manager" "context" .) | nindent 8 }} + azure.workload.identity/use: "true" + spec: + serviceAccountName: {{ include "scope.serviceAccountName" . }} + {{- with .Values.global.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + containers: + - name: token-manager + image: {{ include "scope.image" (dict "image" .Values.tokenManager.image "context" .) }} + imagePullPolicy: {{ .Values.global.imagePullPolicy }} + ports: + - name: http + containerPort: {{ .Values.tokenManager.port }} + envFrom: + - configMapRef: + name: {{ include "scope.componentName" (dict "component" "common-env" "context" .) }} + - secretRef: + name: {{ .Values.secrets.k8sSecretName }} + env: + - name: PORT + value: {{ .Values.tokenManager.port | quote }} + # How often token-manager re-validates registered keys against + # their upstream provider. + - name: VALIDATION_INTERVAL_MS + value: "300000" + volumeMounts: + - name: kv-secrets + mountPath: /mnt/secrets + readOnly: true + readinessProbe: + httpGet: { path: /health, port: http } + initialDelaySeconds: 5 + periodSeconds: 10 + livenessProbe: + httpGet: { path: /health, port: http } + initialDelaySeconds: 15 + periodSeconds: 20 + resources: + {{- toYaml .Values.tokenManager.resources | nindent 12 }} + volumes: + - name: kv-secrets + csi: + driver: secrets-store.csi.k8s.io + readOnly: true + volumeAttributes: + secretProviderClass: {{ include "scope.componentName" (dict "component" "azure-kv" "context" .) }} +{{- end -}} diff --git a/deploy/helm/scope/templates/token-manager/service.yaml b/deploy/helm/scope/templates/token-manager/service.yaml new file mode 100644 index 000000000..c0d9a900b --- /dev/null +++ b/deploy/helm/scope/templates/token-manager/service.yaml @@ -0,0 +1,17 @@ +{{- if .Values.tokenManager.enabled -}} +apiVersion: v1 +kind: Service +metadata: + name: {{ include "scope.componentName" (dict "component" "token-manager" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + app.kubernetes.io/component: token-manager +spec: + selector: + {{- include "scope.selectorLabels" (dict "component" "token-manager" "context" .) | nindent 4 }} + ports: + - name: http + port: {{ .Values.tokenManager.port }} + targetPort: http +{{- end -}} diff --git a/deploy/helm/scope/templates/workers/report-generator/deployment.yaml b/deploy/helm/scope/templates/workers/report-generator/deployment.yaml new file mode 100644 index 000000000..deafb1917 --- /dev/null +++ b/deploy/helm/scope/templates/workers/report-generator/deployment.yaml @@ -0,0 +1,54 @@ +{{- if .Values.workers.reportGenerator.enabled -}} +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "scope.componentName" (dict "component" "report-generator" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + app.kubernetes.io/component: report-generator +spec: + replicas: {{ .Values.workers.reportGenerator.replicas }} + selector: + matchLabels: + {{- include "scope.selectorLabels" (dict "component" "report-generator" "context" .) | nindent 6 }} + template: + metadata: + labels: + {{- include "scope.selectorLabels" (dict "component" "report-generator" "context" .) | nindent 8 }} + azure.workload.identity/use: "true" + spec: + serviceAccountName: {{ include "scope.serviceAccountName" . }} + {{- with .Values.global.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + containers: + # Pure background Storage Queue consumer — no HTTP server/port, so + # no Service or readiness/liveness HTTP probes (see + # apps/workers/report-generator/src/index.ts). + - name: report-generator + image: {{ include "scope.image" (dict "image" .Values.workers.reportGenerator.image "context" .) }} + imagePullPolicy: {{ .Values.global.imagePullPolicy }} + envFrom: + - configMapRef: + name: {{ include "scope.componentName" (dict "component" "common-env" "context" .) }} + - secretRef: + name: {{ .Values.secrets.k8sSecretName }} + env: + - name: REPORT_MODEL + value: {{ .Values.workers.reportGenerator.reportModel | quote }} + volumeMounts: + - name: kv-secrets + mountPath: /mnt/secrets + readOnly: true + resources: + {{- toYaml .Values.workers.reportGenerator.resources | nindent 12 }} + volumes: + - name: kv-secrets + csi: + driver: secrets-store.csi.k8s.io + readOnly: true + volumeAttributes: + secretProviderClass: {{ include "scope.componentName" (dict "component" "azure-kv" "context" .) }} +{{- end -}} diff --git a/deploy/helm/scope/values.yaml b/deploy/helm/scope/values.yaml index 1985ee764..723da14fe 100644 --- a/deploy/helm/scope/values.yaml +++ b/deploy/helm/scope/values.yaml @@ -14,6 +14,7 @@ # azure.serviceAccountName <- output workloadIdentityServiceAccountName # azure.tenantId <- your Azure AD tenant ID # azure.storageAccountName <- output storageAccountName +# azure.redisHostName <- output redisHostName # # This repo ships ONE values.yaml with sane open-source defaults. There are no # per-environment overlay files in-repo — layer your own `-f my-values.yaml` @@ -26,13 +27,19 @@ global: # (output workloadIdentityServiceAccountNamespace, default "scope"). namespace: scope - # Container registry the service images are pulled from. Point this at the - # Bicep-provisioned ACR's login server (output acrLoginServer) in - # production. AKS node kubelet identity already has AcrPull on that - # registry, so no imagePullSecrets are required. Leave empty to use images - # exactly as named in each service's `image.repository` (e.g. for a local - # registry or GHCR during development). - imageRegistry: "" + # Container registry the core app service images (api, portal, judge, + # tokenManager, scheduler, workers.reportGenerator) are pulled from. + # Defaults to this repo's GHCR package namespace — AKS can pull public GHCR + # images with no imagePullSecrets. Override to your own fork's GHCR + # namespace (or any registry you've published these 6 images to, including + # the Bicep-provisioned ACR) with `--set global.imageRegistry=...`. + # + # The two ACP coding-agent workers (workers.coderAcpCopilot/ + # coderAcpClaudeCode) and the db-migrate job are NOT published here — they + # are built/pushed to the Bicep-provisioned ACR as a separate, later step + # (see deploy/helm/scope/README.md "Phase 3"), and use their own + # `image.registry` override below instead of this global default. + imageRegistry: "ghcr.io/microsoft" imagePullPolicy: IfNotPresent imagePullSecrets: [] @@ -46,6 +53,11 @@ global: azure: tenantId: "" keyVaultName: "" + # Full Key Vault URI (output keyVaultUri), e.g. https://.vault.azure.net/. + # Used directly by token-manager's Key Vault SDK client (DefaultAzureCredential + # via workload identity) — distinct from keyVaultName, which the Secrets + # Store CSI driver's SecretProviderClass wants instead. + keyVaultUri: "" # Client ID of the user-assigned managed identity the Bicep template # created and federated to the Kubernetes ServiceAccount below. workloadIdentityClientId: "" @@ -53,17 +65,29 @@ azure: # (system:serviceaccount::). serviceAccountName: scope-workload-identity storageAccountName: "" + # Hostname of the Azure Cache for Redis instance (output redisHostName). + # Non-secret — goes straight into the common-env ConfigMap as REDIS_HOST. + # Azure Cache for Redis only accepts TLS on 6380, so redisPort/redisTls + # below are fixed for that output; override only if you point this chart + # at a non-Azure Redis (e.g. plain 6379, no TLS). + redisHostName: "" + redisPort: 6380 + redisTls: true # Secret object names as they exist in Key Vault, synced into a single K8s # Secret (named by `secrets.k8sSecretName`) via the Secrets Store CSI # driver's secretObjects feature. `mongoConnectionString` and -# `redisConnectionString` are pre-populated by the Bicep template; the rest -# are bring-your-own (add them to the Key Vault before or after install). +# `redisPassword` are pre-populated by the Bicep template (as +# `cosmos-connection-string` and `redis-primary-key` respectively — Redis is +# an access *key*, not a connection string, so it pairs with +# `azure.redisHostName`/`redisPort`/`redisTls` above to form the discrete +# REDIS_HOST/PORT/PASSWORD/TLS env vars every service actually reads). The +# rest are bring-your-own (add them to the Key Vault before or after install). secrets: k8sSecretName: scope-secrets keyVaultSecretNames: mongoConnectionString: cosmos-connection-string - redisConnectionString: redis-connection-string + redisPassword: redis-primary-key githubToken: github-token anthropicApiKey: anthropic-api-key @@ -99,6 +123,14 @@ portal: resources: requests: { cpu: 100m, memory: 64Mi } limits: { cpu: 200m, memory: 128Mi } + # Runtime-only toggles (see apps/portal/docker-entrypoint.sh) — these do + # NOT require rebuilding the portal image. The actual IdP tenant/client + # config (VITE_AUTH_*) is baked into the image at build time instead (see + # docs/architecture/auth-rbac.md §8); rebuild+republish your own portal + # image if you need a different Entra app registration than this chart's + # default GHCR image ships with. + authEnabled: true + docsBaseUrl: "" ingress: enabled: false className: "" @@ -148,12 +180,24 @@ workers: repository: scope-report-generator tag: "" replicas: 1 + reportModel: gpt-5.4-mini resources: requests: { cpu: 250m, memory: 256Mi } limits: { cpu: "500m", memory: 512Mi } + # ACP coding-agent workers are intentionally disabled by default and kept + # out of the main `helm install`. They are not published to GHCR (their + # images need the Kubedock/MCP-gateway sidecar story in + # docs/architecture/kubedock.md, which this chart does not yet template) + # and are built/pushed to the Bicep-provisioned ACR as "Phase 3" — see + # deploy/helm/scope/README.md. Set `enabled: true` and `image.registry` to + # your ACR login server (`az acr build` pushes there) only once you're + # ready to deploy them. coderAcpCopilot: enabled: false image: + # Bicep-provisioned ACR login server, e.g. myacr.azurecr.io. Not GHCR — + # this image is never published there. + registry: "" repository: scope-coder-acp-copilot tag: "" replicas: 1 @@ -163,6 +207,7 @@ workers: coderAcpClaudeCode: enabled: false image: + registry: "" repository: scope-coder-acp-claude-code tag: "" replicas: 1 @@ -175,9 +220,22 @@ jobs: enabled: true image: repository: scope-api - tag: "" + # packages/db-migrations' migrate:up script runs its .ts migrations + # directly via tsx; the production `scope-api:latest` image only ships + # compiled dist/ for db-migrations (no src/, no tsx), so this job must + # run against the Dockerfile's `builder` stage instead. The GHCR + # publish workflow pushes that stage as a distinct `:builder` tag of + # the same scope-api image. + tag: "builder" + # NOT YET IMPLEMENTED as a chart template. Registering the ACP + # coding-agent types (scripts/register-agent.sh + each worker's + # agent.yaml) only matters once those workers are actually deployed — + # which, per this chart's split, happens in Phase 3 (see + # deploy/helm/scope/README.md), alongside building/pushing their images + # to ACR. Run scripts/register-agent.sh manually (or curl the API) at + # that point instead of via `helm install`. registerAgents: - enabled: true + enabled: false image: repository: alpine tag: "3.21" From 622761e22e2c6f95047e6d152d6cd15e93b41d99 Mon Sep 17 00:00:00 2001 From: Josh Duffney Date: Fri, 9 Oct 2026 10:12:31 -0500 Subject: [PATCH 03/18] fix(helm): resolve real-deployment bugs found validating chart on AKS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found and fixed while running `helm install` against a real Bicep-provisioned test AKS cluster: 1. db-migrate Job hook-ordering bug: pre-install/pre-upgrade hooks run before any regular (non-hook) resource exists, so the Job's ServiceAccount and SecretProviderClass weren't present yet. Switched to post-install/ post-upgrade and added the serviceAccountName, workload-identity label, and kv-secrets CSI volume (mirrors api Deployment) so the Job can trigger the CSI driver's secret sync. 2. Portal nginx upstream resolution: apps/portal/nginx.conf hardcodes `proxy_pass http://api:80` at image-build time with no templating hook. The chart was naming the api Service "-api", which nginx could never resolve. Renamed the Service to the literal "api" and updated the few other templates (ingress backend, SCOPE_MT_API_URL, CRITERIA_API_URL, NOTES.txt) that referenced the old componentName. 3. SecretProviderClass all-or-nothing secret sync: the Azure Key Vault CSI provider fails the *entire* volume mount if any single listed secret is missing, even for pods that don't need it. Made each secrets.keyVaultSecretNames. entry skippable via an empty string, so optional secrets (githubToken, anthropicApiKey — needed only by Phase 3 workers / portal AI features) can be omitted until populated. Verified end-to-end: `helm install` reaches STATUS: deployed, all 6 core Deployments reach 1/1 Running, the post-install db-migrate Job completes, and api confirms a live MongoDB (Cosmos) connection. Also relayed two infra-level bugs discovered during this validation to the Deploy-to-Azure Bicep session (PR #1477): a missing dependsOn causing redis-primary-key to never be written to Key Vault, and a misnamed Cosmos private DNS zone (privatelink.mongo.cosmos.azure.net instead of .com) that caused Cosmos DNS to resolve to the public IP and get firewalled. Both are being fixed on that branch, not this one. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff --- deploy/helm/scope/templates/NOTES.txt | 2 +- deploy/helm/scope/templates/api/ingress.yaml | 3 +- deploy/helm/scope/templates/api/service.yaml | 7 +++- .../scope/templates/configmap-common-env.yaml | 2 +- .../helm/scope/templates/jobs/db-migrate.yaml | 35 +++++++++++++++++-- .../scope/templates/judge/deployment.yaml | 2 +- .../scope/templates/secretproviderclass.yaml | 13 ++++++- 7 files changed, 55 insertions(+), 9 deletions(-) diff --git a/deploy/helm/scope/templates/NOTES.txt b/deploy/helm/scope/templates/NOTES.txt index 95532acda..3a3c4971c 100644 --- a/deploy/helm/scope/templates/NOTES.txt +++ b/deploy/helm/scope/templates/NOTES.txt @@ -4,7 +4,7 @@ Scope has been deployed as release "{{ .Release.Name }}" in namespace {{- if .Values.api.ingress.enabled }} API: https://{{ .Values.api.ingress.host }} {{- else }} -API: kubectl -n {{ .Values.global.namespace }} port-forward svc/{{ include "scope.componentName" (dict "component" "api" "context" .) }} 3100:{{ .Values.api.port }} +API: kubectl -n {{ .Values.global.namespace }} port-forward svc/api 3100:{{ .Values.api.port }} {{- end }} {{- if .Values.portal.ingress.enabled }} Portal: https://{{ .Values.portal.ingress.host }} diff --git a/deploy/helm/scope/templates/api/ingress.yaml b/deploy/helm/scope/templates/api/ingress.yaml index 07bf95397..c6912e9a5 100644 --- a/deploy/helm/scope/templates/api/ingress.yaml +++ b/deploy/helm/scope/templates/api/ingress.yaml @@ -23,7 +23,8 @@ spec: pathType: Prefix backend: service: - name: {{ include "scope.componentName" (dict "component" "api" "context" .) }} + # Must match the literal "api" Service name (see api/service.yaml). + name: api port: number: {{ .Values.api.port }} {{- end -}} diff --git a/deploy/helm/scope/templates/api/service.yaml b/deploy/helm/scope/templates/api/service.yaml index 271ca0d5c..1d930ccb5 100644 --- a/deploy/helm/scope/templates/api/service.yaml +++ b/deploy/helm/scope/templates/api/service.yaml @@ -2,7 +2,12 @@ apiVersion: v1 kind: Service metadata: - name: {{ include "scope.componentName" (dict "component" "api" "context" .) }} + # Hardcoded to literal "api" (NOT the usual "-api" componentName + # prefix): the portal image's apps/portal/nginx.conf bakes in + # `proxy_pass http://api:80` at build time with no templating hook, so + # the Service DNS name this chart creates must match it exactly, + # regardless of Helm release name. + name: api namespace: {{ .Values.global.namespace }} labels: {{- include "scope.labels" . | nindent 4 }} diff --git a/deploy/helm/scope/templates/configmap-common-env.yaml b/deploy/helm/scope/templates/configmap-common-env.yaml index f61a76a3f..287e983f1 100644 --- a/deploy/helm/scope/templates/configmap-common-env.yaml +++ b/deploy/helm/scope/templates/configmap-common-env.yaml @@ -18,7 +18,7 @@ data: AZURE_KEYVAULT_URI: {{ required "azure.keyVaultUri is required (see deploy/azure output keyVaultUri)" .Values.azure.keyVaultUri | quote }} TOKEN_MANAGER_URL: "http://{{ include "scope.componentName" (dict "component" "token-manager" "context" .) }}:{{ .Values.tokenManager.port }}" JUDGE_SERVICE_URL: "http://{{ include "scope.componentName" (dict "component" "judge" "context" .) }}:{{ .Values.judge.port }}" - SCOPE_MT_API_URL: "http://{{ include "scope.componentName" (dict "component" "api" "context" .) }}:{{ .Values.api.port }}" + SCOPE_MT_API_URL: "http://api:{{ .Values.api.port }}" AZURE_STORAGE_QUEUE_REPORT: report-queue # Discrete Redis connection settings (Azure Cache for Redis: TLS-only on # 6380). REDIS_PASSWORD (the access key) comes from the KV-synced diff --git a/deploy/helm/scope/templates/jobs/db-migrate.yaml b/deploy/helm/scope/templates/jobs/db-migrate.yaml index e5ba99d77..546e98290 100644 --- a/deploy/helm/scope/templates/jobs/db-migrate.yaml +++ b/deploy/helm/scope/templates/jobs/db-migrate.yaml @@ -8,11 +8,23 @@ metadata: {{- include "scope.labels" . | nindent 4 }} app.kubernetes.io/component: db-migrate annotations: - # Runs before api/judge/scheduler/token-manager come up, mirroring - # docker-compose's `depends_on: db-migrate: condition: + # Mirrors docker-compose's `depends_on: db-migrate: condition: # service_completed_successfully`. Re-run on every `helm upgrade` too, # since new migrations can ship with any release. - "helm.sh/hook": pre-install,pre-upgrade + # + # Must be post-install/post-upgrade, NOT pre-install/pre-upgrade: Helm + # creates every pre-install/pre-upgrade hook BEFORE any regular release + # resource exists, so a pre-install db-migrate Job would run before the + # ServiceAccount (workload identity) and SecretProviderClass (Key Vault + # CSI sync) templates below even exist — the pod can't authenticate or + # mount `kv-secrets`, and `envFrom.secretRef: scope-secrets` fails with + # "secret not found". Running post-install/post-upgrade instead means + # those regular resources are already applied, at the cost of api/ + # judge/scheduler/token-manager pods starting up slightly before + # migrations finish (acceptable: these services retry transient DB + # errors — see the `resilience` skill's retry/backoff conventions used + # throughout this codebase). + "helm.sh/hook": post-install,post-upgrade "helm.sh/hook-weight": "0" "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: @@ -21,8 +33,10 @@ spec: metadata: labels: {{- include "scope.selectorLabels" (dict "component" "db-migrate" "context" .) | nindent 8 }} + azure.workload.identity/use: "true" spec: restartPolicy: Never + serviceAccountName: {{ include "scope.serviceAccountName" . }} {{- with .Values.global.imagePullSecrets }} imagePullSecrets: {{- toYaml . | nindent 8 }} @@ -42,4 +56,19 @@ spec: envFrom: - secretRef: name: {{ .Values.secrets.k8sSecretName }} + volumeMounts: + # Not read directly by migrate:up — mounting the CSI volume is + # what makes the Secrets Store CSI driver sync the Key Vault + # secrets into the `scope-secrets` K8s Secret before this + # container starts, which `envFrom.secretRef` below depends on. + - name: kv-secrets + mountPath: /mnt/secrets + readOnly: true + volumes: + - name: kv-secrets + csi: + driver: secrets-store.csi.k8s.io + readOnly: true + volumeAttributes: + secretProviderClass: {{ include "scope.componentName" (dict "component" "azure-kv" "context" .) }} {{- end -}} diff --git a/deploy/helm/scope/templates/judge/deployment.yaml b/deploy/helm/scope/templates/judge/deployment.yaml index 9ccda568f..5c4038f69 100644 --- a/deploy/helm/scope/templates/judge/deployment.yaml +++ b/deploy/helm/scope/templates/judge/deployment.yaml @@ -42,7 +42,7 @@ spec: # (CRITERIA_DIR) is unused in prod since this image ships no # /app/config/criteria — kept only for parity with dev. - name: CRITERIA_API_URL - value: "http://{{ include "scope.componentName" (dict "component" "api" "context" .) }}:{{ .Values.api.port }}" + value: "http://api:{{ .Values.api.port }}" - name: JUDGE_MODEL value: {{ .Values.judge.judgeModel | quote }} - name: FEEDBACK_MODEL diff --git a/deploy/helm/scope/templates/secretproviderclass.yaml b/deploy/helm/scope/templates/secretproviderclass.yaml index e2b42bc91..b61396082 100644 --- a/deploy/helm/scope/templates/secretproviderclass.yaml +++ b/deploy/helm/scope/templates/secretproviderclass.yaml @@ -10,6 +10,15 @@ continuously in the background. Uses workload identity (clientID) rather than pod identity or the VM's system-assigned identity, matching the Bicep template's AKS configuration. + +Any `secrets.keyVaultSecretNames.` entry set to an empty string is +skipped entirely (not requested from Key Vault, not synced into the K8s +Secret). This lets optional secrets (e.g. `githubToken`, `anthropicApiKey`, +needed only by the Phase 3 worker images / portal AI features) be omitted +with `--set secrets.keyVaultSecretNames.anthropicApiKey=""` until they're +actually created in the vault — otherwise the Azure provider fails the +*entire* CSI mount (for every secret, not just the missing one) if any +listed object doesn't exist. */}} {{- $logicalToEnvKey := dict "mongoConnectionString" "MONGO_CONNECTION_STRING" @@ -32,7 +41,7 @@ spec: data: {{- range $logical, $kvObjectName := .Values.secrets.keyVaultSecretNames }} {{- $envKey := get $logicalToEnvKey $logical }} - {{- if $envKey }} + {{- if and $envKey $kvObjectName }} - objectName: {{ $kvObjectName }} key: {{ $envKey }} {{- end }} @@ -46,7 +55,9 @@ spec: objects: | array: {{- range $logical, $kvObjectName := .Values.secrets.keyVaultSecretNames }} + {{- if $kvObjectName }} - | objectName: {{ $kvObjectName }} objectType: secret {{- end }} + {{- end }} From 0e28bbdc65f9b9231b0def1cb7e3ee307626bda0 Mon Sep 17 00:00:00 2001 From: Josh Duffney Date: Fri, 9 Oct 2026 10:20:46 -0500 Subject: [PATCH 04/18] fix(helm): default portal.authEnabled to false MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The chart previously defaulted portal.authEnabled to true, matching the portal image's own 'secure by default' fail-safe. But the publicly published GHCR portal image has no Entra app registration baked in (VITE_AUTH_* are build-time-only values — see docs/architecture/auth-rbac.md §8), so a fresh `helm install` with no overrides landed every user on RequireAuth.tsx's dead-end "Authentication not configured" page with no way to sign in or disable it short of a values override. Flipped the chart default to false so a vanilla install is usable out of the box; set portal.authEnabled=true (and rebuild/republish the portal image with real VITE_AUTH_* values) once a real IdP is wired up. Verified live: helm upgrade on the test AKS env rolled the portal pod, config.js now reports authEnabled: false, and the portal UI loads (HTTP 200) instead of the auth-not-configured screen. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff --- deploy/helm/scope/values.yaml | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/deploy/helm/scope/values.yaml b/deploy/helm/scope/values.yaml index 723da14fe..459fb1bbc 100644 --- a/deploy/helm/scope/values.yaml +++ b/deploy/helm/scope/values.yaml @@ -129,7 +129,15 @@ portal: # docs/architecture/auth-rbac.md §8); rebuild+republish your own portal # image if you need a different Entra app registration than this chart's # default GHCR image ships with. - authEnabled: true + # + # Defaults to false (unlike the portal image's own "secure by default" + # fallback): the publicly-published GHCR portal image has no Entra app + # registration baked in, so setting this true out of the box makes every + # fresh Helm install land on a dead-end "Authentication not configured" + # page (apps/portal/src/components/auth/RequireAuth.tsx) with no way to + # sign in. Set to true (and rebuild the portal image with your own + # VITE_AUTH_* values) once you've wired up a real IdP. + authEnabled: false docsBaseUrl: "" ingress: enabled: false From 71db4a9c6af2c76919927f435add8f029fa5c367 Mon Sep 17 00:00:00 2001 From: Josh Duffney Date: Fri, 9 Oct 2026 10:41:10 -0500 Subject: [PATCH 05/18] fix: restore missing scripts/get-copilot-token.ts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit package.json has referenced this file (the 'get-copilot-token' pnpm script) since before this repo was split from the internal scope-core-internal repo, but the file itself was never copied over, leaving `pnpm get-copilot-token` / `pnpm -s get-copilot-token` broken with ERR_MODULE_NOT_FOUND. Restored verbatim from growth-ecosystems/scope-core-internal. Reviewed for anything that shouldn't be public: none — it only uses VS Code's well-known public OAuth client_id for the GitHub device-code flow and calls public github.com OAuth endpoints, no secrets or internal URLs. Verified: `pnpm -s get-copilot-token` now starts the device-code flow correctly instead of failing to resolve the module. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff --- scripts/get-copilot-token.ts | 87 ++++++++++++++++++++++++++++++++++++ 1 file changed, 87 insertions(+) create mode 100644 scripts/get-copilot-token.ts diff --git a/scripts/get-copilot-token.ts b/scripts/get-copilot-token.ts new file mode 100644 index 000000000..8cb1cf04d --- /dev/null +++ b/scripts/get-copilot-token.ts @@ -0,0 +1,87 @@ +#!/usr/bin/env npx tsx +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +/** + * Get a GitHub OAuth token for Copilot via device-code flow. + * + * Uses VS Code's OAuth app (client_id: 01ab8ac9400c4e429b23) with NO scopes. + * The resulting scopeless token can be exchanged for a VSCODE_COPILOT_CHAT_TOKEN. + * + * Usage: npx tsx scripts/get-copilot-token.ts + * Output: prints the token to stdout + * + * Example: export COPILOT_GITHUB_TOKEN=$(pnpm -s get-copilot-token) + * + * Based on: vscode-copilot-evaluation/src/proxy/ghOAuthTokenFetcher.ts + */ +import { execSync } from "child_process"; + +const CLIENT_ID = "01ab8ac9400c4e429b23"; +const DEVICE_CODE_URL = "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/login/device/code"; +const ACCESS_TOKEN_URL = "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/login/oauth/access_token"; + +async function main() { + // Step 1: Request device code + const codeResponse = await fetch(DEVICE_CODE_URL, { + method: "POST", + headers: { Accept: "application/json", "Content-Type": "application/json" }, + body: JSON.stringify({ client_id: CLIENT_ID }), + }); + const codeData = await codeResponse.json(); + + console.error("=".repeat(80)); + console.error("ATTENTION: To authenticate with GitHub for Copilot:"); + console.error(); + console.error(` 1. Copy this code: ${codeData.user_code}`); + console.error(` 2. Open: ${codeData.verification_uri}`); + console.error(` 3. Paste the code and approve access.`); + console.error(); + + // Try to open the browser + try { + const openCmd = process.platform === "darwin" ? "open" : process.platform === "win32" ? "start" : "xdg-open"; + execSync(`${openCmd} ${codeData.verification_uri}`, { stdio: "ignore" }); + console.error("(Browser opened automatically)"); + } catch { + console.error("(Could not open browser — please navigate manually)"); + } + console.error(); + console.error(`Polling for approval (up to ${codeData.expires_in}s)...`); + + // Step 2: Poll for access token + let expiresIn = codeData.expires_in; + let accessToken; + while (expiresIn > 0) { + await new Promise((r) => setTimeout(r, codeData.interval * 1000)); + expiresIn -= codeData.interval; + + const tokenResponse = await fetch(ACCESS_TOKEN_URL, { + method: "POST", + headers: { Accept: "application/json", "Content-Type": "application/json" }, + body: JSON.stringify({ + client_id: CLIENT_ID, + device_code: codeData.device_code, + grant_type: "urn:ietf:params:oauth:grant-type:device_code", + }), + }); + const tokenData = await tokenResponse.json(); + + if (tokenData.access_token) { + accessToken = tokenData.access_token; + break; + } + } + + if (!accessToken) { + console.error("❌ Failed to obtain access token within the time limit."); + process.exit(1); + } + + console.log(accessToken); +} + +main().catch((err) => { + console.error("❌", err.message); + process.exit(1); +}); From 6bb5924a4b9f53470b60e16cb8c38650f4fcda19 Mon Sep 17 00:00:00 2001 From: Josh Duffney Date: Fri, 9 Oct 2026 10:49:51 -0500 Subject: [PATCH 06/18] Add Helm Deployment templates for ACP coding-agent workers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds templates/workers/{coder-acp-copilot,coder-acp-claude-code}/deployment.yaml, gated on workers..enabled (default false). Mirrors the existing report-generator pattern: pure background Storage Queue consumers (no HTTP port/probes), common-env ConfigMap + scope-secrets envFrom, Key Vault CSI volume mount, shared workload identity ServiceAccount. New values.yaml fields: workers..queueName and .agentVersion, defaulted to each worker's agent-version.dev.yaml contents for chart smoke-testing; real deployments should register a non-dev version manifest and point agentVersion at it. Explicitly deferred (documented via code comments + README): MCPJungle MCP-gateway sidecar, AI Gateway/DevProxy HAR-capture sidecar, and Kubedock in-task Docker access — none are templated yet. Credential provisioning for these workers needs no new chart plumbing: both workers acquire GitHub/Anthropic credentials dynamically from Token Manager at runtime (POST /api/v1/keys via the Portal's "Create Token" page or direct API call, with the static scope-secrets env vars as an optional fallback only. Updated README's Phase 3 section to describe the now-templated worker Deployments and the agentVersion registration contract. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff EOF ) --- deploy/helm/scope/README.md | 38 +++++++-- .../coder-acp-claude-code/deployment.yaml | 79 +++++++++++++++++++ .../workers/coder-acp-copilot/deployment.yaml | 78 ++++++++++++++++++ deploy/helm/scope/values.yaml | 12 +++ 4 files changed, 200 insertions(+), 7 deletions(-) create mode 100644 deploy/helm/scope/templates/workers/coder-acp-claude-code/deployment.yaml create mode 100644 deploy/helm/scope/templates/workers/coder-acp-copilot/deployment.yaml diff --git a/deploy/helm/scope/README.md b/deploy/helm/scope/README.md index a063713a3..ce88614cb 100644 --- a/deploy/helm/scope/README.md +++ b/deploy/helm/scope/README.md @@ -181,10 +181,34 @@ az acr build --registry --image scope-coder-acp-claude-code:latest \ --build-arg CLAUDE_AGENT_SDK_VERSION=$(grep CLAUDE_AGENT_SDK_VERSION apps/workers/coder-acp-claude-code/versions.env | cut -d= -f2) . ``` -Then register the GitHub/Anthropic credentials the workers need (see -[Secrets](#secrets) above — either Key Vault directly or the token-manager -API) and register each agent type with the API -(`scripts/register-agent.sh`, pointed at `API_URL`/`TOKEN_MANAGER_URL`). -Deploying the worker Pods themselves (with their Kubedock/MCP-gateway -sidecars) is not yet templated in this chart — see -`docs/architecture/kubedock.md` for the intended production pattern. +Then register the GitHub/Anthropic credentials the workers need. The +recommended path is the Portal's **Create Token** page (or a direct +`POST /api/v1/keys` call against the API) — this writes straight to Key +Vault via the token-manager's workload identity and requires no chart +changes. Populating `secrets.keyVaultSecretNames.githubToken`/ +`.anthropicApiKey` in Key Vault yourself works too, as a static fallback the +worker falls back to only if Token Manager has no credential registered. + +Point each worker's `values.yaml` entry at the image you just built +(`workers.coderAcpCopilot.image.registry`/`workers.coderAcpClaudeCode.image.registry` += your ACR login server, `.tag` = the tag you pushed), set `.enabled: true`, +and `helm upgrade` — this chart does template a Deployment for each worker +(`templates/workers/coder-acp-copilot/`, +`templates/workers/coder-acp-claude-code/`), reusing the common env +ConfigMap/Secret and Key Vault CSI mount the rest of the app uses. These are +minimal background Storage Queue consumers only: no MCPJungle MCP-gateway +sidecar (MCP server support), no AI Gateway/DevProxy sidecar (HAR capture), +and no Kubedock (in-task `docker build`/`docker run` support) — see +`docs/architecture/kubedock.md` for the intended production pattern for +that last one. Scenarios needing those features aren't supported by this +chart yet. + +Finally, register each agent type with the API +(`scripts/register-agent.sh `). +`workers.coderAcpCopilot.agentVersion`/`.coderAcpClaudeCode.agentVersion` +must match the `agentVersion` in whatever version manifest you register — +the defaults point at the `-dev` manifests under each worker's directory, +intended for chart smoke-testing only; use a real version manifest (and +matching `imageTag`/`queueName`/`gitCommit`/`buildTime`) for anything else. +This registration step is not run automatically by `helm install`/`upgrade` +— rerun it manually whenever you roll a new worker image or version. diff --git a/deploy/helm/scope/templates/workers/coder-acp-claude-code/deployment.yaml b/deploy/helm/scope/templates/workers/coder-acp-claude-code/deployment.yaml new file mode 100644 index 000000000..cabb9de64 --- /dev/null +++ b/deploy/helm/scope/templates/workers/coder-acp-claude-code/deployment.yaml @@ -0,0 +1,79 @@ +{{- if .Values.workers.coderAcpClaudeCode.enabled -}} +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "scope.componentName" (dict "component" "coder-acp-claude-code" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + app.kubernetes.io/component: coder-acp-claude-code +spec: + replicas: {{ .Values.workers.coderAcpClaudeCode.replicas }} + selector: + matchLabels: + {{- include "scope.selectorLabels" (dict "component" "coder-acp-claude-code" "context" .) | nindent 6 }} + template: + metadata: + labels: + {{- include "scope.selectorLabels" (dict "component" "coder-acp-claude-code" "context" .) | nindent 8 }} + azure.workload.identity/use: "true" + spec: + serviceAccountName: {{ include "scope.serviceAccountName" . }} + {{- with .Values.global.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + containers: + # Pure background Storage Queue consumer — no HTTP server/port, so + # no Service or readiness/liveness HTTP probes (see + # apps/workers/coder-acp-claude-code/src/index.ts). + # + # NOT templated here (see docs/architecture/worker-requirements.md + # and docs/architecture/kubedock.md): the MCPJungle MCP-gateway + # sidecar (MCP server support), the AI Gateway/DevProxy sidecar (HAR + # capture), and Kubedock (in-task `docker build`/`docker run` + # support). The worker runs without them — scenarios that need MCP + # servers or in-task Docker access are not yet supported by this + # chart. Follow up once those sidecars have a templated pattern. + - name: coder-acp-claude-code + image: {{ include "scope.image" (dict "image" .Values.workers.coderAcpClaudeCode.image "context" .) }} + imagePullPolicy: {{ .Values.global.imagePullPolicy }} + envFrom: + - configMapRef: + name: {{ include "scope.componentName" (dict "component" "common-env" "context" .) }} + - secretRef: + # Supplies MONGO_CONNECTION_STRING/REDIS_PASSWORD always, and + # ANTHROPIC_API_KEY as a fallback only if + # secrets.keyVaultSecretNames.anthropicApiKey is populated. + # The worker prefers acquiring its Anthropic credential + # dynamically from Token Manager at runtime (see "Read + # credentials via Token Manager" in + # docs/architecture/worker-requirements.md) — register one + # via the Portal's "Create Token" page or `POST + # /api/v1/keys` instead of populating this secret, unless + # you specifically want the static-env-var fallback. + name: {{ .Values.secrets.k8sSecretName }} + env: + - name: WORKER_NAME + value: coder-acp-claude-code + # Must exactly match the agentVersion registered via + # scripts/register-agent.sh (see deploy/helm/scope/README.md + # "Phase 3") — the queue processor refuses to start otherwise. + - name: SCOPE_AGENT_VERSION + value: {{ .Values.workers.coderAcpClaudeCode.agentVersion | quote }} + - name: QUEUE_NAME + value: {{ .Values.workers.coderAcpClaudeCode.queueName | quote }} + volumeMounts: + - name: kv-secrets + mountPath: /mnt/secrets + readOnly: true + resources: + {{- toYaml .Values.workers.coderAcpClaudeCode.resources | nindent 12 }} + volumes: + - name: kv-secrets + csi: + driver: secrets-store.csi.k8s.io + readOnly: true + volumeAttributes: + secretProviderClass: {{ include "scope.componentName" (dict "component" "azure-kv" "context" .) }} +{{- end -}} diff --git a/deploy/helm/scope/templates/workers/coder-acp-copilot/deployment.yaml b/deploy/helm/scope/templates/workers/coder-acp-copilot/deployment.yaml new file mode 100644 index 000000000..c47fa84d2 --- /dev/null +++ b/deploy/helm/scope/templates/workers/coder-acp-copilot/deployment.yaml @@ -0,0 +1,78 @@ +{{- if .Values.workers.coderAcpCopilot.enabled -}} +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "scope.componentName" (dict "component" "coder-acp-copilot" "context" .) }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + app.kubernetes.io/component: coder-acp-copilot +spec: + replicas: {{ .Values.workers.coderAcpCopilot.replicas }} + selector: + matchLabels: + {{- include "scope.selectorLabels" (dict "component" "coder-acp-copilot" "context" .) | nindent 6 }} + template: + metadata: + labels: + {{- include "scope.selectorLabels" (dict "component" "coder-acp-copilot" "context" .) | nindent 8 }} + azure.workload.identity/use: "true" + spec: + serviceAccountName: {{ include "scope.serviceAccountName" . }} + {{- with .Values.global.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + containers: + # Pure background Storage Queue consumer — no HTTP server/port, so + # no Service or readiness/liveness HTTP probes (see + # apps/workers/coder-acp-copilot/src/index.ts). + # + # NOT templated here (see docs/architecture/worker-requirements.md + # and docs/architecture/kubedock.md): the MCPJungle MCP-gateway + # sidecar (MCP server support), the AI Gateway/DevProxy sidecar (HAR + # capture), and Kubedock (in-task `docker build`/`docker run` + # support). The worker runs without them — scenarios that need MCP + # servers or in-task Docker access are not yet supported by this + # chart. Follow up once those sidecars have a templated pattern. + - name: coder-acp-copilot + image: {{ include "scope.image" (dict "image" .Values.workers.coderAcpCopilot.image "context" .) }} + imagePullPolicy: {{ .Values.global.imagePullPolicy }} + envFrom: + - configMapRef: + name: {{ include "scope.componentName" (dict "component" "common-env" "context" .) }} + - secretRef: + # Supplies MONGO_CONNECTION_STRING/REDIS_PASSWORD always, and + # GITHUB_TOKEN as a fallback only if secrets.keyVaultSecretNames.githubToken + # is populated. The worker prefers acquiring its GitHub + # credential dynamically from Token Manager at runtime (see + # "Read credentials via Token Manager" in + # docs/architecture/worker-requirements.md) — register one + # via the Portal's "Create Token" page or `POST + # /api/v1/keys` instead of populating this secret, unless + # you specifically want the static-env-var fallback. + name: {{ .Values.secrets.k8sSecretName }} + env: + - name: WORKER_NAME + value: coder-acp-copilot + # Must exactly match the agentVersion registered via + # scripts/register-agent.sh (see deploy/helm/scope/README.md + # "Phase 3") — the queue processor refuses to start otherwise. + - name: SCOPE_AGENT_VERSION + value: {{ .Values.workers.coderAcpCopilot.agentVersion | quote }} + - name: QUEUE_NAME + value: {{ .Values.workers.coderAcpCopilot.queueName | quote }} + volumeMounts: + - name: kv-secrets + mountPath: /mnt/secrets + readOnly: true + resources: + {{- toYaml .Values.workers.coderAcpCopilot.resources | nindent 12 }} + volumes: + - name: kv-secrets + csi: + driver: secrets-store.csi.k8s.io + readOnly: true + volumeAttributes: + secretProviderClass: {{ include "scope.componentName" (dict "component" "azure-kv" "context" .) }} +{{- end -}} diff --git a/deploy/helm/scope/values.yaml b/deploy/helm/scope/values.yaml index 459fb1bbc..36515693c 100644 --- a/deploy/helm/scope/values.yaml +++ b/deploy/helm/scope/values.yaml @@ -208,6 +208,14 @@ workers: registry: "" repository: scope-coder-acp-copilot tag: "" + queueName: queue-coder-acp-copilot + # Must match the `agentVersion` you register via + # scripts/register-agent.sh (see deploy/helm/scope/README.md "Phase 3"). + # apps/workers/coder-acp-copilot/agent-version.dev.yaml's "copilot-dev" + # is a dev-only placeholder — set this to a real version string + # (and register a matching, non-dev agent-version manifest) for + # anything beyond chart smoke-testing. + agentVersion: copilot-dev replicas: 1 resources: requests: { cpu: 500m, memory: 512Mi } @@ -218,6 +226,10 @@ workers: registry: "" repository: scope-coder-acp-claude-code tag: "" + queueName: queue-coder-acp-claude-code + # See workers.coderAcpCopilot.agentVersion above for the registration + # contract this must satisfy. + agentVersion: claude-agent-acp-dev-sdk-dev replicas: 1 resources: requests: { cpu: 500m, memory: 512Mi } From f94998bd13249013d191d9f6dfb3d6f2aaf82327 Mon Sep 17 00:00:00 2001 From: Josh Duffney Date: Fri, 9 Oct 2026 11:08:55 -0500 Subject: [PATCH 07/18] Add bootstrap-workers.sh to automate Phase 3 ACP worker setup Wraps building/pushing both ACP worker images, prompting for and registering GitHub/Anthropic credentials via the Token Manager API, helm upgrade to enable the worker Deployments, and agent registration into a single idempotent command, eliminating the prior multi-step manual az acr build + helm upgrade --set + register-agent.sh flow. Validated end-to-end against the live test AKS cluster: both worker images built via az acr build, helm upgrade enabled both Deployments, both pods came up Running and connected to Cosmos DB/Redis/Storage queues, and both agent types registered successfully via the API. Updated deploy/helm/scope/README.md's Phase 3 section to recommend the script as the primary path, keeping the underlying helm upgrade mechanics as a "what it does under the hood" reference. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff --- deploy/helm/scope/README.md | 100 +++++------ scripts/bootstrap-workers.sh | 317 +++++++++++++++++++++++++++++++++++ 2 files changed, 369 insertions(+), 48 deletions(-) create mode 100755 scripts/bootstrap-workers.sh diff --git a/deploy/helm/scope/README.md b/deploy/helm/scope/README.md index ce88614cb..f519b9c7d 100644 --- a/deploy/helm/scope/README.md +++ b/deploy/helm/scope/README.md @@ -16,17 +16,16 @@ Deploying Scope end to end is three separate steps: [`.github/workflows/publish-images.yml`](../../../.github/workflows/publish-images.yml)) by default, so no registry build step is required for this phase — AKS pulls these public images directly. -3. **Build/push the ACP coding-agent workers separately** — - `coder-acp-copilot` and `coder-acp-claude-code` are *not* installed by - this chart (they need the Kubedock/MCP-gateway sidecar pattern described - in [`docs/architecture/kubedock.md`](../../docs/architecture/kubedock.md), - which isn't templated here yet). Build and push those two images to the - Bicep-provisioned **ACR** (`az acr build`), then register the - GitHub/Anthropic credentials they need — either directly in Key Vault or - via the running token-manager's admin API — and enable them with - `helm upgrade` once their manifests exist. See - ["Phase 3: ACP workers"](#phase-3-acp-coding-agent-workers-not-automated) - below. +3. **Bring the ACP coding-agent workers online separately** — + `coder-acp-copilot` and `coder-acp-claude-code` are disabled by default + in this chart (no MCPJungle/AI Gateway/Kubedock sidecars are templated + yet — see [`docs/architecture/kubedock.md`](../../docs/architecture/kubedock.md) + for the intended production pattern). Run + `scripts/bootstrap-workers.sh --acr ` to build/push both worker + images, prompt for and register their GitHub/Anthropic credentials, + `helm upgrade` to enable the worker Deployments this chart already ships, + and register both agent types with the API — all in one command. See + ["Phase 3: ACP workers"](#phase-3-acp-coding-agent-workers) below. This chart only automates phase 2. @@ -36,12 +35,11 @@ This chart only automates phase 2. - Populating bring-your-own secrets in Key Vault (GitHub token, Anthropic API key, etc.) — add those yourself before or after installing this chart (see [Secrets](#secrets) below). -- Building/pushing the two ACP coding-agent worker images, or deploying them - — that's the separate "Phase 3" step above; this chart disables both by - default (`workers.coderAcpCopilot.enabled` / `workers.coderAcpClaudeCode.enabled`) - and ships no Deployment templates for them yet. -- Registering agent types with the API (`scripts/register-agent.sh`) — a - manual Phase 3 step alongside enabling the workers. +- Building/pushing the two ACP coding-agent worker images, enabling their + Deployments, or registering their agent types with the API — that's the + separate "Phase 3" step above, disabled by default + (`workers.coderAcpCopilot.enabled` / `workers.coderAcpClaudeCode.enabled`) + and automated end-to-end by `scripts/bootstrap-workers.sh`. - TLS/cert-manager or DNS — bring your own ingress controller and certificates. ## Prerequisites @@ -166,36 +164,41 @@ All Azure-dependent Pods run under the shared ServiceAccount named by `azure.serviceAccountName`, which must match the Bicep template's federated subject. -## Phase 3: ACP coding-agent workers (not automated) +## Phase 3: ACP coding-agent workers -Once the core app is up, bring the two ACP workers online separately: +Once the core app is up, bring the two ACP workers online with one command: ```bash -az acr build --registry --image scope-coder-acp-copilot:latest \ - -f apps/workers/coder-acp-copilot/Dockerfile \ - --build-arg COPILOT_CLI_VERSION=$(grep COPILOT_CLI_VERSION apps/workers/coder-acp-copilot/versions.env | cut -d= -f2) . - -az acr build --registry --image scope-coder-acp-claude-code:latest \ - -f apps/workers/coder-acp-claude-code/Dockerfile \ - --build-arg CLAUDE_CODE_ACP_VERSION=$(grep CLAUDE_CODE_ACP_VERSION apps/workers/coder-acp-claude-code/versions.env | cut -d= -f2) \ - --build-arg CLAUDE_AGENT_SDK_VERSION=$(grep CLAUDE_AGENT_SDK_VERSION apps/workers/coder-acp-claude-code/versions.env | cut -d= -f2) . +./scripts/bootstrap-workers.sh --acr ``` -Then register the GitHub/Anthropic credentials the workers need. The -recommended path is the Portal's **Create Token** page (or a direct -`POST /api/v1/keys` call against the API) — this writes straight to Key -Vault via the token-manager's workload identity and requires no chart -changes. Populating `secrets.keyVaultSecretNames.githubToken`/ -`.anthropicApiKey` in Key Vault yourself works too, as a static fallback the -worker falls back to only if Token Manager has no credential registered. - -Point each worker's `values.yaml` entry at the image you just built -(`workers.coderAcpCopilot.image.registry`/`workers.coderAcpClaudeCode.image.registry` -= your ACR login server, `.tag` = the tag you pushed), set `.enabled: true`, -and `helm upgrade` — this chart does template a Deployment for each worker +This builds/pushes both worker images via `az acr build`, interactively +prompts for (and registers via the Token Manager API) the GitHub PAT and +Anthropic API key they need, runs `helm upgrade --reuse-values` to point +each worker's `values.yaml` image fields at the image it just built and +flip `workers..enabled: true`, waits for both Deployments to roll +out, and registers both agent types with the API +(`scripts/register-agent.sh`). Run `./scripts/bootstrap-workers.sh --help` +for all flags (`--skip-build`, `--skip-secrets`, `--skip-copilot`, +`--skip-claude-code`, `--skip-agent-registration`, `-y`/`--yes` for +non-interactive use, custom `--namespace`/`--release`/`--chart`/`--image-tag`). + +Credentials can also be registered manually instead of through the script's +prompts: the Portal's **Create Token** page (or a direct `POST /api/v1/keys` +call against the API) writes straight to Key Vault via the token-manager's +workload identity and requires no chart changes. Populating +`secrets.keyVaultSecretNames.githubToken`/`.anthropicApiKey` in Key Vault +yourself works too, as a static fallback the worker falls back to only if +Token Manager has no credential registered. + +### What the script does under the hood + +Each worker's Deployment is templated by this chart (`templates/workers/coder-acp-copilot/`, `templates/workers/coder-acp-claude-code/`), reusing the common env -ConfigMap/Secret and Key Vault CSI mount the rest of the app uses. These are +ConfigMap/Secret and Key Vault CSI mount the rest of the app uses — the +script just flips `workers..enabled: true` and sets +`workers..image.registry`/`.tag` via `helm upgrade --set`. These are minimal background Storage Queue consumers only: no MCPJungle MCP-gateway sidecar (MCP server support), no AI Gateway/DevProxy sidecar (HAR capture), and no Kubedock (in-task `docker build`/`docker run` support) — see @@ -203,12 +206,13 @@ and no Kubedock (in-task `docker build`/`docker run` support) — see that last one. Scenarios needing those features aren't supported by this chart yet. -Finally, register each agent type with the API -(`scripts/register-agent.sh `). `workers.coderAcpCopilot.agentVersion`/`.coderAcpClaudeCode.agentVersion` -must match the `agentVersion` in whatever version manifest you register — -the defaults point at the `-dev` manifests under each worker's directory, -intended for chart smoke-testing only; use a real version manifest (and -matching `imageTag`/`queueName`/`gitCommit`/`buildTime`) for anything else. -This registration step is not run automatically by `helm install`/`upgrade` -— rerun it manually whenever you roll a new worker image or version. +must match the `agentVersion` in whatever version manifest gets registered +— the script's defaults point at the `-dev` manifests under each worker's +directory, intended for chart smoke-testing only; pass a real version +manifest (and matching `imageTag`/`queueName`/`gitCommit`/`buildTime`) for +anything beyond that, and keep `values.yaml`'s `agentVersion` fields in +sync. Agent registration is not run automatically by `helm install`/ +`upgrade` on its own — rerun `scripts/bootstrap-workers.sh` (or +`scripts/register-agent.sh` directly) whenever you roll a new worker image +or version. diff --git a/scripts/bootstrap-workers.sh b/scripts/bootstrap-workers.sh new file mode 100755 index 000000000..dadd80029 --- /dev/null +++ b/scripts/bootstrap-workers.sh @@ -0,0 +1,317 @@ +#!/usr/bin/env bash +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# One-shot setup for the two ACP coding-agent workers (coder-acp-copilot, +# coder-acp-claude-code) on top of an already-`helm install`ed Scope +# release (see deploy/helm/scope/README.md "Phase 1/2"). This is the +# single entry point for "Phase 3": it builds + pushes both worker images +# to your ACR, registers the GitHub/Anthropic credentials they need with +# Token Manager, flips them on via `helm upgrade`, and registers each agent +# with the API. You should not need to run any `az acr build`/`helm +# upgrade --set workers...`/`register-agent.sh` commands by hand — this +# script wraps all of them. +# +# Usage: +# scripts/bootstrap-workers.sh --acr [options] +# +# Options: +# --acr ACR name or login server (required; e.g. myacr +# or myacr.azurecr.io) +# --namespace Kubernetes namespace (default: scope) +# --release Helm release name (default: scope) +# --chart Helm chart path (default: deploy/helm/scope, +# relative to repo root) +# --image-tag Tag to build/push/deploy (default: latest) +# --skip-copilot Skip the coder-acp-copilot worker entirely +# --skip-claude-code Skip the coder-acp-claude-code worker entirely +# --skip-build Reuse already-pushed images; skip `az acr build` +# --skip-secrets Skip the credential prompts/registration step +# --skip-agent-registration +# Skip the scripts/register-agent.sh step +# -y, --yes Non-interactive: never prompt (requires +# GITHUB_TOKEN/ANTHROPIC_API_KEY env vars, or +# use --skip-secrets) +# +# Environment variables (read instead of prompting, if set): +# GITHUB_TOKEN, ANTHROPIC_API_KEY +# +# Credentials are sent once to the already-running Token Manager over a +# temporary `kubectl port-forward` to the `api` Service and are never +# written to disk by this script. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" + +NAMESPACE="scope" +RELEASE_NAME="scope" +CHART_PATH="deploy/helm/scope" +IMAGE_TAG="latest" +ACR_NAME="" +SKIP_COPILOT=false +SKIP_CLAUDE_CODE=false +SKIP_BUILD=false +SKIP_SECRETS=false +SKIP_AGENT_REGISTRATION=false +NON_INTERACTIVE=false + +log() { printf '\n==> %s\n' "$1"; } +die() { + printf 'error: %s\n' "$1" >&2 + exit 1 +} + +usage() { + sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' +} + +while [ "$#" -gt 0 ]; do + case "$1" in + --acr) + ACR_NAME="$2" + shift 2 + ;; + --namespace) + NAMESPACE="$2" + shift 2 + ;; + --release) + RELEASE_NAME="$2" + shift 2 + ;; + --chart) + CHART_PATH="$2" + shift 2 + ;; + --image-tag) + IMAGE_TAG="$2" + shift 2 + ;; + --skip-copilot) + SKIP_COPILOT=true + shift + ;; + --skip-claude-code) + SKIP_CLAUDE_CODE=true + shift + ;; + --skip-build) + SKIP_BUILD=true + shift + ;; + --skip-secrets) + SKIP_SECRETS=true + shift + ;; + --skip-agent-registration) + SKIP_AGENT_REGISTRATION=true + shift + ;; + -y | --yes) + NON_INTERACTIVE=true + shift + ;; + -h | --help) + usage + exit 0 + ;; + *) + die "unknown argument: $1 (see --help)" + ;; + esac +done + +[ -n "$ACR_NAME" ] || die "--acr is required (ACR name or login server)" +ACR_LOGIN_SERVER="$ACR_NAME" +case "$ACR_LOGIN_SERVER" in + *.azurecr.io) ;; + *) ACR_LOGIN_SERVER="${ACR_NAME}.azurecr.io" ;; +esac +ACR_SHORT_NAME="${ACR_LOGIN_SERVER%%.*}" + +for bin in az helm kubectl curl node; do + command -v "$bin" >/dev/null 2>&1 || die "$bin is required on PATH" +done +if [ "$SKIP_AGENT_REGISTRATION" = false ] && ! command -v yq >/dev/null 2>&1; then + die "yq is required for agent registration (brew install yq), or pass --skip-agent-registration" +fi + +# ----------------------------------------------------------------------------- +# 1. Prompt for worker credentials up front (before any build/deploy work), +# so the whole run can proceed unattended once started. +# ----------------------------------------------------------------------------- +if [ "$SKIP_SECRETS" = false ]; then + if [ -z "${GITHUB_TOKEN:-}" ] && [ "$SKIP_COPILOT" = false ]; then + if [ "$NON_INTERACTIVE" = true ]; then + die "GITHUB_TOKEN is required (non-interactive mode); export it, use --skip-copilot, or --skip-secrets" + fi + echo "coder-acp-copilot needs a GitHub token (classic or fine-grained PAT) to drive the Copilot CLI." + echo "Leave blank to skip registering a credential now (you can add one later via the Portal's 'Create Token' page)." + read -r -s -p "GitHub token: " GITHUB_TOKEN + echo + fi + if [ -z "${ANTHROPIC_API_KEY:-}" ] && [ "$SKIP_CLAUDE_CODE" = false ]; then + if [ "$NON_INTERACTIVE" = true ]; then + die "ANTHROPIC_API_KEY is required (non-interactive mode); export it, use --skip-claude-code, or --skip-secrets" + fi + echo "coder-acp-claude-code needs an Anthropic API key to drive the Claude Agent SDK." + echo "Leave blank to skip registering a credential now (you can add one later via the Portal's 'Create Token' page)." + read -r -s -p "Anthropic API key: " ANTHROPIC_API_KEY + echo + fi +fi +GITHUB_TOKEN="${GITHUB_TOKEN:-}" +ANTHROPIC_API_KEY="${ANTHROPIC_API_KEY:-}" + +cd "$REPO_ROOT" + +# ----------------------------------------------------------------------------- +# 2. Build + push worker images to ACR (az acr build — no local Docker +# needed). +# ----------------------------------------------------------------------------- +if [ "$SKIP_BUILD" = false ]; then + if [ "$SKIP_COPILOT" = false ]; then + log "Building coder-acp-copilot:$IMAGE_TAG on $ACR_SHORT_NAME" + COPILOT_CLI_VERSION=$(grep '^COPILOT_CLI_VERSION=' apps/workers/coder-acp-copilot/versions.env | cut -d= -f2) + az acr build --registry "$ACR_SHORT_NAME" --image "scope-coder-acp-copilot:$IMAGE_TAG" \ + -f apps/workers/coder-acp-copilot/Dockerfile \ + --build-arg "COPILOT_CLI_VERSION=$COPILOT_CLI_VERSION" . + fi + if [ "$SKIP_CLAUDE_CODE" = false ]; then + log "Building coder-acp-claude-code:$IMAGE_TAG on $ACR_SHORT_NAME" + CLAUDE_CODE_ACP_VERSION=$(grep '^CLAUDE_CODE_ACP_VERSION=' apps/workers/coder-acp-claude-code/versions.env | cut -d= -f2) + CLAUDE_AGENT_SDK_VERSION=$(grep '^CLAUDE_AGENT_SDK_VERSION=' apps/workers/coder-acp-claude-code/versions.env | cut -d= -f2) + az acr build --registry "$ACR_SHORT_NAME" --image "scope-coder-acp-claude-code:$IMAGE_TAG" \ + -f apps/workers/coder-acp-claude-code/Dockerfile \ + --build-arg "CLAUDE_CODE_ACP_VERSION=$CLAUDE_CODE_ACP_VERSION" \ + --build-arg "CLAUDE_AGENT_SDK_VERSION=$CLAUDE_AGENT_SDK_VERSION" . + fi +else + log "Skipping image build (--skip-build); assuming $ACR_LOGIN_SERVER already has the images tagged $IMAGE_TAG" +fi + +# ----------------------------------------------------------------------------- +# 3. Temporary port-forward to the API, used for both credential +# registration and agent registration below. +# ----------------------------------------------------------------------------- +API_LOCAL_PORT=18080 +API_URL="http://127.0.0.1:$API_LOCAL_PORT" +PORT_FORWARD_PID="" + +start_port_forward() { + [ -n "$PORT_FORWARD_PID" ] && return 0 + log "Port-forwarding svc/api -n $NAMESPACE -> localhost:$API_LOCAL_PORT" + kubectl -n "$NAMESPACE" port-forward svc/api "$API_LOCAL_PORT:80" >/tmp/scope-bootstrap-workers-pf.log 2>&1 & + PORT_FORWARD_PID=$! + for _ in $(seq 1 30); do + if curl -sf -o /dev/null "$API_URL/health"; then + return 0 + fi + sleep 1 + done + die "api did not become reachable through the port-forward (see /tmp/scope-bootstrap-workers-pf.log)" +} + +stop_port_forward() { + if [ -n "$PORT_FORWARD_PID" ]; then + kill "$PORT_FORWARD_PID" >/dev/null 2>&1 || true + wait "$PORT_FORWARD_PID" 2>/dev/null || true + PORT_FORWARD_PID="" + fi +} +trap stop_port_forward EXIT + +register_secret() { + type="$1" + value="$2" + label="$3" + [ -n "$value" ] || { + echo "No $label provided — skipping credential registration (register one later via the Portal's 'Create Token' page)." + return 0 + } + start_port_forward + log "Registering $label with Token Manager" + payload=$(SCOPE_SECRET_TYPE="$type" SCOPE_SECRET_VALUE="$value" node -e \ + 'process.stdout.write(JSON.stringify({type: process.env.SCOPE_SECRET_TYPE, value: process.env.SCOPE_SECRET_VALUE}))') + status=$(curl -sS -o /tmp/scope-bootstrap-workers-register.json -w '%{http_code}' \ + -X POST "$API_URL/api/v1/keys" \ + -H 'Content-Type: application/json' \ + --data-binary "$payload") + case "$status" in + 2??) echo "Registered $label." ;; + *) + echo "Failed to register $label (HTTP $status):" >&2 + cat /tmp/scope-bootstrap-workers-register.json >&2 + return 1 + ;; + esac +} + +if [ "$SKIP_SECRETS" = false ]; then + [ "$SKIP_COPILOT" = false ] && register_secret "github-pat-classic" "$GITHUB_TOKEN" "GitHub token" + [ "$SKIP_CLAUDE_CODE" = false ] && register_secret "anthropic-api-key" "$ANTHROPIC_API_KEY" "Anthropic API key" +fi + +# ----------------------------------------------------------------------------- +# 4. Enable the worker Deployments via `helm upgrade` (chart already has +# them templated, gated on workers..enabled — see +# deploy/helm/scope/templates/workers/). +# ----------------------------------------------------------------------------- +HELM_SET_ARGS=() +if [ "$SKIP_COPILOT" = false ]; then + HELM_SET_ARGS+=( + --set "workers.coderAcpCopilot.enabled=true" + --set "workers.coderAcpCopilot.image.registry=$ACR_LOGIN_SERVER" + --set "workers.coderAcpCopilot.image.tag=$IMAGE_TAG" + ) +fi +if [ "$SKIP_CLAUDE_CODE" = false ]; then + HELM_SET_ARGS+=( + --set "workers.coderAcpClaudeCode.enabled=true" + --set "workers.coderAcpClaudeCode.image.registry=$ACR_LOGIN_SERVER" + --set "workers.coderAcpClaudeCode.image.tag=$IMAGE_TAG" + ) +fi + +if [ "${#HELM_SET_ARGS[@]}" -gt 0 ]; then + log "helm upgrade $RELEASE_NAME (enabling workers)" + helm upgrade "$RELEASE_NAME" "$CHART_PATH" -n "$NAMESPACE" --reuse-values "${HELM_SET_ARGS[@]}" --wait --timeout 5m + + log "Waiting for worker Deployments to roll out" + [ "$SKIP_COPILOT" = false ] && kubectl -n "$NAMESPACE" rollout status deployment/"$RELEASE_NAME"-coder-acp-copilot --timeout=5m + [ "$SKIP_CLAUDE_CODE" = false ] && kubectl -n "$NAMESPACE" rollout status deployment/"$RELEASE_NAME"-coder-acp-claude-code --timeout=5m +fi + +# ----------------------------------------------------------------------------- +# 5. Register each enabled agent with the API. Uses the *-dev.yaml version +# manifests shipped in-repo — fine for chart smoke-testing; for a real +# deployment, register a version manifest whose imageTag/queueName match +# what you actually built above, and keep +# workers..agentVersion (values.yaml) in sync with its agentVersion. +# ----------------------------------------------------------------------------- +if [ "$SKIP_AGENT_REGISTRATION" = false ]; then + start_port_forward + if [ "$SKIP_COPILOT" = false ]; then + log "Registering coder-acp-copilot agent" + "$REPO_ROOT/scripts/register-agent.sh" "$API_URL" \ + apps/workers/coder-acp-copilot/agent.yaml \ + --available true \ + apps/workers/coder-acp-copilot/agent-version.dev.yaml + fi + if [ "$SKIP_CLAUDE_CODE" = false ]; then + log "Registering coder-acp-claude-code agent" + "$REPO_ROOT/scripts/register-agent.sh" "$API_URL" \ + apps/workers/coder-acp-claude-code/agent.yaml \ + --available true \ + apps/workers/coder-acp-claude-code/agent-version.dev.yaml + fi +fi + +stop_port_forward +trap - EXIT + +log "Done." +[ "$SKIP_COPILOT" = false ] && echo " coder-acp-copilot: $ACR_LOGIN_SERVER/scope-coder-acp-copilot:$IMAGE_TAG" +[ "$SKIP_CLAUDE_CODE" = false ] && echo " coder-acp-claude-code: $ACR_LOGIN_SERVER/scope-coder-acp-claude-code:$IMAGE_TAG" +echo "Check pod status with: kubectl -n $NAMESPACE get pods -l 'app.kubernetes.io/component in (coder-acp-copilot,coder-acp-claude-code)'" From c9613a05da3874028610ce82ed8dce8e7159da48 Mon Sep 17 00:00:00 2001 From: Josh Duffney Date: Fri, 9 Oct 2026 11:20:02 -0500 Subject: [PATCH 08/18] Fix criteria/prompt-feature import missing projectId; doc sample-scenario quickstart MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `scope criteria import` and `scope prompt-feature import` never sent the projectId the API's /criteria/seed and /prompt-features/seed routes require (ProjectIdQuerySchema), so both commands always failed with a 400 against any real deployment. Confirmed live against the test cluster before and after the fix. Added `--project ` following the same requireProjectId pattern already used by every other criteria/prompt-feature subcommand in these files, and verified by importing all 14 config/criteria samples into the test env's "Initial Project" end to end. Also documents, in deploy/helm/scope/README.md, how to try a sample scenario after install — explicitly NOT as a Helm hook/Job: criteria are project-scoped business data, not infrastructure, and no project exists until db-migrate's "Initial Project" migration has run, so baking a project assumption into the chart would be wrong for anyone bringing their own projects. Scenarios/personas aren't server-side data at all (read straight from config/ by `run submit`), so there's nothing to import for those either way. Fixed a stale doc bug in passing: README said db-migrate is a pre-install,pre-upgrade hook; the template (and its own inline comment) say post-install,post-upgrade. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff --- .../__snapshots__/cli-commands.test.ts.snap | 2 + apps/cli/src/commands/criteria.ts | 3 + apps/cli/src/commands/prompt-feature.ts | 3 + deploy/helm/scope/README.md | 56 ++++++++++++++++++- 4 files changed, 62 insertions(+), 2 deletions(-) diff --git a/apps/cli/src/__snapshots__/cli-commands.test.ts.snap b/apps/cli/src/__snapshots__/cli-commands.test.ts.snap index 656b9031c..d2938fe48 100644 --- a/apps/cli/src/__snapshots__/cli-commands.test.ts.snap +++ b/apps/cli/src/__snapshots__/cli-commands.test.ts.snap @@ -171,6 +171,7 @@ exports[`CLI command registration > full command tree matches snapshot 1`] = ` "import": { "options": [ "--dry-run", + "--project", "--url", ], }, @@ -533,6 +534,7 @@ exports[`CLI command registration > full command tree matches snapshot 1`] = ` "import": { "options": [ "--dry-run", + "--project", "--url", ], }, diff --git a/apps/cli/src/commands/criteria.ts b/apps/cli/src/commands/criteria.ts index e59d0d79e..467f91eed 100644 --- a/apps/cli/src/commands/criteria.ts +++ b/apps/cli/src/commands/criteria.ts @@ -427,8 +427,10 @@ criteria .argument("", "Path to a .yaml file or a directory of .yaml files") .option("--dry-run", "Preview what would be imported without sending to API") .option("-u, --url ", "API base URL", getDefaultApiUrl()) + .option("--project ", "Project ID for scoped operations (overrides SCOPE_PROJECT and the saved selection)") .action(async (inputPath: string, options) => { try { + const projectId = requireProjectId(options.project); const absPath = resolve(process.env.INIT_CWD || process.cwd(), inputPath); if (!existsSync(absPath)) { console.error(errorText(`Path not found: ${absPath}`)); @@ -512,6 +514,7 @@ criteria method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ criteria: allCriteria }), + projectId, }); if (!response.ok) { diff --git a/apps/cli/src/commands/prompt-feature.ts b/apps/cli/src/commands/prompt-feature.ts index 24b3a8339..56d1e4a07 100644 --- a/apps/cli/src/commands/prompt-feature.ts +++ b/apps/cli/src/commands/prompt-feature.ts @@ -242,8 +242,10 @@ promptFeature .argument("", "Path to a .yaml file or a directory of .yaml files") .option("--dry-run", "Preview what would be imported without sending to API") .option("-u, --url ", "API base URL", getDefaultApiUrl()) + .option("--project ", "Project ID for scoped operations (overrides SCOPE_PROJECT and the saved selection)") .action(async (inputPath: string, options) => { try { + const projectId = requireProjectId(options.project); const absPath = resolve(inputPath); if (!existsSync(absPath)) { console.error(errorText(`Path not found: ${absPath}`)); @@ -320,6 +322,7 @@ promptFeature method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ features: allFeatures }), + projectId, }); if (!response.ok) { diff --git a/deploy/helm/scope/README.md b/deploy/helm/scope/README.md index f519b9c7d..c30ffd20e 100644 --- a/deploy/helm/scope/README.md +++ b/deploy/helm/scope/README.md @@ -99,8 +99,8 @@ helm install scope deploy/helm/scope -n scope -f my-values.yaml This pulls all 6 core images from `global.imageRegistry` (defaults to `ghcr.io/microsoft`, i.e. `ghcr.io/microsoft/scope-api`, etc. — override with `--set global.imageRegistry=ghcr.io/` if you publish your -own fork's images). The `db-migrate` Job (a `pre-install,pre-upgrade` Helm -hook) runs automatically before the Deployments start. +own fork's images). The `db-migrate` Job (a `post-install,post-upgrade` Helm +hook) runs automatically after the Deployments are created. ## Configuration @@ -216,3 +216,55 @@ sync. Agent registration is not run automatically by `helm install`/ `upgrade` on its own — rerun `scripts/bootstrap-workers.sh` (or `scripts/register-agent.sh` directly) whenever you roll a new worker image or version. + +## Quickstart: run a sample scenario + +Scenarios and personas (`config/scenarios/`, `config/personas/`) are **not** +server-side data — `scope run submit --scenario --persona ` +reads them straight off disk, so there's nothing to import for those. The +one piece that *is* server-side and project-scoped is **criteria**: several +scenarios (the `version: v2` ones) reference reusable criteria IDs like +`has_react`/`has_typescript` instead of inlining their own, and those IDs +must exist in the target project before the judge can evaluate a run +against them. + +This seed step is deliberately **not** templated as a Helm hook. Unlike +`db-migrate`, it's project-scoped business data, not infrastructure, and no +project exists until the `db-migrate` Job's "Initial Project" migration has +run — baking a project assumption into the chart would be surprising for +anyone bringing their own projects. Run it yourself once the core app (and +at least one Phase 3 worker, to actually execute the run) is up: + +```bash +kubectl -n scope port-forward svc/api 18080:80 & + +# Discover the project to seed into (the "Initial Project" created by +# db-migrate on first install, or one of your own) +scope project list --url http://localhost:18080 + +# Import every reusable criterion (upsert — safe to rerun) +scope criteria import config/criteria --project --url http://localhost:18080 + +# Try the simplest sample scenario end to end +scope run submit \ + --scenario config/scenarios/hello-world-express-v2.yaml \ + --persona config/personas/vibe-coder.yaml \ + --worker coder-acp-copilot \ + --project \ + --url http://localhost:18080 +``` + +`--worker` must be one of the agent IDs registered in Phase 3 +(`scope agent list --url http://localhost:18080`). `hello-world-express-v2.yaml` +only needs `has_azure_doc`/`has_azure_azd` (plus their `has_iac`/`has_azure`/ +`has_cloud` ancestors); importing the whole `config/criteria` directory is +simpler than cherry-picking a dependency closure and covers every sample +scenario, including `react-snake-game-v2.yaml`. + +> **Note:** this quickstart surfaced (and this change fixes) a bug where +> `scope criteria import`/`scope prompt-feature import` never sent the +> `projectId` the API's `/criteria/seed`/`/prompt-features/seed` routes +> require, so both commands always failed with a 400. Make sure your CLI +> build includes the `--project` option on `criteria import` / +> `prompt-feature import` before following the steps above. + From 9fe2df63671f876429bd6f33528a8a0d61a109e5 Mon Sep 17 00:00:00 2001 From: Josh Duffney Date: Fri, 9 Oct 2026 12:08:04 -0500 Subject: [PATCH 09/18] Retry blob writes on transient authorization errors; fix stale --reuse-values in bootstrap-workers.sh BlobStorage: add a narrow withContainerRetry() wrapper (via the existing withRetry utility) around every snapshots-container write, retrying only on AuthorizationPermissionMismatch / 403 "not authorized" errors (never on other 4xx) with capped exponential backoff. This guards against transient RBAC-propagation races (e.g. right after createIfNotExists auto-creates a container for the first time). Note: this does NOT fix the deterministic "Snapshot upload failed" AuthorizationPermissionMismatch seen on uploadWorkspaceSnapshot / uploadSnapshotFromDirectory. Root-caused separately on the test AKS cluster: those two methods set Blob Index Tags (x-ms-tags) on upload, which requires the blobs/tags/write data action that Storage Blob Data Contributor does not grant. Confirmed via live reproduction (plain PUT succeeds, same PUT with x-ms-tags fails with 403 every time). Fix is a Bicep role upgrade (Storage Blob Data Contributor -> Storage Blob Data Owner for the workload identity) tracked in the deploy-azure-button-infra PR, not addressed by this commit. bootstrap-workers.sh: --values-file lets helm upgrade re-merge the chart's current values.yaml instead of freezing a historical --reuse-values snapshot, which was silently dropping new default keys (agentVersion, queueName) from reaching deployed workers. --- .../shared/src/storage/blob-storage.test.ts | 67 +++++++++++ packages/shared/src/storage/blob-storage.ts | 111 ++++++++++++------ scripts/bootstrap-workers.sh | 36 +++++- 3 files changed, 180 insertions(+), 34 deletions(-) diff --git a/packages/shared/src/storage/blob-storage.test.ts b/packages/shared/src/storage/blob-storage.test.ts index 651c34fe8..a72865e1f 100644 --- a/packages/shared/src/storage/blob-storage.test.ts +++ b/packages/shared/src/storage/blob-storage.test.ts @@ -17,8 +17,15 @@ const mockLogsContainerClient = { getAppendBlobClient: vi.fn(() => mockAppendBlobClient), }; +const mockBlockBlobClient = { + upload: vi.fn().mockResolvedValue(undefined), + uploadData: vi.fn().mockResolvedValue(undefined), + url: "https://test.blob.core.windows.net/snapshots/mock-blob", +}; + const mockSnapshotsContainerClient = { createIfNotExists: vi.fn().mockResolvedValue(undefined), + getBlockBlobClient: vi.fn(() => mockBlockBlobClient), }; const mockBlobServiceClient = { @@ -264,3 +271,63 @@ describe("BlobStorage — log helpers", () => { }); }); }); + +describe("BlobStorage — container-creation authorization retry", () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + function makeAuthzError(): Error & { statusCode: number; code: string } { + return Object.assign( + new Error( + "This request is not authorized to perform this operation using this permission.", + ), + { statusCode: 403, code: "AuthorizationPermissionMismatch" }, + ); + } + + it("retries uploadJson on a transient AuthorizationPermissionMismatch and succeeds", async () => { + mockBlockBlobClient.uploadData + .mockRejectedValueOnce(makeAuthzError()) + .mockResolvedValueOnce(undefined); + + const storage = makeStorage(); + const url = await storage.uploadJson("some/blob.json", { ok: true }); + + expect(mockBlockBlobClient.uploadData).toHaveBeenCalledTimes(2); + expect(url).toBe(mockBlockBlobClient.url); + }); + + it("does not retry uploadJson on an unrelated error", async () => { + const otherError = Object.assign(new Error("Boom"), { statusCode: 500 }); + mockBlockBlobClient.uploadData.mockRejectedValueOnce(otherError); + + const storage = makeStorage(); + await expect(storage.uploadJson("some/blob.json", { ok: true })).rejects.toThrow("Boom"); + + expect(mockBlockBlobClient.uploadData).toHaveBeenCalledTimes(1); + }); + + it("gives up after repeated AuthorizationPermissionMismatch failures", async () => { + vi.useFakeTimers(); + try { + mockBlockBlobClient.uploadData.mockRejectedValue(makeAuthzError()); + + const storage = makeStorage(); + const pending = storage.uploadText("some/blob.txt", "hello"); + // Attach the rejection assertion before advancing timers so the + // rejection handler is registered before fake timers resolve it. + const assertion = expect(pending).rejects.toThrow("not authorized to perform this operation"); + // Lets the retry policy's backoff timers (up to 4 retries) fire without + // waiting for real wall-clock time. + await vi.runAllTimersAsync(); + await assertion; + + // maxRetries: 4 => up to 5 total attempts + expect(mockBlockBlobClient.uploadData.mock.calls.length).toBeLessThanOrEqual(5); + expect(mockBlockBlobClient.uploadData.mock.calls.length).toBeGreaterThan(1); + } finally { + vi.useRealTimers(); + } + }); +}); diff --git a/packages/shared/src/storage/blob-storage.ts b/packages/shared/src/storage/blob-storage.ts index c9b50038b..cb97557dc 100644 --- a/packages/shared/src/storage/blob-storage.ts +++ b/packages/shared/src/storage/blob-storage.ts @@ -16,6 +16,7 @@ import { createGunzip } from "zlib"; import { extract } from "tar"; import type { LogEvent } from "../types/types.js"; import type { ToolCall } from "../har/types.js"; +import { withRetry } from "../utils/retry.js"; const SNAPSHOTS_CONTAINER = "snapshots"; const LOGS_CONTAINER = "logs"; @@ -40,6 +41,22 @@ export interface BlobStorageConfig { storageConnectionString?: string; // For local Azurite } +/** + * True for Azure Storage's `AuthorizationPermissionMismatch` error. In an + * RBAC-only (workload identity) setup this shows up as a transient failure — + * most commonly in the few seconds/minutes right after `createIfNotExists` + * auto-creates a container for the first time, where the data-plane + * authorization check for the new container hasn't fully propagated yet even + * though the account-level role assignment has existed for a long time. + * Retrying with backoff resolves it without any code or infra change. + */ +function isAuthorizationPropagationError(error: unknown): boolean { + const err = error as { statusCode?: number; code?: string; message?: string } | undefined; + if (!err) return false; + if (err.code === "AuthorizationPermissionMismatch") return true; + return err.statusCode === 403 && /not authorized to perform this operation/i.test(err.message ?? ""); +} + export class BlobStorage { private containerClient: ContainerClient; private logsContainerClient: ContainerClient; @@ -73,6 +90,22 @@ export class BlobStorage { await this.containerClient.createIfNotExists(); } + /** + * Wraps a snapshots-container write with a short retry for the + * `AuthorizationPermissionMismatch` container-creation propagation race + * described on {@link isAuthorizationPropagationError}. Non-idempotent + * writes here (blob uploads) are safe to retry: each overwrites the same + * blob name, so a retried attempt simply re-sends identical content. + */ + private async withContainerRetry(fn: () => Promise): Promise { + return withRetry(fn, { + maxRetries: 4, + baseDelayMs: 1000, + maxDelayMs: 8000, + isRetryable: isAuthorizationPropagationError, + }); + } + /** * Ensures the logs container exists. Lazily initialized — the HTTP call is made * at most once per BlobStorage instance, regardless of concurrent callers. @@ -219,9 +252,11 @@ export class BlobStorage { const blobName = `${requestId}/runs/${runId}/iteration-${iteration}/tool-calls.jsonl`; const blockBlobClient = this.containerClient.getBlockBlobClient(blobName); const body = toolCalls.map((tc) => JSON.stringify(tc)).join("\n") + "\n"; - await blockBlobClient.upload(body, Buffer.byteLength(body), { - blobHTTPHeaders: { blobContentType: "application/x-ndjson" }, - }); + await this.withContainerRetry(() => + blockBlobClient.upload(body, Buffer.byteLength(body), { + blobHTTPHeaders: { blobContentType: "application/x-ndjson" }, + }), + ); } /** @@ -328,16 +363,18 @@ export class BlobStorage { ); // Upload to blob storage - await blockBlobClient.uploadFile(archivePath, { - blobHTTPHeaders: { - blobContentType: "application/gzip", - }, - tags: { - requestId, - runId, - iteration: String(iteration), - }, - }); + await this.withContainerRetry(() => + blockBlobClient.uploadFile(archivePath, { + blobHTTPHeaders: { + blobContentType: "application/gzip", + }, + tags: { + requestId, + runId, + iteration: String(iteration), + }, + }), + ); return blockBlobClient.url; } finally { @@ -377,15 +414,17 @@ export class BlobStorage { ); // Upload to blob storage - await blockBlobClient.uploadFile(archivePath, { - blobHTTPHeaders: { - blobContentType: "application/gzip", - }, - tags: { - requestId, - iteration: String(iteration), - }, - }); + await this.withContainerRetry(() => + blockBlobClient.uploadFile(archivePath, { + blobHTTPHeaders: { + blobContentType: "application/gzip", + }, + tags: { + requestId, + iteration: String(iteration), + }, + }), + ); return blockBlobClient.url; } finally { @@ -406,11 +445,13 @@ export class BlobStorage { await this.ensureContainer(); const blockBlobClient = this.containerClient.getBlockBlobClient(blobName); - await blockBlobClient.uploadFile(filePath, { - blobHTTPHeaders: { - blobContentType: contentType, - }, - }); + await this.withContainerRetry(() => + blockBlobClient.uploadFile(filePath, { + blobHTTPHeaders: { + blobContentType: contentType, + }, + }), + ); return blockBlobClient.url; } @@ -432,9 +473,11 @@ export class BlobStorage { const body = typeof data === "string" ? data : JSON.stringify(data); const buf = Buffer.from(body, "utf-8"); const blockBlobClient = this.containerClient.getBlockBlobClient(blobName); - await blockBlobClient.uploadData(buf, { - blobHTTPHeaders: { blobContentType: "application/json" }, - }); + await this.withContainerRetry(() => + blockBlobClient.uploadData(buf, { + blobHTTPHeaders: { blobContentType: "application/json" }, + }), + ); return blockBlobClient.url; } @@ -454,9 +497,11 @@ export class BlobStorage { const buf = Buffer.from(text, "utf-8"); const blockBlobClient = this.containerClient.getBlockBlobClient(blobName); - await blockBlobClient.uploadData(buf, { - blobHTTPHeaders: { blobContentType: contentType }, - }); + await this.withContainerRetry(() => + blockBlobClient.uploadData(buf, { + blobHTTPHeaders: { blobContentType: contentType }, + }), + ); return blockBlobClient.url; } diff --git a/scripts/bootstrap-workers.sh b/scripts/bootstrap-workers.sh index dadd80029..d870d2790 100755 --- a/scripts/bootstrap-workers.sh +++ b/scripts/bootstrap-workers.sh @@ -23,6 +23,16 @@ # --chart Helm chart path (default: deploy/helm/scope, # relative to repo root) # --image-tag Tag to build/push/deploy (default: latest) +# --values-file Same values override file passed to the +# original `helm install ... -f ` (see +# README.md "Install (Phase 2)"). Strongly +# recommended: without it, this script falls back +# to `helm upgrade --reuse-values`, which freezes +# the release's previously computed values and +# will NOT pick up new default keys added to the +# chart's values.yaml since the initial install +# (this silently broke worker agentVersion/ +# queueName in testing — see README "Phase 3"). # --skip-copilot Skip the coder-acp-copilot worker entirely # --skip-claude-code Skip the coder-acp-claude-code worker entirely # --skip-build Reuse already-pushed images; skip `az acr build` @@ -48,6 +58,7 @@ NAMESPACE="scope" RELEASE_NAME="scope" CHART_PATH="deploy/helm/scope" IMAGE_TAG="latest" +VALUES_FILE="" ACR_NAME="" SKIP_COPILOT=false SKIP_CLAUDE_CODE=false @@ -88,6 +99,10 @@ while [ "$#" -gt 0 ]; do IMAGE_TAG="$2" shift 2 ;; + --values-file) + VALUES_FILE="$2" + shift 2 + ;; --skip-copilot) SKIP_COPILOT=true shift @@ -130,6 +145,10 @@ case "$ACR_LOGIN_SERVER" in esac ACR_SHORT_NAME="${ACR_LOGIN_SERVER%%.*}" +if [ -n "$VALUES_FILE" ] && [ ! -f "$VALUES_FILE" ]; then + die "--values-file not found: $VALUES_FILE" +fi + for bin in az helm kubectl curl node; do command -v "$bin" >/dev/null 2>&1 || die "$bin is required on PATH" done @@ -276,7 +295,22 @@ fi if [ "${#HELM_SET_ARGS[@]}" -gt 0 ]; then log "helm upgrade $RELEASE_NAME (enabling workers)" - helm upgrade "$RELEASE_NAME" "$CHART_PATH" -n "$NAMESPACE" --reuse-values "${HELM_SET_ARGS[@]}" --wait --timeout 5m + if [ -n "$VALUES_FILE" ]; then + # Re-merge the chart's current values.yaml with the same override file + # used at install time. This is deliberately NOT `--reuse-values`: + # that flag freezes the release's previously *computed* values and + # silently drops any new default key added to values.yaml since install + # (e.g. workers..agentVersion/queueName), because the frozen + # object simply never had that key to begin with. + helm upgrade "$RELEASE_NAME" "$CHART_PATH" -n "$NAMESPACE" -f "$VALUES_FILE" "${HELM_SET_ARGS[@]}" --wait --timeout 5m + else + echo "warning: --values-file not provided; falling back to --reuse-values." >&2 + echo " This reuses this release's frozen computed values and will NOT" >&2 + echo " pick up new chart defaults (e.g. agentVersion/queueName) added" >&2 + echo " to values.yaml since the original install. Re-run with" >&2 + echo " --values-file to avoid this." >&2 + helm upgrade "$RELEASE_NAME" "$CHART_PATH" -n "$NAMESPACE" --reuse-values "${HELM_SET_ARGS[@]}" --wait --timeout 5m + fi log "Waiting for worker Deployments to roll out" [ "$SKIP_COPILOT" = false ] && kubectl -n "$NAMESPACE" rollout status deployment/"$RELEASE_NAME"-coder-acp-copilot --timeout=5m From 59e9a9a7ac284a84447bc6f2156439da45423b01 Mon Sep 17 00:00:00 2001 From: Josh Duffney Date: Fri, 9 Oct 2026 14:11:38 -0500 Subject: [PATCH 10/18] Split the Helm chart README into a lean quickstart plus detail docs The README had grown to cover install steps, secrets architecture, worker bootstrapping internals, and a sample-scenario walkthrough all in one file. Split it so the README focuses purely on the steps needed to get the app running, and links out to detail docs for anyone who wants to go deeper: - README.md: three-phase overview, prerequisites, install steps, verify (including the port-forward commands needed to reach the API/portal without an ingress), and links to the detail docs. - CONFIGURATION.md (new): values.yaml conventions, image registry strategy, and the two bring-your-own secrets (GitHub token, Anthropic API key) - trimmed to just what's actionable for getting the app running, not the full three-pattern secrets architecture. - WORKERS.md (new): bootstrap-workers.sh details, how the chart templates worker Deployments, agent-version rules, and the Token Manager API as an alternative way to register credentials. - SAMPLE-SCENARIO.md (new): the criteria-seed + sample-run walkthrough. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff --- deploy/helm/scope/CONFIGURATION.md | 74 +++++++ deploy/helm/scope/README.md | 313 +++++++-------------------- deploy/helm/scope/SAMPLE-SCENARIO.md | 52 +++++ deploy/helm/scope/WORKERS.md | 75 +++++++ 4 files changed, 282 insertions(+), 232 deletions(-) create mode 100644 deploy/helm/scope/CONFIGURATION.md create mode 100644 deploy/helm/scope/SAMPLE-SCENARIO.md create mode 100644 deploy/helm/scope/WORKERS.md diff --git a/deploy/helm/scope/CONFIGURATION.md b/deploy/helm/scope/CONFIGURATION.md new file mode 100644 index 000000000..104ed7160 --- /dev/null +++ b/deploy/helm/scope/CONFIGURATION.md @@ -0,0 +1,74 @@ +# Configuration reference + +Details on values.yaml conventions and image registries. For the install +steps, see the [README](./README.md). + +## Values conventions + +See [`values.yaml`](./values.yaml) for the full set of configurable values. +Every service section supports: + +- `enabled` +- `image.repository`, `image.tag`, `image.registry` +- `replicas` +- `resources` + +`image.registry` overrides `global.imageRegistry` for a single image. The +workers use this (they pull from ACR, not GHCR), as does `jobs.dbMigrate` +(which needs a separate `:builder`-tagged image — see +`templates/jobs/db-migrate.yaml`). + +## Image registry strategy + +By default, the six core services (api, portal, judge, token-manager, +scheduler, report-generator) pull from `global.imageRegistry` +(`ghcr.io/microsoft` by default — public images published by +[`.github/workflows/publish-images.yml`](../../../.github/workflows/publish-images.yml)). +No registry build step is required for these. + +If you publish your own fork's images, override the registry: + +```bash +helm install scope deploy/helm/scope -n scope \ + -f my-values.yaml \ + --set global.imageRegistry=ghcr.io/ +``` + +The two ACP worker images are different — they're never published to GHCR. +`scripts/bootstrap-workers.sh` builds and pushes them to the ACR instance +`deploy/azure/` provisions. See [Workers](./WORKERS.md). + +## Bring your own secrets + +`deploy/azure/` pre-populates Key Vault with everything the core app +needs — Cosmos DB connection string, Redis access key, and so on. Two +secrets are bring-your-own, since they depend on your accounts, not the +infrastructure: + +- `githubToken` (Key Vault secret name: `github-token`) +- `anthropicApiKey` (Key Vault secret name: `anthropic-api-key`) + +The ACP workers (see [Workers](./WORKERS.md)) and the portal's GitHub +Models fallback read these directly as plain `GITHUB_TOKEN`/ +`ANTHROPIC_API_KEY` env vars. Set them with: + +```bash +az keyvault secret set --vault-name --name github-token --value +az keyvault secret set --vault-name --name anthropic-api-key --value +``` + +Then restart the pods that read them, since Key Vault values only sync on +pod start: + +```bash +kubectl rollout restart deployment -n scope +``` + +`scripts/bootstrap-workers.sh` prompts for and registers these credentials +for you — see [Workers](./WORKERS.md) for that (and a more capable +alternative through the Token Manager API). + +## Not covered + +Bring your own ingress controller, TLS/cert-manager, and DNS — this chart +doesn't configure any of them. diff --git a/deploy/helm/scope/README.md b/deploy/helm/scope/README.md index c30ffd20e..c8e5a8d71 100644 --- a/deploy/helm/scope/README.md +++ b/deploy/helm/scope/README.md @@ -1,270 +1,119 @@ # Scope Helm chart -Installs the Scope application onto the AKS cluster provisioned by -[`deploy/azure/`](../azure/README.md) (the "Deploy to Azure" Bicep template). +Deploys the Scope application onto the AKS cluster that +[`deploy/azure/`](../azure/README.md) provisions. -## Three-phase deployment +Getting Scope running is three phases: -Deploying Scope end to end is three separate steps: +| Phase | What it does | Tooling | +|---|---|---| +| 1. Infrastructure | AKS, Cosmos DB, Key Vault, Redis, Storage, ACR, networking | `deploy/azure/` (Bicep) | +| 2. Core app | api, portal, judge, token-manager, scheduler, report-generator | This chart | +| 3. Coding-agent workers | Build worker images, register secrets and agents | `scripts/bootstrap-workers.sh` | -1. **Deploy Azure infrastructure** — `deploy/azure/` (the "Deploy to Azure" - Bicep template) provisions AKS, Cosmos DB, Key Vault, Redis, Storage, ACR, - and networking. It does **not** install the app. -2. **`helm install` the core app** (this chart) — api, portal, judge, - token-manager, scheduler, and report-generator. Images are pulled from - **GHCR** (`ghcr.io//scope-`, published by - [`.github/workflows/publish-images.yml`](../../../.github/workflows/publish-images.yml)) - by default, so no registry build step is required for this phase — AKS - pulls these public images directly. -3. **Bring the ACP coding-agent workers online separately** — - `coder-acp-copilot` and `coder-acp-claude-code` are disabled by default - in this chart (no MCPJungle/AI Gateway/Kubedock sidecars are templated - yet — see [`docs/architecture/kubedock.md`](../../docs/architecture/kubedock.md) - for the intended production pattern). Run - `scripts/bootstrap-workers.sh --acr ` to build/push both worker - images, prompt for and register their GitHub/Anthropic credentials, - `helm upgrade` to enable the worker Deployments this chart already ships, - and register both agent types with the API — all in one command. See - ["Phase 3: ACP workers"](#phase-3-acp-coding-agent-workers) below. - -This chart only automates phase 2. - -## What this does **NOT** automate - -- Provisioning Azure infrastructure — that's `deploy/azure/`. -- Populating bring-your-own secrets in Key Vault (GitHub token, Anthropic API - key, etc.) — add those yourself before or after installing this chart (see - [Secrets](#secrets) below). -- Building/pushing the two ACP coding-agent worker images, enabling their - Deployments, or registering their agent types with the API — that's the - separate "Phase 3" step above, disabled by default - (`workers.coderAcpCopilot.enabled` / `workers.coderAcpClaudeCode.enabled`) - and automated end-to-end by `scripts/bootstrap-workers.sh`. -- TLS/cert-manager or DNS — bring your own ingress controller and certificates. +This README covers phases 2 and 3. For infrastructure, see the +[Bicep README](../azure/README.md). ## Prerequisites -- An AKS cluster with: - - The **Azure Key Vault Secrets Provider** CSI driver add-on enabled - (`--enable-addons azure-keyvault-secrets-provider`) - - **OIDC issuer** + **workload identity** enabled - - A user-assigned managed identity federated to - `system:serviceaccount::`, granted - `Key Vault Secrets User` on the Key Vault and - `Storage Blob Data Contributor` + `Storage Queue Data Contributor` on the - Storage account - - All of the above is exactly what `deploy/azure/main.bicep` provisions. - -- `helm` and `kubectl` configured against the target cluster. - -## Install (Phase 2) - -Read the Bicep deployment's outputs and populate a values override file: - -```bash -az deployment group show \ - --resource-group \ - --name \ - --query properties.outputs -o json -``` - -```yaml -# my-values.yaml -azure: - tenantId: - keyVaultName: - keyVaultUri: - workloadIdentityClientId: - serviceAccountName: - storageAccountName: - redisHostName: - -api: - ingress: - enabled: true - host: api.example.com - -portal: - ingress: - enabled: true - host: app.example.com -``` - -```bash -kubectl create namespace scope -helm install scope deploy/helm/scope -n scope -f my-values.yaml -``` +- Azure infrastructure from `deploy/azure/`, already deployed. It + provisions everything phase 2 needs: an AKS cluster with the Key Vault + Secrets Provider CSI driver and workload identity enabled, a federated + managed identity, and Key Vault/Storage/Cosmos DB/Redis. +- `helm` and `kubectl`, configured against the target cluster. -This pulls all 6 core images from `global.imageRegistry` (defaults to -`ghcr.io/microsoft`, i.e. `ghcr.io/microsoft/scope-api`, etc. — override with -`--set global.imageRegistry=ghcr.io/` if you publish your -own fork's images). The `db-migrate` Job (a `post-install,post-upgrade` Helm -hook) runs automatically after the Deployments are created. +## Deploy the core app -## Configuration - -See [`values.yaml`](./values.yaml) for the full set of configurable values — -every service section supports `enabled`, `image.{repository,tag,registry}`, -`replicas`, and `resources`. `image.registry` overrides `global.imageRegistry` -per-image — used by the workers (which point at ACR, not GHCR) and -`jobs.dbMigrate` (which needs a distinct `:builder`-tagged image; see -`templates/jobs/db-migrate.yaml`). - -### Secrets - -Two Azure auth patterns are used side by side: - -1. **CSI-synced secrets** (static values like connection strings and API - keys): listed in `secrets.keyVaultSecretNames`, synced from Key Vault into - a single Kubernetes `Secret` (`secrets.k8sSecretName`) by the - `SecretProviderClass`, and consumed via `envFrom.secretRef`. - `mongoConnectionString` (`cosmos-connection-string`, a full connection - string) and `redisPassword` (`redis-primary-key`, a bare access key — - pairs with the non-secret `azure.redisHostName`/`redisPort`/`redisTls` - values to form `REDIS_HOST`/`REDIS_PORT`/`REDIS_PASSWORD`/`REDIS_TLS`) are - pre-populated in Key Vault by `deploy/azure/`. `githubToken` - (`github-token`) and `anthropicApiKey` (`anthropic-api-key`) are - bring-your-own and are what the ACP workers (Phase 3) and the portal's - GitHub Models fallback (`apps/api/src/llm-token.ts`) read directly as - plain `GITHUB_TOKEN`/`ANTHROPIC_API_KEY` env vars — this is the simplest - path and bypasses token-manager entirely: +1. Get the Bicep deployment's outputs: ```bash - az keyvault secret set --vault-name --name github-token --value - az keyvault secret set --vault-name --name anthropic-api-key --value + az deployment group show \ + --resource-group \ + --name \ + --query properties.outputs -o json ``` - Pods pick these up on their next restart (`kubectl rollout restart` after - setting a secret, since the CSI driver only re-syncs on pod start/the - configured polling interval). +2. Create a values file from those outputs: + + ```yaml + # my-values.yaml + azure: + tenantId: + keyVaultName: + keyVaultUri: + workloadIdentityClientId: + serviceAccountName: + storageAccountName: + redisHostName: + + api: + ingress: + enabled: true + host: api.example.com + + portal: + ingress: + enabled: true + host: app.example.com + ``` -2. **Via the token-manager API** (more capable: multiple keys per - capability, round-robin, rotation, OAuth tokens) — register credentials - directly against the running `token-manager` service instead of writing - fixed-name Key Vault secrets: +3. Install the chart: ```bash - kubectl port-forward -n scope svc/scope-token-manager 3300:80 - curl -X POST http://localhost:3300/api/v1/keys \ - -H 'Content-Type: application/json' \ - -d '{"type":"github-oauth","value":"","capabilities":["github-models","copilot-cli"]}' + kubectl create namespace scope + helm install scope deploy/helm/scope -n scope -f my-values.yaml ``` - `token-manager` stores these itself (its own `DefaultAzureCredential` - Key Vault client — see `azure.keyVaultUri`), independent of the - `secrets.keyVaultSecretNames` list above. See `apps/token-manager/src/routes.ts` - for the full `type`/`capabilities` enum. + Images come from `ghcr.io/microsoft` by default. To use your own fork's + published images instead, add + `--set global.imageRegistry=ghcr.io/`. -3. **Direct workload identity** (in-process `DefaultAzureCredential` calls, - no secret at all): used for Blob/Queue Storage access (RBAC-only) and by - `token-manager` itself for its live Key Vault calls (option 2 above). + Database migrations run automatically as part of the install. -All Azure-dependent Pods run under the shared ServiceAccount named by -`azure.serviceAccountName`, which must match the Bicep template's federated -subject. +4. Add any bring-your-own secrets (GitHub token, Anthropic API key) that + Bicep doesn't provision. See + [Bring your own secrets](./CONFIGURATION.md#bring-your-own-secrets) for + your options. -## Phase 3: ACP coding-agent workers +## Bring workers online -Once the core app is up, bring the two ACP workers online with one command: +The ACP coding-agent workers are disabled by default. Enable them with: ```bash ./scripts/bootstrap-workers.sh --acr ``` -This builds/pushes both worker images via `az acr build`, interactively -prompts for (and registers via the Token Manager API) the GitHub PAT and -Anthropic API key they need, runs `helm upgrade --reuse-values` to point -each worker's `values.yaml` image fields at the image it just built and -flip `workers..enabled: true`, waits for both Deployments to roll -out, and registers both agent types with the API -(`scripts/register-agent.sh`). Run `./scripts/bootstrap-workers.sh --help` -for all flags (`--skip-build`, `--skip-secrets`, `--skip-copilot`, -`--skip-claude-code`, `--skip-agent-registration`, `-y`/`--yes` for -non-interactive use, custom `--namespace`/`--release`/`--chart`/`--image-tag`). - -Credentials can also be registered manually instead of through the script's -prompts: the Portal's **Create Token** page (or a direct `POST /api/v1/keys` -call against the API) writes straight to Key Vault via the token-manager's -workload identity and requires no chart changes. Populating -`secrets.keyVaultSecretNames.githubToken`/`.anthropicApiKey` in Key Vault -yourself works too, as a static fallback the worker falls back to only if -Token Manager has no credential registered. - -### What the script does under the hood - -Each worker's Deployment is templated by this chart -(`templates/workers/coder-acp-copilot/`, -`templates/workers/coder-acp-claude-code/`), reusing the common env -ConfigMap/Secret and Key Vault CSI mount the rest of the app uses — the -script just flips `workers..enabled: true` and sets -`workers..image.registry`/`.tag` via `helm upgrade --set`. These are -minimal background Storage Queue consumers only: no MCPJungle MCP-gateway -sidecar (MCP server support), no AI Gateway/DevProxy sidecar (HAR capture), -and no Kubedock (in-task `docker build`/`docker run` support) — see -`docs/architecture/kubedock.md` for the intended production pattern for -that last one. Scenarios needing those features aren't supported by this -chart yet. +This builds and pushes both worker images, prompts for their credentials, +enables their Deployments, and registers both agent types with the API. +See [Workers](./WORKERS.md) for details and script flags. -`workers.coderAcpCopilot.agentVersion`/`.coderAcpClaudeCode.agentVersion` -must match the `agentVersion` in whatever version manifest gets registered -— the script's defaults point at the `-dev` manifests under each worker's -directory, intended for chart smoke-testing only; pass a real version -manifest (and matching `imageTag`/`queueName`/`gitCommit`/`buildTime`) for -anything beyond that, and keep `values.yaml`'s `agentVersion` fields in -sync. Agent registration is not run automatically by `helm install`/ -`upgrade` on its own — rerun `scripts/bootstrap-workers.sh` (or -`scripts/register-agent.sh` directly) whenever you roll a new worker image -or version. - -## Quickstart: run a sample scenario - -Scenarios and personas (`config/scenarios/`, `config/personas/`) are **not** -server-side data — `scope run submit --scenario --persona ` -reads them straight off disk, so there's nothing to import for those. The -one piece that *is* server-side and project-scoped is **criteria**: several -scenarios (the `version: v2` ones) reference reusable criteria IDs like -`has_react`/`has_typescript` instead of inlining their own, and those IDs -must exist in the target project before the judge can evaluate a run -against them. - -This seed step is deliberately **not** templated as a Helm hook. Unlike -`db-migrate`, it's project-scoped business data, not infrastructure, and no -project exists until the `db-migrate` Job's "Initial Project" migration has -run — baking a project assumption into the chart would be surprising for -anyone bringing their own projects. Run it yourself once the core app (and -at least one Phase 3 worker, to actually execute the run) is up: +## Verify ```bash -kubectl -n scope port-forward svc/api 18080:80 & +kubectl get pods -n scope -# Discover the project to seed into (the "Initial Project" created by -# db-migrate on first install, or one of your own) -scope project list --url http://localhost:18080 +# Port-forward the API so the CLI can reach it without an ingress. +kubectl -n scope port-forward svc/api 18080:80 & +scope agent list --url http://localhost:18080 +``` -# Import every reusable criterion (upsert — safe to rerun) -scope criteria import config/criteria --project --url http://localhost:18080 +No ingress yet? Reach the portal the same way: -# Try the simplest sample scenario end to end -scope run submit \ - --scenario config/scenarios/hello-world-express-v2.yaml \ - --persona config/personas/vibe-coder.yaml \ - --worker coder-acp-copilot \ - --project \ - --url http://localhost:18080 +```bash +kubectl -n scope port-forward svc/scope-portal 5100:80 & ``` -`--worker` must be one of the agent IDs registered in Phase 3 -(`scope agent list --url http://localhost:18080`). `hello-world-express-v2.yaml` -only needs `has_azure_doc`/`has_azure_azd` (plus their `has_iac`/`has_azure`/ -`has_cloud` ancestors); importing the whole `config/criteria` directory is -simpler than cherry-picking a dependency closure and covers every sample -scenario, including `react-snake-game-v2.yaml`. +Then open . + +Then try an end-to-end run — see +[Run a sample scenario](./SAMPLE-SCENARIO.md). -> **Note:** this quickstart surfaced (and this change fixes) a bug where -> `scope criteria import`/`scope prompt-feature import` never sent the -> `projectId` the API's `/criteria/seed`/`/prompt-features/seed` routes -> require, so both commands always failed with a 400. Make sure your CLI -> build includes the `--project` option on `criteria import` / -> `prompt-feature import` before following the steps above. +## Learn more +- [Configuration reference](./CONFIGURATION.md) — values.yaml conventions, + image registry strategy, secrets options +- [Workers](./WORKERS.md) — bootstrapping the ACP coding-agent workers +- [Run a sample scenario](./SAMPLE-SCENARIO.md) — seed criteria and submit + a run +- [`values.yaml`](./values.yaml) — every configurable value +- [Bicep infrastructure README](../azure/README.md) diff --git a/deploy/helm/scope/SAMPLE-SCENARIO.md b/deploy/helm/scope/SAMPLE-SCENARIO.md new file mode 100644 index 000000000..c63964669 --- /dev/null +++ b/deploy/helm/scope/SAMPLE-SCENARIO.md @@ -0,0 +1,52 @@ +# Run a sample scenario + +Once the core app and at least one [worker](./WORKERS.md) are up, try an +end-to-end run. + +## Why you need to seed criteria first + +Scenarios and personas (`config/scenarios/`, `config/personas/`) aren't +server-side data — `scope run submit --scenario --persona ` +reads them straight off disk, so there's nothing to import for those. + +Criteria are different: they're server-side and project-scoped. Several +scenarios (the `version: v2` ones) reference reusable criteria IDs like +`has_react`/`has_typescript` instead of inlining their own, and those IDs +must exist in the target project before the judge can evaluate a run +against them. + +This seed step isn't a Helm hook. Unlike `db-migrate`, it's project-scoped +business data, not infrastructure, and no project exists until the +`db-migrate` Job's "Initial Project" migration has run. Baking a project +assumption into the chart would surprise anyone bringing their own +projects — so run it yourself, once. + +## Steps + +```bash +kubectl -n scope port-forward svc/api 18080:80 & + +# Find the project to seed into: the "Initial Project" created by +# db-migrate on first install, or one of your own. +scope project list --url http://localhost:18080 + +# Import every reusable criterion. This is an upsert, so it's safe to rerun. +scope criteria import config/criteria --project --url http://localhost:18080 + +# Submit the simplest sample scenario. +scope run submit \ + --scenario config/scenarios/hello-world-express-v2.yaml \ + --persona config/personas/vibe-coder.yaml \ + --worker coder-acp-copilot \ + --project \ + --url http://localhost:18080 +``` + +`--worker` must be an agent ID registered in [Workers](./WORKERS.md) — +check with `scope agent list --url http://localhost:18080`. + +`hello-world-express-v2.yaml` only needs `has_azure_doc`/`has_azure_azd` +(plus their `has_iac`/`has_azure`/`has_cloud` ancestors). Importing the +whole `config/criteria` directory is simpler than cherry-picking a +dependency closure, and it covers every sample scenario, including +`react-snake-game-v2.yaml`. diff --git a/deploy/helm/scope/WORKERS.md b/deploy/helm/scope/WORKERS.md new file mode 100644 index 000000000..71df680a6 --- /dev/null +++ b/deploy/helm/scope/WORKERS.md @@ -0,0 +1,75 @@ +# Workers + +The ACP coding-agent workers (`coder-acp-copilot`, `coder-acp-claude-code`) +are disabled by default in this chart — no MCPJungle/AI Gateway/Kubedock +sidecars are templated yet (see +[`docs/architecture/kubedock.md`](../../docs/architecture/kubedock.md) for +the intended production pattern). Bring them online with one command: + +```bash +./scripts/bootstrap-workers.sh --acr +``` + +This: + +1. Builds and pushes both worker images with `az acr build`. +2. Prompts for the GitHub PAT and Anthropic API key they need, and + registers them through the Token Manager API. +3. Runs `helm upgrade` to point each worker's image fields at the image it + just built, and flips `workers..enabled: true`. +4. Waits for both Deployments to roll out. +5. Registers both agent types with the API (`scripts/register-agent.sh`). + +Run `./scripts/bootstrap-workers.sh --help` for all flags, including +`--skip-build`, `--skip-secrets`, `--skip-copilot`, `--skip-claude-code`, +`--skip-agent-registration`, `-y`/`--yes` for non-interactive use, and +custom `--namespace`/`--release`/`--chart`/`--image-tag`. + +> Pass `--values-file ` so the upgrade re-merges your original values +> override file against the chart's current defaults, instead of freezing a +> historical `--reuse-values` snapshot (which silently drops any default key +> added to `values.yaml` after your first `helm install`). + +## Registering credentials another way + +You don't have to go through the script's prompts: + +- The Portal's **Create Token** page, or a direct `POST /api/v1/keys` call + against the API, writes straight to Key Vault through token-manager's + workload identity. No chart changes needed. +- Populating `secrets.keyVaultSecretNames.githubToken`/`.anthropicApiKey` in + Key Vault yourself also works, as a static fallback — the worker only + falls back to it when Token Manager has no credential registered. + +## How the chart templates workers + +Each worker's Deployment is templated by this chart +(`templates/workers/coder-acp-copilot/`, +`templates/workers/coder-acp-claude-code/`), reusing the common env +ConfigMap/Secret and Key Vault CSI mount the rest of the app uses. The +script just flips `workers..enabled: true` and sets +`workers..image.registry`/`.tag` with `helm upgrade --set`. + +These are minimal background Storage Queue consumers only: + +- No MCPJungle MCP-gateway sidecar (MCP server support). +- No AI Gateway/DevProxy sidecar (HAR capture). +- No Kubedock (in-task `docker build`/`docker run` support — see + [`docs/architecture/kubedock.md`](../../docs/architecture/kubedock.md) + for the intended production pattern). + +Scenarios needing those features aren't supported by this chart yet. + +## Agent versions + +`workers.coderAcpCopilot.agentVersion`/`.coderAcpClaudeCode.agentVersion` +must match the `agentVersion` in whatever version manifest you register. +The script's defaults point at the `-dev` manifests under each worker's +directory — intended for chart smoke-testing only. For anything beyond +that, pass a real version manifest (and matching +`imageTag`/`queueName`/`gitCommit`/`buildTime`), and keep `values.yaml`'s +`agentVersion` fields in sync. + +Agent registration doesn't run automatically on `helm install`/`upgrade` — +rerun `scripts/bootstrap-workers.sh` (or `scripts/register-agent.sh` +directly) whenever you roll a new worker image or version. From 46caf15767020bfee7027ebb7488442db9552c57 Mon Sep 17 00:00:00 2001 From: Josh Duffney Date: Mon, 12 Oct 2026 10:17:27 -0500 Subject: [PATCH 11/18] Add MCP gateway sidecar, bootstrap-core script, and doc fixes - Template a native-sidecar MCP gateway (MCPJungle) for both ACP coding agent workers via initContainers with restartPolicy: Always, mirroring the production pattern. Adds a shared /workspace emptyDir volume, MCP_GATEWAY_URL=http://localhost:8080 env, and a readiness probe so the gateway is ready before the worker's first task. - Add workers..mcpGateway.{enabled,image,resources} to values.yaml. - Add scripts/bootstrap-core.sh: queries `az deployment group show` for a user-selected deployment name/resource group, generates a values file from the Bicep outputs, and runs helm install/upgrade in one step. - Rewrite README/CONFIGURATION/WORKERS docs: move "bring your own secrets" to the top of CONFIGURATION.md with simple Portal instructions, document the bootstrap-core.sh workflow and deployment-name selection in README, and document the new MCP gateway sidecar mechanism in WORKERS.md. - Minor stale-comment fix: "Create Token" -> Portal's Secrets tab / Register key, for consistency with the updated docs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff --- deploy/helm/scope/CONFIGURATION.md | 60 +++--- deploy/helm/scope/README.md | 49 ++--- deploy/helm/scope/WORKERS.md | 38 +++- .../coder-acp-claude-code/deployment.yaml | 63 +++++- .../workers/coder-acp-copilot/deployment.yaml | 63 +++++- deploy/helm/scope/values.yaml | 69 ++++-- scripts/bootstrap-core.sh | 201 ++++++++++++++++++ 7 files changed, 434 insertions(+), 109 deletions(-) create mode 100755 scripts/bootstrap-core.sh diff --git a/deploy/helm/scope/CONFIGURATION.md b/deploy/helm/scope/CONFIGURATION.md index 104ed7160..bb4842c60 100644 --- a/deploy/helm/scope/CONFIGURATION.md +++ b/deploy/helm/scope/CONFIGURATION.md @@ -3,6 +3,36 @@ Details on values.yaml conventions and image registries. For the install steps, see the [README](./README.md). +## Bring your own secrets + +Port-forward the Portal if it isn't already running (see README "Verify"): + +```bash +kubectl -n scope port-forward svc/scope-portal 5100:80 +``` + +Then, at : + +1. Go to the **Secrets** tab. +2. Click **Register key**. +3. Add your GitHub token and Anthropic API key. + +> See [Workers](./WORKERS.md#registering-credentials-another-way) for other +> ways to register these credentials (a raw Key Vault secret, or the API +> directly). + +### GitHub OAuth token for the Copilot worker + +The `coder-acp-copilot` worker needs a scopeless GitHub OAuth token (not a +personal access token). Generate one with: + +```bash +pnpm get-copilot-token +``` + +Follow the device-code prompt to approve access, then register the printed +token as the GitHub secret key in the Portal's **Secrets** tab above. + ## Values conventions See [`values.yaml`](./values.yaml) for the full set of configurable values. @@ -38,36 +68,6 @@ The two ACP worker images are different — they're never published to GHCR. `scripts/bootstrap-workers.sh` builds and pushes them to the ACR instance `deploy/azure/` provisions. See [Workers](./WORKERS.md). -## Bring your own secrets - -`deploy/azure/` pre-populates Key Vault with everything the core app -needs — Cosmos DB connection string, Redis access key, and so on. Two -secrets are bring-your-own, since they depend on your accounts, not the -infrastructure: - -- `githubToken` (Key Vault secret name: `github-token`) -- `anthropicApiKey` (Key Vault secret name: `anthropic-api-key`) - -The ACP workers (see [Workers](./WORKERS.md)) and the portal's GitHub -Models fallback read these directly as plain `GITHUB_TOKEN`/ -`ANTHROPIC_API_KEY` env vars. Set them with: - -```bash -az keyvault secret set --vault-name --name github-token --value -az keyvault secret set --vault-name --name anthropic-api-key --value -``` - -Then restart the pods that read them, since Key Vault values only sync on -pod start: - -```bash -kubectl rollout restart deployment -n scope -``` - -`scripts/bootstrap-workers.sh` prompts for and registers these credentials -for you — see [Workers](./WORKERS.md) for that (and a more capable -alternative through the Token Manager API). - ## Not covered Bring your own ingress controller, TLS/cert-manager, and DNS — this chart diff --git a/deploy/helm/scope/README.md b/deploy/helm/scope/README.md index c8e5a8d71..a60269cc4 100644 --- a/deploy/helm/scope/README.md +++ b/deploy/helm/scope/README.md @@ -24,53 +24,28 @@ This README covers phases 2 and 3. For infrastructure, see the ## Deploy the core app -1. Get the Bicep deployment's outputs: +1. List deployments in your resource group and pick one: ```bash - az deployment group show \ + az deployment group list \ --resource-group \ - --name \ - --query properties.outputs -o json + --query "[].{name:name, timestamp:properties.timestamp}" -o table ``` -2. Create a values file from those outputs: - - ```yaml - # my-values.yaml - azure: - tenantId: - keyVaultName: - keyVaultUri: - workloadIdentityClientId: - serviceAccountName: - storageAccountName: - redisHostName: - - api: - ingress: - enabled: true - host: api.example.com - - portal: - ingress: - enabled: true - host: app.example.com - ``` - -3. Install the chart: +2. Run the install script with both the resource group and the deployment + name you chose. It reads that deployment's outputs, generates a values + file from them, and installs the chart: ```bash - kubectl create namespace scope - helm install scope deploy/helm/scope -n scope -f my-values.yaml + ./scripts/bootstrap-core.sh \ + --resource-group \ + --deployment-name ``` - Images come from `ghcr.io/microsoft` by default. To use your own fork's - published images instead, add - `--set global.imageRegistry=ghcr.io/`. - - Database migrations run automatically as part of the install. + Run `./scripts/bootstrap-core.sh --help` for all options (custom image + registry, reusing an edited values file, etc.). -4. Add any bring-your-own secrets (GitHub token, Anthropic API key) that +3. Add any bring-your-own secrets (GitHub token, Anthropic API key) that Bicep doesn't provision. See [Bring your own secrets](./CONFIGURATION.md#bring-your-own-secrets) for your options. diff --git a/deploy/helm/scope/WORKERS.md b/deploy/helm/scope/WORKERS.md index 71df680a6..68de60218 100644 --- a/deploy/helm/scope/WORKERS.md +++ b/deploy/helm/scope/WORKERS.md @@ -1,8 +1,10 @@ # Workers The ACP coding-agent workers (`coder-acp-copilot`, `coder-acp-claude-code`) -are disabled by default in this chart — no MCPJungle/AI Gateway/Kubedock -sidecars are templated yet (see +are disabled by default in this chart. Each gets an MCPJungle MCP-gateway +sidecar (`workers..mcpGateway`, enabled by default — see "MCP-gateway +sidecar" below); the AI Gateway/DevProxy sidecar and Kubedock are not +templated yet (see [`docs/architecture/kubedock.md`](../../docs/architecture/kubedock.md) for the intended production pattern). Bring them online with one command: @@ -34,9 +36,9 @@ custom `--namespace`/`--release`/`--chart`/`--image-tag`. You don't have to go through the script's prompts: -- The Portal's **Create Token** page, or a direct `POST /api/v1/keys` call - against the API, writes straight to Key Vault through token-manager's - workload identity. No chart changes needed. +- The Portal's **Secrets** tab (**Register key**), or a direct + `POST /api/v1/keys` call against the API, writes straight to Key Vault + through token-manager's workload identity. No chart changes needed. - Populating `secrets.keyVaultSecretNames.githubToken`/`.anthropicApiKey` in Key Vault yourself also works, as a static fallback — the worker only falls back to it when Token Manager has no credential registered. @@ -50,15 +52,35 @@ ConfigMap/Secret and Key Vault CSI mount the rest of the app uses. The script just flips `workers..enabled: true` and sets `workers..image.registry`/`.tag` with `helm upgrade --set`. -These are minimal background Storage Queue consumers only: +These are minimal background Storage Queue consumers, plus an MCPJungle +MCP-gateway sidecar: -- No MCPJungle MCP-gateway sidecar (MCP server support). - No AI Gateway/DevProxy sidecar (HAR capture). - No Kubedock (in-task `docker build`/`docker run` support — see [`docs/architecture/kubedock.md`](../../docs/architecture/kubedock.md) for the intended production pattern). -Scenarios needing those features aren't supported by this chart yet. +Scenarios needing those two features aren't supported by this chart yet. + +## MCP-gateway sidecar + +Each worker Pod runs an [MCPJungle](https://github.com/mcpjungle/mcpjungle) +sidecar (`workers..mcpGateway`, enabled by default) that aggregates +MCP servers for the worker, mirroring docker-compose's +`mcp-gateway-copilot`/`mcp-gateway-claude-code` services: + +- Templated as a **native sidecar** (`initContainers` entry with + `restartPolicy: Always`, GA since Kubernetes 1.29) rather than a regular + container, so kubelet starts it first and holds the Pod at `NotReady` + until its `/health` readiness probe passes — the worker's first task + never races a gateway that isn't listening yet. +- Shares an `emptyDir` volume mounted at `/workspace` with the worker + container (compose's `copilot_workspace`/`claude_workspace` volumes). +- The worker reaches it at `http://localhost:8080` (same Pod — no + `Service` object needed). + +Set `workers..mcpGateway.enabled: false` only if your scenarios never +configure MCP servers. ## Agent versions diff --git a/deploy/helm/scope/templates/workers/coder-acp-claude-code/deployment.yaml b/deploy/helm/scope/templates/workers/coder-acp-claude-code/deployment.yaml index cabb9de64..9ae3d3577 100644 --- a/deploy/helm/scope/templates/workers/coder-acp-claude-code/deployment.yaml +++ b/deploy/helm/scope/templates/workers/coder-acp-claude-code/deployment.yaml @@ -23,18 +23,47 @@ spec: imagePullSecrets: {{- toYaml . | nindent 8 }} {{- end }} + {{- if .Values.workers.coderAcpClaudeCode.mcpGateway.enabled }} + # Native sidecar (GA since K8s 1.29): kubelet starts this before the + # main container and holds the Pod at NotReady until its readiness + # probe passes, so the worker's first task never races an MCP-gateway + # that isn't listening yet. Mirrors compose's + # mcp-gateway-claude-code `depends_on: service_healthy`. + initContainers: + - name: mcp-gateway + restartPolicy: Always + image: {{ include "scope.image" (dict "image" .Values.workers.coderAcpClaudeCode.mcpGateway.image "context" .) }} + imagePullPolicy: {{ .Values.global.imagePullPolicy }} + # Suppress MCPJungle's ASCII-art banner (written to stderr). + command: ["/bin/sh", "-c"] + args: ["/mcpjungle start 2>&1 | grep --line-buffered -v '[█═╗╚║╔╝]' >&2"] + env: + - name: SESSION_IDLE_TIMEOUT_SEC + value: "0" + volumeMounts: + - name: workspace + mountPath: /workspace + readinessProbe: + httpGet: + path: /health + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 60 + resources: + {{- toYaml .Values.workers.coderAcpClaudeCode.mcpGateway.resources | nindent 12 }} + {{- end }} containers: # Pure background Storage Queue consumer — no HTTP server/port, so # no Service or readiness/liveness HTTP probes (see # apps/workers/coder-acp-claude-code/src/index.ts). # # NOT templated here (see docs/architecture/worker-requirements.md - # and docs/architecture/kubedock.md): the MCPJungle MCP-gateway - # sidecar (MCP server support), the AI Gateway/DevProxy sidecar (HAR - # capture), and Kubedock (in-task `docker build`/`docker run` - # support). The worker runs without them — scenarios that need MCP - # servers or in-task Docker access are not yet supported by this - # chart. Follow up once those sidecars have a templated pattern. + # and docs/architecture/kubedock.md): the AI Gateway/DevProxy + # sidecar (HAR capture) and Kubedock (in-task `docker + # build`/`docker run` support). The worker runs without them — + # scenarios that need in-task Docker access are not yet supported + # by this chart. Follow up once those sidecars have a templated + # pattern. - name: coder-acp-claude-code image: {{ include "scope.image" (dict "image" .Values.workers.coderAcpClaudeCode.image "context" .) }} imagePullPolicy: {{ .Values.global.imagePullPolicy }} @@ -49,9 +78,9 @@ spec: # dynamically from Token Manager at runtime (see "Read # credentials via Token Manager" in # docs/architecture/worker-requirements.md) — register one - # via the Portal's "Create Token" page or `POST - # /api/v1/keys` instead of populating this secret, unless - # you specifically want the static-env-var fallback. + # via the Portal's **Secrets** tab (**Register key**) or + # `POST /api/v1/keys` instead of populating this secret, + # unless you specifically want the static-env-var fallback. name: {{ .Values.secrets.k8sSecretName }} env: - name: WORKER_NAME @@ -63,10 +92,20 @@ spec: value: {{ .Values.workers.coderAcpClaudeCode.agentVersion | quote }} - name: QUEUE_NAME value: {{ .Values.workers.coderAcpClaudeCode.queueName | quote }} + {{- if .Values.workers.coderAcpClaudeCode.mcpGateway.enabled }} + # Same Pod, so the sidecar is reachable over loopback — no + # Service object needed. + - name: MCP_GATEWAY_URL + value: http://localhost:8080 + {{- end }} volumeMounts: - name: kv-secrets mountPath: /mnt/secrets readOnly: true + {{- if .Values.workers.coderAcpClaudeCode.mcpGateway.enabled }} + - name: workspace + mountPath: /workspace + {{- end }} resources: {{- toYaml .Values.workers.coderAcpClaudeCode.resources | nindent 12 }} volumes: @@ -76,4 +115,10 @@ spec: readOnly: true volumeAttributes: secretProviderClass: {{ include "scope.componentName" (dict "component" "azure-kv" "context" .) }} + {{- if .Values.workers.coderAcpClaudeCode.mcpGateway.enabled }} + # Shared with the mcp-gateway sidecar — mirrors compose's + # claude_workspace named volume. + - name: workspace + emptyDir: {} + {{- end }} {{- end -}} diff --git a/deploy/helm/scope/templates/workers/coder-acp-copilot/deployment.yaml b/deploy/helm/scope/templates/workers/coder-acp-copilot/deployment.yaml index c47fa84d2..d0aa985f8 100644 --- a/deploy/helm/scope/templates/workers/coder-acp-copilot/deployment.yaml +++ b/deploy/helm/scope/templates/workers/coder-acp-copilot/deployment.yaml @@ -23,18 +23,47 @@ spec: imagePullSecrets: {{- toYaml . | nindent 8 }} {{- end }} + {{- if .Values.workers.coderAcpCopilot.mcpGateway.enabled }} + # Native sidecar (GA since K8s 1.29): kubelet starts this before the + # main container and holds the Pod at NotReady until its readiness + # probe passes, so the worker's first task never races an MCP-gateway + # that isn't listening yet. Mirrors compose's + # mcp-gateway-copilot `depends_on: service_healthy`. + initContainers: + - name: mcp-gateway + restartPolicy: Always + image: {{ include "scope.image" (dict "image" .Values.workers.coderAcpCopilot.mcpGateway.image "context" .) }} + imagePullPolicy: {{ .Values.global.imagePullPolicy }} + # Suppress MCPJungle's ASCII-art banner (written to stderr). + command: ["/bin/sh", "-c"] + args: ["/mcpjungle start 2>&1 | grep --line-buffered -v '[█═╗╚║╔╝]' >&2"] + env: + - name: SESSION_IDLE_TIMEOUT_SEC + value: "0" + volumeMounts: + - name: workspace + mountPath: /workspace + readinessProbe: + httpGet: + path: /health + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 60 + resources: + {{- toYaml .Values.workers.coderAcpCopilot.mcpGateway.resources | nindent 12 }} + {{- end }} containers: # Pure background Storage Queue consumer — no HTTP server/port, so # no Service or readiness/liveness HTTP probes (see # apps/workers/coder-acp-copilot/src/index.ts). # # NOT templated here (see docs/architecture/worker-requirements.md - # and docs/architecture/kubedock.md): the MCPJungle MCP-gateway - # sidecar (MCP server support), the AI Gateway/DevProxy sidecar (HAR - # capture), and Kubedock (in-task `docker build`/`docker run` - # support). The worker runs without them — scenarios that need MCP - # servers or in-task Docker access are not yet supported by this - # chart. Follow up once those sidecars have a templated pattern. + # and docs/architecture/kubedock.md): the AI Gateway/DevProxy + # sidecar (HAR capture) and Kubedock (in-task `docker + # build`/`docker run` support). The worker runs without them — + # scenarios that need in-task Docker access are not yet supported + # by this chart. Follow up once those sidecars have a templated + # pattern. - name: coder-acp-copilot image: {{ include "scope.image" (dict "image" .Values.workers.coderAcpCopilot.image "context" .) }} imagePullPolicy: {{ .Values.global.imagePullPolicy }} @@ -48,9 +77,9 @@ spec: # credential dynamically from Token Manager at runtime (see # "Read credentials via Token Manager" in # docs/architecture/worker-requirements.md) — register one - # via the Portal's "Create Token" page or `POST - # /api/v1/keys` instead of populating this secret, unless - # you specifically want the static-env-var fallback. + # via the Portal's **Secrets** tab (**Register key**) or + # `POST /api/v1/keys` instead of populating this secret, + # unless you specifically want the static-env-var fallback. name: {{ .Values.secrets.k8sSecretName }} env: - name: WORKER_NAME @@ -62,10 +91,20 @@ spec: value: {{ .Values.workers.coderAcpCopilot.agentVersion | quote }} - name: QUEUE_NAME value: {{ .Values.workers.coderAcpCopilot.queueName | quote }} + {{- if .Values.workers.coderAcpCopilot.mcpGateway.enabled }} + # Same Pod, so the sidecar is reachable over loopback — no + # Service object needed. + - name: MCP_GATEWAY_URL + value: http://localhost:8080 + {{- end }} volumeMounts: - name: kv-secrets mountPath: /mnt/secrets readOnly: true + {{- if .Values.workers.coderAcpCopilot.mcpGateway.enabled }} + - name: workspace + mountPath: /workspace + {{- end }} resources: {{- toYaml .Values.workers.coderAcpCopilot.resources | nindent 12 }} volumes: @@ -75,4 +114,10 @@ spec: readOnly: true volumeAttributes: secretProviderClass: {{ include "scope.componentName" (dict "component" "azure-kv" "context" .) }} + {{- if .Values.workers.coderAcpCopilot.mcpGateway.enabled }} + # Shared with the mcp-gateway sidecar — mirrors compose's + # copilot_workspace named volume. + - name: workspace + emptyDir: {} + {{- end }} {{- end -}} diff --git a/deploy/helm/scope/values.yaml b/deploy/helm/scope/values.yaml index 36515693c..757de4cae 100644 --- a/deploy/helm/scope/values.yaml +++ b/deploy/helm/scope/values.yaml @@ -3,7 +3,10 @@ # # This chart is designed to be deployed against the AKS cluster provisioned by # deploy/azure/ (the "Deploy to Azure" Bicep template). Values under `azure:` -# correspond directly to that template's outputs: +# correspond directly to that template's outputs. Run +# scripts/bootstrap-core.sh --resource-group to generate a values file +# from those outputs automatically (see README.md "Deploy the core app") +# instead of transcribing them by hand: # # az deployment group show -g -n --query properties.outputs # @@ -15,6 +18,7 @@ # azure.tenantId <- your Azure AD tenant ID # azure.storageAccountName <- output storageAccountName # azure.redisHostName <- output redisHostName +# azure.redisPort <- output redisPort # # This repo ships ONE values.yaml with sane open-source defaults. There are no # per-environment overlay files in-repo — layer your own `-f my-values.yaml` @@ -65,13 +69,14 @@ azure: # (system:serviceaccount::). serviceAccountName: scope-workload-identity storageAccountName: "" - # Hostname of the Azure Cache for Redis instance (output redisHostName). - # Non-secret — goes straight into the common-env ConfigMap as REDIS_HOST. - # Azure Cache for Redis only accepts TLS on 6380, so redisPort/redisTls - # below are fixed for that output; override only if you point this chart - # at a non-Azure Redis (e.g. plain 6379, no TLS). + # Hostname of the Azure Managed Redis (redisEnterprise) instance (output + # redisHostName). Non-secret — goes straight into the common-env ConfigMap + # as REDIS_HOST. Always TLS-only, but the port is per-deployment (the + # Bicep template's `output redisPort` varies by SKU/region) — always set + # this from that output rather than relying on the default below, which is + # only a placeholder. redisHostName: "" - redisPort: 6380 + redisPort: 10000 redisTls: true # Secret object names as they exist in Key Vault, synced into a single K8s @@ -82,14 +87,22 @@ azure: # an access *key*, not a connection string, so it pairs with # `azure.redisHostName`/`redisPort`/`redisTls` above to form the discrete # REDIS_HOST/PORT/PASSWORD/TLS env vars every service actually reads). The -# rest are bring-your-own (add them to the Key Vault before or after install). +# rest are bring-your-own, and default to "" (skipped — see the comment in +# templates/secretproviderclass.yaml) because they don't exist in Key Vault +# yet on a fresh install. The Azure provider fails the *entire* CSI mount if +# any listed object is missing, which would otherwise block every pod that +# mounts kv-secrets (api, judge, token-manager, scheduler, report-generator, +# and the db-migrate hook) — not just the workers that actually need these. +# Once you've created the Key Vault secrets (see CONFIGURATION.md "Bring +# your own secrets"), set these to the real object names with `helm upgrade` +# to start syncing them. secrets: k8sSecretName: scope-secrets keyVaultSecretNames: mongoConnectionString: cosmos-connection-string redisPassword: redis-primary-key - githubToken: github-token - anthropicApiKey: anthropic-api-key + githubToken: "" + anthropicApiKey: "" # ----------------------------------------------------------------------------- # Per-service configuration. Each service supports `enabled`, `image`, @@ -194,12 +207,11 @@ workers: limits: { cpu: "500m", memory: 512Mi } # ACP coding-agent workers are intentionally disabled by default and kept # out of the main `helm install`. They are not published to GHCR (their - # images need the Kubedock/MCP-gateway sidecar story in - # docs/architecture/kubedock.md, which this chart does not yet template) - # and are built/pushed to the Bicep-provisioned ACR as "Phase 3" — see - # deploy/helm/scope/README.md. Set `enabled: true` and `image.registry` to - # your ACR login server (`az acr build` pushes there) only once you're - # ready to deploy them. + # images need the Kubedock sidecar story in docs/architecture/kubedock.md, + # which this chart does not yet template) and are built/pushed to the + # Bicep-provisioned ACR as "Phase 3" — see deploy/helm/scope/README.md. + # Set `enabled: true` and `image.registry` to your ACR login server + # (`az acr build` pushes there) only once you're ready to deploy them. coderAcpCopilot: enabled: false image: @@ -220,6 +232,21 @@ workers: resources: requests: { cpu: 500m, memory: 512Mi } limits: { cpu: "1", memory: 1Gi } + # MCPJungle sidecar (MCP-server aggregation) — mirrors docker-compose's + # mcp-gateway-copilot service. Templated as a native sidecar + # (initContainer with restartPolicy: Always, GA since K8s 1.29) so + # kubelet holds the Pod at NotReady until the gateway's health check + # passes, instead of racing the worker's first task. Disable only if + # your scenarios never configure MCP servers. + mcpGateway: + enabled: true + image: + registry: ghcr.io + repository: mcpjungle/mcpjungle + tag: latest-stdio + resources: + requests: { cpu: 250m, memory: 256Mi } + limits: { cpu: "500m", memory: 512Mi } coderAcpClaudeCode: enabled: false image: @@ -234,6 +261,16 @@ workers: resources: requests: { cpu: 500m, memory: 512Mi } limits: { cpu: "1", memory: 1Gi } + # See workers.coderAcpCopilot.mcpGateway above. + mcpGateway: + enabled: true + image: + registry: ghcr.io + repository: mcpjungle/mcpjungle + tag: latest-stdio + resources: + requests: { cpu: 250m, memory: 256Mi } + limits: { cpu: "500m", memory: 512Mi } jobs: dbMigrate: diff --git a/scripts/bootstrap-core.sh b/scripts/bootstrap-core.sh new file mode 100755 index 000000000..744a7d6fb --- /dev/null +++ b/scripts/bootstrap-core.sh @@ -0,0 +1,201 @@ +#!/usr/bin/env bash +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# One-shot setup for the core Scope app (Phase 2): reads the outputs of an +# already-provisioned `deploy/azure/` Bicep deployment, generates a Helm +# values file from them, and installs/upgrades the chart. Closes the gap +# between Bicep's outputs and a hand-authored values.yaml, which is where +# transcription mistakes (wrong Redis port, stale Key Vault name, etc.) have +# historically crept in. +# +# Usage: +# scripts/bootstrap-core.sh --resource-group [options] +# +# Options: +# --resource-group Resource group the Bicep template deployed into +# (required unless --skip-generate). +# --deployment-name Deployment within that resource group to read +# outputs from (required unless --skip-generate). +# This is never auto-selected — a resource group +# can contain multiple deployments (re-runs, +# nested templates), and guessing could silently +# read the wrong one's outputs. List candidates +# first and choose explicitly: +# az deployment group list -g \ +# --query "[].{name:name, timestamp:properties.timestamp}" \ +# -o table +# --namespace Kubernetes namespace (default: scope) +# --release Helm release name (default: scope) +# --chart Helm chart path (default: deploy/helm/scope, +# relative to repo root) +# --values-file Where to write the generated values file +# (default: my-values.yaml). If this file already +# exists, the script refuses to overwrite it +# unless --force is also given. +# --image-registry Baked into the generated file as +# global.imageRegistry (e.g. your own fork's GHCR +# namespace, or the Bicep-provisioned ACR). Omit +# to use the chart's ghcr.io/microsoft default. +# Always written into the file itself — never +# passed as an ad-hoc `helm --set` — so a later +# `helm upgrade -f ` (e.g. from +# bootstrap-workers.sh) can't silently drop it. +# --force Overwrite --values-file if it already exists. +# --skip-generate Skip the Azure lookups entirely and just +# install/upgrade using an existing --values-file +# (e.g. after you've hand-edited it for ingress +# hosts or other overrides). +# --dry-run Generate the values file but skip the Helm +# install/upgrade step. +# --set Extra Helm --set override; repeatable. +# -h, --help Show this help. +# +# Requires: az (logged in, correct subscription active), jq, helm, kubectl. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" + +RESOURCE_GROUP="" +DEPLOYMENT_NAME="" +NAMESPACE="scope" +RELEASE_NAME="scope" +CHART_PATH="deploy/helm/scope" +VALUES_FILE="my-values.yaml" +IMAGE_REGISTRY="" +FORCE=false +SKIP_GENERATE=false +DRY_RUN=false +EXTRA_SET=() + +usage() { + sed -n '2,50p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --resource-group) RESOURCE_GROUP="$2"; shift 2 ;; + --deployment-name) DEPLOYMENT_NAME="$2"; shift 2 ;; + --namespace) NAMESPACE="$2"; shift 2 ;; + --release) RELEASE_NAME="$2"; shift 2 ;; + --chart) CHART_PATH="$2"; shift 2 ;; + --values-file) VALUES_FILE="$2"; shift 2 ;; + --image-registry) IMAGE_REGISTRY="$2"; shift 2 ;; + --force) FORCE=true; shift ;; + --skip-generate) SKIP_GENERATE=true; shift ;; + --dry-run) DRY_RUN=true; shift ;; + --set) EXTRA_SET+=("--set" "$2"); shift 2 ;; + -h|--help) usage; exit 0 ;; + *) echo "Unknown option: $1" >&2; usage; exit 1 ;; + esac +done + +cd "$REPO_ROOT" + +for bin in helm kubectl; do + command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: $bin not found on PATH" >&2; exit 1; } +done + +if [[ "$SKIP_GENERATE" != true ]]; then + for bin in az jq; do + command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: $bin not found on PATH" >&2; exit 1; } + done + + [[ -n "$RESOURCE_GROUP" ]] || { echo "ERROR: --resource-group is required (or pass --skip-generate to reuse an existing values file)" >&2; usage; exit 1; } + + if [[ -z "$DEPLOYMENT_NAME" ]]; then + echo "ERROR: --deployment-name is required. List deployments in '$RESOURCE_GROUP' and pick one:" >&2 + echo " az deployment group list -g $RESOURCE_GROUP --query \"[].{name:name, timestamp:properties.timestamp}\" -o table" >&2 + exit 1 + fi + + if [[ -f "$VALUES_FILE" && "$FORCE" != true ]]; then + echo "ERROR: $VALUES_FILE already exists. Pass --force to overwrite it, --values-file to write elsewhere, or --skip-generate to install using it as-is." >&2 + exit 1 + fi + + echo "==> Reading Bicep outputs from $RESOURCE_GROUP/$DEPLOYMENT_NAME" + OUTPUTS_JSON="$(az deployment group show -g "$RESOURCE_GROUP" -n "$DEPLOYMENT_NAME" --query properties.outputs -o json)" + + get_output() { + jq -r --arg k "$1" '.[$k].value // empty' <<<"$OUTPUTS_JSON" + } + + TENANT_ID="$(az account show --query tenantId -o tsv)" + KEY_VAULT_NAME="$(get_output keyVaultName)" + KEY_VAULT_URI="$(get_output keyVaultUri)" + WORKLOAD_IDENTITY_CLIENT_ID="$(get_output workloadIdentityClientId)" + SERVICE_ACCOUNT_NAME="$(get_output workloadIdentityServiceAccountName)" + STORAGE_ACCOUNT_NAME="$(get_output storageAccountName)" + REDIS_HOST_NAME="$(get_output redisHostName)" + REDIS_PORT="$(get_output redisPort)" + + for pair in \ + "tenantId:$TENANT_ID" \ + "keyVaultName:$KEY_VAULT_NAME" \ + "keyVaultUri:$KEY_VAULT_URI" \ + "workloadIdentityClientId:$WORKLOAD_IDENTITY_CLIENT_ID" \ + "workloadIdentityServiceAccountName:$SERVICE_ACCOUNT_NAME" \ + "storageAccountName:$STORAGE_ACCOUNT_NAME" \ + "redisHostName:$REDIS_HOST_NAME" \ + "redisPort:$REDIS_PORT" + do + key="${pair%%:*}" + val="${pair#*:}" + if [[ -z "$val" ]]; then + echo "ERROR: value for '$key' was empty. Check --resource-group/--deployment-name point at the right Bicep deployment." >&2 + exit 1 + fi + done + + echo "==> Writing $VALUES_FILE" + { + if [[ -n "$IMAGE_REGISTRY" ]]; then + cat < "$VALUES_FILE" + + echo " Done. Review/edit $VALUES_FILE (e.g. to add ingress hosts) before re-running if needed." +else + [[ -f "$VALUES_FILE" ]] || { echo "ERROR: --skip-generate given but $VALUES_FILE does not exist" >&2; exit 1; } + echo "==> --skip-generate set; reusing existing $VALUES_FILE" +fi + +if [[ "$DRY_RUN" == true ]]; then + echo "==> --dry-run set; skipping helm install/upgrade." + exit 0 +fi + +echo "==> Installing/upgrading Helm release '$RELEASE_NAME' in namespace '$NAMESPACE'" +helm upgrade --install "$RELEASE_NAME" "$CHART_PATH" \ + --namespace "$NAMESPACE" --create-namespace \ + -f "$VALUES_FILE" \ + "${EXTRA_SET[@]}" \ + --wait --timeout 5m + +cat < Date: Mon, 12 Oct 2026 11:02:20 -0500 Subject: [PATCH 12/18] Split bootstrap-core.sh into values-only generator, fix set -u array bug - Rename scripts/bootstrap-core.sh to generate-values-from-bicep.sh and strip out the helm install/upgrade step entirely. It now only reads Bicep outputs and writes a values file, with no opinion on how the chart gets installed, giving a cleaner separation for people not using the Azure deployment, who can now see the plain helm upgrade --install command directly in the README instead of it being buried in a script. - Move deployment-name auto-detection into the script itself (was previously a copy/paste jq command from the README): when --deployment-name is omitted, it filters az deployment group list for the most recent deployment whose outputs include keyVaultName, redisHostName, and workloadIdentityClientId together, a combination only the root deployment has, since nested Bicep modules (e.g. a keyvault module) can re-expose a subset of the same output names. - Fix "${EXTRA_SET[@]}" throwing unbound variable under set -u on bash 3.2 (macOS default shell) when no --set flags are given. Moot now that EXTRA_SET is not part of this script, but noting it since the README's new plain helm upgrade command sidesteps the whole issue for the common case. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff --- deploy/helm/scope/README.md | 56 +++---- scripts/bootstrap-core.sh | 201 -------------------------- scripts/generate-values-from-bicep.sh | 153 ++++++++++++++++++++ 3 files changed, 184 insertions(+), 226 deletions(-) delete mode 100755 scripts/bootstrap-core.sh create mode 100755 scripts/generate-values-from-bicep.sh diff --git a/deploy/helm/scope/README.md b/deploy/helm/scope/README.md index a60269cc4..61ce89e74 100644 --- a/deploy/helm/scope/README.md +++ b/deploy/helm/scope/README.md @@ -24,31 +24,37 @@ This README covers phases 2 and 3. For infrastructure, see the ## Deploy the core app -1. List deployments in your resource group and pick one: - - ```bash - az deployment group list \ - --resource-group \ - --query "[].{name:name, timestamp:properties.timestamp}" -o table - ``` - -2. Run the install script with both the resource group and the deployment - name you chose. It reads that deployment's outputs, generates a values - file from them, and installs the chart: - - ```bash - ./scripts/bootstrap-core.sh \ - --resource-group \ - --deployment-name - ``` - - Run `./scripts/bootstrap-core.sh --help` for all options (custom image - registry, reusing an edited values file, etc.). - -3. Add any bring-your-own secrets (GitHub token, Anthropic API key) that - Bicep doesn't provision. See - [Bring your own secrets](./CONFIGURATION.md#bring-your-own-secrets) for - your options. +Deployed with `deploy/azure/`'s Bicep? Generate `my-values.yaml` from its +outputs: + +```bash +./scripts/generate-values-from-bicep.sh --resource-group +``` + +It auto-picks the right deployment — nested Bicep modules (e.g. a +`keyvault` module) can show up as separate deployments and re-expose a +subset of the same output names, so it filters on a combination only the +root deployment has, then takes the most recent match. Pass +`--deployment-name ` to override. Run +`./scripts/generate-values-from-bicep.sh --help` for all options (custom +image registry, output path, etc.). + +Not using the Bicep template? Hand-author `my-values.yaml` instead — see +[Configuration](./CONFIGURATION.md). + +Either way, install the chart the same way: + +```bash +helm upgrade --install scope deploy/helm/scope \ + --namespace scope --create-namespace \ + -f my-values.yaml \ + --wait --timeout 5m +``` + +Then add any bring-your-own secrets (GitHub token, Anthropic API key) that +Bicep doesn't provision. See +[Bring your own secrets](./CONFIGURATION.md#bring-your-own-secrets) for +your options. ## Bring workers online diff --git a/scripts/bootstrap-core.sh b/scripts/bootstrap-core.sh deleted file mode 100755 index 744a7d6fb..000000000 --- a/scripts/bootstrap-core.sh +++ /dev/null @@ -1,201 +0,0 @@ -#!/usr/bin/env bash -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# -# One-shot setup for the core Scope app (Phase 2): reads the outputs of an -# already-provisioned `deploy/azure/` Bicep deployment, generates a Helm -# values file from them, and installs/upgrades the chart. Closes the gap -# between Bicep's outputs and a hand-authored values.yaml, which is where -# transcription mistakes (wrong Redis port, stale Key Vault name, etc.) have -# historically crept in. -# -# Usage: -# scripts/bootstrap-core.sh --resource-group [options] -# -# Options: -# --resource-group Resource group the Bicep template deployed into -# (required unless --skip-generate). -# --deployment-name Deployment within that resource group to read -# outputs from (required unless --skip-generate). -# This is never auto-selected — a resource group -# can contain multiple deployments (re-runs, -# nested templates), and guessing could silently -# read the wrong one's outputs. List candidates -# first and choose explicitly: -# az deployment group list -g \ -# --query "[].{name:name, timestamp:properties.timestamp}" \ -# -o table -# --namespace Kubernetes namespace (default: scope) -# --release Helm release name (default: scope) -# --chart Helm chart path (default: deploy/helm/scope, -# relative to repo root) -# --values-file Where to write the generated values file -# (default: my-values.yaml). If this file already -# exists, the script refuses to overwrite it -# unless --force is also given. -# --image-registry Baked into the generated file as -# global.imageRegistry (e.g. your own fork's GHCR -# namespace, or the Bicep-provisioned ACR). Omit -# to use the chart's ghcr.io/microsoft default. -# Always written into the file itself — never -# passed as an ad-hoc `helm --set` — so a later -# `helm upgrade -f ` (e.g. from -# bootstrap-workers.sh) can't silently drop it. -# --force Overwrite --values-file if it already exists. -# --skip-generate Skip the Azure lookups entirely and just -# install/upgrade using an existing --values-file -# (e.g. after you've hand-edited it for ingress -# hosts or other overrides). -# --dry-run Generate the values file but skip the Helm -# install/upgrade step. -# --set Extra Helm --set override; repeatable. -# -h, --help Show this help. -# -# Requires: az (logged in, correct subscription active), jq, helm, kubectl. -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" - -RESOURCE_GROUP="" -DEPLOYMENT_NAME="" -NAMESPACE="scope" -RELEASE_NAME="scope" -CHART_PATH="deploy/helm/scope" -VALUES_FILE="my-values.yaml" -IMAGE_REGISTRY="" -FORCE=false -SKIP_GENERATE=false -DRY_RUN=false -EXTRA_SET=() - -usage() { - sed -n '2,50p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' -} - -while [[ $# -gt 0 ]]; do - case "$1" in - --resource-group) RESOURCE_GROUP="$2"; shift 2 ;; - --deployment-name) DEPLOYMENT_NAME="$2"; shift 2 ;; - --namespace) NAMESPACE="$2"; shift 2 ;; - --release) RELEASE_NAME="$2"; shift 2 ;; - --chart) CHART_PATH="$2"; shift 2 ;; - --values-file) VALUES_FILE="$2"; shift 2 ;; - --image-registry) IMAGE_REGISTRY="$2"; shift 2 ;; - --force) FORCE=true; shift ;; - --skip-generate) SKIP_GENERATE=true; shift ;; - --dry-run) DRY_RUN=true; shift ;; - --set) EXTRA_SET+=("--set" "$2"); shift 2 ;; - -h|--help) usage; exit 0 ;; - *) echo "Unknown option: $1" >&2; usage; exit 1 ;; - esac -done - -cd "$REPO_ROOT" - -for bin in helm kubectl; do - command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: $bin not found on PATH" >&2; exit 1; } -done - -if [[ "$SKIP_GENERATE" != true ]]; then - for bin in az jq; do - command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: $bin not found on PATH" >&2; exit 1; } - done - - [[ -n "$RESOURCE_GROUP" ]] || { echo "ERROR: --resource-group is required (or pass --skip-generate to reuse an existing values file)" >&2; usage; exit 1; } - - if [[ -z "$DEPLOYMENT_NAME" ]]; then - echo "ERROR: --deployment-name is required. List deployments in '$RESOURCE_GROUP' and pick one:" >&2 - echo " az deployment group list -g $RESOURCE_GROUP --query \"[].{name:name, timestamp:properties.timestamp}\" -o table" >&2 - exit 1 - fi - - if [[ -f "$VALUES_FILE" && "$FORCE" != true ]]; then - echo "ERROR: $VALUES_FILE already exists. Pass --force to overwrite it, --values-file to write elsewhere, or --skip-generate to install using it as-is." >&2 - exit 1 - fi - - echo "==> Reading Bicep outputs from $RESOURCE_GROUP/$DEPLOYMENT_NAME" - OUTPUTS_JSON="$(az deployment group show -g "$RESOURCE_GROUP" -n "$DEPLOYMENT_NAME" --query properties.outputs -o json)" - - get_output() { - jq -r --arg k "$1" '.[$k].value // empty' <<<"$OUTPUTS_JSON" - } - - TENANT_ID="$(az account show --query tenantId -o tsv)" - KEY_VAULT_NAME="$(get_output keyVaultName)" - KEY_VAULT_URI="$(get_output keyVaultUri)" - WORKLOAD_IDENTITY_CLIENT_ID="$(get_output workloadIdentityClientId)" - SERVICE_ACCOUNT_NAME="$(get_output workloadIdentityServiceAccountName)" - STORAGE_ACCOUNT_NAME="$(get_output storageAccountName)" - REDIS_HOST_NAME="$(get_output redisHostName)" - REDIS_PORT="$(get_output redisPort)" - - for pair in \ - "tenantId:$TENANT_ID" \ - "keyVaultName:$KEY_VAULT_NAME" \ - "keyVaultUri:$KEY_VAULT_URI" \ - "workloadIdentityClientId:$WORKLOAD_IDENTITY_CLIENT_ID" \ - "workloadIdentityServiceAccountName:$SERVICE_ACCOUNT_NAME" \ - "storageAccountName:$STORAGE_ACCOUNT_NAME" \ - "redisHostName:$REDIS_HOST_NAME" \ - "redisPort:$REDIS_PORT" - do - key="${pair%%:*}" - val="${pair#*:}" - if [[ -z "$val" ]]; then - echo "ERROR: value for '$key' was empty. Check --resource-group/--deployment-name point at the right Bicep deployment." >&2 - exit 1 - fi - done - - echo "==> Writing $VALUES_FILE" - { - if [[ -n "$IMAGE_REGISTRY" ]]; then - cat < "$VALUES_FILE" - - echo " Done. Review/edit $VALUES_FILE (e.g. to add ingress hosts) before re-running if needed." -else - [[ -f "$VALUES_FILE" ]] || { echo "ERROR: --skip-generate given but $VALUES_FILE does not exist" >&2; exit 1; } - echo "==> --skip-generate set; reusing existing $VALUES_FILE" -fi - -if [[ "$DRY_RUN" == true ]]; then - echo "==> --dry-run set; skipping helm install/upgrade." - exit 0 -fi - -echo "==> Installing/upgrading Helm release '$RELEASE_NAME' in namespace '$NAMESPACE'" -helm upgrade --install "$RELEASE_NAME" "$CHART_PATH" \ - --namespace "$NAMESPACE" --create-namespace \ - -f "$VALUES_FILE" \ - "${EXTRA_SET[@]}" \ - --wait --timeout 5m - -cat < [options] +# +# Options: +# --resource-group Resource group the Bicep template deployed into +# (required). +# --deployment-name Deployment within that resource group to read +# outputs from. Optional: if omitted, the most +# recent deployment whose outputs match this +# template (keyVaultName, redisHostName, +# workloadIdentityClientId all present) is used. +# This combination excludes nested module +# deployments (e.g. a `keyvault` module), which +# can re-expose a subset of the same output names. +# Pass this explicitly to pick a different one. +# --values-file Where to write the generated values file +# (default: my-values.yaml). If this file already +# exists, the script refuses to overwrite it +# unless --force is also given. +# --image-registry Baked into the generated file as +# global.imageRegistry (e.g. your own fork's GHCR +# namespace, or the Bicep-provisioned ACR). Omit +# to use the chart's ghcr.io/microsoft default. +# --force Overwrite --values-file if it already exists. +# -h, --help Show this help. +# +# Requires: az (logged in, correct subscription active), jq. +set -euo pipefail + +RESOURCE_GROUP="" +DEPLOYMENT_NAME="" +VALUES_FILE="my-values.yaml" +IMAGE_REGISTRY="" +FORCE=false + +usage() { + sed -n '2,38p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --resource-group) RESOURCE_GROUP="$2"; shift 2 ;; + --deployment-name) DEPLOYMENT_NAME="$2"; shift 2 ;; + --values-file) VALUES_FILE="$2"; shift 2 ;; + --image-registry) IMAGE_REGISTRY="$2"; shift 2 ;; + --force) FORCE=true; shift ;; + -h|--help) usage; exit 0 ;; + *) echo "Unknown option: $1" >&2; usage; exit 1 ;; + esac +done + +for bin in az jq; do + command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: $bin not found on PATH" >&2; exit 1; } +done + +[[ -n "$RESOURCE_GROUP" ]] || { echo "ERROR: --resource-group is required" >&2; usage; exit 1; } + +if [[ -f "$VALUES_FILE" && "$FORCE" != true ]]; then + echo "ERROR: $VALUES_FILE already exists. Pass --force to overwrite it or --values-file to write elsewhere." >&2 + exit 1 +fi + +if [[ -z "$DEPLOYMENT_NAME" ]]; then + echo "==> --deployment-name not given; finding the most recent matching deployment in $RESOURCE_GROUP" + DEPLOYMENT_NAME="$( + az deployment group list --resource-group "$RESOURCE_GROUP" -o json \ + | jq -r ' + [.[] | select( + .properties.outputs.keyVaultName != null + and .properties.outputs.redisHostName != null + and .properties.outputs.workloadIdentityClientId != null + )] + | sort_by(.properties.timestamp) + | reverse + | .[0].name // empty + ' + )" + [[ -n "$DEPLOYMENT_NAME" ]] || { + echo "ERROR: no deployment in '$RESOURCE_GROUP' has this template's outputs. Pass --deployment-name explicitly." >&2 + exit 1 + } + echo " Using deployment: $DEPLOYMENT_NAME" +fi + +echo "==> Reading Bicep outputs from $RESOURCE_GROUP/$DEPLOYMENT_NAME" +OUTPUTS_JSON="$(az deployment group show -g "$RESOURCE_GROUP" -n "$DEPLOYMENT_NAME" --query properties.outputs -o json)" + +get_output() { + jq -r --arg k "$1" '.[$k].value // empty' <<<"$OUTPUTS_JSON" +} + +TENANT_ID="$(az account show --query tenantId -o tsv)" +KEY_VAULT_NAME="$(get_output keyVaultName)" +KEY_VAULT_URI="$(get_output keyVaultUri)" +WORKLOAD_IDENTITY_CLIENT_ID="$(get_output workloadIdentityClientId)" +SERVICE_ACCOUNT_NAME="$(get_output workloadIdentityServiceAccountName)" +STORAGE_ACCOUNT_NAME="$(get_output storageAccountName)" +REDIS_HOST_NAME="$(get_output redisHostName)" +REDIS_PORT="$(get_output redisPort)" + +for pair in \ + "tenantId:$TENANT_ID" \ + "keyVaultName:$KEY_VAULT_NAME" \ + "keyVaultUri:$KEY_VAULT_URI" \ + "workloadIdentityClientId:$WORKLOAD_IDENTITY_CLIENT_ID" \ + "workloadIdentityServiceAccountName:$SERVICE_ACCOUNT_NAME" \ + "storageAccountName:$STORAGE_ACCOUNT_NAME" \ + "redisHostName:$REDIS_HOST_NAME" \ + "redisPort:$REDIS_PORT" +do + key="${pair%%:*}" + val="${pair#*:}" + if [[ -z "$val" ]]; then + echo "ERROR: value for '$key' was empty. Check --resource-group/--deployment-name point at the right Bicep deployment." >&2 + exit 1 + fi +done + +echo "==> Writing $VALUES_FILE" +{ + if [[ -n "$IMAGE_REGISTRY" ]]; then + cat < "$VALUES_FILE" + +echo " Done. Review/edit $VALUES_FILE (e.g. to add ingress hosts), then install" +echo " the chart — see deploy/helm/scope/README.md." From 91e286321b60aaa89bf0ef027b44742cc1602ef1 Mon Sep 17 00:00:00 2001 From: Josh Duffney Date: Mon, 12 Oct 2026 12:07:15 -0500 Subject: [PATCH 13/18] docs(helm): tighten chart README for conciseness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Trim deploy/helm/scope/README.md from 100 to 78 lines: number the phase-2 steps instead of prose paragraphs, drop the deployment-name auto-detection explanation (discoverable via --help), and merge the two verify port-forwards into one block. No content removed — just condensed; details remain in CONFIGURATION.md/WORKERS.md. --- deploy/helm/scope/README.md | 64 ++++++++++++------------------------- 1 file changed, 21 insertions(+), 43 deletions(-) diff --git a/deploy/helm/scope/README.md b/deploy/helm/scope/README.md index 61ce89e74..43ea38d64 100644 --- a/deploy/helm/scope/README.md +++ b/deploy/helm/scope/README.md @@ -3,46 +3,32 @@ Deploys the Scope application onto the AKS cluster that [`deploy/azure/`](../azure/README.md) provisions. -Getting Scope running is three phases: - | Phase | What it does | Tooling | |---|---|---| | 1. Infrastructure | AKS, Cosmos DB, Key Vault, Redis, Storage, ACR, networking | `deploy/azure/` (Bicep) | | 2. Core app | api, portal, judge, token-manager, scheduler, report-generator | This chart | -| 3. Coding-agent workers | Build worker images, register secrets and agents | `scripts/bootstrap-workers.sh` | +| 3. Coding-agent workers | Worker images, secrets, agent registration | `scripts/bootstrap-workers.sh` | -This README covers phases 2 and 3. For infrastructure, see the -[Bicep README](../azure/README.md). +This README covers phases 2 and 3. ## Prerequisites -- Azure infrastructure from `deploy/azure/`, already deployed. It - provisions everything phase 2 needs: an AKS cluster with the Key Vault - Secrets Provider CSI driver and workload identity enabled, a federated - managed identity, and Key Vault/Storage/Cosmos DB/Redis. +- Phase 1 infrastructure already deployed (AKS with Key Vault Secrets + Provider CSI driver + workload identity, Key Vault, Storage, Cosmos DB, + Redis, ACR). - `helm` and `kubectl`, configured against the target cluster. -## Deploy the core app +## Phase 2: Deploy the core app -Deployed with `deploy/azure/`'s Bicep? Generate `my-values.yaml` from its -outputs: +**1. Get a values file.** Deployed via `deploy/azure/`'s Bicep: ```bash ./scripts/generate-values-from-bicep.sh --resource-group ``` -It auto-picks the right deployment — nested Bicep modules (e.g. a -`keyvault` module) can show up as separate deployments and re-expose a -subset of the same output names, so it filters on a combination only the -root deployment has, then takes the most recent match. Pass -`--deployment-name ` to override. Run -`./scripts/generate-values-from-bicep.sh --help` for all options (custom -image registry, output path, etc.). - -Not using the Bicep template? Hand-author `my-values.yaml` instead — see -[Configuration](./CONFIGURATION.md). +Otherwise, hand-author `my-values.yaml` — see [Configuration](./CONFIGURATION.md). -Either way, install the chart the same way: +**2. Install the chart:** ```bash helm upgrade --install scope deploy/helm/scope \ @@ -51,43 +37,35 @@ helm upgrade --install scope deploy/helm/scope \ --wait --timeout 5m ``` -Then add any bring-your-own secrets (GitHub token, Anthropic API key) that -Bicep doesn't provision. See -[Bring your own secrets](./CONFIGURATION.md#bring-your-own-secrets) for -your options. +**3. Add any bring-your-own secrets** (GitHub token, Anthropic API key) +Bicep doesn't provision — see +[Bring your own secrets](./CONFIGURATION.md#bring-your-own-secrets). -## Bring workers online +## Phase 3: Bring workers online -The ACP coding-agent workers are disabled by default. Enable them with: +ACP coding-agent workers are disabled by default: ```bash ./scripts/bootstrap-workers.sh --acr ``` -This builds and pushes both worker images, prompts for their credentials, -enables their Deployments, and registers both agent types with the API. -See [Workers](./WORKERS.md) for details and script flags. +Builds + pushes both worker images, registers their credentials, enables +their Deployments, and registers both agents with the API. See +[Workers](./WORKERS.md) for flags and details. ## Verify ```bash kubectl get pods -n scope -# Port-forward the API so the CLI can reach it without an ingress. +# No ingress yet — reach services via port-forward. kubectl -n scope port-forward svc/api 18080:80 & -scope agent list --url http://localhost:18080 -``` - -No ingress yet? Reach the portal the same way: - -```bash kubectl -n scope port-forward svc/scope-portal 5100:80 & -``` -Then open . +scope agent list --url http://localhost:18080 # or open http://localhost:5100 +``` -Then try an end-to-end run — see -[Run a sample scenario](./SAMPLE-SCENARIO.md). +Then try an end-to-end run — see [Run a sample scenario](./SAMPLE-SCENARIO.md). ## Learn more From c99275f55b963b73dba47c796856d6657ab7ba5c Mon Sep 17 00:00:00 2001 From: Josh Duffney Date: Mon, 12 Oct 2026 12:29:24 -0500 Subject: [PATCH 14/18] helm: remove kv-secrets CSI mount and workload identity from ACP workers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both ACP workers run benchmark-controlled agent code and should not share the backend workload identity (Key Vault Secrets Officer) or its mounted secrets file tree with untrusted execution. - Drop the azure.workload.identity/use label and kv-secrets CSI volume/mount from both worker Deployments, so no federated Azure AD token or Key Vault file tree reaches the untrusted agent subprocess. - Keep the common-secret envFrom (Mongo/Redis creds), since the worker still needs those to claim/lock Storage Queue work — the queue-processor and agent executor are one process today, so a chart change alone can't separate them further. - Document the residual limitation and a worker-scoped managed identity (Storage Blob/Queue only, planned on the Bicep side) in WORKERS.md. Addresses part of the review finding in https://github.com/microsoft/scope/pull/1481#discussion_r4248323314. Full trusted/untrusted separation needs worker code changes and a dedicated worker identity from deploy/azure/, which is still being worked out on the Bicep PR (#1477) after an earlier attempt there was reverted. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff --- deploy/helm/scope/WORKERS.md | 24 +++++++++++++++++ .../coder-acp-claude-code/deployment.yaml | 27 +++++++++---------- .../workers/coder-acp-copilot/deployment.yaml | 27 +++++++++---------- 3 files changed, 48 insertions(+), 30 deletions(-) diff --git a/deploy/helm/scope/WORKERS.md b/deploy/helm/scope/WORKERS.md index 68de60218..3e53a514b 100644 --- a/deploy/helm/scope/WORKERS.md +++ b/deploy/helm/scope/WORKERS.md @@ -82,6 +82,30 @@ MCP servers for the worker, mirroring docker-compose's Set `workers..mcpGateway.enabled: false` only if your scenarios never configure MCP servers. +## Worker identity isolation + +Both ACP workers run benchmark-controlled agent code, so their pods don't +carry the `azure.workload.identity/use: "true"` label or the `kv-secrets` +CSI mount the rest of the app uses — no federated Azure AD token or Key +Vault file tree is exposed to that untrusted execution. They still read +`MONGO_CONNECTION_STRING`/`REDIS_PASSWORD` via `envFrom.secretRef` (needed to +claim/lock queue work) and share the common ServiceAccount otherwise. + +**Known limitation**: full isolation needs worker code changes this chart +can't make alone. `coder-acp-copilot`/`coder-acp-claude-code` are a single +process today — the same container that claims Storage Queue work via Mongo +and Redis also spawns the benchmark agent subprocess, so the pod can't be +handed queue credentials without also handing them to the agent's +environment/filesystem. True separation (a trusted queue-processor process +with no agent execution, and an executor with zero backend credentials — +`automountServiceAccountToken: false`, no Mongo/Redis/API-key env) is +tracked as follow-up work, not part of this chart. + +A dedicated worker-scoped managed identity (Storage Blob/Queue only, no Key +Vault access) is planned on the Bicep side (`deploy/azure/`); once it lands, +this chart will add a second ServiceAccount for the workers instead of +reusing the shared one. + ## Agent versions `workers.coderAcpCopilot.agentVersion`/`.coderAcpClaudeCode.agentVersion` diff --git a/deploy/helm/scope/templates/workers/coder-acp-claude-code/deployment.yaml b/deploy/helm/scope/templates/workers/coder-acp-claude-code/deployment.yaml index 9ae3d3577..f9f4e10f2 100644 --- a/deploy/helm/scope/templates/workers/coder-acp-claude-code/deployment.yaml +++ b/deploy/helm/scope/templates/workers/coder-acp-claude-code/deployment.yaml @@ -16,7 +16,12 @@ spec: metadata: labels: {{- include "scope.selectorLabels" (dict "component" "coder-acp-claude-code" "context" .) | nindent 8 }} - azure.workload.identity/use: "true" + # No azure.workload.identity/use label: this pod runs + # benchmark-controlled agent code and must not receive a federated + # Azure AD token (see deploy/helm/scope/WORKERS.md "Worker identity + # isolation"). It still runs under the shared ServiceAccount below, + # but without the label the workload-identity webhook never injects + # a token into it. spec: serviceAccountName: {{ include "scope.serviceAccountName" . }} {{- with .Values.global.imagePullSecrets }} @@ -98,27 +103,19 @@ spec: - name: MCP_GATEWAY_URL value: http://localhost:8080 {{- end }} + {{- if .Values.workers.coderAcpClaudeCode.mcpGateway.enabled }} volumeMounts: - - name: kv-secrets - mountPath: /mnt/secrets - readOnly: true - {{- if .Values.workers.coderAcpClaudeCode.mcpGateway.enabled }} - name: workspace mountPath: /workspace - {{- end }} + {{- end }} resources: {{- toYaml .Values.workers.coderAcpClaudeCode.resources | nindent 12 }} + {{- if .Values.workers.coderAcpClaudeCode.mcpGateway.enabled }} volumes: - - name: kv-secrets - csi: - driver: secrets-store.csi.k8s.io - readOnly: true - volumeAttributes: - secretProviderClass: {{ include "scope.componentName" (dict "component" "azure-kv" "context" .) }} - {{- if .Values.workers.coderAcpClaudeCode.mcpGateway.enabled }} # Shared with the mcp-gateway sidecar — mirrors compose's - # claude_workspace named volume. + # claude_workspace named volume. No kv-secrets CSI volume here: + # see the identity note above. - name: workspace emptyDir: {} - {{- end }} + {{- end }} {{- end -}} diff --git a/deploy/helm/scope/templates/workers/coder-acp-copilot/deployment.yaml b/deploy/helm/scope/templates/workers/coder-acp-copilot/deployment.yaml index d0aa985f8..cc06dffcb 100644 --- a/deploy/helm/scope/templates/workers/coder-acp-copilot/deployment.yaml +++ b/deploy/helm/scope/templates/workers/coder-acp-copilot/deployment.yaml @@ -16,7 +16,12 @@ spec: metadata: labels: {{- include "scope.selectorLabels" (dict "component" "coder-acp-copilot" "context" .) | nindent 8 }} - azure.workload.identity/use: "true" + # No azure.workload.identity/use label: this pod runs + # benchmark-controlled agent code and must not receive a federated + # Azure AD token (see deploy/helm/scope/WORKERS.md "Worker identity + # isolation"). It still runs under the shared ServiceAccount below, + # but without the label the workload-identity webhook never injects + # a token into it. spec: serviceAccountName: {{ include "scope.serviceAccountName" . }} {{- with .Values.global.imagePullSecrets }} @@ -97,27 +102,19 @@ spec: - name: MCP_GATEWAY_URL value: http://localhost:8080 {{- end }} + {{- if .Values.workers.coderAcpCopilot.mcpGateway.enabled }} volumeMounts: - - name: kv-secrets - mountPath: /mnt/secrets - readOnly: true - {{- if .Values.workers.coderAcpCopilot.mcpGateway.enabled }} - name: workspace mountPath: /workspace - {{- end }} + {{- end }} resources: {{- toYaml .Values.workers.coderAcpCopilot.resources | nindent 12 }} + {{- if .Values.workers.coderAcpCopilot.mcpGateway.enabled }} volumes: - - name: kv-secrets - csi: - driver: secrets-store.csi.k8s.io - readOnly: true - volumeAttributes: - secretProviderClass: {{ include "scope.componentName" (dict "component" "azure-kv" "context" .) }} - {{- if .Values.workers.coderAcpCopilot.mcpGateway.enabled }} # Shared with the mcp-gateway sidecar — mirrors compose's - # copilot_workspace named volume. + # copilot_workspace named volume. No kv-secrets CSI volume here: + # see the identity note above. - name: workspace emptyDir: {} - {{- end }} + {{- end }} {{- end -}} From 6b4c847d32a66b49aadeec0ad95115261425d183 Mon Sep 17 00:00:00 2001 From: Josh Duffney Date: Mon, 12 Oct 2026 12:36:33 -0500 Subject: [PATCH 15/18] helm: give ACP workers their own worker-scoped identity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bicep PR #1477 landed a worker-specific managed identity (Storage Blob Data Owner + Queue Data Contributor only — no Key Vault access), exposed as workerWorkloadIdentityClientId/ workerWorkloadIdentityServiceAccountName. - values.yaml: add azure.workerWorkloadIdentityClientId and azure.workerServiceAccountName (default scope-worker-workload-identity). - templates/serviceaccount.yaml: add a second ServiceAccount for the worker identity, rendered only when a worker is enabled. - _helpers.tpl: add scope.workerServiceAccountName helper. - Both ACP worker Deployments now use the worker ServiceAccount (with azure.workload.identity/use re-added, needed for their direct Storage Blob/Queue SDK access via DefaultAzureCredential) instead of the shared backend ServiceAccount. The kv-secrets CSI mount stays removed from their own pods — the worker's GITHUB_TOKEN/ ANTHROPIC_API_KEY fallback values already reach them via envFrom.secretRef on the common K8s Secret, which the api pod's own CSI mount keeps synced; the workers don't need a mount of their own for that. - scripts/generate-values-from-bicep.sh: read the two new outputs, tolerating deployments that don't have them (older Bicep deployments, or ones that never provisioned the worker identity). - WORKERS.md: document the real worker-identity wiring and keep the known-limitation note (full trusted/untrusted process separation needs worker code changes). Continues the fix for https://github.com/microsoft/scope/pull/1481#discussion_r4248323314. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff --- deploy/helm/scope/WORKERS.md | 32 +++++++++------ deploy/helm/scope/templates/_helpers.tpl | 9 ++++ .../helm/scope/templates/serviceaccount.yaml | 17 ++++++++ .../coder-acp-claude-code/deployment.yaml | 13 +++--- .../workers/coder-acp-copilot/deployment.yaml | 13 +++--- deploy/helm/scope/values.yaml | 41 +++++++++++++------ scripts/generate-values-from-bicep.sh | 22 ++++++++++ 7 files changed, 108 insertions(+), 39 deletions(-) diff --git a/deploy/helm/scope/WORKERS.md b/deploy/helm/scope/WORKERS.md index 3e53a514b..9b926d670 100644 --- a/deploy/helm/scope/WORKERS.md +++ b/deploy/helm/scope/WORKERS.md @@ -48,8 +48,10 @@ You don't have to go through the script's prompts: Each worker's Deployment is templated by this chart (`templates/workers/coder-acp-copilot/`, `templates/workers/coder-acp-claude-code/`), reusing the common env -ConfigMap/Secret and Key Vault CSI mount the rest of the app uses. The -script just flips `workers..enabled: true` and sets +ConfigMap/Secret the rest of the app uses, but under their own +narrower-identity ServiceAccount (see "Worker identity isolation" below — +no Key Vault CSI mount). The script just flips +`workers..enabled: true` and sets `workers..image.registry`/`.tag` with `helm upgrade --set`. These are minimal background Storage Queue consumers, plus an MCPJungle @@ -84,12 +86,21 @@ configure MCP servers. ## Worker identity isolation -Both ACP workers run benchmark-controlled agent code, so their pods don't -carry the `azure.workload.identity/use: "true"` label or the `kv-secrets` -CSI mount the rest of the app uses — no federated Azure AD token or Key -Vault file tree is exposed to that untrusted execution. They still read -`MONGO_CONNECTION_STRING`/`REDIS_PASSWORD` via `envFrom.secretRef` (needed to -claim/lock queue work) and share the common ServiceAccount otherwise. +Both ACP workers run benchmark-controlled agent code, so they get a +separate, narrower `ServiceAccount`/identity than the rest of the app: +`azure.workerWorkloadIdentityClientId`/`azure.workerServiceAccountName` +(Storage Blob/Queue only — no Key Vault access), instead of the shared +identity's Key Vault Secrets Officer role. Their pods also don't mount the +`kv-secrets` CSI volume. They still read `GITHUB_TOKEN`/`ANTHROPIC_API_KEY` +fallback values and `MONGO_CONNECTION_STRING`/`REDIS_PASSWORD` (needed to +claim/lock queue work) via `envFrom.secretRef` on the common K8s Secret — +that secret is synced by the `api` pod's own CSI mount, so the workers don't +need their own mount to receive it. + +`scripts/generate-values-from-bicep.sh` picks up the worker identity's +outputs automatically if your Bicep deployment provisioned one; otherwise +those two values stay blank and are only required once you set +`workers..enabled: true`. **Known limitation**: full isolation needs worker code changes this chart can't make alone. `coder-acp-copilot`/`coder-acp-claude-code` are a single @@ -101,11 +112,6 @@ with no agent execution, and an executor with zero backend credentials — `automountServiceAccountToken: false`, no Mongo/Redis/API-key env) is tracked as follow-up work, not part of this chart. -A dedicated worker-scoped managed identity (Storage Blob/Queue only, no Key -Vault access) is planned on the Bicep side (`deploy/azure/`); once it lands, -this chart will add a second ServiceAccount for the workers instead of -reusing the shared one. - ## Agent versions `workers.coderAcpCopilot.agentVersion`/`.coderAcpClaudeCode.agentVersion` diff --git a/deploy/helm/scope/templates/_helpers.tpl b/deploy/helm/scope/templates/_helpers.tpl index 3037b5cf0..5dfcc7381 100644 --- a/deploy/helm/scope/templates/_helpers.tpl +++ b/deploy/helm/scope/templates/_helpers.tpl @@ -65,3 +65,12 @@ template's federated subject. {{- define "scope.serviceAccountName" -}} {{- .Values.azure.serviceAccountName -}} {{- end -}} + +{{/* +Name of the worker-scoped workload-identity ServiceAccount (ACP workers +only). Narrower identity than scope.serviceAccountName — see WORKERS.md +"Worker identity isolation". +*/}} +{{- define "scope.workerServiceAccountName" -}} +{{- .Values.azure.workerServiceAccountName -}} +{{- end -}} diff --git a/deploy/helm/scope/templates/serviceaccount.yaml b/deploy/helm/scope/templates/serviceaccount.yaml index de7fd15f1..d22d25d70 100644 --- a/deploy/helm/scope/templates/serviceaccount.yaml +++ b/deploy/helm/scope/templates/serviceaccount.yaml @@ -8,3 +8,20 @@ metadata: azure.workload.identity/use: "true" annotations: azure.workload.identity/client-id: {{ required "azure.workloadIdentityClientId is required (see deploy/azure output workloadIdentityClientId)" .Values.azure.workloadIdentityClientId | quote }} +{{- if or .Values.workers.coderAcpCopilot.enabled .Values.workers.coderAcpClaudeCode.enabled }} +--- +# Narrower identity for the ACP workers (Storage Blob/Queue only — no Key +# Vault access), so benchmark-controlled agent code never runs under the +# backend's Key Vault Secrets Officer identity. See WORKERS.md "Worker +# identity isolation". +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "scope.workerServiceAccountName" . }} + namespace: {{ .Values.global.namespace }} + labels: + {{- include "scope.labels" . | nindent 4 }} + azure.workload.identity/use: "true" + annotations: + azure.workload.identity/client-id: {{ required "azure.workerWorkloadIdentityClientId is required when workers are enabled (see deploy/azure output workerWorkloadIdentityClientId)" .Values.azure.workerWorkloadIdentityClientId | quote }} +{{- end }} diff --git a/deploy/helm/scope/templates/workers/coder-acp-claude-code/deployment.yaml b/deploy/helm/scope/templates/workers/coder-acp-claude-code/deployment.yaml index f9f4e10f2..60512ee54 100644 --- a/deploy/helm/scope/templates/workers/coder-acp-claude-code/deployment.yaml +++ b/deploy/helm/scope/templates/workers/coder-acp-claude-code/deployment.yaml @@ -16,14 +16,13 @@ spec: metadata: labels: {{- include "scope.selectorLabels" (dict "component" "coder-acp-claude-code" "context" .) | nindent 8 }} - # No azure.workload.identity/use label: this pod runs - # benchmark-controlled agent code and must not receive a federated - # Azure AD token (see deploy/helm/scope/WORKERS.md "Worker identity - # isolation"). It still runs under the shared ServiceAccount below, - # but without the label the workload-identity webhook never injects - # a token into it. + azure.workload.identity/use: "true" spec: - serviceAccountName: {{ include "scope.serviceAccountName" . }} + # Worker-scoped identity (Storage Blob/Queue only — no Key Vault + # access), not the shared backend ServiceAccount: this pod runs + # benchmark-controlled agent code and must not be able to reach Key + # Vault. See WORKERS.md "Worker identity isolation". + serviceAccountName: {{ include "scope.workerServiceAccountName" . }} {{- with .Values.global.imagePullSecrets }} imagePullSecrets: {{- toYaml . | nindent 8 }} diff --git a/deploy/helm/scope/templates/workers/coder-acp-copilot/deployment.yaml b/deploy/helm/scope/templates/workers/coder-acp-copilot/deployment.yaml index cc06dffcb..c99c4d3d7 100644 --- a/deploy/helm/scope/templates/workers/coder-acp-copilot/deployment.yaml +++ b/deploy/helm/scope/templates/workers/coder-acp-copilot/deployment.yaml @@ -16,14 +16,13 @@ spec: metadata: labels: {{- include "scope.selectorLabels" (dict "component" "coder-acp-copilot" "context" .) | nindent 8 }} - # No azure.workload.identity/use label: this pod runs - # benchmark-controlled agent code and must not receive a federated - # Azure AD token (see deploy/helm/scope/WORKERS.md "Worker identity - # isolation"). It still runs under the shared ServiceAccount below, - # but without the label the workload-identity webhook never injects - # a token into it. + azure.workload.identity/use: "true" spec: - serviceAccountName: {{ include "scope.serviceAccountName" . }} + # Worker-scoped identity (Storage Blob/Queue only — no Key Vault + # access), not the shared backend ServiceAccount: this pod runs + # benchmark-controlled agent code and must not be able to reach Key + # Vault. See WORKERS.md "Worker identity isolation". + serviceAccountName: {{ include "scope.workerServiceAccountName" . }} {{- with .Values.global.imagePullSecrets }} imagePullSecrets: {{- toYaml . | nindent 8 }} diff --git a/deploy/helm/scope/values.yaml b/deploy/helm/scope/values.yaml index 757de4cae..092f985b3 100644 --- a/deploy/helm/scope/values.yaml +++ b/deploy/helm/scope/values.yaml @@ -4,21 +4,28 @@ # This chart is designed to be deployed against the AKS cluster provisioned by # deploy/azure/ (the "Deploy to Azure" Bicep template). Values under `azure:` # correspond directly to that template's outputs. Run -# scripts/bootstrap-core.sh --resource-group to generate a values file -# from those outputs automatically (see README.md "Deploy the core app") -# instead of transcribing them by hand: +# scripts/generate-values-from-bicep.sh --resource-group to generate a +# values file from those outputs automatically (see README.md "Deploy the +# core app") instead of transcribing them by hand: # # az deployment group show -g -n --query properties.outputs # -# azure.acrLoginServer <- output acrLoginServer -# azure.keyVaultName <- output keyVaultName -# azure.keyVaultUri <- output keyVaultUri -# azure.workloadIdentityClientId <- output workloadIdentityClientId -# azure.serviceAccountName <- output workloadIdentityServiceAccountName -# azure.tenantId <- your Azure AD tenant ID -# azure.storageAccountName <- output storageAccountName -# azure.redisHostName <- output redisHostName -# azure.redisPort <- output redisPort +# azure.acrLoginServer <- output acrLoginServer +# azure.keyVaultName <- output keyVaultName +# azure.keyVaultUri <- output keyVaultUri +# azure.workloadIdentityClientId <- output workloadIdentityClientId +# azure.serviceAccountName <- output workloadIdentityServiceAccountName +# azure.tenantId <- your Azure AD tenant ID +# azure.storageAccountName <- output storageAccountName +# azure.redisHostName <- output redisHostName +# azure.redisPort <- output redisPort +# azure.workerWorkloadIdentityClientId <- output workerWorkloadIdentityClientId +# azure.workerServiceAccountName <- output workerWorkloadIdentityServiceAccountName +# +# The last two are only needed if you plan to bring the ACP workers online +# (see WORKERS.md) — they're a separate, narrower identity (Storage Blob/Queue +# only) than the shared one above, so untrusted benchmark agent code running +# in the worker pods never gets the backend's Key Vault access. # # This repo ships ONE values.yaml with sane open-source defaults. There are no # per-environment overlay files in-repo — layer your own `-f my-values.yaml` @@ -78,6 +85,16 @@ azure: redisHostName: "" redisPort: 10000 redisTls: true + # Client ID of the worker-scoped managed identity (Storage Blob/Queue only + # — no Key Vault access), federated to the ServiceAccount below. Only + # needed if you bring the ACP workers online (see WORKERS.md); leave blank + # otherwise. Kept separate from workloadIdentityClientId above so + # benchmark-controlled agent code running in the worker pods never gets + # the backend's Key Vault access. + workerWorkloadIdentityClientId: "" + # Must exactly match the Bicep's federated subject + # (system:serviceaccount::). + workerServiceAccountName: scope-worker-workload-identity # Secret object names as they exist in Key Vault, synced into a single K8s # Secret (named by `secrets.k8sSecretName`) via the Secrets Store CSI diff --git a/scripts/generate-values-from-bicep.sh b/scripts/generate-values-from-bicep.sh index ee8cae173..120284fca 100755 --- a/scripts/generate-values-from-bicep.sh +++ b/scripts/generate-values-from-bicep.sh @@ -35,6 +35,14 @@ # --force Overwrite --values-file if it already exists. # -h, --help Show this help. # +# If the deployment also has worker-identity outputs +# (workerWorkloadIdentityClientId/workerWorkloadIdentityServiceAccountName — +# only present if you provisioned the ACP worker identity), those are +# written too. Missing is not an error: older deployments, or ones that +# never provisioned worker identity, just won't get the worker.* keys, and +# the chart falls back to its defaults (empty client ID, required only once +# you enable a worker — see WORKERS.md). +# # Requires: az (logged in, correct subscription active), jq. set -euo pipefail @@ -108,6 +116,10 @@ SERVICE_ACCOUNT_NAME="$(get_output workloadIdentityServiceAccountName)" STORAGE_ACCOUNT_NAME="$(get_output storageAccountName)" REDIS_HOST_NAME="$(get_output redisHostName)" REDIS_PORT="$(get_output redisPort)" +# Optional: only present if the Bicep deployment provisioned the ACP worker +# identity. Not required below — see WORKERS.md. +WORKER_WORKLOAD_IDENTITY_CLIENT_ID="$(get_output workerWorkloadIdentityClientId)" +WORKER_SERVICE_ACCOUNT_NAME="$(get_output workerWorkloadIdentityServiceAccountName)" for pair in \ "tenantId:$TENANT_ID" \ @@ -127,6 +139,10 @@ do fi done +if [[ -z "$WORKER_WORKLOAD_IDENTITY_CLIENT_ID" || -z "$WORKER_SERVICE_ACCOUNT_NAME" ]]; then + echo " Note: worker-identity outputs not found on this deployment — skipping worker.* values. Bring the ACP workers online only after re-provisioning with that identity (see WORKERS.md)." +fi + echo "==> Writing $VALUES_FILE" { if [[ -n "$IMAGE_REGISTRY" ]]; then @@ -147,6 +163,12 @@ azure: redisHostName: $REDIS_HOST_NAME redisPort: $REDIS_PORT EOF + if [[ -n "$WORKER_WORKLOAD_IDENTITY_CLIENT_ID" && -n "$WORKER_SERVICE_ACCOUNT_NAME" ]]; then + cat < "$VALUES_FILE" echo " Done. Review/edit $VALUES_FILE (e.g. to add ingress hosts), then install" From a0243d1fc01207d8c98b8d4a46ac9a37db3ab71c Mon Sep 17 00:00:00 2001 From: Josh Duffney Date: Mon, 12 Oct 2026 12:54:26 -0500 Subject: [PATCH 16/18] scripts: remove secret bootstrap logic from bootstrap-workers.sh Drop the interactive GitHub token/Anthropic API key prompts and the register_secret() helper that POSTed them to Token Manager over a port-forward. That flow put the plaintext credential in curl's argument list (visible to other local users via process listing while the request is pending), and non-interactive mode required the secret to sit in a shell variable/env var either way. Workers now rely entirely on the Portal's Secrets tab (or a direct POST /api/v1/keys call) to register their GitHub/Anthropic credentials after the Deployments are up. Also drops the now-unused --skip-secrets and -y/--yes flags and the node runtime dependency (only needed for building the removed request's JSON body). Addresses https://github.com/microsoft/scope/pull/1481#discussion_r4248323339 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff --- deploy/helm/scope/README.md | 9 ++-- deploy/helm/scope/WORKERS.md | 25 +++++---- scripts/bootstrap-workers.sh | 101 +++++------------------------------ 3 files changed, 33 insertions(+), 102 deletions(-) diff --git a/deploy/helm/scope/README.md b/deploy/helm/scope/README.md index 43ea38d64..a5d3baf96 100644 --- a/deploy/helm/scope/README.md +++ b/deploy/helm/scope/README.md @@ -7,7 +7,7 @@ Deploys the Scope application onto the AKS cluster that |---|---|---| | 1. Infrastructure | AKS, Cosmos DB, Key Vault, Redis, Storage, ACR, networking | `deploy/azure/` (Bicep) | | 2. Core app | api, portal, judge, token-manager, scheduler, report-generator | This chart | -| 3. Coding-agent workers | Worker images, secrets, agent registration | `scripts/bootstrap-workers.sh` | +| 3. Coding-agent workers | Worker images, agent registration | `scripts/bootstrap-workers.sh` | This README covers phases 2 and 3. @@ -49,9 +49,10 @@ ACP coding-agent workers are disabled by default: ./scripts/bootstrap-workers.sh --acr ``` -Builds + pushes both worker images, registers their credentials, enables -their Deployments, and registers both agents with the API. See -[Workers](./WORKERS.md) for flags and details. +Builds + pushes both worker images, enables their Deployments, and +registers both agents with the API. Register the GitHub token/Anthropic +API key they need yourself via the Portal's **Secrets** tab (or +`POST /api/v1/keys`) — see [Workers](./WORKERS.md) for flags and details. ## Verify diff --git a/deploy/helm/scope/WORKERS.md b/deploy/helm/scope/WORKERS.md index 9b926d670..3294298b9 100644 --- a/deploy/helm/scope/WORKERS.md +++ b/deploy/helm/scope/WORKERS.md @@ -15,26 +15,31 @@ the intended production pattern). Bring them online with one command: This: 1. Builds and pushes both worker images with `az acr build`. -2. Prompts for the GitHub PAT and Anthropic API key they need, and - registers them through the Token Manager API. -3. Runs `helm upgrade` to point each worker's image fields at the image it +2. Runs `helm upgrade` to point each worker's image fields at the image it just built, and flips `workers..enabled: true`. -4. Waits for both Deployments to roll out. -5. Registers both agent types with the API (`scripts/register-agent.sh`). +3. Waits for both Deployments to roll out. +4. Registers both agent types with the API (`scripts/register-agent.sh`). + +The script does **not** register worker credentials. Add the GitHub PAT +and/or Anthropic API key each worker needs through the Portal's **Secrets** +tab (**Register key**) after the workers are up — see "Registering +credentials" below. Run `./scripts/bootstrap-workers.sh --help` for all flags, including -`--skip-build`, `--skip-secrets`, `--skip-copilot`, `--skip-claude-code`, -`--skip-agent-registration`, `-y`/`--yes` for non-interactive use, and -custom `--namespace`/`--release`/`--chart`/`--image-tag`. +`--skip-build`, `--skip-copilot`, `--skip-claude-code`, +`--skip-agent-registration`, and custom +`--namespace`/`--release`/`--chart`/`--image-tag`. > Pass `--values-file ` so the upgrade re-merges your original values > override file against the chart's current defaults, instead of freezing a > historical `--reuse-values` snapshot (which silently drops any default key > added to `values.yaml` after your first `helm install`). -## Registering credentials another way +## Registering credentials -You don't have to go through the script's prompts: +Workers need a GitHub PAT (`coder-acp-copilot`) and/or an Anthropic API key +(`coder-acp-claude-code`) to drive their respective CLIs/SDKs. Register +these yourself, through either: - The Portal's **Secrets** tab (**Register key**), or a direct `POST /api/v1/keys` call against the API, writes straight to Key Vault diff --git a/scripts/bootstrap-workers.sh b/scripts/bootstrap-workers.sh index d870d2790..8aee9226f 100755 --- a/scripts/bootstrap-workers.sh +++ b/scripts/bootstrap-workers.sh @@ -6,12 +6,16 @@ # coder-acp-claude-code) on top of an already-`helm install`ed Scope # release (see deploy/helm/scope/README.md "Phase 1/2"). This is the # single entry point for "Phase 3": it builds + pushes both worker images -# to your ACR, registers the GitHub/Anthropic credentials they need with -# Token Manager, flips them on via `helm upgrade`, and registers each agent +# to your ACR, flips them on via `helm upgrade`, and registers each agent # with the API. You should not need to run any `az acr build`/`helm # upgrade --set workers...`/`register-agent.sh` commands by hand — this # script wraps all of them. # +# This script does NOT register worker credentials (GitHub token, +# Anthropic API key). Add those yourself via the Portal's "Create Token" +# page (or POST /api/v1/keys directly) after the workers are up — see +# deploy/helm/scope/WORKERS.md. +# # Usage: # scripts/bootstrap-workers.sh --acr [options] # @@ -36,19 +40,8 @@ # --skip-copilot Skip the coder-acp-copilot worker entirely # --skip-claude-code Skip the coder-acp-claude-code worker entirely # --skip-build Reuse already-pushed images; skip `az acr build` -# --skip-secrets Skip the credential prompts/registration step # --skip-agent-registration # Skip the scripts/register-agent.sh step -# -y, --yes Non-interactive: never prompt (requires -# GITHUB_TOKEN/ANTHROPIC_API_KEY env vars, or -# use --skip-secrets) -# -# Environment variables (read instead of prompting, if set): -# GITHUB_TOKEN, ANTHROPIC_API_KEY -# -# Credentials are sent once to the already-running Token Manager over a -# temporary `kubectl port-forward` to the `api` Service and are never -# written to disk by this script. set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -63,9 +56,7 @@ ACR_NAME="" SKIP_COPILOT=false SKIP_CLAUDE_CODE=false SKIP_BUILD=false -SKIP_SECRETS=false SKIP_AGENT_REGISTRATION=false -NON_INTERACTIVE=false log() { printf '\n==> %s\n' "$1"; } die() { @@ -74,7 +65,7 @@ die() { } usage() { - sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' + sed -n '2,44p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' } while [ "$#" -gt 0 ]; do @@ -115,18 +106,10 @@ while [ "$#" -gt 0 ]; do SKIP_BUILD=true shift ;; - --skip-secrets) - SKIP_SECRETS=true - shift - ;; --skip-agent-registration) SKIP_AGENT_REGISTRATION=true shift ;; - -y | --yes) - NON_INTERACTIVE=true - shift - ;; -h | --help) usage exit 0 @@ -149,44 +132,17 @@ if [ -n "$VALUES_FILE" ] && [ ! -f "$VALUES_FILE" ]; then die "--values-file not found: $VALUES_FILE" fi -for bin in az helm kubectl curl node; do +for bin in az helm kubectl curl; do command -v "$bin" >/dev/null 2>&1 || die "$bin is required on PATH" done if [ "$SKIP_AGENT_REGISTRATION" = false ] && ! command -v yq >/dev/null 2>&1; then die "yq is required for agent registration (brew install yq), or pass --skip-agent-registration" fi -# ----------------------------------------------------------------------------- -# 1. Prompt for worker credentials up front (before any build/deploy work), -# so the whole run can proceed unattended once started. -# ----------------------------------------------------------------------------- -if [ "$SKIP_SECRETS" = false ]; then - if [ -z "${GITHUB_TOKEN:-}" ] && [ "$SKIP_COPILOT" = false ]; then - if [ "$NON_INTERACTIVE" = true ]; then - die "GITHUB_TOKEN is required (non-interactive mode); export it, use --skip-copilot, or --skip-secrets" - fi - echo "coder-acp-copilot needs a GitHub token (classic or fine-grained PAT) to drive the Copilot CLI." - echo "Leave blank to skip registering a credential now (you can add one later via the Portal's 'Create Token' page)." - read -r -s -p "GitHub token: " GITHUB_TOKEN - echo - fi - if [ -z "${ANTHROPIC_API_KEY:-}" ] && [ "$SKIP_CLAUDE_CODE" = false ]; then - if [ "$NON_INTERACTIVE" = true ]; then - die "ANTHROPIC_API_KEY is required (non-interactive mode); export it, use --skip-claude-code, or --skip-secrets" - fi - echo "coder-acp-claude-code needs an Anthropic API key to drive the Claude Agent SDK." - echo "Leave blank to skip registering a credential now (you can add one later via the Portal's 'Create Token' page)." - read -r -s -p "Anthropic API key: " ANTHROPIC_API_KEY - echo - fi -fi -GITHUB_TOKEN="${GITHUB_TOKEN:-}" -ANTHROPIC_API_KEY="${ANTHROPIC_API_KEY:-}" - cd "$REPO_ROOT" # ----------------------------------------------------------------------------- -# 2. Build + push worker images to ACR (az acr build — no local Docker +# 1. Build + push worker images to ACR (az acr build — no local Docker # needed). # ----------------------------------------------------------------------------- if [ "$SKIP_BUILD" = false ]; then @@ -211,8 +167,8 @@ else fi # ----------------------------------------------------------------------------- -# 3. Temporary port-forward to the API, used for both credential -# registration and agent registration below. +# 2. Temporary port-forward to the API, used for agent registration +# below. # ----------------------------------------------------------------------------- API_LOCAL_PORT=18080 API_URL="http://127.0.0.1:$API_LOCAL_PORT" @@ -241,39 +197,8 @@ stop_port_forward() { } trap stop_port_forward EXIT -register_secret() { - type="$1" - value="$2" - label="$3" - [ -n "$value" ] || { - echo "No $label provided — skipping credential registration (register one later via the Portal's 'Create Token' page)." - return 0 - } - start_port_forward - log "Registering $label with Token Manager" - payload=$(SCOPE_SECRET_TYPE="$type" SCOPE_SECRET_VALUE="$value" node -e \ - 'process.stdout.write(JSON.stringify({type: process.env.SCOPE_SECRET_TYPE, value: process.env.SCOPE_SECRET_VALUE}))') - status=$(curl -sS -o /tmp/scope-bootstrap-workers-register.json -w '%{http_code}' \ - -X POST "$API_URL/api/v1/keys" \ - -H 'Content-Type: application/json' \ - --data-binary "$payload") - case "$status" in - 2??) echo "Registered $label." ;; - *) - echo "Failed to register $label (HTTP $status):" >&2 - cat /tmp/scope-bootstrap-workers-register.json >&2 - return 1 - ;; - esac -} - -if [ "$SKIP_SECRETS" = false ]; then - [ "$SKIP_COPILOT" = false ] && register_secret "github-pat-classic" "$GITHUB_TOKEN" "GitHub token" - [ "$SKIP_CLAUDE_CODE" = false ] && register_secret "anthropic-api-key" "$ANTHROPIC_API_KEY" "Anthropic API key" -fi - # ----------------------------------------------------------------------------- -# 4. Enable the worker Deployments via `helm upgrade` (chart already has +# 3. Enable the worker Deployments via `helm upgrade` (chart already has # them templated, gated on workers..enabled — see # deploy/helm/scope/templates/workers/). # ----------------------------------------------------------------------------- @@ -318,7 +243,7 @@ if [ "${#HELM_SET_ARGS[@]}" -gt 0 ]; then fi # ----------------------------------------------------------------------------- -# 5. Register each enabled agent with the API. Uses the *-dev.yaml version +# 4. Register each enabled agent with the API. Uses the *-dev.yaml version # manifests shipped in-repo — fine for chart smoke-testing; for a real # deployment, register a version manifest whose imageTag/queueName match # what you actually built above, and keep From 4bda1fb5774eb6b7160bf104c893fb97f4c2790f Mon Sep 17 00:00:00 2001 From: Josh Duffney Date: Mon, 12 Oct 2026 13:38:23 -0500 Subject: [PATCH 17/18] fix(helm): default worker bootstrap image tag to git SHA, not latest Rerunning bootstrap-workers.sh with the mutable 'latest' tag could push new worker image content without changing the rendered Deployment PodTemplate, so Helm would leave the existing pods running under stale code even after a successful az acr build + helm upgrade. Default --image-tag to the current short commit SHA (with a '-dirty' suffix for uncommitted changes), falling back to a timestamp outside a git checkout, so each bootstrap run produces a distinct tag and actually rolls out. Addresses microsoft/scope#1481 (review thread 4248323413). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff --- scripts/bootstrap-workers.sh | 25 ++++++++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/scripts/bootstrap-workers.sh b/scripts/bootstrap-workers.sh index 8aee9226f..77df926e4 100755 --- a/scripts/bootstrap-workers.sh +++ b/scripts/bootstrap-workers.sh @@ -26,7 +26,14 @@ # --release Helm release name (default: scope) # --chart Helm chart path (default: deploy/helm/scope, # relative to repo root) -# --image-tag Tag to build/push/deploy (default: latest) +# --image-tag Tag to build/push/deploy (default: a unique tag +# derived from the current git commit, e.g. +# a1b2c3d or a1b2c3d-dirty). Rerunning this script +# with the mutable "latest" tag would push new +# image content without changing the rendered +# Deployment PodTemplate, so Helm would leave the +# old pods running — see deploy/helm/scope/ +# README.md "Phase 3" for details. # --values-file Same values override file passed to the # original `helm install ... -f ` (see # README.md "Install (Phase 2)"). Strongly @@ -50,7 +57,19 @@ REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" NAMESPACE="scope" RELEASE_NAME="scope" CHART_PATH="deploy/helm/scope" -IMAGE_TAG="latest" +# Default to a unique tag derived from the current commit so each run +# produces a distinct tag and the rendered Deployment PodTemplate actually +# changes (a mutable tag like "latest" lets Helm think nothing changed and +# leave old pods running — see --image-tag help above). Falls back to a +# timestamp if this isn't a git checkout. +if git -C "$REPO_ROOT" rev-parse --short HEAD >/dev/null 2>&1; then + IMAGE_TAG="$(git -C "$REPO_ROOT" rev-parse --short HEAD)" + if ! git -C "$REPO_ROOT" diff --quiet HEAD -- 2>/dev/null; then + IMAGE_TAG="${IMAGE_TAG}-dirty" + fi +else + IMAGE_TAG="$(date +%Y%m%d%H%M%S)" +fi VALUES_FILE="" ACR_NAME="" SKIP_COPILOT=false @@ -65,7 +84,7 @@ die() { } usage() { - sed -n '2,44p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' + sed -n '2,51p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' } while [ "$#" -gt 0 ]; do From d85ed7a687a953c594c6a6cbb70b1f862dce4b3a Mon Sep 17 00:00:00 2001 From: Josh Duffney Date: Mon, 12 Oct 2026 13:40:27 -0500 Subject: [PATCH 18/18] fix(helm): pin migrations to the app's revision; widen publish-images.yml filters - publish-images.yml: add packages/telemetry/** and tsconfig.base.json to the push path filters. All 5 backend Dockerfiles (api, judge, token-manager, scheduler, report-generator) COPY both, so a telemetry-only or root-tsconfig change previously didn't trigger a republish, leaving deployments on stale code until an unrelated matching change happened. - db-migrate Job: derive its image tag from global.imageTag via a new scope.dbMigrateTag helper instead of a hardcoded 'builder'. Setting global.imageTag= now also pins the migration Job to 'builder-' (already published by publish-images.yml's second build-push step), so application and schema changes stay on the same revision; defaults to the mutable 'builder' tag only when global.imageTag is left at its 'latest' default. jobs.dbMigrate.image.tag still wins if explicitly set. Addresses microsoft/scope#1481 (review threads 4248323498, 4248323472). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff --- .github/workflows/publish-images.yml | 2 ++ deploy/helm/scope/templates/_helpers.tpl | 23 +++++++++++++++++++ .../helm/scope/templates/jobs/db-migrate.yaml | 2 +- deploy/helm/scope/values.yaml | 11 ++++++--- 4 files changed, 34 insertions(+), 4 deletions(-) diff --git a/.github/workflows/publish-images.yml b/.github/workflows/publish-images.yml index 079e286cd..03eaf19c3 100644 --- a/.github/workflows/publish-images.yml +++ b/.github/workflows/publish-images.yml @@ -16,7 +16,9 @@ on: - "apps/portal/**" - "apps/workers/report-generator/**" - "packages/shared/**" + - "packages/telemetry/**" - "packages/db-migrations/**" + - "tsconfig.base.json" - "pnpm-workspace.yaml" - "pnpm-lock.yaml" - ".github/workflows/publish-images.yml" diff --git a/deploy/helm/scope/templates/_helpers.tpl b/deploy/helm/scope/templates/_helpers.tpl index 5dfcc7381..a3635c7f6 100644 --- a/deploy/helm/scope/templates/_helpers.tpl +++ b/deploy/helm/scope/templates/_helpers.tpl @@ -58,6 +58,29 @@ is unset, and to a bare (unregistered) name when no registry applies at all. {{- end -}} {{- end -}} +{{/* +The db-migrate Job's image tag. GHCR only publishes the api Dockerfile's +"builder" stage as "builder" (latest) and "builder-" — never a bare + or "latest" tag (see .github/workflows/publish-images.yml) — so this +can't reuse "scope.image"'s plain default-to-global.imageTag fallback. +`jobs.dbMigrate.image.tag` (empty by default) wins if explicitly set; +otherwise this derives "builder-" from whatever revision +the rest of the app is pinned to via `global.imageTag`, falling back to the +mutable "builder" tag when `global.imageTag` is still the default "latest" +(matching the one `latest`-equivalent tag GHCR actually publishes). This +keeps migrations and the application on the same revision when +`global.imageTag` is pinned to a commit SHA. +*/}} +{{- define "scope.dbMigrateTag" -}} +{{- if .Values.jobs.dbMigrate.image.tag -}} +{{- .Values.jobs.dbMigrate.image.tag -}} +{{- else if or (not .Values.global.imageTag) (eq .Values.global.imageTag "latest") -}} +{{- "builder" -}} +{{- else -}} +{{- printf "builder-%s" .Values.global.imageTag -}} +{{- end -}} +{{- end -}} + {{/* Name of the shared workload-identity ServiceAccount. Must match the Bicep template's federated subject. diff --git a/deploy/helm/scope/templates/jobs/db-migrate.yaml b/deploy/helm/scope/templates/jobs/db-migrate.yaml index 546e98290..00b725daa 100644 --- a/deploy/helm/scope/templates/jobs/db-migrate.yaml +++ b/deploy/helm/scope/templates/jobs/db-migrate.yaml @@ -49,7 +49,7 @@ spec: # tsx) — see jobs.dbMigrate.image.tag default ("builder") in # values.yaml. - name: db-migrate - image: {{ include "scope.image" (dict "image" .Values.jobs.dbMigrate.image "context" .) }} + image: {{ include "scope.image" (dict "image" (merge (dict "tag" (include "scope.dbMigrateTag" .)) .Values.jobs.dbMigrate.image) "context" .) }} imagePullPolicy: {{ .Values.global.imagePullPolicy }} workingDir: /app command: ["pnpm", "--filter", "db-migrations", "migrate:up"] diff --git a/deploy/helm/scope/values.yaml b/deploy/helm/scope/values.yaml index 092f985b3..f2d359eaa 100644 --- a/deploy/helm/scope/values.yaml +++ b/deploy/helm/scope/values.yaml @@ -298,9 +298,14 @@ jobs: # directly via tsx; the production `scope-api:latest` image only ships # compiled dist/ for db-migrations (no src/, no tsx), so this job must # run against the Dockerfile's `builder` stage instead. The GHCR - # publish workflow pushes that stage as a distinct `:builder` tag of - # the same scope-api image. - tag: "builder" + # publish workflow pushes that stage as "builder" (mutable) and + # "builder-" (immutable) variants of the same scope-api image. + # Left empty so `scope.dbMigrateTag` (templates/_helpers.tpl) derives + # "builder-" automatically when you pin + # `global.imageTag` to a commit SHA, keeping migrations on the same + # revision as the rest of the app — set this explicitly only to + # override that derivation. + tag: "" # NOT YET IMPLEMENTED as a chart template. Registering the ACP # coding-agent types (scripts/register-agent.sh + each worker's # agent.yaml) only matters once those workers are actually deployed —