diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8cdec8fda..a5df5b542 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -524,6 +524,9 @@ jobs: - name: Prove docs-only failed-canary rollback run: infrastructure/deployment/scripts/test-deploy-docs-rollback.sh + - name: Prove OpenFGA model rollout and rollback + run: infrastructure/deployment/scripts/test-deploy-openfga-model-rollout.sh + - name: Validate publication verifier run: python3 -m py_compile infrastructure/deployment/scripts/verify-docs-publication.py diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 080efceed..7ca0b1afc 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -413,6 +413,13 @@ git. `scripts/bootstrap-openfga.ps1` creates a development store/model, imports demo relationships, and writes non-secret local identifiers after the compose service is available. +Production keeps one durable OpenFGA store and pins every application request +to an immutable authorization model ID. First-store bootstrap records that ID +and the SHA-256 of the repository model. Each product deployment writes and +atomically pins a new model before recreating API and worker containers only +when the repository model digest changes; unchanged models are no-ops. Failed +deployment rollback restores the previous images and previous model pin. + ## Build And Run ```powershell diff --git a/docs/decisions/0017-pin-openfga-models-to-product-releases.md b/docs/decisions/0017-pin-openfga-models-to-product-releases.md new file mode 100644 index 000000000..cb16aa3fe --- /dev/null +++ b/docs/decisions/0017-pin-openfga-models-to-product-releases.md @@ -0,0 +1,74 @@ +# 0017 — Pin OpenFGA Models To Product Releases + +## Status + +Accepted on 2026-07-29 by explicit project-owner direction. + +The repository-required independent Claude challenge was unavailable because +the configured account remained over quota. The project owner had already +directed the delivery loop to continue without that reviewer and then reported +the production authorization failure for repair. The proposal, strongest +counterargument, repository evidence, current official OpenFGA guidance, and +rollback test are recorded in the +[increment design](../increments/active/2026-07-29-openfga-model-rollout/design.md). + +## Context + +OpenFGA authorization models are immutable. Writing one produces a new model +ID, while tuples remain in the store. Production OrgMemory requests explicitly +send one configured model ID so every binary uses a known policy version. + +The initial deployment created one store and model, then persisted both IDs. +Subsequent deployments updated application images and the repository model but +never wrote another model version. Code could therefore start checking a +relation that did not exist in the pinned production model. This happened when +`can_manage_ai` shipped: authorization correctly failed closed, but valid +organization administrators lost access to the new AI settings endpoints. + +## Decision + +The repository OpenFGA model is a versioned product-release input. + +- First-store bootstrap persists the store ID, model ID, and SHA-256 of the + model bytes. +- A production deployment compares the release model digest with the pinned + digest. +- A missing or changed digest writes a new immutable model into the same store + before application containers are recreated. +- The deployment atomically persists the returned model ID and digest, and all + application calls remain explicitly pinned to that model ID. +- An unchanged model is a no-op and does not create another immutable version. +- Failed deployment rollback restores the previous images, model ID, and + digest. A newly written but unused model may remain in the store. + +Tuple migration remains an explicit concern for model changes that add, rename, +or remove tuple-bearing relations. The deployment mechanism orders and pins the +model; it does not invent or rewrite tuples. + +## Strongest Counterargument + +Omit `authorization_model_id` and let OpenFGA select the latest model. That +removes the configuration update and would have hidden this deployment bug. + +This is rejected because a model write would then change authorization for +running replicas independently of their binary version. An accidental write +could affect production immediately, gradual rollout would be impossible, and +application rollback would not restore the prior policy. OpenFGA recommends +pinning a specific model ID in production. + +## Consequences + +- Application and authorization policy rollback are one environment rollback. +- Legacy environments intentionally write one current model because they have + no stored digest. +- Identical product releases do not accumulate model versions. +- Model changes must keep the immediately previous binary/model combination + rollback-safe or explicitly use a staged migration. +- Deployment CI must test upgrade ordering, unchanged-model no-op, and rollback + of the model pin. + +## References + +- [OpenFGA immutable authorization models](https://openfga.dev/docs/getting-started/immutable-models) +- [OpenFGA model migrations](https://openfga.dev/docs/modeling/migrating/migrating-models) +- [OpenFGA CLI model versions](https://openfga.dev/docs/getting-started/cli) diff --git a/docs/increments/active/2026-07-29-openfga-model-rollout/design.md b/docs/increments/active/2026-07-29-openfga-model-rollout/design.md new file mode 100644 index 000000000..82634bde7 --- /dev/null +++ b/docs/increments/active/2026-07-29-openfga-model-rollout/design.md @@ -0,0 +1,97 @@ +# OpenFGA Model Rollout Repair + +## Problem + +The production application pins every authorization request to +`ORGMEMORY_OPENFGA_AUTHORIZATION_MODEL_ID`, but the deployment lifecycle writes +that identifier only during first-store bootstrap. Later releases update the +repository-owned `model.fga` without writing a new immutable model version or +changing the pinned identifier. + +This became user-visible when the multi-provider control plane added +`organization#can_manage_ai`. The web and API images deployed successfully, +while the production API continued checking an older model that did not contain +that relation. Organization administrators could enter the admin shell but the +Language Models and Index Settings requests failed closed. + +Direct SSH evidence was unavailable during diagnosis because the ZM host timed +out from the current workstation. Repository and workflow evidence still proves +the lifecycle defect: + +- `bootstrap-openfga.sh` creates a store and model only when both IDs are empty; +- `deploy.sh` requires and reuses the existing model ID without writing the + current `model.fga`; +- production successfully deployed the application commit containing + `can_manage_ai`. + +## Selected Design + +Treat the authorization model as a versioned release input: + +1. Keep one durable OpenFGA store and all existing tuples. +2. Compute SHA-256 over the repository model used by the release. +3. If that digest differs from the digest pinned in the host environment, write + the model into the existing store with the official OpenFGA CLI. +4. Parse and validate the returned `authorization_model_id`. +5. Atomically persist the new model ID and digest before recreating API and + worker containers. +6. Keep every application request explicitly pinned to that model ID. +7. On a failed deployment, restore the prior environment and recreate the prior + image set with its prior model ID. The unused immutable model version may + remain in OpenFGA. + +First-store bootstrap writes both the initial model ID and digest. Existing +installations have no digest, intentionally forcing one model write on the first +deployment containing this repair. + +The official OpenFGA guidance says models are immutable, each write creates a +new version, production clients should pin a specific model ID, and adding a +relation requires writing the model before application code starts using it. + +## Strongest Counterargument + +The application could stop sending an authorization model ID and let OpenFGA +use the latest version. That would make a newly written model visible without +updating application configuration. + +This is rejected because "latest" disconnects a running binary from the policy +version it was tested against. A later or accidental model write could change +authorization for every replica immediately, and rollback of the application +would not restore its compatible policy. Explicit pinning is the safer +production contract. + +Writing a model on every deployment is also rejected. OpenFGA models are +immutable and cannot be deleted, so identical releases would accumulate +unnecessary versions. The digest makes unchanged model delivery a no-op while +forcing legacy installations through one repair write. + +## Architecture Challenge + +This changes the authorization deployment boundary and therefore requires an +independent challenge. The configured Claude reviewer remained unavailable due +to the previously reported quota limit. The project owner had already directed +this session to continue without the Claude discussion step and explicitly +asked for the production bug to be fixed. The counterargument above, repository +evidence, official OpenFGA lifecycle guidance, rollback behavior, and negative +tests are recorded here in place of that unavailable review. + +## Scope + +- production Compose operations service for writing the repository model; +- first-store bootstrap model digest; +- production deployment model write, atomic pin, no-op, and rollback; +- deterministic shell regression coverage; +- deployment runbook, architecture, and authorization coverage updates. + +No OpenFGA relation, tuple, application role, or browser authorization bypass is +changed by this repair. + +## Exit Gates + +- OpenFGA model validation and store tests pass; +- production Compose interpolation and shellcheck pass; +- deterministic tests prove upgrade, unchanged-model no-op, and failed-canary + rollback to the prior model ID; +- documentation checks pass; +- PR CI passes, the PR merges, production deploys the immutable release, and an + authenticated administrator can load both affected screens. diff --git a/docs/increments/active/2026-07-29-openfga-model-rollout/plan.md b/docs/increments/active/2026-07-29-openfga-model-rollout/plan.md new file mode 100644 index 000000000..7c960de82 --- /dev/null +++ b/docs/increments/active/2026-07-29-openfga-model-rollout/plan.md @@ -0,0 +1,17 @@ +# OpenFGA Model Rollout Repair Plan + +- [x] Trace the UI denial through the API guard and production deployment model + pin. +- [x] Verify OpenFGA model-write and immutable-version behavior against current + official documentation. +- [x] Add a model-write operation to production Compose. +- [x] Persist the model digest during first-store bootstrap. +- [x] Write and atomically pin a changed model before API/worker recreation. +- [x] Preserve the previous model ID and digest across failed deployment + rollback. +- [x] Add deterministic upgrade, no-op, and rollback tests to deployment CI. +- [x] Reconcile architecture, deployment runbook, authorization spec/coverage, + and roadmap. +- [x] Run OpenFGA, deployment, documentation, and repository hygiene gates. +- [ ] Open the PR, resolve actionable review/CI findings, merge, deploy, and + verify the two administrator screens. diff --git a/docs/roadmap.md b/docs/roadmap.md index bb108b7b5..7669aaa61 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -29,6 +29,7 @@ The table is a delivery index, not a second description of current behavior. | Increment | Status | Remaining gate | | --- | --- | --- | +| [OpenFGA model rollout repair](increments/active/2026-07-29-openfga-model-rollout/plan.md) | active | version and pin the repository model during deployment, then verify the affected admin screens | | [Production CI/CD and ZM runtime](increments/active/2026-07-25-production-cicd-zm/plan.md) | active | shared-PostgreSQL cutover, restore proof, end-to-end runtime and rollback gates | | [Reproducible demo bootstrap](increments/active/2026-07-22-reproducible-demo-bootstrap/plan.md) | active | public ingestion and permission-evaluation run | | [Slack connector live proof](increments/active/2026-07-23-slack-connector-live/plan.md) | active | live workspace crawl and next-crawl revocation | diff --git a/docs/runbooks/production-zm-deployment.md b/docs/runbooks/production-zm-deployment.md index 942b55921..03c1ade7c 100644 --- a/docs/runbooks/production-zm-deployment.md +++ b/docs/runbooks/production-zm-deployment.md @@ -66,8 +66,8 @@ Build or pull the exact production images, then initialize OpenFGA: ``` The command creates one store from the repository authorization model and saves -the store/model IDs to `.env.production`. It refuses to create a second store -when identifiers already exist. +the store ID, immutable model ID, and model SHA-256 to `.env.production`. It +refuses to create a second store when identifiers already exist. ## Nginx Proxy Manager @@ -168,9 +168,12 @@ The deployment: 5. idempotently checks database roles/databases; 6. backs up OrgMemory, OpenFGA, and Keycloak; 7. runs OpenFGA migration and API-owned Flyway migration; -8. starts the private runtime; -9. checks web, API, MCP, Keycloak, and optionally the public endpoints; -10. restores the previous image references when a gate fails. +8. when the repository authorization-model digest changed or the legacy digest + is absent, writes a new immutable model into the existing store and + atomically pins its ID before application recreation; +9. starts the private runtime; +10. checks web, API, MCP, Keycloak, and optionally the public endpoints; +11. restores the previous image references and model pin when a gate fails. `ORGMEMORY_BACKUP_UID` and `ORGMEMORY_BACKUP_GID` must match the owner of `ORGMEMORY_BACKUP_DIRECTORY`. The one-shot backup container drops all Linux @@ -181,6 +184,12 @@ Database migrations must remain backward compatible with the immediately previous application image. The rollback does not reverse a committed database migration. +OpenFGA models are also immutable and remain in the store after a failed +canary. Rollback makes that unused version inert by restoring the previous +model ID. Model changes that need tuple migration must stage that migration +explicitly; the release script orders and pins models but does not synthesize +tuples. + Logical backup rotation is an operator responsibility, not part of the transactional deployment script. A scheduled retention job may prune old timestamped directories only after a newer `SHA256SUMS` set has passed a restore diff --git a/docs/specs/domains/ai-model-control-plane.md b/docs/specs/domains/ai-model-control-plane.md index d5c2138c8..62eb8830a 100644 --- a/docs/specs/domains/ai-model-control-plane.md +++ b/docs/specs/domains/ai-model-control-plane.md @@ -4,7 +4,7 @@ Source: `core/src/main/java/com/orgmemory/core/ai`, `integrations/ai-openai-compatible`, `apps/api/.../AdminAiModelController`, and `apps/web/src/features/admin/components/admin-language-models-page.tsx`. -Reconciled: `2026-07-29-multi-provider-model-control-plane (d7ca979)`. +Reconciled: `2026-07-29-openfga-model-rollout (c9a366b)`. ## Current Behavior @@ -44,6 +44,12 @@ Index Settings is a separate read-only surface. The embedding provider, model, dimensions, and cosine metric cannot be mutated through the chat control plane; a geometry change requires a versioned embedding profile and reindex lifecycle. +Both administration surfaces require OpenFGA `organization#can_manage_ai`. +Production writes and pins the repository authorization model before a release +whose model digest changed starts application containers. A legacy deployment +with no stored digest writes the current model once. A failed release restores +the previous model ID with its previous image set. + ## Source Modules - `core.ai` @@ -55,3 +61,4 @@ a geometry change requires a versioned embedding profile and reindex lifecycle. - [0006](../../decisions/0006-ai-tasks-route-through-provider-adapters.md) - [0008](../../decisions/0008-worker-owns-ingestion-and-derived-indexes.md) +- [0017](../../decisions/0017-pin-openfga-models-to-product-releases.md) diff --git a/docs/tests/domains/ai-model-control-plane.md b/docs/tests/domains/ai-model-control-plane.md index cf521712d..9be128bf8 100644 --- a/docs/tests/domains/ai-model-control-plane.md +++ b/docs/tests/domains/ai-model-control-plane.md @@ -5,11 +5,12 @@ Source: `core/src/test/java/com/orgmemory/core/ai`, `apps/api/src/test/java/com/orgmemory/api/admin`, `integrations/authorization-openfga/src/test/openfga`, and the admin web build. -Reconciled: `2026-07-29-multi-provider-model-control-plane (d7ca979)`. +Reconciled: `2026-07-29-openfga-model-rollout (c9a366b)`. | Behavior | Evidence | Status | | --- | --- | --- | | Only organization administrators receive `can_manage_ai` | OpenFGA `store.fga.yaml`, `PermissionsAdminIntegrationTests` | covered | +| Production writes and pins a changed OpenFGA model before application recreation, skips identical bytes, and restores the previous pin on failed canary | `test-deploy-openfga-model-rollout.sh` | covered | | Secrets are encrypted and absent from views/log rendering | `AiGatewayAdministrationServiceTests#storesOnlyCiphertextAndKeepsCredentialsOutOfViewsAndLogs` | covered | | Cross-tenant profile IDs are opaque and cannot rotate credentials | `AiGatewayAdministrationServiceTests#aProfileIdFromAnotherOrganizationIsOpaqueAndCannotRotateASecret` | covered | | Profile, credential, and route actor FKs cannot cross tenant boundaries | `PermissionsAdminIntegrationTests#aiControlPlaneActorReferencesCannotCrossTenantBoundaries` | covered | diff --git a/infrastructure/deployment/compose.production.yaml b/infrastructure/deployment/compose.production.yaml index 8a50a90dd..b0fc64329 100644 --- a/infrastructure/deployment/compose.production.yaml +++ b/infrastructure/deployment/compose.production.yaml @@ -187,6 +187,35 @@ services: cap_drop: - ALL + openfga-model-write: + image: openfga/cli:v0.7.19@sha256:2e0e250043ef480a9162623dbf1ff7a62a1a2cb96a79cb20577b144994ab114d + profiles: + - ops + command: + - model + - write + - --store-id + - ${ORGMEMORY_OPENFGA_STORE_ID:-} + - --file + - /model/model.fga + - --format + - fga + - --api-url + - http://openfga:8080 + depends_on: + openfga-ready: + condition: service_completed_successfully + networks: + - orgmemory-internal + volumes: + - ../../integrations/authorization-openfga/src/main/openfga/model.fga:/model/model.fga:ro + restart: "no" + read_only: true + security_opt: + - no-new-privileges:true + cap_drop: + - ALL + postgres-backup: image: ${ORGMEMORY_POSTGRES_IMAGE:?Set ORGMEMORY_POSTGRES_IMAGE} profiles: diff --git a/infrastructure/deployment/production.env.example b/infrastructure/deployment/production.env.example index 729196d4e..6ce3dd0d8 100644 --- a/infrastructure/deployment/production.env.example +++ b/infrastructure/deployment/production.env.example @@ -26,6 +26,9 @@ OPENFGA_DB_PASSWORD= OPENFGA_DATASTORE_URI=postgres://openfga:@postgres:5432/openfga?sslmode=disable ORGMEMORY_OPENFGA_STORE_ID= ORGMEMORY_OPENFGA_AUTHORIZATION_MODEL_ID= +# SHA-256 of the repository model pinned by the model ID above. The bootstrap +# and deployment scripts own this value. +ORGMEMORY_OPENFGA_MODEL_SHA256= KEYCLOAK_DB_NAME=keycloak KEYCLOAK_DB_USER=keycloak diff --git a/infrastructure/deployment/scripts/bootstrap-openfga.sh b/infrastructure/deployment/scripts/bootstrap-openfga.sh index de71efb84..7bbac651c 100755 --- a/infrastructure/deployment/scripts/bootstrap-openfga.sh +++ b/infrastructure/deployment/scripts/bootstrap-openfga.sh @@ -1,9 +1,10 @@ #!/usr/bin/env bash set -Eeuo pipefail -repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" -compose_file="$repo_root/infrastructure/deployment/compose.production.yaml" +repo_root="${ORGMEMORY_REPO_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)}" +compose_file="${ORGMEMORY_COMPOSE_FILE:-$repo_root/infrastructure/deployment/compose.production.yaml}" environment_file="${ORGMEMORY_ENV_FILE:-$repo_root/.env.production}" +model_file="${ORGMEMORY_OPENFGA_MODEL_FILE:-$repo_root/integrations/authorization-openfga/src/main/openfga/model.fga}" if [[ ! -f "$environment_file" ]]; then printf 'Missing production environment file: %s\n' "$environment_file" >&2 @@ -77,24 +78,38 @@ PY store_id="${identifiers[0]}" model_id="${identifiers[1]}" +model_sha256="$(sha256sum "$model_file" | awk '{ print $1 }')" -update_environment_key() { - local key="$1" - local value="$2" +update_environment_model() { + local store_id_value="$1" + local model_id_value="$2" + local model_sha256_value="$3" local temporary_file temporary_file="$(mktemp)" - awk -v key="$key" -v value="$value" ' - BEGIN { found = 0 } - $0 ~ "^" key "=" { - print key "=" value - found = 1 - next + awk \ + -v store_id="$store_id_value" \ + -v model_id="$model_id_value" \ + -v model_sha256="$model_sha256_value" ' + BEGIN { + values["ORGMEMORY_OPENFGA_STORE_ID"] = store_id + values["ORGMEMORY_OPENFGA_AUTHORIZATION_MODEL_ID"] = model_id + values["ORGMEMORY_OPENFGA_MODEL_SHA256"] = model_sha256 + } + { + split($0, parts, "=") + if (parts[1] in values) { + print parts[1] "=" values[parts[1]] + seen[parts[1]] = 1 + } else { + print + } } - { print } END { - if (!found) { - print key "=" value + for (key in values) { + if (!seen[key]) { + print key "=" values[key] + } } } ' "$environment_file" > "$temporary_file" @@ -103,8 +118,7 @@ update_environment_key() { rm -f "$temporary_file" } -update_environment_key ORGMEMORY_OPENFGA_STORE_ID "$store_id" -update_environment_key ORGMEMORY_OPENFGA_AUTHORIZATION_MODEL_ID "$model_id" +update_environment_model "$store_id" "$model_id" "$model_sha256" -printf 'OpenFGA store and model created. Identifiers were saved to %s.\n' \ +printf 'OpenFGA store and model created. The pinned model configuration was saved to %s.\n' \ "$environment_file" diff --git a/infrastructure/deployment/scripts/deploy.sh b/infrastructure/deployment/scripts/deploy.sh index d98524e27..402c7e433 100755 --- a/infrastructure/deployment/scripts/deploy.sh +++ b/infrastructure/deployment/scripts/deploy.sh @@ -7,10 +7,13 @@ if [[ "$#" -ne 1 || ! "$1" =~ ^[0-9a-f]{40}$ ]]; then fi commit_sha="$1" -repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" -compose_file="$repo_root/infrastructure/deployment/compose.production.yaml" +repo_root="${ORGMEMORY_REPO_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)}" +compose_file="${ORGMEMORY_COMPOSE_FILE:-$repo_root/infrastructure/deployment/compose.production.yaml}" environment_file="${ORGMEMORY_ENV_FILE:-$repo_root/.env.production}" runtime_root="${ORGMEMORY_RUNTIME_ROOT:-/apps/orgmemory-runtime}" +model_file="${ORGMEMORY_OPENFGA_MODEL_FILE:-$repo_root/integrations/authorization-openfga/src/main/openfga/model.fga}" +keycloak_configuration_script="${ORGMEMORY_KEYCLOAK_CONFIGURATION_SCRIPT:-$repo_root/infrastructure/deployment/scripts/configure-keycloak-mcp.sh}" +smoke_script="${ORGMEMORY_SMOKE_SCRIPT:-$repo_root/infrastructure/deployment/scripts/smoke-production.sh}" lock_file="$runtime_root/deploy.lock" release_stamp="$(date -u +%Y%m%dT%H%M%SZ)" release_environment="$runtime_root/releases/$release_stamp.env" @@ -99,6 +102,39 @@ replace_image_references() { rm -f "$temporary_file" } +update_openfga_model_configuration() { + local model_id="$1" + local model_sha256="$2" + local temporary_file + temporary_file="$(mktemp)" + + awk -v model_id="$model_id" -v model_sha256="$model_sha256" ' + BEGIN { + values["ORGMEMORY_OPENFGA_AUTHORIZATION_MODEL_ID"] = model_id + values["ORGMEMORY_OPENFGA_MODEL_SHA256"] = model_sha256 + } + { + split($0, parts, "=") + if (parts[1] in values) { + print parts[1] "=" values[parts[1]] + seen[parts[1]] = 1 + } else { + print + } + } + END { + for (key in values) { + if (!seen[key]) { + print key "=" values[key] + } + } + } + ' "$environment_file" > "$temporary_file" + + install -m 0600 "$temporary_file" "$environment_file" + rm -f "$temporary_file" +} + rollback() { local exit_code="$?" trap - ERR @@ -126,10 +162,11 @@ rollback() { trap rollback ERR replace_image_references -install -m 0600 "$environment_file" "$release_environment" openfga_store_id="$(read_environment_value ORGMEMORY_OPENFGA_STORE_ID)" openfga_model_id="$(read_environment_value ORGMEMORY_OPENFGA_AUTHORIZATION_MODEL_ID)" +openfga_model_sha256="$(read_environment_value ORGMEMORY_OPENFGA_MODEL_SHA256)" +release_model_sha256="$(sha256sum "$model_file" | awk '{ print $1 }')" public_smoke="${ORGMEMORY_REQUIRE_PUBLIC_SMOKE:-$(read_environment_value ORGMEMORY_REQUIRE_PUBLIC_SMOKE)}" if [[ -z "$openfga_store_id" || -z "$openfga_model_id" ]]; then @@ -147,6 +184,32 @@ compose=( "${compose[@]}" pull "${compose[@]}" run --rm postgres-bootstrap "${compose[@]}" --profile ops run --rm postgres-backup +"${compose[@]}" up -d openfga +"${compose[@]}" run --rm --no-deps openfga-ready + +if [[ "$openfga_model_sha256" != "$release_model_sha256" ]]; then + model_write_json="$( + "${compose[@]}" --profile ops run --rm --no-deps openfga-model-write + )" + new_openfga_model_id="$( + MODEL_WRITE_JSON="$model_write_json" python3 - <<'PY' +import json +import os + +payload = json.loads(os.environ["MODEL_WRITE_JSON"]) +model_id = payload.get("authorization_model_id") or payload.get("id") +if not model_id: + raise SystemExit("OpenFGA CLI response did not contain an authorization model id") +print(model_id) +PY + )" + update_openfga_model_configuration \ + "$new_openfga_model_id" \ + "$release_model_sha256" +fi + +install -m 0600 "$environment_file" "$release_environment" + "${compose[@]}" up \ -d \ --wait \ @@ -154,11 +217,11 @@ compose=( --remove-orphans ORGMEMORY_ENV_FILE="$environment_file" \ - "$repo_root/infrastructure/deployment/scripts/configure-keycloak-mcp.sh" + "$keycloak_configuration_script" ORGMEMORY_ENV_FILE="$environment_file" \ ORGMEMORY_REQUIRE_PUBLIC_SMOKE="${public_smoke:-true}" \ - "$repo_root/infrastructure/deployment/scripts/smoke-production.sh" + "$smoke_script" printf '%s\n' "$commit_sha" > "$current_commit_file" trap - ERR diff --git a/infrastructure/deployment/scripts/test-deploy-openfga-model-rollout.sh b/infrastructure/deployment/scripts/test-deploy-openfga-model-rollout.sh new file mode 100755 index 000000000..7441458a4 --- /dev/null +++ b/infrastructure/deployment/scripts/test-deploy-openfga-model-rollout.sh @@ -0,0 +1,251 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" +deploy_script="$repo_root/infrastructure/deployment/scripts/deploy.sh" +bootstrap_script="$repo_root/infrastructure/deployment/scripts/bootstrap-openfga.sh" +temporary_root="$(mktemp -d)" + +cleanup() { + if [[ -n "${temporary_root:-}" && "$temporary_root" == /tmp/* ]]; then + rm -rf -- "$temporary_root" + fi +} +trap cleanup EXIT + +old_sha="1111111111111111111111111111111111111111" +candidate_sha="2222222222222222222222222222222222222222" +second_candidate_sha="3333333333333333333333333333333333333333" +old_model_id="01J00000000000000000000000" +new_model_id="01J11111111111111111111111" +store_id="01J22222222222222222222222" +old_model_sha256="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" +model_file="$temporary_root/model.fga" +compose_file="$temporary_root/compose.production.yaml" +stub_bin="$temporary_root/bin" +docker_log="$temporary_root/docker.log" +keycloak_script="$temporary_root/configure-keycloak.sh" +smoke_success_script="$temporary_root/smoke-success.sh" +smoke_rollback_script="$temporary_root/smoke-rollback.sh" +smoke_count="$temporary_root/smoke.count" + +install -d -m 0700 "$stub_bin" +printf 'model\n schema 1.1\n\ntype user\n' > "$model_file" +printf 'services: {}\n' > "$compose_file" + +cat > "$stub_bin/docker" <<'SH' +#!/usr/bin/env bash +set -euo pipefail +printf '%s\n' "$*" >> "$ORGMEMORY_TEST_DOCKER_LOG" +if [[ "$*" == *"openfga-model-write"* ]]; then + printf '{"authorization_model_id":"%s"}\n' "$ORGMEMORY_TEST_NEW_MODEL_ID" +elif [[ "$*" == *"openfga-bootstrap"* ]]; then + printf \ + '{"store":{"id":"%s"},"model":{"authorization_model_id":"%s"}}\n' \ + "$ORGMEMORY_TEST_STORE_ID" \ + "$ORGMEMORY_TEST_NEW_MODEL_ID" +fi +SH +chmod +x "$stub_bin/docker" + +cat > "$keycloak_script" <<'SH' +#!/usr/bin/env bash +set -euo pipefail +SH +chmod +x "$keycloak_script" + +cat > "$smoke_success_script" <<'SH' +#!/usr/bin/env bash +set -euo pipefail +SH +chmod +x "$smoke_success_script" + +cat > "$smoke_rollback_script" <<'SH' +#!/usr/bin/env bash +set -euo pipefail +count=0 +if [[ -f "$ORGMEMORY_TEST_SMOKE_COUNT" ]]; then + count="$(cat "$ORGMEMORY_TEST_SMOKE_COUNT")" +fi +count=$((count + 1)) +printf '%s\n' "$count" > "$ORGMEMORY_TEST_SMOKE_COUNT" +if [[ "$count" -eq 1 ]]; then + exit 1 +fi +SH +chmod +x "$smoke_rollback_script" + +write_environment() { + local path="$1" + local model_sha256="${2:-}" + + cat > "$path" < "$bootstrap_environment" <<'EOF' +ORGMEMORY_OPENFGA_STORE_ID= +ORGMEMORY_OPENFGA_AUTHORIZATION_MODEL_ID= +ORGMEMORY_OPENFGA_MODEL_SHA256= +EOF +chmod 0600 "$bootstrap_environment" +: > "$docker_log" +run_bootstrap "$bootstrap_environment" +grep -Fxq "ORGMEMORY_OPENFGA_STORE_ID=$store_id" "$bootstrap_environment" +assert_model_configuration \ + "$bootstrap_environment" \ + "$new_model_id" \ + "$release_model_sha256" + +# A legacy environment has no model digest. Its first deployment must write and +# pin the repository model before the application stack is recreated. +upgrade_root="$temporary_root/upgrade-runtime" +upgrade_environment="$temporary_root/upgrade.env" +install -d -m 0700 "$upgrade_root/releases" +write_environment "$upgrade_environment" +: > "$docker_log" +run_deploy \ + "$upgrade_environment" \ + "$upgrade_root" \ + "$smoke_success_script" \ + "$candidate_sha" + +assert_model_configuration \ + "$upgrade_environment" \ + "$new_model_id" \ + "$release_model_sha256" +grep -Fxq "$candidate_sha" "$upgrade_root/current-commit" +grep -q 'openfga-model-write' "$docker_log" + +model_write_line="$(grep -n 'openfga-model-write' "$docker_log" | head -1 | cut -d: -f1)" +application_up_line="$( + grep -n -- 'up -d --wait --wait-timeout 240 --remove-orphans' "$docker_log" \ + | head -1 \ + | cut -d: -f1 +)" +if [[ "$model_write_line" -ge "$application_up_line" ]]; then + printf 'The application stack started before the changed model was pinned.\n' >&2 + exit 1 +fi + +# A second release with identical model bytes must retain the pinned version and +# avoid creating an unnecessary immutable model. +no_op_root="$temporary_root/no-op-runtime" +no_op_environment="$temporary_root/no-op.env" +install -d -m 0700 "$no_op_root/releases" +install -m 0600 "$upgrade_environment" "$no_op_environment" +printf '%s\n' "$candidate_sha" > "$no_op_root/current-commit" +: > "$docker_log" +run_deploy \ + "$no_op_environment" \ + "$no_op_root" \ + "$smoke_success_script" \ + "$second_candidate_sha" + +assert_model_configuration \ + "$no_op_environment" \ + "$new_model_id" \ + "$release_model_sha256" +if grep -q 'openfga-model-write' "$docker_log"; then + printf 'An unchanged authorization model created another immutable version.\n' >&2 + exit 1 +fi + +# A failed canary after a changed model write must restore the previous image, +# model ID, and digest. The newly written immutable model remains inert. +rollback_root="$temporary_root/rollback-runtime" +rollback_environment="$temporary_root/rollback.env" +install -d -m 0700 "$rollback_root/releases" +write_environment "$rollback_environment" "$old_model_sha256" +printf '%s\n' "$old_sha" > "$rollback_root/current-commit" +: > "$docker_log" +: > "$smoke_count" + +set +e +run_deploy \ + "$rollback_environment" \ + "$rollback_root" \ + "$smoke_rollback_script" \ + "$candidate_sha" +status="$?" +set -e + +if [[ "$status" -eq 0 ]]; then + printf 'Expected the forced production canary to fail.\n' >&2 + exit 1 +fi + +assert_model_configuration \ + "$rollback_environment" \ + "$old_model_id" \ + "$old_model_sha256" +grep -Fxq \ + "ORGMEMORY_API_IMAGE=ghcr.io/kl3init/orgmemory-api:sha-$old_sha" \ + "$rollback_environment" +grep -Fxq "1" "$smoke_count" +grep -q 'openfga-model-write' "$docker_log" +grep -q 'up -d --remove-orphans' "$docker_log" + +printf 'OpenFGA model rollout, no-op, and rollback contracts passed.\n'