kl3inIT · Jul 29, 2026 · Jul 29, 2026 · Jul 29, 2026 · coderabbitai · Jul 29, 2026
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -524,6 +524,9 @@ jobs:
       - name: Prove docs-only failed-canary rollback
         run: infrastructure/deployment/scripts/test-deploy-docs-rollback.sh
 
+      - name: Prove OpenFGA model rollout and rollback
+        run: infrastructure/deployment/scripts/test-deploy-openfga-model-rollout.sh
+
       - name: Validate publication verifier
         run: python3 -m py_compile infrastructure/deployment/scripts/verify-docs-publication.py
 

diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
@@ -413,6 +413,13 @@ git. `scripts/bootstrap-openfga.ps1` creates a development store/model, imports
 demo relationships, and writes non-secret local identifiers after the compose
 service is available.
 
+Production keeps one durable OpenFGA store and pins every application request
+to an immutable authorization model ID. First-store bootstrap records that ID
+and the SHA-256 of the repository model. Each product deployment writes and
+atomically pins a new model before recreating API and worker containers only
+when the repository model digest changes; unchanged models are no-ops. Failed
+deployment rollback restores the previous images and previous model pin.
+
 ## Build And Run
 
 ```powershell

diff --git a/docs/decisions/0017-pin-openfga-models-to-product-releases.md b/docs/decisions/0017-pin-openfga-models-to-product-releases.md
@@ -0,0 +1,74 @@
+# 0017 — Pin OpenFGA Models To Product Releases
+
+## Status
+
+Accepted on 2026-07-29 by explicit project-owner direction.
+
+The repository-required independent Claude challenge was unavailable because
+the configured account remained over quota. The project owner had already
+directed the delivery loop to continue without that reviewer and then reported
+the production authorization failure for repair. The proposal, strongest
+counterargument, repository evidence, current official OpenFGA guidance, and
+rollback test are recorded in the
+[increment design](../increments/active/2026-07-29-openfga-model-rollout/design.md).
+
+## Context
+
+OpenFGA authorization models are immutable. Writing one produces a new model
+ID, while tuples remain in the store. Production OrgMemory requests explicitly
+send one configured model ID so every binary uses a known policy version.
+
+The initial deployment created one store and model, then persisted both IDs.
+Subsequent deployments updated application images and the repository model but
+never wrote another model version. Code could therefore start checking a
+relation that did not exist in the pinned production model. This happened when
+`can_manage_ai` shipped: authorization correctly failed closed, but valid
+organization administrators lost access to the new AI settings endpoints.
+
+## Decision
+
+The repository OpenFGA model is a versioned product-release input.
+
+- First-store bootstrap persists the store ID, model ID, and SHA-256 of the
+  model bytes.
+- A production deployment compares the release model digest with the pinned
+  digest.
+- A missing or changed digest writes a new immutable model into the same store
+  before application containers are recreated.
+- The deployment atomically persists the returned model ID and digest, and all
+  application calls remain explicitly pinned to that model ID.
+- An unchanged model is a no-op and does not create another immutable version.
+- Failed deployment rollback restores the previous images, model ID, and
+  digest. A newly written but unused model may remain in the store.
+
+Tuple migration remains an explicit concern for model changes that add, rename,
+or remove tuple-bearing relations. The deployment mechanism orders and pins the
+model; it does not invent or rewrite tuples.
+
+## Strongest Counterargument
+
+Omit `authorization_model_id` and let OpenFGA select the latest model. That
+removes the configuration update and would have hidden this deployment bug.
+
+This is rejected because a model write would then change authorization for
+running replicas independently of their binary version. An accidental write
+could affect production immediately, gradual rollout would be impossible, and
+application rollback would not restore the prior policy. OpenFGA recommends
+pinning a specific model ID in production.
+
+## Consequences
+
+- Application and authorization policy rollback are one environment rollback.
+- Legacy environments intentionally write one current model because they have
+  no stored digest.
+- Identical product releases do not accumulate model versions.
+- Model changes must keep the immediately previous binary/model combination
+  rollback-safe or explicitly use a staged migration.
+- Deployment CI must test upgrade ordering, unchanged-model no-op, and rollback
+  of the model pin.
+
+## References
+
+- [OpenFGA immutable authorization models](https://openfga.dev/docs/getting-started/immutable-models)
+- [OpenFGA model migrations](https://openfga.dev/docs/modeling/migrating/migrating-models)
+- [OpenFGA CLI model versions](https://openfga.dev/docs/getting-started/cli)
diff --git a/docs/increments/active/2026-07-29-openfga-model-rollout/design.md b/docs/increments/active/2026-07-29-openfga-model-rollout/design.md
@@ -0,0 +1,97 @@
+# OpenFGA Model Rollout Repair
+
+## Problem
+
+The production application pins every authorization request to
+`ORGMEMORY_OPENFGA_AUTHORIZATION_MODEL_ID`, but the deployment lifecycle writes
+that identifier only during first-store bootstrap. Later releases update the
+repository-owned `model.fga` without writing a new immutable model version or
+changing the pinned identifier.
+
+This became user-visible when the multi-provider control plane added
+`organization#can_manage_ai`. The web and API images deployed successfully,
+while the production API continued checking an older model that did not contain
+that relation. Organization administrators could enter the admin shell but the
+Language Models and Index Settings requests failed closed.
+
+Direct SSH evidence was unavailable during diagnosis because the ZM host timed
+out from the current workstation. Repository and workflow evidence still proves
+the lifecycle defect:
+
+- `bootstrap-openfga.sh` creates a store and model only when both IDs are empty;
+- `deploy.sh` requires and reuses the existing model ID without writing the
+  current `model.fga`;
+- production successfully deployed the application commit containing
+  `can_manage_ai`.
+
+## Selected Design
+
+Treat the authorization model as a versioned release input:
+
+1. Keep one durable OpenFGA store and all existing tuples.
+2. Compute SHA-256 over the repository model used by the release.
+3. If that digest differs from the digest pinned in the host environment, write
+   the model into the existing store with the official OpenFGA CLI.
+4. Parse and validate the returned `authorization_model_id`.
+5. Atomically persist the new model ID and digest before recreating API and
+   worker containers.
+6. Keep every application request explicitly pinned to that model ID.
+7. On a failed deployment, restore the prior environment and recreate the prior
+   image set with its prior model ID. The unused immutable model version may
+   remain in OpenFGA.
+
+First-store bootstrap writes both the initial model ID and digest. Existing
+installations have no digest, intentionally forcing one model write on the first
+deployment containing this repair.
+
+The official OpenFGA guidance says models are immutable, each write creates a
+new version, production clients should pin a specific model ID, and adding a
+relation requires writing the model before application code starts using it.
+
+## Strongest Counterargument
+
+The application could stop sending an authorization model ID and let OpenFGA
+use the latest version. That would make a newly written model visible without
+updating application configuration.
+
+This is rejected because "latest" disconnects a running binary from the policy
+version it was tested against. A later or accidental model write could change
+authorization for every replica immediately, and rollback of the application
+would not restore its compatible policy. Explicit pinning is the safer
+production contract.
+
+Writing a model on every deployment is also rejected. OpenFGA models are
+immutable and cannot be deleted, so identical releases would accumulate
+unnecessary versions. The digest makes unchanged model delivery a no-op while
+forcing legacy installations through one repair write.
+
+## Architecture Challenge
+
+This changes the authorization deployment boundary and therefore requires an
+independent challenge. The configured Claude reviewer remained unavailable due
+to the previously reported quota limit. The project owner had already directed
+this session to continue without the Claude discussion step and explicitly
+asked for the production bug to be fixed. The counterargument above, repository
+evidence, official OpenFGA lifecycle guidance, rollback behavior, and negative
+tests are recorded here in place of that unavailable review.
+
+## Scope
+
+- production Compose operations service for writing the repository model;
+- first-store bootstrap model digest;
+- production deployment model write, atomic pin, no-op, and rollback;
+- deterministic shell regression coverage;
+- deployment runbook, architecture, and authorization coverage updates.
+
+No OpenFGA relation, tuple, application role, or browser authorization bypass is
+changed by this repair.
+
+## Exit Gates
+
+- OpenFGA model validation and store tests pass;
+- production Compose interpolation and shellcheck pass;
+- deterministic tests prove upgrade, unchanged-model no-op, and failed-canary
+  rollback to the prior model ID;
+- documentation checks pass;
+- PR CI passes, the PR merges, production deploys the immutable release, and an
+  authenticated administrator can load both affected screens.
diff --git a/docs/increments/active/2026-07-29-openfga-model-rollout/plan.md b/docs/increments/active/2026-07-29-openfga-model-rollout/plan.md
@@ -0,0 +1,17 @@
+# OpenFGA Model Rollout Repair Plan
+
+- [x] Trace the UI denial through the API guard and production deployment model
+  pin.
+- [x] Verify OpenFGA model-write and immutable-version behavior against current
+  official documentation.
+- [x] Add a model-write operation to production Compose.
+- [x] Persist the model digest during first-store bootstrap.
+- [x] Write and atomically pin a changed model before API/worker recreation.
+- [x] Preserve the previous model ID and digest across failed deployment
+  rollback.
+- [x] Add deterministic upgrade, no-op, and rollback tests to deployment CI.
+- [x] Reconcile architecture, deployment runbook, authorization spec/coverage,
+  and roadmap.
+- [x] Run OpenFGA, deployment, documentation, and repository hygiene gates.
+- [ ] Open the PR, resolve actionable review/CI findings, merge, deploy, and
+  verify the two administrator screens.
diff --git a/docs/roadmap.md b/docs/roadmap.md
@@ -29,6 +29,7 @@ The table is a delivery index, not a second description of current behavior.
 
 | Increment | Status | Remaining gate |
 | --- | --- | --- |
+| [OpenFGA model rollout repair](increments/active/2026-07-29-openfga-model-rollout/plan.md) | active | version and pin the repository model during deployment, then verify the affected admin screens |
 | [Production CI/CD and ZM runtime](increments/active/2026-07-25-production-cicd-zm/plan.md) | active | shared-PostgreSQL cutover, restore proof, end-to-end runtime and rollback gates |
 | [Reproducible demo bootstrap](increments/active/2026-07-22-reproducible-demo-bootstrap/plan.md) | active | public ingestion and permission-evaluation run |
 | [Slack connector live proof](increments/active/2026-07-23-slack-connector-live/plan.md) | active | live workspace crawl and next-crawl revocation |

diff --git a/docs/runbooks/production-zm-deployment.md b/docs/runbooks/production-zm-deployment.md
@@ -66,8 +66,8 @@ Build or pull the exact production images, then initialize OpenFGA:
 ```
 
 The command creates one store from the repository authorization model and saves
-the store/model IDs to `.env.production`. It refuses to create a second store
-when identifiers already exist.
+the store ID, immutable model ID, and model SHA-256 to `.env.production`. It
+refuses to create a second store when identifiers already exist.
 
 ## Nginx Proxy Manager
 
@@ -168,9 +168,12 @@ The deployment:
 5. idempotently checks database roles/databases;
 6. backs up OrgMemory, OpenFGA, and Keycloak;
 7. runs OpenFGA migration and API-owned Flyway migration;
-8. starts the private runtime;
-9. checks web, API, MCP, Keycloak, and optionally the public endpoints;
-10. restores the previous image references when a gate fails.
+8. when the repository authorization-model digest changed or the legacy digest
+   is absent, writes a new immutable model into the existing store and
+   atomically pins its ID before application recreation;
+9. starts the private runtime;
+10. checks web, API, MCP, Keycloak, and optionally the public endpoints;
+11. restores the previous image references and model pin when a gate fails.
 
 `ORGMEMORY_BACKUP_UID` and `ORGMEMORY_BACKUP_GID` must match the owner of
 `ORGMEMORY_BACKUP_DIRECTORY`. The one-shot backup container drops all Linux
@@ -181,6 +184,12 @@ Database migrations must remain backward compatible with the immediately
 previous application image. The rollback does not reverse a committed database
 migration.
 
+OpenFGA models are also immutable and remain in the store after a failed
+canary. Rollback makes that unused version inert by restoring the previous
+model ID. Model changes that need tuple migration must stage that migration
+explicitly; the release script orders and pins models but does not synthesize
+tuples.
+
 Logical backup rotation is an operator responsibility, not part of the
 transactional deployment script. A scheduled retention job may prune old
 timestamped directories only after a newer `SHA256SUMS` set has passed a restore

diff --git a/docs/specs/domains/ai-model-control-plane.md b/docs/specs/domains/ai-model-control-plane.md
@@ -4,7 +4,7 @@ Source: `core/src/main/java/com/orgmemory/core/ai`,
 `integrations/ai-openai-compatible`, `apps/api/.../AdminAiModelController`, and
 `apps/web/src/features/admin/components/admin-language-models-page.tsx`.
 
-Reconciled: `2026-07-29-multi-provider-model-control-plane (d7ca979)`.
+Reconciled: `2026-07-29-openfga-model-rollout (c9a366b)`.
 
 ## Current Behavior
 
@@ -44,6 +44,12 @@ Index Settings is a separate read-only surface. The embedding provider, model,
 dimensions, and cosine metric cannot be mutated through the chat control plane;
 a geometry change requires a versioned embedding profile and reindex lifecycle.
 
+Both administration surfaces require OpenFGA `organization#can_manage_ai`.
+Production writes and pins the repository authorization model before a release
+whose model digest changed starts application containers. A legacy deployment
+with no stored digest writes the current model once. A failed release restores
+the previous model ID with its previous image set.
+
 ## Source Modules
 
 - `core.ai`
@@ -55,3 +61,4 @@ a geometry change requires a versioned embedding profile and reindex lifecycle.
 
 - [0006](../../decisions/0006-ai-tasks-route-through-provider-adapters.md)
 - [0008](../../decisions/0008-worker-owns-ingestion-and-derived-indexes.md)
+- [0017](../../decisions/0017-pin-openfga-models-to-product-releases.md)
diff --git a/docs/tests/domains/ai-model-control-plane.md b/docs/tests/domains/ai-model-control-plane.md
@@ -5,11 +5,12 @@ Source: `core/src/test/java/com/orgmemory/core/ai`,
 `apps/api/src/test/java/com/orgmemory/api/admin`,
 `integrations/authorization-openfga/src/test/openfga`, and the admin web build.
 
-Reconciled: `2026-07-29-multi-provider-model-control-plane (d7ca979)`.
+Reconciled: `2026-07-29-openfga-model-rollout (c9a366b)`.
 
 | Behavior | Evidence | Status |
 | --- | --- | --- |
 | Only organization administrators receive `can_manage_ai` | OpenFGA `store.fga.yaml`, `PermissionsAdminIntegrationTests` | covered |
+| Production writes and pins a changed OpenFGA model before application recreation, skips identical bytes, and restores the previous pin on failed canary | `test-deploy-openfga-model-rollout.sh` | covered |
 | Secrets are encrypted and absent from views/log rendering | `AiGatewayAdministrationServiceTests#storesOnlyCiphertextAndKeepsCredentialsOutOfViewsAndLogs` | covered |
 | Cross-tenant profile IDs are opaque and cannot rotate credentials | `AiGatewayAdministrationServiceTests#aProfileIdFromAnotherOrganizationIsOpaqueAndCannotRotateASecret` | covered |
 | Profile, credential, and route actor FKs cannot cross tenant boundaries | `PermissionsAdminIntegrationTests#aiControlPlaneActorReferencesCannotCrossTenantBoundaries` | covered |

diff --git a/infrastructure/deployment/compose.production.yaml b/infrastructure/deployment/compose.production.yaml
@@ -187,6 +187,35 @@ services:
     cap_drop:
       - ALL
 
+  openfga-model-write:
+    image: openfga/cli:v0.7.19@sha256:2e0e250043ef480a9162623dbf1ff7a62a1a2cb96a79cb20577b144994ab114d
+    profiles:
+      - ops
+    command:
+      - model
+      - write
+      - --store-id
+      - ${ORGMEMORY_OPENFGA_STORE_ID:-}
+      - --file
+      - /model/model.fga
+      - --format
+      - fga
+      - --api-url
+      - http://openfga:8080
+    depends_on:
+      openfga-ready:
+        condition: service_completed_successfully
+    networks:
+      - orgmemory-internal
+    volumes:
+      - ../../integrations/authorization-openfga/src/main/openfga/model.fga:/model/model.fga:ro
+    restart: "no"
+    read_only: true
+    security_opt:
+      - no-new-privileges:true
+    cap_drop:
+      - ALL
+
-  openfga-model-write:
-    image: openfga/cli:v0.7.19@sha256:2e0e250043ef480a9162623dbf1ff7a62a1a2cb96a79cb20577b144994ab114d
-    profiles:
-      - ops
-    command:
-      - model
-      - write
-      - --store-id
-      - ${ORGMEMORY_OPENFGA_STORE_ID:-}
-      - --file
-      - /model/model.fga
-      - --format
-      - fga
-      - --api-url
-      - http://openfga:8080
-    depends_on:
-      openfga-ready:
-        condition: service_completed_successfully
-    networks:
-      - orgmemory-internal
-    volumes:
-      - ../../integrations/authorization-openfga/src/main/openfga/model.fga:/model/model.fga:ro
-    restart: "no"
-    read_only: true
-    security_opt:
-      - no-new-privileges:true
-    cap_drop:
-      - ALL
+  openfga-model-write:
+    image: openfga/cli:v0.7.19@sha256:2e0e250043ef480a9162623dbf1ff7a62a1a2cb96a79cb20577b144994ab114d
+    profiles:
+      - ops
+    command:
+      - model
+      - write
+      - --store-id
+      - ${ORGMEMORY_OPENFGA_STORE_ID:?Set ORGMEMORY_OPENFGA_STORE_ID}
+      - --file
+      - /model/model.fga
+      - --format
+      - fga
+      - --api-url
+      - http://openfga:8080
+    depends_on:
+      openfga-ready:
+        condition: service_completed_successfully
+    networks:
+      - orgmemory-internal
+    volumes:
+      - ../../integrations/authorization-openfga/src/main/openfga/model.fga:/model/model.fga:ro
+    restart: "no"
+    read_only: true
+    security_opt:
+      - no-new-privileges:true
+    cap_drop:
+      - ALL
-  openfga-model-write:
-    image: openfga/cli:v0.7.19@sha256:2e0e250043ef480a9162623dbf1ff7a62a1a2cb96a79cb20577b144994ab114d
-    profiles:
-      - ops
-    command:
-      - model
-      - write
-      - --store-id
-      - ${ORGMEMORY_OPENFGA_STORE_ID:-}
-      - --file
-      - /model/model.fga
-      - --format
-      - fga
-      - --api-url
-      - http://openfga:8080
-    depends_on:
-      openfga-ready:
-        condition: service_completed_successfully
-    networks:
-      - orgmemory-internal
-    volumes:
-      - ../../integrations/authorization-openfga/src/main/openfga/model.fga:/model/model.fga:ro
-    restart: "no"
-    read_only: true
-    security_opt:
-      - no-new-privileges:true
-    cap_drop:
-      - ALL
+  openfga-model-write:
+    image: openfga/cli:v0.7.19@sha256:2e0e250043ef480a9162623dbf1ff7a62a1a2cb96a79cb20577b144994ab114d
+    profiles:
+      - ops
+    command:
+      - model
+      - write
+      - --store-id
+      - ${ORGMEMORY_OPENFGA_STORE_ID:?Set ORGMEMORY_OPENFGA_STORE_ID}
+      - --file
+      - /model/model.fga
+      - --format
+      - fga
+      - --api-url
+      - http://openfga:8080
+    depends_on:
+      openfga-ready:
+        condition: service_completed_successfully
+    networks:
+      - orgmemory-internal
+    volumes:
+      - ../../integrations/authorization-openfga/src/main/openfga/model.fga:/model/model.fga:ro
+    restart: "no"
+    read_only: true
+    security_opt:
+      - no-new-privileges:true
+    cap_drop:
+      - ALL
   postgres-backup:
     image: ${ORGMEMORY_POSTGRES_IMAGE:?Set ORGMEMORY_POSTGRES_IMAGE}
     profiles:

diff --git a/infrastructure/deployment/production.env.example b/infrastructure/deployment/production.env.example
@@ -26,6 +26,9 @@ OPENFGA_DB_PASSWORD=
 OPENFGA_DATASTORE_URI=postgres://openfga:<url-encoded-password>@postgres:5432/openfga?sslmode=disable
 ORGMEMORY_OPENFGA_STORE_ID=
 ORGMEMORY_OPENFGA_AUTHORIZATION_MODEL_ID=
+# SHA-256 of the repository model pinned by the model ID above. The bootstrap
+# and deployment scripts own this value.
+ORGMEMORY_OPENFGA_MODEL_SHA256=
 
 KEYCLOAK_DB_NAME=keycloak
 KEYCLOAK_DB_USER=keycloak