Fix: resolved RBAC issue for hackathon env, tested locally, need to test on the platform - #455
Closed
ajakupov-microsoft wants to merge 6 commits into
Closed
Fix: resolved RBAC issue for hackathon env, tested locally, need to test on the platform#455ajakupov-microsoft wants to merge 6 commits into
ajakupov-microsoft wants to merge 6 commits into
Conversation
Three defects found by running deploy-lab.ps1 against a live subscription. Each one alone left the lab unusable for participants. 1. ARM output names are not case-preserving, so Get-OutVal read nothing. A template output declared APIM_GATEWAY_URL is returned by Azure as 'apiM_GATEWAY_URL' (the leading run of capitals is lowercased except the last character), and APPLICATIONINSIGHTS_CONNECTION_STRING comes back as 'applicationinsightS_CONNECTION_STRING'. The outputs collection is a case-sensitive dictionary, so the literal ContainsKey() lookup missed all 52 requested keys. The deployment "succeeded" while every participant RBAC grant was skipped and all 42 dashboard credentials were emitted empty. Get-OutVal now falls back to a case-insensitive match. 2. Key Vault Secrets Officer had a wrong role definition GUID (...eccb2c155090 instead of ...eccb2c155cd7), so both Key Vault grants failed with RoleDefinitionDoesNotExist and failed the whole provision. All seven role GUIDs were re-verified against the live role definitions. 3. Deployment required the standalone Bicep CLI on the host. Az PowerShell shells out to it for -TemplateFile *.bicep and cannot see the az-bundled binary in ~/.azure/bin, so deployment died before creating anything. The committed infra/resources.json exists precisely to avoid this dependency but was never used; it is now preferred, with the .bicep source as fallback. Verified byte-identical to a fresh compile of the Bicep. Also: run-local.ps1 now shims Test-MhhDeploymentFailureRetryable, which the platform injects but a local run did not have. Without it every failure looked retryable and a host-local error was retried across all three regions. Noted that Azure renamed 'Azure AI User' to 'Foundry User' and 'Azure AI Project Manager' to 'Foundry Project Manager'; matching is by GUID, but the portal shows the new names when verifying access. Verified against a live deployment in swedencentral: 15/15 RBAC grants, 42/42 credentials populated, 0 errors, 0 warnings, and re-runs are idempotent. Gateway smoke test passed - all 6 models listed, chat and embeddings 200, missing model defaults to gpt-4.1, unsupported model 400, missing key 401. All 10 notebook azd keys resolve from challenges/workshop. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
…ai-governance-lab-automation fix(06_AI_Governance): make lab deployment work end to end
…s expect (#3) Challenge 1 failed on its very first cell because LLM_BACKEND_CONFIG was both encoded and shaped wrong. * Encoding: the output was wrapped in base64(), but notebooks 1 and 8 call json.loads() on it directly, raising JSONDecodeError immediately. Now emitted as raw JSON via string(). * Schema: the output was a flat model catalogue of { name, publisher, endpoint, models } - one entry per model. Notebook 1 cell 15 requires { backendId, backendType, endpoint, supportedModels }, so it raised KeyError: 'backendId'. Replaced with the contract documented in challenges/bicep/infra/llm-backend-onboarding/main.bicep (lines 30-68): a single ai-foundry backend whose supportedModels carries the per-model metadata (sku, capacity, modelFormat, modelVersion, retirementDate) derived from the aiFoundryModels variable, so the config can no longer drift from what is actually deployed. * Added A2A_FOUNDRY_ACCOUNT_NAME output (spoke account) so notebook 8 has a supported override for its "-0" backendId convention, which our aif-hub-/aif-spoke- naming does not satisfy. * Updated the LlmBackendConfig credential note and setup-notebook-env.ps1 help, both of which still said "base64". Regenerated infra/resources.json, since deploy-lab.ps1 deploys the compiled ARM template rather than the .bicep source. Verified against rg-agentic-gov-dryrun-ajk01 (swedencentral): notebook 1 cells 2 and 15 now run unmodified and list 1 backend / 6 models; gateway regression clean (models/chat/embeddings 200, no-key 401, bad model 400). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
) * fix(06_AI_Governance): emit LLM_BACKEND_CONFIG in the schema notebooks expect Challenge 1 failed on its very first cell because LLM_BACKEND_CONFIG was both encoded and shaped wrong. * Encoding: the output was wrapped in base64(), but notebooks 1 and 8 call json.loads() on it directly, raising JSONDecodeError immediately. Now emitted as raw JSON via string(). * Schema: the output was a flat model catalogue of { name, publisher, endpoint, models } - one entry per model. Notebook 1 cell 15 requires { backendId, backendType, endpoint, supportedModels }, so it raised KeyError: 'backendId'. Replaced with the contract documented in challenges/bicep/infra/llm-backend-onboarding/main.bicep (lines 30-68): a single ai-foundry backend whose supportedModels carries the per-model metadata (sku, capacity, modelFormat, modelVersion, retirementDate) derived from the aiFoundryModels variable, so the config can no longer drift from what is actually deployed. * Added A2A_FOUNDRY_ACCOUNT_NAME output (spoke account) so notebook 8 has a supported override for its "-0" backendId convention, which our aif-hub-/aif-spoke- naming does not satisfy. * Updated the LlmBackendConfig credential note and setup-notebook-env.ps1 help, both of which still said "base64". Regenerated infra/resources.json, since deploy-lab.ps1 deploys the compiled ARM template rather than the .bicep source. Verified against rg-agentic-gov-dryrun-ajk01 (swedencentral): notebook 1 cells 2 and 15 now run unmodified and list 1 backend / 6 models; gateway regression clean (models/chat/embeddings 200, no-key 401, bad model 400). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58 * feat(06_AI_Governance): make azd optional for notebook configuration Participants get RG access and a credentials dashboard, not a provisioned hackbox - so every tool the notebooks need is a prerequisite we impose on their own laptop. Requiring azd (plus pwsh, to run setup-notebook-env.ps1) just to read config was the largest avoidable failure surface in the lab. Notebooks 1-7 now resolve configuration from os.environ first and fall back to `azd env get-value`, so both routes work and the existing azd bridge is unaffected. This follows the workshop's own direction rather than diverging from it: notebooks 8 and 9 - the newest ones upstream - already use the os.environ-first / tolerate-missing-azd pattern. Notebooks 1-7 were simply the older, brittle variant that hard-raised on a missing azd. The .env loader is deliberately dependency-free (a ~15 line parser walking up from cwd) rather than python-dotenv. An early version used dotenv and failed silently when the package was absent, producing a confusing missing -key error; the error now also reports whether a .env was found at all. Notably this fixes challenge 8, which was quietly broken: it resolves the agent-hosting account by looking for a backendId ending in "-0", which our aif-hub-*/aif-spoke-* naming never produces. Its documented override reads os.environ only, so the azd bridge could not supply it. Adds the A2aFoundryAccountName dashboard credential and documents the override. Manual entry from the Azure Portal was considered and rejected: LLM_BACKEND_CONFIG is a synthesized JSON document that appears nowhere in the portal, so it cannot be copied from there - and a hand-written .env would still have required azd to read it. - .env.template documents all 10 notebook keys against their exact dashboard credential names (verified: no key the notebooks read is missing) - README, challenge-01 Part A, per-challenge troubleshooting rows, the nine walkthrough solutions and labautomation/README updated; "unmodified notebooks" claims corrected Verified against rg-agentic-gov-dryrun-ajk01 on system Python 3.9 with no third-party packages: config resolves with azd absent from PATH (.env only) and with .env absent (azd only). .env stays gitignored. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
…cy (#5) The lab deployed every chat/embedding model at capacity 100 (100K TPM). With labsPerSubscription: 8 that costs 800 units per model per region against a typical 1000-unit GlobalStandard quota. Measured against the live subscription in swedencentral, three models could not have fitted 8 labs: text-embedding-3-large 300 free vs 800 needed gpt-5.4-mini 700 free vs 800 needed Mistral-Large-3 700 free vs 800 needed and gpt-4.1/gpt-5.2 had exactly zero margin. Labs would have deployed fine until roughly the fourth attendee and then failed mid-event. Introduce a modelCapacity parameter (default 20) instead of the hardcoded 100, so a full subscription costs 160 units per model. Phi-4 stays at 1. Verified by redeploying the live dry-run RG: scale-down succeeded, all six deployments now report capacity 20, and the gateway smoke test still passes (models 200, chat 200 across gpt-4.1/gpt-5.4-mini/Mistral-Large-3, default-model routing 200, embeddings 200, unknown model 400, missing key 401). Tightest model now has 380 free vs 160 needed. Also document that challenges 2/3/5/6 require challenge 1 to have been *run*, not merely for the .env to be populated. Their generated product policies include APIM policy fragments (set-llm-requested-model, validate-model-access, ...) that are created by notebook 1's llm-policy-fragments.bicep. The hub template deliberately ships only the minimal get-available-models fragment, so starting at challenge 3 fails with "Policy fragment not found". The existing prerequisite links pointed only at Challenge 1 Part A (environment setup). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
The retirementDate values in aiFoundryModels were hardcoded and stale. They are surfaced to participants through LLM_BACKEND_CONFIG.supportedModels, so they were actively misleading: gpt-5.4-mini claimed 2026-09-30, i.e. 33 days from now, which would look like the lab was about to break mid-event. Azure actually reports 2027-09-21 for that model. Corrected all six against live `az cognitiveservices model list` data for the exact pinned versions, and verified every value matches: gpt-4.1 2026-10-14 -> 2027-04-14 gpt-5.4-mini 2026-09-30 -> 2027-09-21 gpt-5.2 2027-02-05 -> 2027-06-08 text-embedding-3-large 2027-04-14 -> 2028-02-09 Mistral-Large-3 2099-12-30 -> 2099-12-31 Phi-4 2099-10-14 -> 2099-12-31 The nearest real retirement is now gpt-4.1 in April 2027, so no model retires anywhere near the event. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 05a828d3-c050-4705-9340-6c65646dfb58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.