Skip to content

Delete the log workspace permanently when tearing down - #173

Merged
jaredthivener merged 1 commit into
mainfrom
destroy-workspace-for-good
Oct 7, 2026
Merged

jaredthivener merged 1 commit into
mainfrom
destroy-workspace-for-good

Conversation

@jaredthivener

Copy link
Copy Markdown
Owner

The first attempt of today's Deploy run failed with:

InvalidOutputTable: Table for output stream 'Microsoft-ContainerLogV2' is not available for destination 'workspace'.

Cause

The environment had been torn down a few hours earlier with az group delete, which is also all the Destroy workflow does. That only soft-deletes the Log Analytics workspace. Per Microsoft's page on deleting and recovering workspaces:

  • Re-creating a workspace with the same name, resource group and region within 14 days recovers the deleted one. The workspace this deploy got has the creation time of the original, 18:22 UTC.
  • After a recovery, schemas can be missing until they are added back.
  • For environments that are deployed repeatedly under the same name, delete the workspace permanently (--force).

What was observed: the tables were missing when the first attempt created the collection rule, about a minute after the recovery, and present eight minutes later, when the workflow's second attempt created the same rule successfully. No Microsoft page I found documents this exact error for this case, so the link between the two is an inference from that timing.

Change

  • The Destroy workflow deletes the group's workspaces with --force before deleting the group.
  • infra/README.md gives the same two commands for a manual teardown, and says what happens otherwise.

Not tested

The new step has not run: it would delete the workspace the live environment is using. Its list query returns the right workspace against the live group. The current deployment needs nothing from this; its retry already got past the error.

🤖 Generated with Claude Code

Deleting the resource group only soft-deletes its Log Analytics
workspace. A deployment within the next 14 days recovers the old
workspace instead of creating one, and for its first minutes the
recovered workspace has no container tables: the log collection rule
fails with InvalidOutputTable.

Microsoft's guidance for environments redeployed under the same name
is to delete the workspace permanently, so the destroy workflow and
the documented manual steps now do that before deleting the group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jaredthivener
jaredthivener merged commit 8383b02 into main Oct 7, 2026
8 checks passed
jaredthivener added a commit that referenced this pull request Oct 7, 2026
…178)

The Destroy workflow failed in the step added by #173. On GitHub's
runners, `az monitor log-analytics workspace delete` crashed inside the
Azure CLI before sending anything:

    deadlock detected by _ModuleLock('requests.structures')

That is Azure/azure-cli#33996, open and unfixed: with the CLI's bundled
Python 3.14, long-running commands create their credential on a worker
thread, and its import of `requests` intermittently collides with the
main thread's.

The step now sends the same permanent delete as a REST call through
`az rest`, which has no worker thread, and waits for the workspace to
be gone before the group is deleted.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant