Path B: route whole-file deletes away from the offline window, correct the reclaim rationale - #346
Conversation
…claim rationale CSS feedback on a live case (customer deleted a number of VMs and wanted the space back) was that the steps before slab consolidation are heavyweight and may not always be needed, and that the key requirement is really about active file handles on the CSV. Reviewing Path B against first-party documentation, most of that is correct, and the guide's stated reason for the offline window was wrong. Changes: - Add a no-downtime pre-branch. Deleting a whole file frees its slabs outright and ReFS returns them to the pool on its own, with no Optimize-Volume. Learn documents this as "a gradual process that can take 15 minutes or so after the files are deleted" and notes that many running workloads make it take longer rather than preventing it. Only interior fragmentation, where live data still occupies part of a slab, needs consolidation and therefore an offline window. The guide already said this in the pool-fill escalation ladder; this promotes it into Path B where the reader making the decision will see it. - Correct the reason for taking VMs offline. The guide said virtual disk file handles must be released, "required for consolidation". No first-party source states that, and FSCTL_FILE_LEVEL_TRIM exists specifically to reclaim ranges inside an open virtual-disk file. The accurate reason is that ReFS allocates on write, so a running VM keeps re-dirtying slabs that consolidation is trying to empty. The step is unchanged; only the rationale is corrected. - Note that consolidation runs at low priority by default, per Optimize-Volume (-NormalPriority: "By default, the priority is low") and defrag /h, while being explicit that this is scheduling priority and not total cost: pool disks are shared by every volume, so hours of relocation I/O can still be felt elsewhere. - Add a caution against moving VM disks between volumes to relieve pressure. For Arc-managed VMs this is storage live migration, which Learn lists among operations that "can lead to Azure Local VMs becoming unmanageable from the Azure portal", because the storage path resource keeps pointing at the old volume. Scoped to Arc VMs, since Move-VMStorage remains supported for traditional clustered Hyper-V VMs, and paired with the supported alternative. - Add the stretched-cluster caveat (TRIM is disabled, so deleted capacity is never returned) and a DisableDeleteNotify precondition check, both of which explain a reclaim that appears to do nothing. - Move the optional checkpoint merge out of the numbered sequence so it stops reading as a required first step, and renumber accordingly. Not changed: the offline window itself. Whether consolidation can run safely with live VMs is a separate question that needs measurement on physical S2D hardware, and is not loosened here. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e36f861-864e-4e08-a04c-b0d7f1b897c4
Located the March 2026 FS engineering thread that the offline-window guidance originally came from, and it does support the mechanism the guide stated. FS engineering's position there is that "pinned unmovable" slabs are mostly caused by applications holding files pinned, that slab consolidation cannot process files that are actively in use, and that the recommended sequence is to stop workloads on the volume, consolidate, and let the ReFS unmap work items reclaim. It also notes that internal testing tended to succeed because workloads were stopped, while production workloads keep slabs pinned and reduce how much a pass recovers. The same thread independently confirms the guide's existing advice not to use -ReTrim on thin-provisioned ReFS. My earlier rewording attributed the need for a quiesced volume to ReFS allocate-on-write re-dirtying slabs. That is a real secondary effect but it is not the primary mechanism, and replacing the in-use rationale with it was wrong. Restored to the accurate mechanism: consolidation relocates live data out of partially used slabs and cannot relocate data belonging to files that are actively in use, so those slabs are reported pinned unmovable and skipped. Also adjusted the Suspend-VM note. Pausing leaves the virtual disk files open and the guest resident, so it is not a reliable substitute for a shutdown. This drops the earlier "not been validated" phrasing, which understated a mechanism that is in fact documented by the owning team. Unchanged from the previous commit: the no-downtime pre-branch for whole-file deletes, which rests on separate Microsoft documentation and is not affected by this correction; the low-priority note; the Arc disk-move caution; and the stretched-cluster and DisableDeleteNotify preconditions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e36f861-864e-4e08-a04c-b0d7f1b897c4
…RIM check Five review passes across two model families found four things worth fixing. Deleting a VM does not delete its disks. Remove-VM "deletes the virtual machine's configuration file, but does not delete any virtual hard drives", so a customer who removed VMs can still be holding every VHDX on the volume. The pre-branch previously said only to confirm the deletions were complete, which would send that customer into a 15 minute wait that cannot reclaim anything and leave them concluding the guide is wrong. It now asks them to confirm the disk files themselves are gone and gives a command to find orphaned ones. The DisableDeleteNotify check described a single value, but the command returns one line per file system and a line can read "is not currently set", which is neither 0 nor 1. Azure Local CSVs are ReFS, so the ReFS line is the one that governs and the check did not say so. It now shows the real output shape, names the file system to read, covers all three states, and says to run it on the CSV owner node. The 0 and 1 mapping itself was correct and is unchanged. The precondition is also re-scoped to gate both paths rather than only the automatic one, since consolidation depends on the same ReFS unmap to return freed slabs. The options table described the no-downtime branch as "delete, then wait" while the ownership gate called that branch always safe to run. Together those made an irreversible delete read as a safe action. The table now says confirm, wait, re-measure, and the gate is explicit that the branch asks for an irreversible delete and is not covered by the safe-to-run exemption. Trimmed the -NormalPriority quote to the clause that carries the meaning. The longer sentence is missing a verb on Learn itself, so quoting it verbatim was faithful but read as an error in this guide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e36f861-864e-4e08-a04c-b0d7f1b897c4
|
Reviewed this against the base rather than just reading the diff, checked every external citation against its source, and traced the renumbering and the routing end to end. Four things needed fixing and are now pushed. 1. Deleting a VM does not delete its disks. This was the important one. The pre-branch routed on "you deleted whole VMs or files" and asked only that the deletions be confirmed complete. 2. The DisableDeleteNotify check described a single value. The command returns one line per file system, and a line can read 3. An irreversible delete read as a safe action. The options table described the no-downtime branch as "delete, then wait" while the ownership gate listed that branch as always safe to run. Read together those made deleting VMs look like it sat under the safe-to-run exemption. The table now says confirm, wait, re-measure, and the gate is explicit that the branch asks for an irreversible delete and is not covered by the exemption. 4. A quote that read as a typo. The What held up: every Learn citation is verbatim and on the page it is attributed to, including the fifteen minute reclamation quote, the workloads clause, the stretched cluster TRIM sentence, and the storage live migration entry in the unsupported operations list. Scoping the disk-move caution to Arc VMs is correct, Still unverified: nobody has run these commands on a 23H2 node in this pass, so the exact |
…un is reasonable Two gaps surfaced by the question "why not just try slab consolidation first". The guide warned against moving VM disks to another CSV on Arc-management grounds but never said the more basic thing: it cannot work anyway. Azure Local uses one storage pool per cluster, so every CSV draws from the same pool and relocating a VHDX between them returns nothing to the pool. For Arc VMs that makes it risk for no benefit, and the same futility applies to the supported non-Arc Move-VMStorage path. Added the one-pool reason, and noted that live-migrating a VM to another node does not help either, since the CSV is cluster-shared so the file stays on the volume and stays in use. Also addressed the probe question directly rather than leaving the guide reading as though the window is mandatory before anything can be attempted. Running consolidation with the workload up to see what it recovers is reasonable: it is non-destructive, it relocates rather than deletes, and it runs at low priority. Two caveats now stated. On a multi-terabyte volume it is hours of relocation I/O felt by every other volume on the shared pool, so it is cheap in risk and not in cost. And on a nearly-full pool it deserves more care, because ReFS allocates on write and whether relocation transiently raises allocation there is not established either way, while pool exhaustion is the one failure in this article that takes VMs offline. The read-only checks above cost seconds and can make the question moot, so they come first. Also states that a probe recovering little is the expected result when the workload still holds its files, not evidence the procedure does not work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e36f861-864e-4e08-a04c-b0d7f1b897c4
I added this earlier in the same review cycle on the strength of a sentence in the Learn thin provisioning article: "Because TRIM is disabled for stretched clusters, storage isn't returned to the pool after data is deleted." The quote is real and it is on the 23H2-named page, which is how it got through. Stretched clusters are not a supported Azure Local 23H2 or later configuration. The stretched-clusters concept page still sits on the legacy azure-stack/hci path, and Microsoft's guidance is that 23H2 does not support them. So the sentence is stale text carried forward onto a renamed page, and repeating it here implied a deployment shape a reader cannot have, and sent them to check a condition that cannot be true on a supported system. Removed the caveat, the two escalation preconditions that asked the operator to certify the cluster is not stretched, and the mention in the closing troubleshooting note. The DisableDeleteNotify check stays, still scoped to the ReFS line and still gating both paths. Verifying that a quote exists is not the same as verifying that the configuration it describes exists on the product the guide targets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e36f861-864e-4e08-a04c-b0d7f1b897c4
|
John Neemes (@1008covingtonlane) could you give this a review please? |
1 participant
Path B currently sends every thin-provisioned reclaim down one heavyweight route: merge checkpoints, take all VMs on the volume offline, run slab consolidation, wait, bring VMs back. CSS raised this from a live case where the customer had deleted a number of VMs and simply wanted the space back, and asked whether all of that is really needed, and whether the real requirement is just about active file handles on the CSV.
The answer turned out to be split. The heavyweight route is right for the case it was written for, but it was being applied to a case that does not need it at all.
The main change: not every reclaim needs consolidation
Two different mechanisms return capacity to the pool, and only one needs an offline window.
When a whole file is deleted, the slabs it occupied become entirely free and ReFS returns them to the pool on its own, with no
Optimize-Volumeat all. The Azure Local thin provisioning FAQ documents this: "Will space be given back to the pool immediately after files are deleted? No. This is a gradual process that can take 15 minutes or so after the files are deleted. If there are many workloads running on the system, it may take longer for all of the space to be returned to the pool." That last clause is the direct answer to the CSS question: running workloads slow reclamation down, they do not block it.Slab consolidation exists for the other case, interior fragmentation, where data was deleted from inside a VHDX and live data still occupies part of each slab, so no whole slab ever frees.
The customer in the CSS case had deleted whole VMs, which is the first case. They did not need the offline window, and Path B gave them no way to discover that. The guide already said as much in the pool-fill escalation ladder ("Audit and prune... the reclaimed space returns to the pool gradually"), but that note is a long way from Path B and cross-links into it. This PR promotes it into an explicit no-downtime pre-branch at the top of Path B: confirm the deletes, wait, re-measure, and only fall through to consolidation if the pool is still above threshold and real interior free space remains.
The offline window itself is correct, and stays
CSS's other point was that the VM shutdown may be unnecessary because consolidation is a low-priority background job. Half of that holds up and half does not.
The priority part is right, and the guide now says so:
Optimize-Volumedocuments-NormalPriorityas "By default, the priority is low", matchingdefrag /h. But that governs scheduling priority, not total cost. Pool physical disks are shared by every volume in the pool, so hours of relocation I/O on a multi-terabyte volume can still be felt by workloads on other volumes. The guide now makes that distinction explicitly.The shutdown requirement, however, is real. I initially rewrote the rationale, on the grounds that no public documentation states a handle prerequisite and that
FSCTL_FILE_LEVEL_TRIMreclaims ranges inside open virtual-disk files. That was wrong, and the second commit here reverts it. Tracking down the Microsoft file-system engineering guidance this section was originally based on confirms the mechanism as the guide had it: pinned-unmovable slabs are mostly caused by applications holding files pinned, slab consolidation cannot process files that are actively in use, and the recommended sequence is to stop workloads on the volume, consolidate, then let the ReFS unmap work items reclaim. The same guidance notes that lab tests tend to succeed precisely because workloads were stopped, while production workloads keep slabs pinned and reduce how much a pass recovers. It independently confirms the existing advice not to use-ReTrimon thin-provisioned ReFS.So the step is unchanged and the rationale is now stated in terms of files in use being unmovable, rather than the vaguer "handles are released". The
Suspend-VMnote is also adjusted: pausing leaves the virtual disk files open and the guest resident, so it is not a reliable substitute for a shutdown.Other changes
Move-VMStorageplus updating the cluster resource is still supported for traditional clustered Hyper-V VMs and a blanket prohibition would be wrong for that audience; it is not scoped to "beforehand", since storage live migration is unsupported at any time; and it names the supported alternative, choosing a storage path on the target volume at creation, because there is no supported in-place move and the operator otherwise has nowhere to go. Learn lists storage live migration among operations that "can lead to Azure Local VMs becoming unmanageable from the Azure portal", and the failure mode is that the storage path resource keeps pointing at the old volume.DisableDeleteNotifyprecondition check, a documented reason a reclaim appears to do nothing.Scope and validation
The offline window for the consolidation path is not loosened. Whether consolidation can run safely with live VMs is a separate question, and worth being honest that the physical validation behind this guide ran on a scratch volume with no VMs on it, so it never tested that. A controlled run that separates an open file from an actively writing one, and that measures the effect on other volumes in the pool rather than only the one being consolidated, has not been done. The experiment design and its required control arms are written up in the internal validation spec so it can be picked up, but to be clear about status, it is not scheduled and no cluster is reserved for it. It is also lower priority than it first appeared: the owning team's guidance already answers whether the window is needed, so what remains is quantifying how much yield is lost under live workload, not deciding the step.
The no-downtime pre-branch is sourced from Microsoft documentation rather than a new lab run, so the metadata block is unchanged and still reflects the last hardware validation.
tsg-forge --lintgrade A: fences balanced, all relative links and in-page anchors resolve, no bare drive-root deletes, prose style clean.