DataON Support Portal

Updated October 7, 2026: Lowered the phantom entry threshold from 1,000 to 250, added an URGENT status for hives over 400 MB, and clarified that VM backups should keep running unless a node shows URGENT. If one node is flagged, treat the whole cluster as affected. If you ran this check before October 7, please re-run it using the current script.

TL;DR: On some Azure Local clusters, leftover shadow copy device entries can build up in a node's SYSTEM registry hive until the node can no longer boot. This usually shows up during a solution update reboot. Run the check below before updating or rebooting any node.

Recommended action:

1. Open an elevated PowerShell session on any cluster node and run the check below. It reads information only and makes no changes. On an affected node, it can take several minutes to finish.

Invoke-Command -ComputerName (Get-ClusterNode).Name -ScriptBlock {

    # All device entries created for VSS shadow copy volumes
    $s = Get-PnpDevice -ErrorAction SilentlyContinue |
         Where-Object -Property InstanceId -Like -Value 'STORAGE\VOLUMESNAPSHOT\*'

    $hiveMB  = [math]::Round((Get-Item -Path "$env:SystemRoot\System32\config\SYSTEM").Length / 1MB, 0)
    $phantom = @($s | Where-Object { -not $_.Present }).Count

    [pscustomobject]@{
        HiveSizeMB            = $hiveMB
        VolumeSnapshotTotal   = @($s).Count
        VolumeSnapshotPhantom = $phantom
        Status                = if ($hiveMB -gt 400) { 'URGENT' }
                                elseif ($hiveMB -gt 100 -or $phantom -gt 250) { 'CONTACT DATAON' }
                                else { 'OK' }
    }

} | Sort-Object -Property PSComputerName |
    Format-Table -Property PSComputerName, HiveSizeMB, VolumeSnapshotTotal, VolumeSnapshotPhantom, Status -AutoSize

2. If every node shows OK, no action is needed. You can go ahead with updates as usual.
3. If any node shows CONTACT DATAON (HiveSizeMB over 100, or VolumeSnapshotPhantom over 250), treat the whole cluster as affected, since all nodes share the same backup activity:
   • Do not reboot any node or apply solution updates to the cluster.
   • Keep your VM backups running. Staying protected is more important than the slow growth of these entries. Reducing an unusually frequent backup schedule is reasonable, but do not disable backups.
   • Contact DataON Support. Include the full output and the name and version of your backup software.
   • Re-run the check weekly and send the output to DataON Support so the trend can be tracked.
4. If any node shows URGENT (HiveSizeMB over 400), follow all of the steps above, and also:
   • Pause VM backup jobs on the cluster until DataON Support has reviewed the results. At this size, additional growth brings the node closer to a point where it may not boot.
   • Contact DataON Support immediately.

Why:

Each time a VSS shadow copy is created, Windows registers a device entry for it. If those entries aren't removed when the snapshot is released, they stay in the SYSTEM registry hive as non-present ("phantom") devices. Over time the hive can grow from tens of megabytes to more than a gigabyte. Once it gets that large, Windows may not be able to load it at boot, and the node fails with a registry error. This has been seen on clusters with frequent VSS-based VM backups. The root cause of the leftover entries is still under investigation.

Going forward:

Run this check before every solution update, before any planned node reboot, and after changes to your backup schedule. A phantom count that keeps rising between checks means entries are still accumulating, even if the node is under the threshold today.

Optional details:

• With automatic restart disabled, an affected node that fails to boot shows: File: \Windows\system32\config\system Status: 0xc000000d. The message says the system registry file is missing or contains errors.
• Reference values: healthy nodes typically show a hive of about 25 to 55 MB and fewer than 100 phantom entries. Affected nodes have shown hives of 500 MB to more than 1 GB, with tens of thousands of phantom entries.
• Each phantom entry adds roughly 15 to 20 KB to the hive, so the phantom count usually rises before the hive size crosses a threshold.
• Don't edit the registry or remove device entries yourself. Cleanup should be done with support guidance, because changes to the SYSTEM hive can leave a node unable to boot.
• After cleanup, the hive file size may not shrink right away. The phantom count is the more reliable measure of progress.