DataON Support Portal

Updated October 7, 2026: Lowered the phantom entry threshold from 1,000 to 250, added an URGENT status for hives over 400 MB, and clarified that VM backups should keep running unless a node shows URGENT. If one node is flagged, treat the whole cluster as affected. If you ran this check before October 7, please re-run it using the current script. No supported cleanup or fix is available yet; see "Going forward" for what to expect.

TL;DR: On some Azure Local clusters, leftover shadow copy device entries can build up in a node's SYSTEM registry hive until the node can no longer boot. This usually shows up during a solution update reboot. Run the check below before updating or rebooting any node.

Recommended action:

1. Open an elevated PowerShell session on any cluster node and run the check below. It reads information only and makes no changes. On an affected node, it can take several minutes to finish.

Invoke-Command -ComputerName (Get-ClusterNode).Name -ScriptBlock {

    # All device entries created for VSS shadow copy volumes
    $s = Get-PnpDevice -ErrorAction SilentlyContinue |
         Where-Object -Property InstanceId -Like -Value 'STORAGE\VOLUMESNAPSHOT\*'

    $hiveMB  = [math]::Round((Get-Item -Path "$env:SystemRoot\System32\config\SYSTEM").Length / 1MB, 0)
    $phantom = @($s | Where-Object { -not $_.Present }).Count

    [pscustomobject]@{
        HiveSizeMB            = $hiveMB
        VolumeSnapshotTotal   = @($s).Count
        VolumeSnapshotPhantom = $phantom
        Status                = if ($hiveMB -gt 400) { 'URGENT' }
                                elseif ($hiveMB -gt 100 -or $phantom -gt 250) { 'CONTACT DATAON' }
                                else { 'OK' }
    }

} | Sort-Object -Property PSComputerName |
    Format-Table -Property PSComputerName, HiveSizeMB, VolumeSnapshotTotal, VolumeSnapshotPhantom, Status -AutoSize

2. If every node shows OK, no action is needed. You can go ahead with updates as usual.
3. If any node shows CONTACT DATAON (HiveSizeMB over 100, or VolumeSnapshotPhantom over 250), treat the whole cluster as affected, since all nodes share the same backup activity:
   • Do not reboot any node or apply solution updates to the cluster.
   • Keep your VM backups running. Staying protected is more important than the slow growth of these entries. Reducing an unusually frequent backup schedule is reasonable, but do not disable backups.
   • Contact DataON Support. Include the full output and the name and version of your backup software.
   • Re-run the check weekly and send the output to DataON Support so the trend can be tracked.
4. If any node shows URGENT (HiveSizeMB over 400), follow all of the steps above, and also:
   • Pause VM backup jobs on the cluster until DataON Support has reviewed the results. At this size, additional growth brings the node closer to a point where it may not boot.
   • Contact DataON Support immediately.

Why:

Each time a VSS shadow copy is created, Windows registers a device entry for it. If those entries aren't removed when the snapshot is released, they stay in the SYSTEM registry hive as non-present ("phantom") devices. Over time the hive can grow from tens of megabytes to more than a gigabyte. Once it gets that large, Windows may not be able to load it at boot, and the node fails with a registry error.

This condition appears to involve the interaction between VSS-based backup software and Azure Local, and has been seen on clusters with frequent VM-level backups. It is not specific to DataON systems: nothing DataON installs on the hosts creates VSS snapshots. The "CONTACT DATAON" status simply routes results to us so we can review them, track affected clusters, and share updates. The root cause is under investigation with Microsoft.

Going forward:

Run this check before every solution update, before any planned node reboot, and after changes to your backup schedule. A phantom count that keeps rising between checks means entries are still accumulating, even if the node is under the threshold today.

What to expect: There is currently no fix or supported cleanup procedure. Microsoft is investigating, and has advised against cleaning up these entries until that investigation is complete. When you contact DataON Support, your ticket is used to track your cluster's status and to share remediation guidance as soon as it becomes available. Until then, the steps above are the recommended way to reduce risk.

Optional details:

• With automatic restart disabled, an affected node that fails to boot shows: File: \Windows\system32\config\system Status: 0xc000000d. The message says the system registry file is missing or contains errors.
• Reference values: healthy nodes typically show a hive of about 25 to 55 MB and fewer than 100 phantom entries. Affected nodes have shown hives of 500 MB to more than 1 GB, with tens of thousands of phantom entries.
• Each phantom entry adds roughly 15 to 20 KB to the hive, so the phantom count usually rises before the hive size crosses a threshold.
• Don't edit the registry or remove device entries yourself. Changes to the SYSTEM hive can leave a node unable to boot. Supported remediation guidance will be shared here once available.