TL;DR: Run Get-StorageQoSFlow from any cluster node to see live IOPS, latency, and bandwidth for every running VM on the cluster. No agents, configuration, or downtime required — the Storage QoS resource that collects this data is built into every Storage Spaces Direct and Azure Local cluster.
Recommended action
Where to run: Every command in this article works the same way in either of these:
- An elevated PowerShell window directly on a cluster node (console or Remote Desktop), or
- A remote session to a cluster node from your management machine:
Enter-PSSession -ComputerName <NodeName>
Each command is labeled with where to run it and what it covers: all VMs in the cluster, one node only, or one VM.
1. Per-VM summary (one row per VM, busiest first)
Run on: any one cluster node, directly or via Enter-PSSession
Covers: all VMs in the cluster — run it once, not per node
A VM with several virtual disks appears as several separate flows. This script rolls them up into one row per VM: IOPS and bandwidth are summed, and latency is IOPS-weighted so a quiet disk doesn't skew the average. Flows are de-duplicated by FlowId first so the same disk is never counted twice (see Optional details).
Get-StorageQoSFlow |
Group-Object InitiatorName |
ForEach-Object {
$flows = @($_.Group | Sort-Object FlowId -Unique)
$iops = ($flows | Measure-Object InitiatorIOPS -Sum).Sum
$wLat = ($flows | ForEach-Object { $_.InitiatorLatency * $_.InitiatorIOPS } | Measure-Object -Sum).Sum
[PSCustomObject]@{
'VM Name' = if ($_.Name) { $_.Name } else { '(non-VM I/O)' }
'Disks' = $flows.Count
'IOPS' = $iops
'Latency (ms)' = if ($iops -gt 0) { [math]::Round($wLat / $iops, 2) } else { 0 }
'Bandwidth (MB/s)' = [math]::Round(($flows | Measure-Object InitiatorBandwidth -Sum).Sum / 1MB, 2)
}
} |
Sort-Object IOPS -Descending |
Format-Table -AutoSizeExample output from a lightly loaded all-flash cluster:
VM Name Disks IOPS Latency (ms) Bandwidth (MB/s)
------- ----- ---- ------------ ----------------
APP-SQL01 2 412 0.58 3.21
FILE01 1 36 0.47 0.29
DC01 1 4 0.47 0.02
MGMT01 1 1 0 0.01A latency of 0 on a VM doing roughly 1 IOPS is normal — at that rate the sample interval may not capture a completion time.
2. Per-virtual-disk detail
Run on: any one cluster node, directly or via Enter-PSSession
Covers: all VMs in the cluster, one row per virtual disk
Use this when a VM in step 1 stands out and you want to know which of its disks, and which host, the load is coming from. The file name is listed last so long names don't push the numeric columns off screen.
Get-StorageQoSFlow |
Sort-Object InitiatorIOPS -Descending |
Format-Table @{n='VM Name';e={$_.InitiatorName}},
@{n='Host';e={$_.InitiatorNodeName.Split('.')[0]}},
@{n='IOPS';e={$_.InitiatorIOPS}},
@{n='Latency (ms)';e={[math]::Round($_.InitiatorLatency,2)}},
@{n='Bandwidth (MB/s)';e={[math]::Round($_.InitiatorBandwidth/1MB,2)}},
Status,
@{n='Virtual Disk';e={Split-Path $_.FilePath -Leaf}} -AutoSize3. A single VM, all fields
Run on: any one cluster node, directly or via Enter-PSSession
Covers: one VM — the one you name, on whichever node it is running
Get-StorageQoSFlow -InitiatorName "VMName" | Format-List *4. Optional: save a snapshot for comparison
Run on: the same node and session as step 1 or 2
Covers: all VMs in the cluster (same as the step 1 or 2 command it is attached to)
Append | Export-Csv -Path C:\Temp\VM-Storage-$(Get-Date -Format yyyyMMdd-HHmm).csv -NoTypeInformation to the step 1 or step 2 command in place of Format-Table to keep a record you can compare against later. The file is written to the cluster node — including when you are connected through Enter-PSSession, in which case it is not on your management machine. The C:\Temp folder must already exist on that node.
Why
On a Storage Spaces Direct cluster, every VM's virtual hard disk I/O passes through the cluster's Storage QoS layer, which tracks each open virtual disk as a flow. Storage QoS runs as a cluster resource, so Get-StorageQoSFlow returns the counters for the whole cluster from whichever node you ask — a cluster-wide, per-VM view without logging into each VM or each node.
Key columns:
- IOPS — I/O operations per second the VM is issuing.
- Latency (ms) — average response time as seen from the host on the VM's behalf. This is the number that reflects what the application experiences.
- Bandwidth (MB/s) — throughput. The raw
InitiatorBandwidthproperty is reported in bytes per second; the scripts above convert it to MB/s. - Status —
Okis normal.InsufficientThroughputmeans a QoS minimum policy is not being met;LostCommunicationorUnknownPolicyIdpoint to a Storage QoS resource or policy problem rather than a performance issue.
Going forward
- These are live values, not history. Run the commands while the issue is happening. A single sample during a quiet period will not explain a slowdown that occurred earlier.
- Look at latency and IOPS together. One VM with very high IOPS and normal latency is a busy workload, not a problem. Rising latency across many VMs at once usually points to a cluster-wide cause (background storage repair jobs, network/RDMA issues, or a degraded drive) rather than a single noisy VM.
- Check for storage jobs first when latency is elevated cluster-wide with
Get-StorageJob. It should return nothing on a healthy, idle cluster. Repair and resync after maintenance or a drive failure will raise latency for all VMs until it completes.
Run on: any one cluster node, directly or via Enter-PSSession
Covers: the whole cluster's storage pool - Avoid comparing against benchmark numbers taken with synthetic tools. Production VMs mix block sizes and read/write ratios, so their IOPS and latency will not match a 4K random test.
Optional details
The same disk listed twice. The per-disk view (step 2) can occasionally show one VM's disk on two rows with identical numbers — sometimes with two different file paths, for example after the VM's storage was moved or a copy of the disk was made. If both rows share the same FlowId, they are the same flow reported twice, not two separate disks. The step 1 summary removes these duplicates automatically. To check a specific VM:
Run on: any one cluster node, directly or via Enter-PSSession
Covers: one VM
Get-StorageQoSFlow -InitiatorName "VMName" | Format-List FlowId, FilePath, InitiatorIOPS.avhdx files. A virtual disk ending in .avhdx means the VM is running on a checkpoint (differencing disk). The VM writes to the checkpoint file rather than its base disk. Checkpoints left in place long-term grow over time and add read overhead; if this is unexpected, review and merge them in Hyper-V Manager, Failover Cluster Manager, or Windows Admin Center.
Initiator vs. storage node values. Each flow also exposes StorageNodeIOPS, StorageNodeLatency, and StorageNodeBandwidth, measured where the I/O lands on the storage side. If initiator latency is much higher than storage node latency for the same flow, the delay is being added between the VM's host and the storage (for example, network or CSV redirected I/O), not at the drives.
Normalized IOPS. Storage QoS counts I/O in normalized 8 KB units: an I/O of 8 KB or smaller counts as one, and larger I/Os count as multiples (a 64 KB I/O counts as eight). VMs doing large sequential I/O, such as backups, will therefore show higher IOPS here than inside the guest OS. Bandwidth is the better comparison metric for those workloads.
Historical data. For trends over hours or days, use the per-VM performance charts in Windows Admin Center or, for Azure Local, the VM metrics in the Azure portal. These charts depend on the cluster's Health resource being online and the ClusterPerformanceHistory volume being present; if the charts are empty, check both before troubleshooting further.
Companion article. For per-VM network throughput and dropped packets, see the Hyper-V article on checking vNIC throughput.
Microsoft reference: Storage Quality of Service overview