vSphere right-sizing: where the waste is in oversized VMs, snapshots and headroom
Eurokommerz, Vienna, since 2006: Private AI/ML · IT Managed Services · Enterprise Training · AI Hardware & Software
- Most oversized VMs still have the vCPU and memory requested at deployment, because sizing is rarely reviewed later
- A VM with more vCPUs than it uses can run slower than the same VM with fewer. The scheduler must keep sibling vCPUs in step, so a wide VM is co-stopped on a busy host; watch CPU ready and co-stop time, not CPU usage
- Forgotten snapshots keep growing and can fill the datastore; Broadcom’s KB 318825 says not to retain a single snapshot for more than 72 hours
- Since licensing follows physical cores (at least 16 per CPU), each host retired before the renewal comes off the licensed core count
- Measure for at least 30 days at 5-minute resolution before you resize anything: a week of data misses month-end and other cyclic peaks, and vCenter’s default roll-ups smooth them away
Eurokommerz × Vixen.UNO: VMware Optimisation Talk to an expert →
Why right-sizing matters under per-core licensing
Right-sizing a vSphere estate means finding capacity that is allocated but not used (oversized VMs, zombie VMs and orphaned disks, forgotten snapshots, cluster headroom nobody re-checked) and giving it back. Since VMware licensing moved to a per-core subscription, the number of hosts you need is a recurring line on the invoice rather than a one-off purchase you already made. Freeing enough load to retire a host takes its cores off the invoice at the next renewal, and every year after that. Before a renewal, our engineering partner Vixen.UNO shows what you pay for and how much of it is in use, so you see the number before you sign; if there is nothing to optimise, we say so. Send us your renewal date and host count.
The second reason is performance. An estate that feels slow may need smaller VMs rather than more hosts, for the scheduling reasons described in the section on CPU ready and co-stop below.
Four places where capacity sits unused
Reclaimable capacity in a vSphere estate usually sits in four places.
| WHERE | WHAT IT LOOKS LIKE | WHAT IT COSTS YOU |
|---|---|---|
| Oversized VMs | a database VM with 16 vCPU that peaks at three | memory consumed and scheduling overhead carried by the cluster, inflated headroom planning; on wide VMs, worse latency than a smaller VM would have |
| Zombie VMs and orphaned disks | powered-on machines nobody has logged into in months; VMDKs left after a VM was removed | full RAM and storage cost for zero work, plus backup windows and licence counts on guest software |
| Snapshots | a snapshot taken “before the upgrade” in March, still there | a delta file that can grow to the full size of its base disk, so a chain can more than double what a VM occupies until the datastore fills; performance degrades as the chain lengthens |
| Stale cluster headroom | N+2 reserved on a cluster whose workloads were right-sized two years ago | a whole host of capacity, licensed and powered, held for a failure scenario that no longer needs it |
The order matters. Snapshots first, because they are a risk and not merely a cost. Zombies second, because they are free to remove and need only evidence of idleness, not sizing data. Right-sizing third, because it needs data. Headroom last, because it needs a decision from someone who owns the risk.
More vCPU, slower VM: CPU ready and co-stop
A VM with more vCPUs than it needs can run slower than the same VM with fewer.
The reason is scheduling. ESXi schedules each vCPU on its own, but it has to keep the vCPUs of one VM roughly in step: when some of a VM’s vCPUs fall behind their siblings by more than a threshold, the ones that are ahead are co-stopped until the laggards catch up (relaxed co-scheduling). A 16-vCPU VM has more siblings to keep in step than a 4-vCPU VM, so on a busy host it is co-stopped more often; its idle vCPUs still cost a little overhead, and if it is wider than a NUMA node it also spans nodes it did not need. Inside the guest this looks like the application being slow while the CPU graph looks calm.
Two numbers expose it. CPU ready time is time a vCPU was ready but got no physical CPU: contention in general, or a CPU limit. Co-stop time exists only for multi-vCPU VMs and is the wide-VM penalty itself. Broadcom’s right-sizing guidance (KB 438023) cites industry-accepted thresholds of below about 5% ready per vCPU as benign and above 10% as a noticeable performance impact, and calls co-stop below about 3% per vCPU normal; sustained co-stop above that typically indicates too many vCPUs. Usage tells you how hard a VM worked; ready and co-stop tell you how long it waited. An estate can show comfortable usage everywhere and still be starving its largest VMs. Look at both per vCPU rather than per VM, since the totals scale with the width of the machine.
Memory has a matching signal: ballooning and swapping at the host level mean the host is reclaiming from guests, and a VM that is swapping has performance problems that no amount of guest tuning will fix.
How long and how to measure before resizing
A few days of data miss the cyclic peaks. Three rules apply.
Thirty days, minimum, at five-minute resolution. Month-end, payroll runs, reporting cycles and backup windows all move the peak, and a week of data will miss at least one of them. If the business has a quarterly close, watch a quarter for anything close to it. Mind where the data comes from: with vCenter’s default collection intervals, 5-minute samples are kept for one day, 30-minute averages for a week and 2-hour averages for a month, so a month of vCenter data has already flattened the peaks you are looking for, and vCenter stores co-stop only from statistics level 2, one above its default. Keep 5-minute samples for the whole window, with VCF Operations (formerly Aria Operations) or an external collector.
Size on percentiles, not on averages or maxima. The average is meaningless for a workload that spikes; the maximum is usually one anomalous sample. Size on the 95th percentile of sustained use and add the headroom you decide on.
Cut in steps, and one dimension at a time. Halve a vCPU count, wait a fortnight, look again; each cut needs a power-off. Changing CPU and memory in the same window means you cannot tell which one caused the complaint.
One exception worth knowing: for guest software licensed per core or per vCPU (some database engines do), the guest software licence may dominate the decision entirely, and the right vCPU count is the one the database vendor charges least for. Check that before you tune anything.
What an audit produces
A useful audit result is a sequenced plan in which every line has a number and a risk attached:
Reclaim now, low risk: snapshots older than the retention policy, orphaned VMDKs confirmed unused by every vCenter and host that sees the datastore, powered-off VMs past their review date. Storage back the same day.
Reclaim after confirmation: VMs with negligible network traffic and no logins for 90 days. The list goes to application owners with the evidence attached.
Right-size on measurement: per-VM proposed vCPU and memory with the percentile data behind each, ordered by how much they release.
Consolidate: what the estate looks like after the above, how many hosts it actually needs, and what that does to the licensed core count over the subscription term. This is the line where a licence saving shows up, if there is one; see how the core count is calculated for why.
Our infrastructure audit works from the exports and documentation you provide, without access to your production systems, and its scope is fixed in the contract. Tell us how many hosts and VMs you run, and our team will help.
A worked example
The following is an illustrative example, not a client case. An eight-host cluster, comfortable on paper: 62% memory, 41% CPU. The complaint was that the ERP system had “got slow”.
In this example, five weeks of measurement found three things. The ERP database VM had 24 vCPU and a 95th percentile of six, with ready and co-stop time high enough to explain the complaint entirely. Nineteen VMs had had no logins and negligible network traffic for over 90 days. And 4.1 TB of datastore was held by eleven snapshots, the oldest fourteen months.
Removing the snapshots returned the storage that week. Retiring the zombies after owner sign-off released memory. Cutting the ERP VM from 24 vCPU to 12 and then to 8 made it faster: the same result the application vendor had been proposing to solve with a bigger host. What was left fitted on six hosts with the same failure tolerance, and two hosts came out of the licensed core count.
What we do
Our infrastructure audit is a separate product with a fixed price and its scope fixed in the contract: a full inventory, actual resource usage with where there is excess and where a shortage, an ROI and TCO model and a roadmap. The roadmap is yours: you can deliver it with us, on your own or with another contractor. Implementation runs through our engineering partner Vixen.UNO under one European contract. See infrastructure audit and VMware optimisation.
FAQ
How long should I measure before right-sizing?
Which metric tells me a VM is oversized?
Is it safe to delete a snapshot that is a year old?
How do I find zombie VMs without breaking something?
Does right-sizing reduce my VMware bill?
We are already tight on capacity. Is an audit still worth it?
Our infrastructure audit shows where your estate has excess capacity and where a shortage, and the roadmap is yours. Describe your task in a few lines and our team will help: we reply within one business day. The first call is free of charge.
Talk to an expertWe reply within one business day