Where the waste actually is in a vSphere estate

Why this is worth doing now

Reclaiming capacity used to be housekeeping. It stopped being housekeeping when licensing moved to a per-core subscription, because the number of hosts you need is now a recurring line on the invoice rather than a one-off purchase you already made. Freeing enough load to retire a host takes its cores off the invoice at the next renewal, and every year after that.

The other reason is less obvious: most estates that feel slow are not short of capacity. They are badly shaped. The fix costs nothing and often makes things measurably faster.

The four places it goes

In the estates we audit, essentially all reclaimable capacity sits in four buckets.

The order matters. Snapshots first, because they are a risk and not merely a cost. Zombies second, because they are free to remove and need no measurement. Right-sizing third, because it needs data. Headroom last, because it needs a decision from someone who owns the risk.

The counter-intuitive one: more vCPU, slower VM

This is the finding that surprises people, so it is worth stating plainly. A VM with more vCPUs than it needs can run slower than the same VM with fewer.

The reason is scheduling. ESXi schedules each vCPU on its own, but it has to keep the vCPUs of one VM roughly in step: when some of a VM’s vCPUs fall behind their siblings by more than a threshold, the ones that are ahead are co-stopped until the laggards catch up (relaxed co-scheduling). A 16-vCPU VM has more siblings to keep in step and more idle vCPUs to service than a 4-vCPU VM, so on a busy host it is co-stopped more often and also spans NUMA nodes it did not need. Inside the guest this looks like the application being slow while the CPU graph looks calm.

Two numbers expose it. CPU ready time is time a vCPU was ready but had no physical core: contention in general. Co-stop time exists only for multi-vCPU VMs and is the wide-VM penalty itself. Broadcom’s right-sizing guidance puts the thresholds at under 5 % ready per vCPU as benign and over 10 % as a real impact, and under 3 % co-stop per vCPU as normal; sustained co-stop above that means too many vCPUs. Usage tells you how hard a VM worked; ready and co-stop tell you how long it waited. An estate can show comfortable usage everywhere and still be starving its largest VMs. Look at both per vCPU rather than per VM, since the totals scale with the width of the machine.

Memory has a matching signal: ballooning and swapping at the host level mean the host is reclaiming from guests, and a VM that is swapping has performance problems that no amount of guest tuning will fix.

Measuring honestly

Right-sizing from a snapshot of last Tuesday is how you cause an incident. Three rules keep it safe.

Thirty days, minimum, at five-minute resolution. Month-end, payroll runs, reporting cycles and backup windows all move the peak, and a week of data will miss at least one of them. If the business has a quarterly close, watch a quarter for anything close to it. Mind where the data comes from: with vCenter’s default statistics levels, anything older than a week is a 30-minute or 2-hour average, and a percentile computed on 2-hour averages has already flattened the peaks you are looking for. Keep 5-minute samples for the whole window, with VCF Operations (Aria) or an external collector.

Size on percentiles, not on averages or maxima. The average is meaningless for a workload that spikes; the maximum is usually one anomalous sample. The 95th percentile of sustained use, with deliberate headroom on top, is the number that survives contact with production.

Cut in steps, and one dimension at a time. Halve a vCPU count, wait a fortnight, look again. Changing CPU and memory in the same window means you cannot tell which one caused the complaint.

One exception worth knowing: for guest software licensed per core or per vCPU (some database engines do), the guest software licence may dominate the decision entirely, and the right vCPU count is the one the database vendor charges least for. Check that before you tune anything.

What an audit actually produces

The output that is useful is not a list of findings. It is a sequenced plan where every line has a number and a risk attached:

Reclaim now, no risk: snapshots older than the retention policy, orphaned VMDKs, powered-off VMs past their review date. Storage back the same day.

Reclaim after confirmation: VMs with no network traffic and no logins for 90 days. Someone has to say the word, so the list goes to application owners with the evidence attached.

Right-size on measurement: per-VM proposed vCPU and memory with the percentile data behind each, ordered by how much they release.

Consolidate: what the estate looks like after the above, how many hosts it actually needs, and what that does to the licensed core count over the subscription term. This is the line that usually pays for the whole exercise; see how the core count is calculated for why.

A worked example

An eight-host cluster, comfortable on paper: 62% memory, 41% CPU. The complaint was that the ERP system had “got slow”.

Measurement over five weeks found three things. The ERP database VM had 24 vCPU and a 95th percentile of six, with ready time high enough to explain the complaint entirely. Nineteen VMs had had no logins and negligible network traffic for over 90 days. And 4.1 TB of datastore was held by eleven snapshots, the oldest fourteen months.

Removing the snapshots returned the storage that week. Retiring the zombies after owner sign-off released memory. Cutting the ERP VM from 24 vCPU to 8 made it faster: the same result the application vendor had been proposing to solve with a bigger host. What was left fitted on six hosts with the same failure tolerance, and two hosts came out of the licensed core count.

No new hardware, no platform change, and the original complaint fixed by removing resources from the machine that was complaining.

What we do

Our infrastructure audit is a fixed-price engagement with a fixed deliverable: the current-state map, the measured right-sizing table, the consolidation options and the licence-core delta for each. You own the report whether or not you do the work with us, and it is written to be handed to another supplier if you prefer. Implementation runs through our engineering partner Vixen.UNO under one European contract. See infrastructure audit and VMware optimisation.

FAQ