Where the waste actually is in a vSphere estate

Why this is worth doing now

Reclaiming capacity used to be housekeeping. It stopped being housekeeping when licensing moved to a per-core subscription, because the number of hosts you need is now a recurring line on the invoice rather than a one-off purchase you already made. Freeing enough load to retire a host takes its cores off the invoice at the next renewal, and every year after that.

The other reason is less obvious: most estates that feel slow are not short of capacity. They are badly shaped. The fix needs no new hardware and often makes things measurably faster.

The four places it goes

In the estates we audit, essentially all reclaimable capacity sits in four buckets.

The order matters. Snapshots first, because they are a risk and not merely a cost. Zombies second, because they are free to remove and need only evidence of idleness, not sizing data. Right-sizing third, because it needs data. Headroom last, because it needs a decision from someone who owns the risk.

The counter-intuitive one: more vCPU, slower VM

This is the finding that surprises people, so it is worth stating plainly. A VM with more vCPUs than it needs can run slower than the same VM with fewer.

The reason is scheduling. ESXi schedules each vCPU on its own, but it has to keep the vCPUs of one VM roughly in step: when some of a VM’s vCPUs fall behind their siblings by more than a threshold, the ones that are ahead are co-stopped until the laggards catch up (relaxed co-scheduling). A 16-vCPU VM has more siblings to keep in step than a 4-vCPU VM, so on a busy host it is co-stopped more often; its idle vCPUs still cost a little overhead, and if it is wider than a NUMA node it also spans nodes it did not need. Inside the guest this looks like the application being slow while the CPU graph looks calm.

Two numbers expose it. CPU ready time is time a vCPU was ready but got no physical CPU: contention in general, or a CPU limit. Co-stop time exists only for multi-vCPU VMs and is the wide-VM penalty itself. Broadcom’s right-sizing guidance (KB 438023) puts the thresholds at under 5 % ready per vCPU as benign and over 10 % as a real impact, and under 3 % co-stop per vCPU as normal; sustained co-stop above that typically indicates too many vCPUs. Usage tells you how hard a VM worked; ready and co-stop tell you how long it waited. An estate can show comfortable usage everywhere and still be starving its largest VMs. Look at both per vCPU rather than per VM, since the totals scale with the width of the machine.

Memory has a matching signal: ballooning and swapping at the host level mean the host is reclaiming from guests, and a VM that is swapping has performance problems that no amount of guest tuning will fix.

Measuring honestly

Right-sizing from last Tuesday’s numbers causes incidents. Three rules keep it safe.

Thirty days, minimum, at five-minute resolution. Month-end, payroll runs, reporting cycles and backup windows all move the peak, and a week of data will miss at least one of them. If the business has a quarterly close, watch a quarter for anything close to it. Mind where the data comes from: with vCenter’s default collection intervals, 5-minute samples are kept for one day, 30-minute averages for a week and 2-hour averages for a month, so a month of vCenter data has already flattened the peaks you are looking for, and vCenter stores co-stop only from statistics level 2, one above its default. Keep 5-minute samples for the whole window, with VCF Operations (formerly Aria Operations) or an external collector.

Size on percentiles, not on averages or maxima. The average is meaningless for a workload that spikes; the maximum is usually one anomalous sample. The 95th percentile of sustained use, with deliberate headroom on top, is the number that survives contact with production.

Cut in steps, and one dimension at a time. Halve a vCPU count, wait a fortnight, look again; each cut needs a power-off. Changing CPU and memory in the same window means you cannot tell which one caused the complaint.

One exception worth knowing: for guest software licensed per core or per vCPU (some database engines do), the guest software licence may dominate the decision entirely, and the right vCPU count is the one the database vendor charges least for. Check that before you tune anything.

What an audit actually produces

The output that is useful is not a list of findings. It is a sequenced plan where every line has a number and a risk attached:

Reclaim now, low risk: snapshots older than the retention policy, orphaned VMDKs confirmed unused by every vCenter and host that sees the datastore, powered-off VMs past their review date. Storage back the same day.

Reclaim after confirmation: VMs with negligible network traffic and no logins for 90 days. The list goes to application owners with the evidence attached.

Right-size on measurement: per-VM proposed vCPU and memory with the percentile data behind each, ordered by how much they release.

Consolidate: what the estate looks like after the above, how many hosts it actually needs, and what that does to the licensed core count over the subscription term. This is the line where a licence saving shows up, if there is one; see how the core count is calculated for why.

A worked example

The following is an illustrative example, not a client case. An eight-host cluster, comfortable on paper: 62% memory, 41% CPU. The complaint was that the ERP system had “got slow”.

In this example, five weeks of measurement found three things. The ERP database VM had 24 vCPU and a 95th percentile of six, with ready and co-stop time high enough to explain the complaint entirely. Nineteen VMs had had no logins and negligible network traffic for over 90 days. And 4.1 TB of datastore was held by eleven snapshots, the oldest fourteen months.

Removing the snapshots returned the storage that week. Retiring the zombies after owner sign-off released memory. Cutting the ERP VM from 24 vCPU to 12 and then to 8 made it faster: the same result the application vendor had been proposing to solve with a bigger host. What was left fitted on six hosts with the same failure tolerance, and two hosts came out of the licensed core count.

No new hardware, no platform change, and the original complaint fixed by removing resources from the machine that was complaining.

What we do

Our infrastructure audit is a fixed-price engagement with its scope fixed in the contract; the report can include the current-state map, the measured right-sizing table, the consolidation options and the licence-core delta for each. The report is yours whether or not you do the work with us, and you can hand it to another supplier. Implementation runs through our engineering partner Vixen.UNO under one European contract. See infrastructure audit and VMware optimisation.

FAQ