vSphere Memory Tiering with NVMe: requirements, ratio, setup and sizing in VCF 9.0 and 9.1

What vSphere Memory Tiering with NVMe does

vSphere Memory Tiering, which Broadcom’s documentation calls Memory Tiering over NVMe, adds memory capacity to an ESX host with NVMe devices installed locally in it. DRAM is Tier 0 and the NVMe device is Tier 1, and ESX keeps the active pages of each VM in DRAM while it moves cold pages to NVMe. The feature is deactivated by default, and the vSphere 9.1 documentation limits the NVMe tier by default to the size of DRAM, up to 4 TB.

Broadcom’s KB 311934 describes Memory Tiering as a Technical Preview in vSphere 8.0 Update 3, and the Performance Best Practices for vSphere 9.1, revision 20260624, state that it is supported beginning in vSphere 9.0. For 9.1, the vSphere 9.1 FAQ of May 2026 lists no reboot to turn it on, device health and wear in the UI, and RAID 1 mirroring. Our overview of what is new in VCF 9.1 covers the rest of the release.

The 9.1 page on considerations and best practices, updated on 12 October 2026, says that “kernel memory is never tiered” and that cold pages start to move when host memory consumption reaches around 80% of DRAM capacity.

Memory Tiering requirements in vSphere 9.0 and 9.1

Broadcom TechDocs, Memory Tiering over NVMe, vSphere 9.0 (24 August 2026) and 9.1 (12 October 2026); Broadcom KB 311934.

The 9.1 documentation describes the device as “a drive similar to vSAN cache (Mixed Use with 3 DWPD)” and refers to the Broadcom Compatibility Guide for supported devices. Hot-plug and prepare to remove are not supported for a tier device, and Memory Tiering must be deactivated on the host before the device is removed or replaced.

On vSAN hosts the tier device is a separate drive, either a new disk added with automatic claiming switched off, a disk left unclaimed, or a disk removed from vSAN after a pre-check.

NVMe-to-DRAM ratio and tier size

In vSphere 9.0 the tier size is the host advanced setting Mem.TierNvmePct, a percentage of total DRAM. Broadcom advises 100% or lower, so that DRAM is at least as large as the tier, limits the tier partition to 4 TB and asks for a reboot after a change.

In vSphere 9.1 the same value is tier_size_pct in the cluster configuration and --tier-size-pct in ESXCLI. The ESXCLI reference reads “Set maximum tiering size with respect to DRAM (Value must be between 1 and 400)”, and the configuration page advises the default of 100 for optimal performance. The 9.1 documentation gives the 4 TB maximum with the default and does not say how it applies above 100. If the device is smaller than DRAM, the smaller capacity is used, and the best practices page states that “In general, NVMe sizes must be at least as large as the DRAM of the system.”

NVMe is slower than DRAM, so Broadcom limits how much active memory may depend on it. The 9.1 Performance Best Practices advise keeping active memory at no more than 50% of DRAM, though some workloads do not suffer until it approaches 75%. For NVMe read latency they advise staying below 200 microseconds, with the same caveat at 400. They also ask for CPU headroom, and the 9.1 TechDocs put the cost at about an eighth of a core on smaller hosts.

VMs and features excluded from Memory Tiering

The rules for VM types differ between the versions. The 9.0 page on VM profiles (24 August 2026) says that five types, listed in the sizing table below, “cannot be placed on an ESX host configured for Memory Tiering”, so on 9.0 they need hosts without tiering. The 9.1 page (12 October 2026) lets them run on a tiered host, but they “are still excluded from the Memory Tiering process”; of these, only VBS VMs can be tiered, with sched.mem.enableNestedTiering set to TRUE. For VMs with more than 512 GB and 64 vCPUs, the 9.1 page warns of performance degradation and advises turning tiering off and reserving their full memory.

On 9.0, VMs with 1 GB pages, preallocated memory, PCI passthrough, UPT or vQAT stay in DRAM automatically. To keep another VM in DRAM, power it off, add sched.mem.enableTiering with the value FALSE and select Reserve all guest memory (All locked), as the 9.1 configuration page describes.

The best practices page says that memory reservations, not pinned memory, can be split between DRAM and NVMe when the reserved VMs consume more than DRAM holds. It also says “Do not use memory overcommit when Memory Tiering is activated”, which we read as configured VM memory within DRAM plus the tier. Ballooning and host swapping are covered in our guide to vSphere memory overcommit, ballooning and swapping.

vMotion “will take longer (1.5X to 2X) to complete”, because pages are first fetched from NVMe, and DRS supports clusters that mix tiered and non-tiered hosts.

The 9.1 page on NVMe failure scenarios says “If a page fault is raised against a page on the failed device and cannot be serviced, the virtual machine powers off.” VMs that use only DRAM keep running, and HA VM restart policies can restart the others on hosts with spare capacity, which argues for mirrored devices and an HA reserve sized with the tier.

Which of your hosts qualify depends on their active memory and on how many VMs must stay in DRAM. Describe your clusters and their memory profile in the form below, and we work through the candidates with you.

How to enable Memory Tiering in vSphere 9.1 and 9.0

In vSphere 9.1, Memory Tiering is set per cluster in the vSphere Client under Configure, Desired State, Configuration: on the Draft tab, under esx, memtier, nvme, Configure Settings, set enable to True and select at least one NVMe device, optionally a mirror device, encryption and tier_size_pct. The Configuration Manager places hosts in maintenance mode automatically, without a reboot. For one host, with the cluster managed by a single image and vSphere Configuration Profiles, the ESXCLI steps are these.

1. Place the host in maintenance mode.
2. Run esxcli memtier device list to find the NVMe device; -a lists only the available ones.
3. Run esxcli memtier enable -d <device_1>; for software mirroring, pass both devices to -d as <device_1,device_2>.
4. Add -c t for encryption of the tier and -r with a percentage to change the tier size, as in the example -c t -r 50.
5. Check the result with esxcli memtier status get and esxcli memtier config get, then exit maintenance mode.

Later changes use esxcli memtier config set on a host in maintenance mode, for example with --tier-size-pct 25 for a 4:1 ratio, and esxcli memtier disable turns tiering off. Broadcom warns that this reduces the memory available on the cluster.

In vSphere 9.0, with the host in maintenance mode and all partitions deleted from the device, esxcli system tierdevice create -d followed by /vmfs/devices/disks/ and the device name creates the tier partition, and esxcli system tierdevice list shows it. VMKernel.Boot.memoryTiering is then set to true under Configure, System, Advanced System Settings, and the host is rebooted. Broadcom’s VCF blog of 25 September 2026 says 9.1 moves all these operations into the esxcli memtier namespace, so check 9.0 scripts before the upgrade.

Monitoring Memory Tiering on ESX hosts

If esxcli memtier config get reports Enable as True while esxcli memtier status get reports Disabled, the 9.1 documentation points to the system logs and alarms. In the vSphere Client, the Memory card on the host’s Summary tab shows its Memory Tiering configuration, and the cluster’s Hosts tab can show a Memory Tiering column that reads Software for configured hosts.

KB 449016 lists four counters in the Memory view of the advanced performance charts: Memory type read bandwidth, read latency, write bandwidth and write latency. We would compare read latency with the 200 microseconds above. Per VM, the KB runs memstats -r vmtier-stats -u mb on the host over SSH, with -s selecting the columns name, memSize, active, tier1Target and tier1Consumed.

Memory Tiering licensing in VVF and VCF

Memory Tiering is part of vSphere 9, and the vSphere 9.1 FAQ states that “vSphere 9.0 and 9.1 features are only available as part of VMware vSphere Foundation 9.0 and 9.1 and VMware Cloud Foundation 9.0 and 9.1.” The same FAQ limits vSphere Standard and vSphere Enterprise Plus to version 8 Update 3 and earlier. As of October 2026 we found no Broadcom document that licenses the NVMe tier, DRAM or Memory Tiering separately.

VVF and VCF are licensed on every physical core of every server where the software is installed, at least 16 per processor, so where memory sets the host count, a larger memory tier can lower the number of hosts and of licensed cores. Our guide to host consolidation under per-core licensing covers HA admission control and vSAN host minimums, which set a floor of their own.

Sizing Memory Tiering for 10 to 60 hosts

Broadcom’s best practices page advises starting with hosts or clusters whose active memory is between 0% and 50% of the existing DRAM, and our guide to vSphere capacity planning explains how to collect 90 days of 5-minute samples per cluster.

Broadcom TechDocs, Memory Tiering Considerations and Best Practices, 9.1 (12 October 2026); Virtual Machine Profiles and Performance with Memory Tiering, 9.0 (24 August 2026) and 9.1 (12 October 2026).

This example is illustrative, not a client case. A cluster of 16 hosts, each with two 32-core CPUs and 1,024 GB of DRAM, runs 1,000 VMs with 14,000 GB of configured memory. The 95th percentiles over 90 days are 12,000 GB consumed, 2,400 GB active and 420 cores of CPU demand. With an 80% ceiling and one host in reserve, memory needs 12,000 ÷ (0.8 × 1,024) = 14.6, rounded up to 15, plus one: all 16 hosts. CPU needs 420 ÷ (0.8 × 64) = 8.2, so 9 plus one: 10 hosts.

With a 1:1 tier, each host offers 2,048 GB, and memory needs 12,000 ÷ (0.8 × 2,048) = 7.3, so 8 plus one: 9 hosts. CPU then sets the count at 10 hosts, or 10 × 64 = 640 licensed cores in place of 16 × 64 = 1,024. With one host down, active memory is 2,400 ÷ 9 = 267 GB per host, about 26% of DRAM, and the 14,000 GB of configured memory fits in 9 × 2,048 = 18,432 GB without overcommit. A mirrored tier needs two mixed-use NVMe devices of at least 1,024 GB per host, and the memory of VMs that must stay in DRAM counts against DRAM alone. Whether that justifies fewer hosts is a calculation on your own numbers; new servers are the topic of our guide to VMware host refresh.

If a memory tier lets a cluster run on fewer hosts, the consolidation is part of our VMware optimisation service and runs in agreed maintenance windows with a rollback plan. Send us the DRAM per host and 90 days of consumed and active memory.

What we do

For Memory Tiering, VMware optimisation starts with the estate and licence audit by our engineering partner Vixen.UNO, which records versions, subscriptions and the actual resource usage of every cluster. From that data the team matches editions and subscriptions to your actual workloads and consolidates hosts and clusters, with all changes in agreed maintenance windows and a rollback plan at every stage. Eurokommerz holds the contract and supplies the hardware under the solution, including the NVMe devices. The first call is free of charge; the price of the technical assessment is fixed before work begins.

FAQ