Every new class of workload seems to arrive with the same claim attached. It’s too important or too performance-sensitive to run in a virtual machine (VM). I heard it about databases, then about Exchange, then about SAP. Each time the argument lasted a few years, and each time virtualization won because the business cared more about agility and lifecycle management than about the last few percent of performance. AI on graphics processing units (GPUs) is the newest workload to get this treatment. At Cloud Field Day 26, Shawn Kelly from VMware by Broadcom made a strong case that the bare-metal argument doesn’t hold up for most enterprise AI.
Virtualization Overhead on GPUs Is Usually Under 2%
Under the hood, nearly every way of attaching a GPU to a VM in VMware Cloud Foundation (VCF) is pass-through. DirectPath I/O hands the whole device to the VM. NVIDIA vGPU adds a manager on the ESX host that presents virtual GPUs, but it uses mediated pass-through only for setup and migration. After that, the VM’s work goes straight to the GPU. The hypervisor isn’t in the data path doing arithmetic. That’s why Kelly reported overhead usually under 2 percent and sometimes a fraction of a percent. In a few cases, the ESX scheduler even beat bare-metal Linux, just as it did with some CPU-based applications. I’d still want to see those numbers on my own workloads before quoting them to a CFO. The direction is hard to argue with, and matches how we resolved all those other “special” cases.
Performance overhead was always the weakest part of the bare-metal case. The real cost of bare metal is operational. A physical GPU server is tied to its OS install, driver versions, and maybe the one person who knows how it was built. Patching means downtime; moving a workload means rebuilding it. Virtualization brings the lifecycle tools we’ve used for VMs for twenty years, such as templates, snapshots, and vMotion. Those tools matter even more when GPUs make each host cost as much as a small house.
Choosing Between Pass-Through and vGPU
The two modes used to differ significantly. Pass-through now supports time slicing and Multi-Instance GPU (MIG), so the gap has narrowed to one main feature. You need vGPU and its NVIDIA AI Enterprise licensing for vMotion; pass-through prevents vMotion, which changes the cost math. Kelly’s guidance follows the availability model of the application. A researcher’s GPU workstation is a pet. If the host needs maintenance, you want to live migrate that VM without killing a week of work in a notebook session, so vGPU makes sense there. Model serving behind an AI gateway is different. The application gets its availability from several Kubernetes nodes, and losing one node to maintenance barely registers. Pass-through is fine, and you skip the extra licensing. This is the same decision we’ve made for traditional applications for years. When an application handles its own availability, you don’t pay for infrastructure to protect it.
Where Bare Metal Still Makes Sense
Bare metal still has a place, but it’s smaller than many people assume. Training a large model from scratch is the case Kelly identified. Most enterprises don’t do that. They run inference and fine-tune existing models, and both fit well on virtualized GPUs. For models too big for one GPU, eight-GPU systems with NVLink bridges and an NVSwitch fabric can hold models approaching 2.8 trillion parameters in a single host. VCF device groups let an administrator use those eight GPUs as one large pool, as pairs, or as individual GPUs, then change the arrangement later. Some of the pool can run fine-tuning when inference is quiet and return to inference at peak times. A bare-metal build tends to lock in one of those choices on the day the server is racked; all eight GPUs are attached to one operating system.
Not every model needs a GPU at all. Kelly suggested central processing units (CPUs) can still serve models under about 15 billion parameters. Many internal tools, like document summarizers and ticket classifiers, use models in that size range, which can lower token cost compared to using a GPU.
Sharing GPUs Between Workloads
GPU sharing with time slicing gives each workload full access to the GPU cores in turn, while memory is divided into fixed shares. An active workload gets more GPU time when its neighbors are idle. Users might get used to borrowed performance and complain when it disappears, something anyone who ran overcommitted vSphere clusters in 2010 will recognize. MIG divides the GPU into isolated slices with dedicated streaming multiprocessors, cache, and frame buffer memory. No noisy neighbors, and the isolation is strong enough for multi-tenant use. Slices come in different sizes, and different GPUs in the same host can run different modes. Newer Blackwell and RTX Pro 6000 cards can even time slice within a MIG slice.
All of this matters because of utilization. Kelly said most customers he talks to run their GPUs below 50 percent utilization. That’s expensive hardware sitting idle for half its working life and an opportunity for optimization. VCF dashboards show temperature, memory, and compute use by cluster, host, and workload, so you can find idle capacity before you buy more GPUs. Reclaiming that capacity is a lot cheaper than another purchase order.
There’s one gap to watch. AMD and Intel GPUs work in pass-through mode, but slicing and a vGPU equivalent for those vendors are still in development. If you’ve standardized on NVIDIA, it won’t affect you. If vendor choice matters to your organization, watch the roadmap.
The bare-metal argument for enterprise AI will fade the same way it did for databases. Performance was never the real issue; running expensive hardware well over its whole life is. A virtualization layer that costs a couple of percent of performance and gives you live migration and GPU sharing looks like a good trade for most enterprise AI.
You can watch Shawn Kelly’s session, along with the rest of the VMware by Broadcom presentations, on the Cloud Field Day 26 appearance page, and find more independent technical content at TechFieldDay.com.
