Configuring SR-IOV Networking for Proxmox VE 8.x Virtual Machines

Introduction

Most Proxmox VE clusters run fine on the default virtio-net path. The bridge does its job, the vhost thread keeps up, and nobody thinks twice about it. Then someone puts a market-data feed, a storage-heavy Ceph client, or a telco-style packet processing VM on the same host, and suddenly that comfortable virtio path is the bottleneck. This is where SR-IOV earns its keep. It lets a single physical NIC present itself as dozens of lightweight virtual devices, each one assigned directly to a guest, bypassing the host's network stack entirely.

I've set this up on Intel X710 and X550 cards more times than I can count, and on Mellanox ConnectX-4/5 gear for a couple of Ceph-adjacent builds. It's not hard once you understand the moving parts, but the failure modes are quiet — a VM boots fine, the interface shows up, and then nothing passes traffic because a spoof-check setting or a missing IOMMU flag is silently dropping every frame. This guide walks through the whole thing on PVE 8.x: diagnosing whether you actually need SR-IOV, enabling it, wiring it into a VM, and the mistakes that cost me an afternoon the first time around.

Problem Overview

Standard Proxmox networking runs every VM's traffic through a Linux bridge (or Open vSwitch) on the host, backed by a vhost-net kernel thread per queue. That thread does real work: it copies packets between the guest's virtio ring and the host's network stack, applies bridge forwarding rules, and hands off to the physical NIC driver. For most workloads this overhead is invisible. For workloads pushing tens of gigabits per second, or anything sensitive to jitter — VoIP media relays, financial tick data, NFV appliances — that extra hop through the host kernel adds latency and burns CPU cycles you'd rather spend on the guest itself.

SR-IOV (Single Root I/O Virtualization) solves this by letting the NIC itself do the multiplexing in hardware. The physical function (PF) spawns a configurable number of virtual functions (VFs), each with its own PCIe configuration space, its own MAC address, and its own set of TX/RX queues. You pass a VF straight into the guest with the same hostpci mechanism used for GPU passthrough, and the guest talks to the NIC directly. No bridge, no vhost thread, no host kernel in the data path.

The tradeoff is real: you lose live migration for VMs holding a VF (barring some newer failover-bond tricks that are more trouble than they're worth on a small cluster), you lose the flexibility of software-defined bridging, and you're now managing hardware-level VF configuration instead of a clean software abstraction. It's the right tool for a specific problem, not a default.

Symptoms

You don't need SR-IOV just because a VM feels slow on the network. Before reaching for it, confirm you're actually looking at host-side network overhead rather than something else entirely. The pattern I look for:

• High si (softirq) time in top or mpstat -P ALL 1 on the Proxmox host, concentrated on the CPU cores handling vhost-net threads, even when the guest itself isn't CPU-bound.
• Throughput inside the VM plateaus well below what ethtool reports as the link speed, and it doesn't improve after tuning virtio queue counts or multiqueue settings.
• Latency-sensitive traffic (RTP streams, market data, cluster heartbeats) shows inconsistent jitter under load, even though the physical link and switch are clean.
• Running iperf3 from the VM to a host on the same L2 segment tops out noticeably lower than the same test run from the Proxmox host itself to that target.

If you're seeing these on a 1GbE deployment, SR-IOV is probably overkill — check multiqueue and CPU pinning first. Where I've actually needed it is 10GbE and above, particularly on hosts running six or more VMs with sustained east-west traffic.

Root Cause

The underlying cause is architectural, not a misconfiguration. Every packet that crosses a virtio-net interface takes this path: guest driver writes to the virtio ring, a vhost-net kernel thread on the host wakes up, copies the packet descriptor, the bridge code does a forwarding lookup, and the physical NIC driver finally puts the frame on the wire. Each of those steps costs CPU cycles and adds latency, and under sustained load the vhost thread for a busy VM can pin an entire host core just moving packets.

Multiqueue virtio-net and adjusting queues= on the network device helps distribute that load across more vhost threads, and it's worth trying first. But it doesn't remove the fundamental copy-and-forward overhead — it just parallelizes it. SR-IOV removes the host from the data path altogether, which is the only way to get genuinely close to bare-metal throughput and latency inside a guest.

There's a secondary cause worth naming: on hosts where IOMMU isn't enabled at all, people sometimes try to reach for PCI passthrough of the whole NIC as a workaround, which then takes the physical port away from every other VM on the host. SR-IOV is the fix for that specific mistake — it gives you passthrough-grade performance while still letting a dozen VMs share one physical port.

Diagnosis

Start by confirming your hardware and BIOS actually support this before touching any Proxmox config. On the host:

lscpu | grep -i virtualization
cat /proc/cpuinfo | grep -Em1 'vmx|svm'

vmx means Intel VT-x, svm means AMD-V — either confirms hardware virtualization is present, but you separately need VT-d (Intel) or AMD-Vi enabled in the BIOS/UEFI setup for IOMMU. There's no software check that substitutes for actually going into the BIOS and turning it on; I've lost time assuming a vendor shipped it enabled by default when it wasn't.

Check whether the kernel sees IOMMU groups after a reboot with the flag set (more on the flag itself below):

dmesg | grep -e DMAR -e IOMMU

On a working Intel host you'll see something like:

[    0.913841] DMAR: IOMMU enabled
[    1.042199] DMAR: Intel(R) Virtualization Technology for Directed I/O

If that's empty, the kernel command line flag either isn't set or didn't take — check /proc/cmdline against what you configured. Next, identify the NIC and confirm it supports SR-IOV:

lspci -nn | grep -i ethernet

01:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572]

Check the driver and confirm SR-IOV capability via ethtool or lspci -vvv (look for the SR-IOV capability block):

ethtool -i enp1s0f0

driver: i40e
version: 5.15.0
firmware-version: 8.50 0x8000b6c1

And check how many VFs the card supports before you configure anything:

cat /sys/class/net/enp1s0f0/device/sriov_totalvfs

On the X710 this typically returns 64, though how many you actually want to create depends on your VM count and how many queues each VF needs — creating the maximum rarely makes sense.

Step-by-Step Solution

1. Enable IOMMU on the host

Edit the kernel command line. On Proxmox 8.x with a UEFI/systemd-boot setup (common on ZFS root installs), that's /etc/kernel/cmdline:

root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt

For AMD hosts, use amd_iommu=on instead. Apply it and reboot:

proxmox-boot-tool refresh
reboot

If your host uses GRUB instead (legacy BIOS boot, or non-ZFS install), edit /etc/default/grub and set GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt", then run update-grub and reboot.

2. Load the required modules

Make sure vfio, vfio_iommu_type1, vfio_pci, and vfio_virqfd are loaded at boot by adding them to /etc/modules, then regenerate initramfs:

update-initramfs -u -k all

3. Create the virtual functions

With IOMMU confirmed active, create VFs on the physical interface. This is a live sysfs write — it doesn't survive a reboot on its own, which trips people up constantly:

echo 4 > /sys/class/net/enp1s0f0/device/sriov_numvfs

Verify they appeared:

lspci -nn | grep -i "Virtual Function"

01:02.0 Ethernet controller [0200]: Intel Corporation X710/X557-AT 10GBASE-T Virtual Function [8086:1889]
01:02.1 Ethernet controller [0200]: Intel Corporation X710/X557-AT 10GBASE-T Virtual Function [8086:1889]

4. Make VF creation persistent

Add a pre-up stanza to /etc/network/interfaces so the VFs get recreated every boot before the parent interface comes up. See the configuration example below for the exact block.

5. Assign a VF to a VM

Find the PCI address of the VF you want to hand out, then attach it as a hostpci device:

qm set 100 -hostpci0 01:02.0,pcie=1

Set pcie=1 so it attaches to a PCIe slot rather than a legacy PCI one — matters for guests expecting PCIe topology, and it's the default I use even when it technically wouldn't matter for a given guest OS.

6. Boot the VM and confirm the interface

Inside the guest, install the appropriate VF driver if it isn't already present (Linux distros generally ship iavf or i40evf in-kernel; Windows needs the Intel PROSet drivers installed manually), then confirm link:

ip link show
ethtool eth0

Commands

Configuration Examples

Persistent VF creation via /etc/network/interfaces:

auto enp1s0f0
iface enp1s0f0 inet manual
    pre-up echo 4 > /sys/class/net/enp1s0f0/device/sriov_numvfs
    mtu 9000

Note the jumbo frame setting — if your fabric supports it, set it on the PF before VFs come up; VFs inherit the PF's MTU ceiling and you can't raise it from the guest side afterward.

/etc/kernel/cmdline for an Intel host on ZFS root:

root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt

VM config snippet (/etc/pve/qemu-server/100.conf) showing a VF alongside a normal virtio management NIC — I generally keep one virtio interface for management/monitoring traffic even on SR-IOV VMs, so you're not locked out if the VF misbehaves:

net0: virtio=BC:24:11:2A:44:01,bridge=vmbr0
hostpci0: 01:02.0,pcie=1
cpu: host
numa: 1

Setting cpu: host matters here — some guest VF drivers check CPU features during initialization and behave inconsistently under the default kvm64 model. I always set it for SR-IOV workloads regardless of migration concerns, since you've already given up migration by attaching a VF.

Mellanox cards follow a slightly different path worth calling out separately, since a fair number of Ceph-adjacent builds end up on ConnectX-4 or ConnectX-5 hardware rather than Intel. VF creation still goes through sriov_numvfs, but Mellanox additionally requires the firmware itself to have SR-IOV enabled before Linux will let you create anything:

mlxconfig -d /dev/mst/mt4119_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=8
mlxfwreset -d /dev/mst/mt4119_pciconf0 reset

That firmware-level toggle trips people up who are used to the Intel workflow, where everything happens in sysfs and the driver. Skip it on a Mellanox card and sriov_numvfs will accept the write but silently cap out at zero usable VFs, which looks a lot like a driver bug until you check mlxconfig -d /dev/mst/mt4119_pciconf0 query and notice SRIOV_EN is still off. The mst tooling comes from Mellanox's mstflint/MFT package, not the Proxmox base install, so you'll need to add it separately before any of this works.

A udev rule to pin persistent VF MAC addresses, useful when your guest OS or automation expects a stable MAC across reboots (/etc/udev/rules.d/70-persistent-vf.rules):

ACTION=="add", SUBSYSTEM=="net", DRIVERS=="i40e", RUN+="/sbin/ip link set enp1s0f0 vf 0 mac 02:00:00:aa:bb:01"

Common Mistakes

The one that costs the most time: forgetting that sriov_numvfs is a runtime sysfs value, not a saved configuration. You test it interactively, it works, you reboot for an unrelated reason a week later, and the VFs are gone because nothing recreated them. Always wire VF creation into /etc/network/interfaces or a systemd unit before you consider the setup done.

Another common one is enabling IOMMU but leaving it in the default passthrough mode without iommu=pt. Without that flag, non-passthrough devices on the host also route through the IOMMU translation layer, which adds unnecessary overhead to devices that were never meant to be passed through. It's a one-word difference in the kernel command line with a real performance cost if you skip it.

People also frequently leave spoof checking enabled on VFs used for anything other than a single static MAC — this silently drops traffic if the guest does its own MAC manipulation (common in NFV or bonded setups inside the VM), and the symptom looks exactly like a broken link rather than a filtering rule.

Last one: assigning a VF to a VM and then trying to migrate it like a normal VM. It'll fail, and the error isn't always obvious about why. hostpci devices tie a VM to the physical host holding that hardware — plan your placement accordingly, or don't use SR-IOV for VMs that need to float across the cluster.

Best Practices

Don't create more VFs than you'll actually use. Each VF consumes a PCIe function and a slice of the NIC's queue resources — creating 64 VFs on a card you're only going to assign to 6 VMs wastes queue depth that those 6 VFs could otherwise use for better multiqueue performance.

Keep a virtio-backed management interface on any VM that also holds a VF. If the VF's driver crashes inside the guest or the physical link flaps, you still have a way in without console access.

Document the PCI address to VF-to-VM mapping somewhere outside of Proxmox's own config — a spreadsheet, a README in your infra repo, whatever you actually maintain. qm config shows you the mapping for one VM at a time, but I've walked into environments where nobody could say at a glance which physical NIC six different VFs across three VMs traced back to.

Test VF failover behavior before you commit to it in production, if your NIC and driver support the bonding/failover model. It behaves differently across vendors, and I wouldn't take a whitepaper's word for it — I've seen Mellanox ConnectX-5 failover behave cleanly in a lab that Intel X710 didn't quite match under the same test.

FAQ

Does SR-IOV work with Open vSwitch on Proxmox?

Not in the way you might expect. A VF bypasses OVS entirely — that's the point of SR-IOV. You can't apply OVS flow rules or bridge logic to VF traffic since it never touches the host's software switch.

Can I live-migrate a VM with a hostpci VF attached?

No, not with plain hostpci passthrough. The VF is tied to the physical host. Some setups work around this with an in-guest bonding driver that fails over to a virtio interface during migration, but that adds real complexity and isn't something I'd set up without a specific need for it.

How many VFs should I create per physical port?

Match it to your actual VM count plus some headroom, not the card's maximum. Four to eight VFs per port is a reasonable starting point for most clusters; go higher only if you've confirmed the queue-per-VF tradeoff still gives you the throughput you need.

Do I need IOMMU for SR-IOV even without GPU passthrough?

Yes. SR-IOV and PCI/GPU passthrough both depend on the same IOMMU infrastructure to isolate the device (or VF) into its own protection domain before handing it to a guest.

Why does my VF show no link inside the guest even though the host sees it as up?

Check spoof checking and VF trust settings first with ip link show enp1s0f0 from the host — a mismatched or unset MAC combined with spoof checking on will drop everything silently. Also confirm the guest actually loaded a VF driver rather than falling back to a generic one.

Conclusion

SR-IOV isn't something to reach for by default — it trades away migration flexibility and adds a layer of hardware-specific configuration that a plain virtio bridge doesn't have. But for the workloads that actually need it, the difference is not subtle. I've taken guests from saturating a host core on vhost-net threads to barely registering on the host's CPU graphs at all, just by moving them onto a VF. Get the IOMMU flag right, make VF creation persistent, and keep a fallback management interface on anything you pass hardware into, and this becomes a reliable part of the toolkit rather than a one-off hack you're afraid to touch again.