Introduction
If you run more than a handful of nodes in a Proxmox VE cluster, you've probably hit the same wall I have: VLANs stop scaling once you need isolated Layer 2 segments that span racks, or worse, span sites connected over a routed WAN link. Trunking every VLAN ID to every node works fine until your switch admin tells you the trunk is full, or until you're building a lab that has no access to switch config at all. That's the gap VXLAN zones in Proxmox VE's Software-Defined Networking stack are built to close.
This piece walks through setting up a working multi-node VXLAN overlay on Proxmox VE 9.2, the failure modes you'll actually hit doing it, and how to read the diagnostic output when a VNet looks configured but traffic silently disappears. I've deployed this in a three-node home lab connected over a flat /24 and in a cluster split across two colocation racks with only a routed underlay between them — the failure patterns were nearly identical in both cases.
Problem Overview
Standard Proxmox networking ties every VM and container to a Linux bridge on the local node, with VLAN tags handled either by the bridge itself (vlan-aware bridge) or by a dedicated bridge per VLAN. That model is simple and it's exactly what most single-site clusters should use. It falls apart in two specific scenarios: you're out of VLAN IDs on the physical switch fabric, or your nodes don't share a common Layer 2 segment at all — think a stretched cluster across two buildings routed over Layer 3.
VXLAN solves this by encapsulating Ethernet frames inside UDP datagrams (destination port 4789) and tunneling them between nodes over whatever Layer 3 path already exists. The VNet you create on top of a VXLAN zone behaves like a private switch that happens to span your entire cluster, with no dependency on the physical switch understanding anything about it. Proxmox VE's SDN stack manages the zone, VNet, and subnet objects for you and pushes the resulting FRR/bridge configuration to every node when you hit Apply.
The catch is that VXLAN is an overlay on top of an underlay you still have to get right. Get the peer list wrong, forget about MTU, or leave a firewall rule blocking UDP 4789 and you'll have a VNet that shows up green in the GUI while VMs on it can't ping each other.
Symptoms
The failure mode is almost never "the zone won't apply." It's usually one of these, discovered well after the fact:
- Two VMs attached to the same VNet, on different nodes, can't ping each other, but VMs on the same node and same VNet work fine.
- ARP requests go out but never get a reply —
arp -non the guest shows an incomplete entry that never resolves. - Everything works for small pings (56 bytes) but breaks the moment you push real traffic — SSH sessions hang mid-handshake, or a file transfer stalls after the first few packets.
- SDN status in the GUI shows the zone and VNet as applied with no errors, which makes it look like a non-issue, but the underlying
vnetXinterface has zero peers configured. - Migrating a VM between two nodes causes it to lose connectivity on the VXLAN VNet, while its other NICs on regular bridges keep working.
That third one — small pings working, everything else breaking — is the tell almost every time. It means the tunnel is up but something downstream is fragmenting or dropping oversized frames.
Root Cause
Three root causes account for nearly everything I've seen go wrong with Proxmox VXLAN zones in production and in the lab.
MTU mismatch. VXLAN adds 50 bytes of encapsulation overhead per frame. If your underlay physical NIC is at the standard 1500 MTU and you don't drop the VNet's effective MTU below that, every frame close to 1500 bytes gets dropped somewhere between the guest and the far node — not by Proxmox, but by whatever router or switch in the path enforces the physical MTU and can't fragment a UDP-encapsulated frame that's already at the wire limit. This is exactly why small ICMP pings succeed and everything else fails: an 84-byte ping fits inside the overhead budget, a 1500-byte TCP segment doesn't.
Incomplete or stale peer list. A VXLAN zone (as opposed to an EVPN zone with a controller) works on statically configured unicast peers — every node needs the IP address of every other node participating in that zone. Add a fourth node to the cluster and forget to add it to the zone's peer list, and VMs on that node are simply unreachable from the VXLAN VNet; there's no dynamic discovery to save you. I've also seen this bite people after a node's management IP changed — the peer list still references the old address.
Firewall dropping UDP 4789 on the underlay. If you've got the Proxmox firewall enabled at the datacenter or host level, or there's a network firewall between racks, UDP port 4789 has to be open between every pair of peer IPs. This one is sneaky because the Proxmox firewall GUI doesn't call out VXLAN traffic by name — it's just UDP traffic on the physical NIC that silently gets dropped by a default-deny host firewall rule.
Diagnosis
Start at the interface level on the node hosting one of the VMs that can't talk. Confirm the VNet interface actually exists and check its MTU:
ip -d link show vnet1You're looking for a line describing the vxlan parameters — something like vxlan id 100 srcport 0 0 dstport 4789 ... along with the interface's MTU. If the MTU reads 1500 and your physical uplink is also 1500, you've already found the problem before running another command.
Next, check whether the tunnel actually has peers configured at the kernel level, not just in the SDN config:
bridge fdb show dev vnet1 | grep -v permanentAn empty or suspiciously short list here, especially missing an entry for the node you're trying to reach, points straight back to the peer list in the zone config. Compare that output against what's actually in /etc/pve/sdn/zones.cfg for that zone.
To confirm whether UDP 4789 is actually leaving the box and getting a response, run a capture on the physical uplink while you ping from the guest:
tcpdump -i vmbr0 -n udp port 4789If you see outbound packets from your node's IP but nothing coming back from the peer, the problem is on the far end — either its firewall or its peer list doesn't include you. If you see nothing leaving at all, the VXLAN interface on the local node isn't actually forwarding, which usually means it isn't in the bridge the VM's NIC is attached to.
For the MTU-specific symptom, this is the fastest confirmation you'll get:
ping -M do -s 1422 <peer-vm-ip>1422 bytes plus the 28-byte ICMP/IP header lands you right at a 1450 MTU. If that fails with "Frag needed" while a plain ping succeeds, you've confirmed the MTU theory without needing to touch a packet capture at all.
Last, check host-level firewall logs if the Proxmox firewall is enabled:
journalctl -u pve-firewall -fand watch for drops while you generate traffic. A DROP entry referencing port 4789 between two node IPs is about as unambiguous as diagnosis gets.
Step-by-Step Solution
Here's the sequence I use to build a VXLAN overlay from scratch across a cluster, in an order that avoids most of the pitfalls above.
- Confirm every node's management or dedicated underlay interface can reach every other node's on the same interface, and note the physical MTU with
ip link show vmbr0. - Decide on a VXLAN MTU that's at least 50 bytes below the smallest physical MTU in the path. For a standard 1500 MTU underlay, use 1450. If you're running jumbo frames end to end at 9000, you can go up to 8950.
- In the GUI, go to Datacenter → SDN → Zones and add a new VXLAN zone. Set the peer address list to every node's underlay IP, including the node you're on.
- Set the MTU field on the zone to your calculated value from step 2. Don't leave this at the interface default.
- Go to Datacenter → SDN → VNets and create a VNet attached to the zone, with a VXLAN tag (VNI) that's unique within the zone — anything from 1 to 16777215.
- Optionally add a Subnet under the VNet if you want Proxmox to manage a gateway and IPAM-assigned addresses for guests on it.
- Click Apply at the top of the SDN panel. This is the step people forget — creating the zone and VNet only stages the config, it isn't live until you apply.
- On each node, verify the VNet bridge came up:
ip -d link show vnet1, and confirm the MTU matches what you set. - Open UDP 4789 between the peer IPs if a host or network firewall sits in the path. If you're using the Proxmox firewall, add the rule at the datacenter level so it applies cluster-wide rather than per node.
- Attach a test VM's NIC to the new VNet on two different nodes and confirm connectivity with both a normal ping and the fragmentation-testing ping from the diagnosis section.
If you're adding a node to an existing zone later, don't forget step 3 has to be redone — update the peer list on the zone and re-apply. That step alone accounts for most of the "worked yesterday, broken today" tickets I've dealt with after someone scaled a cluster.
Commands
These are the commands you'll use most often building and troubleshooting VXLAN zones, whether from the CLI directly or scripting the setup with pvesh.
# Create a VXLAN zone via the API
pvesh create /cluster/sdn/zones --zone labzone --type vxlan --peers 10.10.0.11,10.10.0.12,10.10.0.13 --mtu 1450
# Create a VNet on that zone
pvesh create /cluster/sdn/vnets --vnet vnet100 --zone labzone --tag 100
# Add a subnet with a gateway and SNAT for outbound access
pvesh create /cluster/sdn/subnets --subnet 10.100.0.0/24 --vnet vnet100 --gateway 10.100.0.1 --snat 1
# Apply the pending SDN configuration cluster-wide
pvesh set /cluster/sdn
# Inspect the resulting interface on a node
ip -d link show vnet100
# Check the forwarding database for known peers/MACs
bridge fdb show dev vnet100
# Confirm UDP encapsulation traffic on the physical uplink
tcpdump -i vmbr0 -n udp port 4789
# Test for MTU-related fragmentation issues
ping -M do -s 1422 10.100.0.20
# Reload network config after manual interface edits
ifreload -a
# Watch firewall drops live while testing
journalctl -u pve-firewall -fConfiguration Examples
The SDN stack stores its configuration as plain text under /etc/pve/sdn, replicated cluster-wide through pmxcfs. You rarely need to hand-edit these, but knowing the format makes the GUI output a lot less mysterious when you're debugging.
A VXLAN zone in /etc/pve/sdn/zones.cfg looks like this:
vxlan: labzone
peers 10.10.0.11,10.10.0.12,10.10.0.13
mtu 1450The corresponding VNet in /etc/pve/sdn/vnets.cfg:
vnet: vnet100
zone labzone
tag 100And a subnet definition in /etc/pve/sdn/subnets.cfg, if you want Proxmox handling IPAM and a gateway for this VNet:
subnet: 10.100.0.0-24
vnet vnet100
gateway 10.100.0.1
snat 1If you'd rather manage the underlying interfaces yourself outside SDN — some environments have change-control processes that don't play well with the pmxcfs-managed config — a manual VXLAN interface in /etc/network/interfaces looks like this:
auto vxlan100
iface vxlan100 inet manual
vxlan-id 100
vxlan-local-tunnelip 10.10.0.11
vxlan_remoteip 10.10.0.12
vxlan_remoteip 10.10.0.13
mtu 1450
auto vmbr100
iface vmbr100 inet manual
bridge-ports vxlan100
bridge-stp off
bridge-fd 0
mtu 1450I'd only reach for the manual approach if you have a specific reason SDN's zone abstraction doesn't fit — for almost everyone, letting Proxmox manage the zone/vnet/subnet objects is less error-prone and survives node reboots and cluster changes without you having to remember to update three separate config stanzas.
Common Mistakes
The mistake I see most is leaving the zone MTU at whatever the GUI defaults to instead of calculating it from the actual physical uplink. It's an easy field to skip past, and the consequences don't show up until someone runs a workload that pushes full-size packets.
A close second is forgetting to re-apply after editing the peer list. The GUI change gets saved to the config file immediately, but it stays pending — shown with a small warning icon on the SDN overview — until you explicitly hit Apply. I've watched someone spend an hour debugging a "broken" VXLAN zone that just needed the Apply button clicked.
People also frequently confuse VXLAN zones with EVPN zones. A VXLAN zone uses static unicast peers you maintain by hand; an EVPN zone uses a routing controller (BGP) to discover peers and handle inter-subnet routing dynamically. If you're building something that'll grow past four or five nodes, or that needs routing between multiple VNets, EVPN is worth the extra setup complexity. Sticking with plain VXLAN and manually updating peer lists on a 20-node cluster gets old fast.
Another one: assuming VXLAN gives you encryption because it's "tunneled." It doesn't — it's cleartext UDP encapsulation. If your overlay crosses an untrusted network between sites, you need a site-to-site VPN or WireGuard underlay carrying the VXLAN traffic, not the VXLAN encapsulation itself.
Finally, don't skip testing with the Proxmox firewall enabled if that's how you'll run in production. Testing with the firewall off, then flipping it on for the production rollout, is how UDP 4789 firewall drops sneak past validation entirely.
Best Practices
Keep VXLAN VNIs documented somewhere outside Proxmox — a spreadsheet, a wiki page, whatever you'll actually maintain. Once you have a handful of zones and VNets across a cluster, remembering which VNI maps to which environment by memory stops working.
Use a dedicated NIC or VLAN for the VXLAN underlay traffic where the hardware allows it, separate from management traffic and separate from storage replication. VXLAN encapsulation overhead plus a busy migration network is a good way to introduce intermittent packet loss that's miserable to diagnose later.
Set the zone MTU deliberately and write down the physical MTU it was calculated from. When someone eventually changes a switch config or adds jumbo frames on the underlay, that MTU needs updating too, and it's not obvious from the GUI that the two are linked.
For anything beyond a lab, prefer EVPN zones over plain VXLAN once your peer count grows. The static peer list on a VXLAN zone is fine for three or four nodes; past that, the maintenance burden of updating it every time the cluster changes outweighs the simplicity of not running BGP.
Test the fragmentation-boundary ping (ping -M do -s <mtu-28>) as part of your rollout checklist, not just when something breaks. It takes ten seconds and catches the single most common misconfiguration before a workload ever gets deployed on the VNet.
If you're stretching a VXLAN zone across sites over the public internet or an untrusted WAN, put a WireGuard tunnel underneath it and encapsulate VXLAN inside that. Proxmox VE 9.2's SDN fabric options can help set up WireGuard as an underlay fabric directly, which saves you from hand-rolling the tunnel interfaces yourself.
FAQ
Does VXLAN require multicast on the physical switches?
No. Proxmox VE's VXLAN zones use statically configured unicast peers, not multicast group discovery, so there's nothing special required from the switch fabric beyond routing UDP traffic between the peer IPs.
Can I mix VXLAN VNets and regular VLAN-aware bridges on the same node?
Yes, they coexist without conflict. A VM can even have separate NICs on each type if it needs to reach both a local VLAN segment and a cluster-wide overlay.
What happens to VXLAN connectivity during a live migration?
Nothing special — the VNet is present on every node in the zone's peer list, so the VM's network config follows it during migration exactly like a local bridge would, as long as the destination node is also in the peer list.
Is there a limit to how many VXLAN zones I can run on one cluster?
No hard limit enforced by Proxmox, though each VNI needs to be unique within its zone and you'll want to keep the total number of zones manageable for your own sanity when auditing peer lists.
Do I need a Subnet object if I'm managing IPs on guests manually?
No. Subnets are optional and mainly useful if you want Proxmox's IPAM to hand out addresses or you need SNAT for outbound guest traffic. A VNet works fine as a pure Layer 2 segment without one.
Why does the GUI show my zone as applied even though it's broken?
"Applied" just means the configuration was pushed to the nodes successfully — it doesn't validate reachability between peers or confirm the underlay firewall allows the traffic through. That validation is on you.
Conclusion
VXLAN zones turn a routed, multi-site, or VLAN-exhausted underlay into something that looks and behaves like a flat switch to your guests, and once it's working correctly it's genuinely low-maintenance. The three failure points — MTU, peer lists, and firewall rules on port 4789 — cover almost every problem you'll run into, and all three are quick to check with the commands above once you know to look for them. Build the habit of validating MTU and peer reachability as part of your rollout instead of after a user reports a hung SSH session, and this becomes one of the more reliable pieces of a Proxmox SDN deployment rather than a recurring source of tickets.