Configuring Redundant Corosync Links for Proxmox VE Cluster Resilience

Introduction

A Proxmox VE cluster lives and dies by corosync. It's the layer that keeps every node agreeing on who's up, who's down, and who owns quorum at any given moment. Most homelab clusters and more than a few production ones run corosync over a single network path, which works fine right up until that path has a bad day. A flaky switch port, a misconfigured VLAN, a NIC that drops into a degraded state — any of these can take corosync traffic down with it, and when corosync goes quiet, nodes start losing quorum and HA starts making decisions you didn't ask for.

This is where corosync link redundancy comes in. Proxmox VE's clustering stack, built on corosync 3 with the kronosnet (knet) transport, supports multiple independent network links between nodes. Lose one link and the cluster keeps talking over the survivor, no fencing, no surprise VM restarts, no 2 a.m. page. Setting this up correctly is one of those tasks that takes twenty minutes and saves you a very bad night later. Here's how to do it properly, what to check before you touch corosync.conf, and where people usually get it wrong.

Problem Overview

By default, when you build a Proxmox cluster with pvecm create or pvecm add, you give it one network address per node and corosync uses that single link for all its heartbeat and membership traffic. That's link0. It's enough to get a cluster running, and for a lot of test environments it's all anyone configures.

The trouble is that corosync is latency-sensitive and unforgiving of packet loss. It's not moving bulk data — it's exchanging small, frequent totem protocol messages to maintain cluster membership and quorum. If those packets stop arriving, corosync doesn't wait around patiently. After the token timeout expires (1000 ms by default, retried across a configurable number of retransmits), the node drops out of the membership list. If that node no longer sees a quorate cluster, and HA is enabled, watchdog-based fencing can kick in and reboot it. Meanwhile the surviving nodes may start relocating HA-managed VMs and containers, assuming the missing node is actually dead rather than just network-isolated.

A single corosync link means a single point of failure for the entire cluster's stability, independent of how redundant your storage or your VM workloads are. You can have a fully mirrored Ceph pool and dual power supplies on every node, and still take an outage because one switch reboot cut corosync traffic on link0.

Symptoms

The failure mode is usually loud and confusing if you haven't seen it before. Typical signs:

• Nodes flip between green and red in the Proxmox web GUI, sometimes for just a few seconds at a time.
• pvecm status shows a node count lower than expected, or reports the cluster as not quorate.
• HA-managed guests migrate or restart on other nodes even though the original node's storage and services were never actually down.
• One or more nodes reboot on their own — this is the watchdog fencer doing its job when a node can't confirm quorum.
• journalctl -u corosync fills with retransmit and "Token has not been received" messages right before a membership change.

What makes this particularly annoying to troubleshoot after the fact is that everything else on the node looks healthy. SSH still works, the storage is fine, the VMs that didn't get evacuated are running normally. It really does look like corosync just decided to panic for no reason, until you go check the switch logs and find a port flapped for eleven seconds.

Root Cause

Almost every case of unexplained corosync instability traces back to one of these:

The cluster network runs over the same physical NIC and switch as VM traffic, storage traffic, or backup traffic, and corosync gets starved out during a burst of unrelated load — a backup job saturating the link, a large Ceph rebalance, a broadcast storm from a misbehaving guest. Corosync packets are small, but if the switch queue is full, they get dropped just like anything else.

There's genuinely only one physical path between nodes for corosync, so any single failure — a bad cable, a switch reboot for firmware, a NIC driver hiccup — takes the whole cluster's control plane down with it.

Less commonly, someone did configure a second link, but routed both through the same switch, or worse, the same physical uplink further upstream. That's redundancy on paper that doesn't actually protect against the failure that matters.

MTU mismatches between nodes on the corosync network segment can also cause intermittent knet link instability that looks a lot like a flaky NIC, especially if one node has jumbo frames enabled on that interface and another doesn't.

Diagnosis

Before you touch the config, confirm what you're actually dealing with. Start with cluster status:

pvecm status

Look at the quorum line and the membership list. If a node is missing or the vote count is short, that tells you corosync currently disagrees with reality. Next, check link-level status with knet's own tooling:

corosync-cfgtool -s

On a healthy single-link node you'll see something like:

Local node ID 2, transport knet
LINK ID 0 udp
	addr	= 10.10.10.2
	status:
		nodeid 1:	connected
		nodeid 3:	connected

If a nodeid shows disconnected under that link, you've confirmed the network path to that specific peer is the problem, not corosync itself misbehaving. Cross-reference with live logs during the event:

journalctl -u corosync -f

Watch for lines like Retransmit List: followed by growing sequence numbers, or Token has not been received in ... ms. A handful of retransmits under heavy load is normal noise. A sustained climb followed by a membership change is your link actually failing.

Rule out the obvious next: run a sustained ping between the corosync interfaces on the affected nodes during normal operation and during whatever load pattern seems to trigger the issue (backup window, rebalance, etc.):

ping -M do -s 1472 10.10.10.3

The -M do -s 1472 combination disables fragmentation and sends a packet sized to fail on a standard 1500 MTU path if there's a jumbo frame mismatch somewhere in between. If that ping fails while a plain ping 10.10.10.3 succeeds, you've found your MTU problem.

Finally, check the switch side if you have access — interface error counters, CRC errors, and port flap history will usually confirm or rule out a physical layer issue in under a minute.

Step-by-Step Solution

Adding a second corosync link doesn't require rebuilding the cluster. You need a second NIC (or at minimum a second VLAN on a genuinely separate physical uplink — though a separate NIC and separate switch is what actually buys you protection) on every node, with connectivity between all nodes on that segment.

1. Plan your addressing. Pick a dedicated subnet for the second link that doesn't overlap with anything else in use, e.g. 10.10.20.0/24. Assign each node a static address on it and confirm they can all reach each other before editing corosync.conf.
2. Wire it for real redundancy. The second link needs to be on hardware that can actually fail independently of link0 — a different NIC, a different switch, ideally a different upstream path. Two links through the same switch protect you from a cable or NIC failure but not from the switch itself.
3. Copy the config before editing it. Never edit /etc/pve/corosync.conf in place. Work on a copy:

   cp /etc/pve/corosync.conf /etc/pve/corosync.conf.new

4. Add the new link. In corosync.conf.new, add a second interface block under totem with linknumber: 1, and add a ring1_addr entry to every node in the nodelist.
5. Bump the version. Increment config_version in the totem block. Corosync ignores a config file that doesn't have a higher version number than what's currently loaded — this is the single most common reason a corosync.conf edit silently does nothing.
6. Apply it atomically. Move the edited file into place rather than editing the live one directly:

   mv /etc/pve/corosync.conf.new /etc/pve/corosync.conf

   pmxcfs validates the file on write and, if it's malformed, rejects the move rather than corrupting the live config that every node is reading.
7. Distribute and reload. Because /etc/pve/ is the cluster filesystem, the change propagates to every node automatically within a few seconds. Trigger a config reload cluster-wide:

   corosync-cfgtool -R

8. Verify on every node. Run corosync-cfgtool -s on each member and confirm LINK ID 1 shows connected to every peer. Also check pvecm status to confirm quorum is unaffected by the change.

Do this one node's worth of verification at a time before you consider it done. I wouldn't trust a redundant link setup until I've actually pulled a cable on link0 in a maintenance window and watched the cluster stay quorate on link1 alone. Configuration that's never been tested against the failure it's meant to survive is a guess, not a solution.

Commands

The commands you'll lean on most for this work, gathered in one place:

A quick sanity check worth running right after any corosync.conf change: cat /etc/pve/.members gives you pmxcfs's own view of node status and IPs, useful for confirming the cluster filesystem itself agrees with what corosync reports.

Configuration Examples

Here's a minimal single-link config, the kind you get from a default pvecm create:

totem {
  version: 2
  cluster_name: pvecluster
  config_version: 4
  transport: knet
  interface {
    linknumber: 0
  }
}

nodelist {
  node {
    name: pve1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.10.10.1
  }
  node {
    name: pve2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.10.10.2
  }
  node {
    name: pve3
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 10.10.10.3
  }
}

quorum {
  provider: corosync_votequorum
}

logging {
  to_syslog: yes
}

And the same cluster after adding a second, physically independent link:

totem {
  version: 2
  cluster_name: pvecluster
  config_version: 5
  transport: knet
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
}

nodelist {
  node {
    name: pve1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.10.10.1
    ring1_addr: 10.10.20.1
  }
  node {
    name: pve2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.10.10.2
    ring1_addr: 10.10.20.2
  }
  node {
    name: pve3
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 10.10.10.3
    ring1_addr: 10.10.20.3
  }
}

quorum {
  provider: corosync_votequorum
}

logging {
  to_syslog: yes
}

Note that only config_version changed on the totem block, plus the new interface stanza and the per-node ring1_addr lines. Everything else stays untouched. Resist the urge to "clean up" other settings while you're in there — an unrelated typo in a config file that every node depends on for quorum is not the kind of mistake you want to discover during an actual outage.

Common Mistakes

Forgetting to bump config_version is the classic one. The file saves fine, pmxcfs replicates it to every node, and corosync just keeps running on the old configuration in memory because nothing told it a newer version exists. You'll only find out during the next real failure, when the redundancy you thought you had isn't there.

Routing both links through the same switch. This passes every sanity check — both links show connected, failover testing with a single cable pull works fine — but a switch reboot or firmware update takes the entire cluster down anyway, because you never actually built independent paths.

Editing /etc/corosync/corosync.conf directly on one node instead of /etc/pve/corosync.conf. The former is a local copy that pmxcfs overwrites from the cluster filesystem; changes made there vanish the next time /etc/pve/ syncs, and in the meantime that one node is running a divergent config from the rest of the cluster.

Skipping the failure test. A redundant link that's never actually been failed over in a controlled way is unverified. I've seen setups where the second link was cabled into the wrong VLAN and looked fine in corosync-cfgtool -s right up until it was needed.

Adding the second link but not updating firewall rules on nodes that run one, if applicable — corosync uses UDP, and a host-level firewall blocking the new subnet will make the link appear configured but never actually pass traffic.

Best Practices

Keep corosync traffic off anything shared with VM, storage, or backup data whenever you can. Even a fast switch can queue-drop small UDP packets during a burst, and corosync doesn't need much bandwidth — it needs low, consistent latency.

Use two links minimum on any cluster you actually care about, on genuinely separate hardware paths. For clusters spanning multiple racks or rooms, that means separate switches and ideally separate upstream paths, not just separate cables into the same switch.

Match MTU across every interface participating in corosync traffic, on every node and every switch port in between. Mismatches here produce intermittent, hard-to-diagnose instability rather than a clean failure.

Leave link_mode at its default (passive) unless you have a specific reason to change it — for corosync's traffic volume, active load-balancing across links adds complexity without meaningful benefit, and passive failover is simpler to reason about during an incident.

Don't touch the default token timeout unless you understand exactly why you're changing it. Extending it to paper over an unstable link hides the underlying problem and slows down legitimate failure detection. If you're running a stretched cluster with real inter-site latency, that's a different conversation and worth its own careful tuning, tested under load before you trust it.

Test failover on a schedule, not just once at setup. Networks change — someone reconfigures a switch, adds a VLAN, replaces a patch panel. A quarterly "pull the primary corosync cable and watch what happens" drill catches configuration drift before it catches you.

FAQ

Do I need identical NICs on every node for a second corosync link?
No, corosync doesn't care about NIC model or speed. It just needs a working IP path between every node on that link's subnet.

How many corosync links can I configure?
Kronosnet supports up to eight links per cluster. Most environments get everything they need from two.

Will adding a second link cause downtime?
No. The config change propagates through pmxcfs and reloads live via corosync-cfgtool -R without interrupting quorum, as long as the config is valid and the version number is correctly incremented.

Can I use the same subnet for both links?
Technically corosync will let you, but it defeats the purpose. Use separate subnets on separate physical infrastructure so a single failure domain can't take out both links at once.

What happens if both corosync links fail at the same time?
The cluster behaves exactly as it would with a single link down — nodes lose quorum, and if HA and watchdog fencing are enabled, isolated nodes may reboot. Redundant links reduce the odds of this, they don't eliminate the possibility.

Conclusion

Corosync redundancy is one of the cheapest insurance policies you can buy for a Proxmox cluster. It's a second NIC, a bit of cabling, and a few minutes editing corosync.conf against maybe an hour of total downtime and a stack of confused HA migration logs the next time a switch has a bad afternoon. If your cluster is still running on a single corosync link, that's worth fixing before you need it rather than after. Pull a cable in a maintenance window, confirm the cluster survives it, and move on to the next thing on your list knowing this one won't wake you up.