Troubleshooting Corosync Cluster Communication Failures in Proxmox VE

Introduction

Every Proxmox VE cluster depends on one quiet, unglamorous service to keep its nodes talking to each other: corosync. It carries the totem protocol traffic that establishes quorum, synchronizes the pmxcfs cluster filesystem, and tells High Availability manager whether a node is alive or gone. When corosync is healthy, administrators rarely think about it. When it starts dropping packets, missing tokens, or losing quorum intermittently, the effects ripple through the entire cluster: nodes flicker between green and red in the GUI, HA-managed virtual machines get fenced and rebooted without warning, and cluster-wide configuration changes stop propagating.

This article walks through a complete, practical process for diagnosing and resolving corosync communication failures in Proxmox VE clusters running current releases (Proxmox VE 7.x and 8.x, using Corosync 3 with the Kronosnet transport). It is written for system administrators, DevOps engineers, and infrastructure teams who manage production or lab clusters and need a repeatable methodology rather than guesswork. We will cover the underlying architecture, the symptoms that point to corosync trouble, root causes seen most often in the field, diagnostic commands, a step-by-step remediation path, and the configuration patterns that prevent recurrence.

Problem Overview

Corosync is the cluster communication layer underneath every Proxmox VE cluster. It implements the totem single-ring (or multi-link, with Kronosnet) protocol to provide reliable, ordered messaging and membership tracking between nodes. On top of corosync sits pmxcfs, the cluster filesystem mounted at /etc/pve, which relies on corosync messages to replicate configuration files such as /etc/pve/qemu-server/*.conf and /etc/pve/corosync.conf itself across every node in real time.

When corosync communication degrades, nodes cannot reliably exchange totem tokens within the configured timeout window. This causes the cluster to lose quorum, split into partitions, or repeatedly re-form membership. Because Proxmox VE's High Availability stack (implemented through pve-ha-crm and pve-ha-lrm) treats loss of quorum as a signal that a node may be unreachable, it triggers watchdog-based self-fencing on affected nodes to prevent split-brain scenarios. In an HA-enabled cluster, this means a flaky corosync link does not just produce cosmetic GUI warnings — it can reboot hosts that are otherwise running fine, interrupting every VM and container on them.

Corosync trouble is disproportionately common in three environments: clusters where corosync shares a network path with VM or storage traffic, clusters spanning multiple sites or unreliable WAN links, and clusters where the underlying switching hardware introduces latency spikes or intermittent packet loss. Understanding which category your environment falls into is the first step toward a durable fix.

Symptoms

Corosync communication problems present in fairly recognizable ways. Common symptoms include:

Nodes flickering between an online (green) and unknown/offline (red or grey) state in the Proxmox VE web GUI, even though the nodes are pingable and running.
pvecm status reporting Quorate: No intermittently, or showing a lower node/vote count than expected.
HA-managed guests being migrated, stopped, or restarted unexpectedly, with logs showing a watchdog reset on the node that briefly lost quorum.
Delays or failures when saving configuration changes through the GUI or API, because writes to /etc/pve require an active quorum.
Repeated corosync[...]: [TOTEM] Retransmit List: or [TOTEM] A processor failed, forming new configuration entries in journalctl -u corosync.
Cluster join or node addition operations timing out or failing with quorum-related errors.
systemctl status pve-cluster showing pmxcfs in a degraded state, unable to write cluster files.
Sporadic SSH or console access issues to /etc/pve files, since the filesystem is FUSE-based and depends on quorum to remain writable.

These symptoms can appear continuously under sustained network problems, or only under load — for example, during backup windows, storage replication, or large file transfers that saturate the same physical link corosync depends on.

Root Cause

In the overwhelming majority of field cases, corosync instability traces back to one or more of the following root causes:

Shared network path with bulk traffic. Corosync is extremely latency-sensitive; its default totem token timeout expects a response within a small number of milliseconds. If the same NIC, bridge, or bond also carries VM traffic, backup traffic, or storage replication, momentary congestion delays corosync packets enough to trigger token loss.
No redundant link configured. A cluster running corosync over a single physical path has no fallback when that path degrades, even briefly (a flapping switch port, a firmware update on a NIC, a cable reseat).
MTU mismatches. If jumbo frames are enabled on some switch ports or bonded interfaces but not consistently end to end, corosync packets can be fragmented or dropped silently.
Firewall interference. When the Proxmox VE firewall is enabled on the corosync-carrying interface, missing rules for the corosync UDP ports block or delay traffic intermittently, especially after firewall rule reloads.
Excessive network latency or jitter. Corosync was designed for low-latency LAN segments. Stretching a cluster across sites connected by a WAN link, VPN tunnel, or oversubscribed uplink routinely produces latency spikes above the totem token timeout.
Time drift between nodes. While corosync itself does not require synchronized clocks to function, significant clock drift can confuse related services and logs, complicating diagnosis and occasionally interacting with certificate validation used by pmxcfs.
Undersized or oversubscribed virtual NICs when corosync runs inside nested or virtualized lab environments, where the underlying hypervisor's vNIC emulation introduces unpredictable latency.

Identifying which of these applies to your cluster requires walking through a structured diagnostic process rather than changing configuration at random.

Diagnosis

Start by confirming the current cluster and quorum state, then narrow down whether the problem is network-layer, configuration-layer, or load-related.

Step 1 — Check quorum and membership.

pvecm status
pvecm nodes
corosync-quorumtool -s

Look for a mismatch between the number of expected votes and the number actually seen, and note which node IDs are missing when a partition occurs.

Step 2 — Check link status per node.

corosync-cfgtool -s

This reports the state of each configured link (ring) to every other node. A link reporting as down or with a high number of connectivity transitions points directly at the network path tied to that link.

Step 3 — Watch live corosync logs during a suspected event.

journalctl -u corosync -f

Repeated Retransmit List messages indicate packet loss or delayed delivery. A processor failed, forming new configuration indicates the token was lost entirely and membership had to be recalculated — a direct trigger for the symptoms described above.

Step 4 — Test the underlying network path independently of corosync.

ping -M do -s 1400 <peer-corosync-ip>
iperf3 -c <peer-corosync-ip>
omping -c 600 -i 1 -F -q node1 node2 node3

The ping -M do -s test with the do-not-fragment flag set exposes MTU mismatches immediately by failing when packet size exceeds the path MTU. iperf3 reveals whether the link is saturated under load. omping is a long-standing tool for measuring loss and latency between cluster members and remains useful for validating a dedicated cluster network segment even though corosync's knet transport uses unicast rather than multicast.

Step 5 — Confirm firewall state on the corosync interface.

pve-firewall status
iptables -L -n -v | grep -i 54

If the Proxmox VE firewall is active on the node, verify that UDP traffic on the corosync ports is explicitly permitted rather than relying on an implicit allow.

Step 6 — Correlate timing with other workloads. Cross-reference the timestamps of corosync token losses against backup jobs, replication schedules, or known high-throughput operations on the same physical interface. If the failures cluster tightly around those windows, a shared network path is confirmed as the root cause.

Step-by-Step Solution

The following sequence resolves the large majority of corosync instability cases. Apply the steps that match what your diagnosis surfaced; not every cluster needs every step.

1. Move corosync onto a dedicated network path. If corosync currently shares an interface or bridge with VM, storage, or backup traffic, give it a dedicated physical NIC or, at minimum, a dedicated VLAN with guaranteed low latency. This single change eliminates the most common root cause.

2. Add a redundant link. Proxmox VE and Corosync 3 support multiple links per node through the ring0_addr/ring1_addr nodelist entries paired with interface blocks per link number in the totem section. Configure a second physical path (ideally on different switching hardware) so that a single point of failure cannot take down cluster communication.

3. Edit corosync.conf through the correct workflow. Never edit /etc/corosync/corosync.conf directly on a single node. Instead, copy the cluster-managed file, edit the copy, increment the version counter, and let pmxcfs validate and propagate it:

cp /etc/pve/corosync.conf /etc/pve/corosync.conf.new
nano /etc/pve/corosync.conf.new
# increment config_version inside the quorum section, then save

pmxcfs detects the .new file, validates the syntax, and if valid, activates it cluster-wide and removes the temporary file automatically. If the file is invalid, it is left in place for correction and the running configuration is untouched.

4. Verify MTU consistency end to end. If jumbo frames are used anywhere on the corosync path, confirm every switch port, bond member, and bridge along that path is configured for the same MTU. When in doubt, keep the corosync network at the standard 1500-byte MTU rather than mixing sizes.

5. Adjust totem timing only if the network genuinely requires it. If diagnosis confirms consistently higher latency than a typical LAN (for example, a stretched cluster across buildings on a slower backbone), increase the totem token timeout modestly rather than aggressively, and re-test.

6. Add firewall rules explicitly if pve-firewall is enabled. Permit UDP traffic on the corosync ports between cluster nodes rather than disabling the firewall outright.

7. Re-run the diagnostic commands after each change to confirm the link is stable and no further retransmits or configuration changes are logged over an extended observation window before considering the issue resolved.

Commands

The following commands form the core toolkit for both diagnosing and validating corosync health on Proxmox VE.

# Cluster and quorum state
pvecm status
pvecm nodes

# Detailed quorum information
corosync-quorumtool -s

# Per-link connectivity status
corosync-cfgtool -s

# Live corosync service logs
journalctl -u corosync -f
journalctl -u corosync --since "1 hour ago"

# pmxcfs (cluster filesystem) service state
systemctl status pve-cluster

# Query corosync's in-memory config/status database
corosync-cmapctl | grep members

# Basic path MTU test between two nodes
ping -M do -s 1400 10.10.10.2

# Throughput test on the corosync network segment
iperf3 -s          # run on one node
iperf3 -c 10.10.10.2   # run on the peer

# Multi-node latency/loss test
omping -c 600 -i 1 -F -q pve1 pve2 pve3

When adding a node to the cluster or re-synchronizing certificates after a network change, the standard commands are:

pvecm add <existing-cluster-ip>
pvecm updatecerts

Configuration Examples

The example below shows a three-node cluster configured with two redundant corosync links: a primary link on a dedicated management VLAN and a secondary link on a separate physical NIC connected to different switching hardware.

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.10.10.1
    ring1_addr: 10.10.20.1
  }
  node {
    name: pve2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.10.10.2
    ring1_addr: 10.10.20.2
  }
  node {
    name: pve3
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 10.10.10.3
    ring1_addr: 10.10.20.3
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: prod-cluster
  config_version: 7
  ip_version: ipv4-6
  secauth: on
  version: 2
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
}

In this layout, ring0_addr carries the primary corosync path and ring1_addr the redundant one. Because both addresses belong to networks not shared with VM traffic, corosync retains reliable low-latency delivery even if one physical path fails or becomes congested.

A minimal firewall rule permitting corosync traffic between cluster nodes, added through the Proxmox VE firewall configuration for the relevant network, should allow UDP traffic on the corosync ports between the node IP addresses used in the nodelist above rather than opening the ports broadly.

Common Mistakes

Editing /etc/corosync/corosync.conf directly on one node instead of going through /etc/pve/corosync.conf.new, causing configuration drift between nodes.
Forgetting to increment config_version when saving changes, which prevents pmxcfs from recognizing and propagating the update.
Placing both redundant links on the same physical switch, defeating the purpose of redundancy when that switch fails or reboots.
Running corosync traffic over the same bond or bridge used for VM or backup traffic without any QoS or traffic separation.
Enabling jumbo frames on only part of the network path, causing silent packet drops that are difficult to distinguish from generic instability.
Increasing the totem token timeout dramatically as a first response without first identifying and fixing the underlying network issue — this masks symptoms temporarily but does not resolve them and slows down legitimate failure detection.
Assuming that because nodes can ping each other, corosync communication must also be healthy; ICMP reachability does not confirm timing behavior under the totem protocol's stricter requirements.
Ignoring watchdog fencing events in the logs as unrelated noise, when they are frequently the clearest evidence of a corosync problem.

Best Practices

Dedicate a physical NIC or an isolated VLAN exclusively to corosync traffic, separate from VM, storage, and backup networks.
Configure at least two redundant links across physically separate switching hardware for any cluster running production workloads.
Keep corosync links on low-latency, low-jitter LAN segments; avoid stretching corosync traffic across WAN links or VPN tunnels.
For two-node clusters, deploy a QDevice on a separate, independent host to provide a tie-breaking vote instead of relying on unusual quorum workarounds.
Monitor pvecm status and corosync logs proactively (through your existing monitoring stack) rather than waiting for HA fencing events to surface a problem.
Keep clocks synchronized across all cluster nodes using a consistent time source to simplify log correlation during troubleshooting.
Test any planned network change (switch firmware update, cable move, VLAN change) against the corosync network in a maintenance window, since even brief interruptions can trigger fencing on HA-enabled clusters.
Document the physical topology of your corosync links so that on-call staff can quickly identify which switch or NIC corresponds to which ring during an incident.

FAQ

What network ports does corosync use?

Corosync communicates over UDP, with the default port for the first link being 5405 and subsequent links using additional ports in that range. When a firewall is active on the corosync interface, these ports must be explicitly permitted between cluster node addresses.

Can corosync run over a WAN link or VPN?

Technically it can be configured to, but it is not recommended for production clusters. Corosync's totem protocol expects consistently low latency and minimal jitter; WAN links introduce variability that leads to frequent token loss and unnecessary HA fencing.

What is a QDevice and when should I use one?

A QDevice is an external arbitrator process that provides an additional quorum vote without being a full cluster member. It is primarily used in two-node clusters, where a tie in votes would otherwise make quorum decisions ambiguous during a split.

How many redundant links does a cluster need?

A minimum of two independent links across separate physical infrastructure is recommended for any cluster running HA-managed workloads. Additional links can be added for very large or critical environments, but two well-separated links resolve the vast majority of single-point-of-failure scenarios.

Does increasing the token timeout fix corosync instability?

It can reduce the frequency of visible symptoms by giving the network more time to deliver a token before declaring it lost, but it does not address the underlying cause. Treat timeout adjustments as a tuning step after the network path itself has been verified and corrected, not as a substitute for fixing it.

Conclusion

Corosync communication failures are among the most disruptive issues a Proxmox VE cluster can experience, precisely because the service operates quietly until something goes wrong, at which point HA fencing turns a network hiccup into unplanned reboots. The fix is rarely mysterious: isolate corosync traffic onto its own low-latency path, configure genuine redundancy across separate hardware, and use the diagnostic commands covered here — pvecm status, corosync-cfgtool -s, and live corosync logs — to confirm root cause before changing timing parameters. Clusters built with a dedicated, redundant corosync network from the outset rarely experience these issues at all, which makes this one of the highest-value pieces of planning in any Proxmox VE deployment.

Vormox is an independent virtualization management platform built on Proxmox VE. Proxmox® and Proxmox VE® are trademarks of Proxmox Server Solutions GmbH. Vormox is not affiliated with, endorsed by, or sponsored by Proxmox.