Proxmox VE 9.2 HA Disarm/Arm: Cluster-Wide Maintenance Without Fencing

Every Proxmox VE cluster admin has hit this problem at some point: you need to touch the network the cluster depends on—replace a switch, change a VLAN, re-cable a corosync link—and the HA stack doesn't know the difference between "the admin is doing planned maintenance" and "this node just fell off the network." Before Proxmox VE 9.2, your options were limited. You could stop pve-ha-lrm and pve-ha-crm on every node, set every HA resource to ignored one by one, or just accept the risk and hope the corosync token timeout was generous enough to survive the switch reboot. None of those were great.

Proxmox VE 9.2 finally gives you a real answer: cluster-wide arm and disarm for the HA Manager. It's a small addition on paper—two new CRM commands—but it changes how you plan network maintenance on any cluster that runs HA-managed VMs or containers. This article walks through what it does, why it exists, how to diagnose whether your cluster needs it, and how to use it without leaving yourself in a half-armed state at 2 a.m.

Problem Overview

HA in Proxmox VE exists to do one job: if a node disappears, fence it and restart its services elsewhere. That's exactly what you want when a node genuinely crashes. It's exactly what you don't want when you're the one causing the disappearance on purpose.

The watchdog-based fencing model that Proxmox uses is unforgiving by design. Each node's Local Resource Manager (LRM) holds a hardware or software watchdog open as long as it can talk to the CRM master and the cluster has quorum. Lose that connection past the corosync token timeout, and the watchdog fires—the node reboots itself, and the CRM reassigns its services to whichever node is still standing. That's correct behavior during a real outage. It's a problem when you're the one who pulled the cable.

Before 9.2, admins worked around this in a few ways, none of them clean:

Setting every HA resource's state to ignored individually with ha-manager set <sid> --state ignored, then remembering to flip each one back afterward.
Stopping the pve-ha-crm and pve-ha-lrm services outright, which works but leaves you outside the normal HA lifecycle and easy to forget about.
Scheduling maintenance windows tight enough that nothing ever crosses the corosync token timeout—fine until the switch firmware update takes longer than the runbook said it would.

The per-node ha-manager crm-command node-maintenance enable <node> command already existed and still has its place for single-node work like a kernel upgrade with a reboot. But it doesn't help when the thing you're changing is the network the whole cluster depends on to reach quorum in the first place. That's a cluster-wide problem, and it needed a cluster-wide switch.

Symptoms

If you've been running HA clusters for a while, you've probably seen this sequence before. A switch reboots for a firmware update, corosync loses its ring, and a minute later you're watching VMs restart on nodes you didn't ask them to move to. The giveaway in the logs looks something like this:

Jul 11 02:14:07 pve02 corosync[1893]:   [KNET  ] link: host: 1 link: 0 is down
Jul 11 02:14:07 pve02 corosync[1893]:   [KNET  ] host: host: 1 has no active links
Jul 11 02:14:10 pve02 corosync[1893]:   [TOTEM ] Token has not been received in 2250 ms
Jul 11 02:14:11 pve02 pve-ha-crm[2201]: node 'pve01': state changed from 'online' => 'unknown'
Jul 11 02:14:41 pve02 pve-ha-crm[2201]: fencing node 'pve01'
Jul 11 02:14:41 pve02 watchdog-mux[1701]: client watchdog expired - disable watchdog updates

A few seconds later, pve01 self-reboots because its watchdog wasn't refreshed in time, and every HA-managed guest that was running there gets restarted somewhere else. If that guest was a database VM mid-transaction, you now have a recovery conversation to have with whoever owns that workload.

Other symptoms that point at the same underlying issue: ha-manager status showing services flipping between nodes during what should have been a quiet maintenance window, unexpected entries in /var/log/pve-firewall.log or the syslog around migration events you didn't trigger, and users reporting brief outages on services you thought were protected by HA, not exposed by it.

Root Cause

The root cause isn't a bug—it's a mismatch between what the HA stack is designed to detect (node failure) and what you're actually doing (planned network change). The CRM and LRM have no built-in concept of "the admin knows what's happening here." They only know quorum, watchdog state, and the last time they heard from their peers. Two watchdog daemons are involved. watchdog-mux multiplexes access to the physical or software watchdog device on each node so multiple consumers (LRM, and in older setups, other services) can share it. pve-ha-lrm refreshes its lease with watchdog-mux as long as it's actively in contact with the CRM master. When corosync communication breaks down past the configured token timeout—3000 ms by default on new clusters—the LRM can no longer safely assume it should keep running services, and it stops refreshing the watchdog. The watchdog then does exactly what it's built to do: reboot the node, on the assumption that a hung or partitioned node is worse than a rebooted one. That's the correct failure mode for an actual outage. The problem is that a switch firmware update, a bonding interface flap, or a VLAN change on the corosync network produces the exact same signal to the CRM as a dead node. The stack has no way to distinguish "administrator is doing something" from "something is broken," which is precisely the gap that disarm-ha and arm-ha close.

Diagnosis

Before you touch anything, confirm what state your cluster is actually in. Don't assume—check.

Start with quorum and corosync link health:

pvecm status
corosync-quorumtool -s

You want to see Quorate: Yes and every expected ring/link listed as active before you start any maintenance that touches the corosync network. If a link is already flaky, fix that first—disarming HA doesn't fix an unstable network, it just stops HA from reacting to it.

Next, check the current HA manager state:

ha-manager status

On a healthy, fully armed cluster you'll see output along these lines:

quorum OK
master pve01 (active, Sat Jul 11 01:52:03 2026)
lrm pve01 (active, Sat Jul 11 01:52:00 2026)
lrm pve02 (active, Sat Jul 11 01:51:58 2026)
lrm pve03 (active, Sat Jul 11 01:52:01 2026)
service vm:100 (pve01, started)
service vm:101 (pve02, started)
service ct:200 (pve03, started)

That confirms every LRM is active and the CRM master is healthy—the state you want to be in before you disarm, not after something has already gone sideways. If a node's LRM is already showing anything other than active, resolve that first; disarming a cluster that's already unhealthy just hides the symptom during your maintenance window instead of preventing it. It's also worth tailing the HA logs in a separate terminal while you work, so you can see the transition happen in real time rather than guessing:

journalctl -u pve-ha-crm -f
journalctl -u pve-ha-lrm -f
journalctl -u watchdog-mux -f

Step-by-Step Solution

Here's the workflow I use for planned network maintenance on an HA cluster running 9.2 or later.

Verify cluster health first. Run pvecm status and ha-manager status as described above. Do not proceed if quorum is marginal or any LRM is unresponsive.
Pick the right resource mode. This is the decision that actually matters. Use freeze if you expect all nodes to stay up and reachable during the work, just with a temporarily unstable network—services stay exactly where they are and the CRM simply stops reacting. Use ignore if you need to manually stop, start, or migrate guests yourself during the maintenance, for example if you're taking individual nodes down one at a time for firmware updates.
Disarm the cluster. Run the command from any single node—it propagates cluster-wide, you don't need to repeat it on every host:
```
ha-manager crm-command disarm-ha freeze
```
Confirm the disarm took effect. Check status again before you start the actual network change:
```
ha-manager status
```
The fencing/CRM line should now report the cluster as disarmed rather than armed. Don't proceed with the network work until you've seen that confirmation—the disarm state propagates through corosync and on a busy cluster it can take a few seconds.
Do the maintenance. Replace the switch, change the VLAN tagging, re-cable the bond, whatever the actual task is. With the HA stack disarmed, corosync token timeouts and watchdog expirations during this window won't trigger fencing.
Re-arm as soon as the network is stable again. Don't leave this for later in the day—re-arm the moment you've confirmed the network change is good:
```
ha-manager crm-command arm-ha
```
Verify quorum and HA state one more time. Run pvecm status and ha-manager status again. Confirm every node shows active and every service is where you expect it. If you used ignore mode and moved guests around manually, double check the CRM has picked up their current location correctly rather than trying to "correct" them back to where they were before you disarmed.

For single-node work—a kernel update with a reboot, say—you still don't need the cluster-wide disarm at all. ha-manager crm-command node-maintenance enable <node> handles that case and is the better tool for it; disarm-ha is for when the risk is to the cluster's shared communication path, not to one node's uptime.

Commands

The full set of commands relevant to this workflow, in one place:

# Check cluster quorum and corosync link status
pvecm status
corosync-quorumtool -s

# Check current HA manager and resource state
ha-manager status
ha-manager config

# Disarm HA cluster-wide before network maintenance
ha-manager crm-command disarm-ha freeze
ha-manager crm-command disarm-ha ignore

# Re-arm HA cluster-wide once maintenance is complete
ha-manager crm-command arm-ha

# Per-node maintenance mode (single node, not cluster-wide)
ha-manager crm-command node-maintenance enable pve02
ha-manager crm-command node-maintenance disable pve02

# Watch HA and watchdog activity live during the change
journalctl -u pve-ha-crm -f
journalctl -u pve-ha-lrm -f
journalctl -u watchdog-mux -f

# Built-in command help if you forget the syntax mid-maintenance
ha-manager help crm-command
ha-manager help crm-command disarm-ha

Configuration Examples

The disarm and arm commands don't require any configuration file changes—they're runtime CRM commands, not persistent settings. But it helps to know what you're protecting. Here's a typical HA resource definition from /etc/pve/ha/resources.cfg on a three-node cluster:

vm: 100
    state started
    group priority-group
    max_relocate 2
    max_restart 1
    failback 1

vm: 101
    state started
    max_relocate 1
    max_restart 1
    comment "billing-db, do not auto-relocate more than once"

ct: 200
    state started
    max_relocate 1

These resource states (started, stopped, disabled, ignored, enabled) are exactly what disarm-ha ignore temporarily overrides at the cluster level—instead of editing this file resource by resource before maintenance and reverting it afterward, the cluster-wide disarm gives you the same effective protection without touching the config at all.

Compare the ha-manager status output before and after a disarm, so you know what you're actually looking for on your own cluster:

Armed (normal operation):

quorum OK
master pve01 (active, Sat Jul 11 09:12:44 2026)
lrm pve01 (active, Sat Jul 11 09:12:41 2026)
lrm pve02 (active, Sat Jul 11 09:12:39 2026)
lrm pve03 (active, Sat Jul 11 09:12:42 2026)
service vm:100 (pve01, started)
service vm:101 (pve02, started)
service ct:200 (pve03, started)

Disarmed (maintenance in progress, freeze mode):

quorum OK
master pve01 (active, Sat Jul 11 09:14:02 2026)
disarmed (freeze)
lrm pve01 (active, Sat Jul 11 09:12:41 2026)
lrm pve02 (active, Sat Jul 11 09:12:39 2026)
lrm pve03 (active, Sat Jul 11 09:12:42 2026)
service vm:100 (pve01, started, freeze)
service vm:101 (pve02, started, freeze)
service ct:200 (pve03, started, freeze)

The freeze marker on each service line is your confirmation that the CRM has stopped acting on state changes for that resource—it's what you check before you trust the network to survive without triggering a fencing event.

Common Mistakes

The mistakes I see with this feature aren't really about the commands—they're about process.

Forgetting to re-arm is the big one. The disarm state doesn't have a timeout by design; it stays disarmed until you explicitly run arm-ha. That's correct behavior, but it means a cluster left disarmed after a late-night maintenance window sits unprotected until someone notices. If a node genuinely fails the next morning, HA won't respond to it. Put re-arming as an explicit, checked step in your runbook, not an assumption.

Picking the wrong mode is the second. ignore is tempting because it feels like it gives you more control, but if you're not actually planning to manually manage guests during the window, freeze is the safer default—it keeps the CRM's view of "where things are supposed to be" intact instead of handing that responsibility to you for the duration. Confusing cluster-wide disarm with per-node maintenance mode is another one. If you're only rebooting a single node for a kernel update, node-maintenance enable is the right tool; reaching for a full cluster-wide disarm for single-node work is using a bigger hammer than the job needs, and it leaves every other node's HA protection off for no reason.

Disarming without checking quorum first is a mistake that compounds the problem it's trying to solve. If the cluster is already partially split before you disarm, you're now flying blind on a cluster that was already in trouble, with the one mechanism that would have caught it turned off.

Finally, running the maintenance without anyone watching the HA and corosync logs. The commands work, but you still want to actually see the transition happen and see the network recover cleanly before you re-arm—don't disarm, walk away, do the switch change, and re-arm on faith.

Best Practices

A few habits make this feature genuinely safer to use in production rather than just theoretically safer:

Test the full disarm/maintenance/re-arm cycle on a lab or staging cluster before you rely on it during a real production network change. Know what the status output looks like on your setup specifically.
Default to freeze unless you have a concrete reason to need ignore. Freeze requires less judgment from you mid-maintenance, which matters more at 2 a.m. than it does in a calm afternoon test.
Write the disarm/re-arm steps into your maintenance runbook as numbered, checked items—not as something the engineer on call is expected to remember unprompted.
Set a monitoring check for the HA disarmed state itself. A cluster sitting disarmed for more than an hour or two outside a known maintenance window is worth an alert, not a surprise discovered days later.
Pair the disarm with your change window, not with the start of your terminal session. Disarm right before the risky step, re-arm right after it's confirmed stable—don't disarm at the start of a two-hour maintenance block and forget about it until the end.
Keep an eye on corosync link redundancy generally. Disarming HA reduces the blast radius of a planned network change, but a cluster with only one corosync ring is still one bad cable away from a real quorum loss, disarmed or not.

FAQ

Does disarm-ha affect quorum itself?

No. Disarming HA only changes whether the CRM reacts to failures with fencing and service recovery. Corosync quorum calculation keeps running independently—you can still lose quorum while disarmed, you just won't get automatic fencing as a response to it.

Do I need to run disarm-ha on every node?

No. Run it once, from any node with access to the cluster filesystem. It propagates cluster-wide through the CRM.

What happens to guests that are mid-migration when I disarm?

An in-progress migration isn't interrupted by disarming HA. The state change applies to how the CRM reacts to future events, not to operations already underway.

Is disarm-ha the same as stopping pve-ha-crm and pve-ha-lrm?

No, and that distinction matters. Stopping those services takes the node out of HA management entirely and can leave resource state inconsistent when you restart them. Disarm-ha keeps the normal HA lifecycle running and just suspends its fencing and recovery reactions, with resource state preserved for when you re-arm.

Can I disarm just to prevent a single risky migration, rather than for a network change?

You can, but it's the wrong tool for that. If you're worried about one specific resource, set that resource's state to ignored with ha-manager set instead. Save cluster-wide disarm for genuinely cluster-wide risk, like changes to the network corosync depends on.

Conclusion

This is one of those Proxmox VE features that doesn't show up in a demo but earns its keep the first time you actually need it. Cluster-wide HA disarm turns "hope the maintenance window is short enough" into an explicit, verifiable state you control—and turns "re-arm afterward" into a step you check rather than a step you remember, if your runbook is written properly. If you're running HA on a cluster of any size and you've been doing network maintenance by shrinking your change window and crossing your fingers, it's worth updating to 9.2 for this feature alone. Test it on a non-production cluster first, get comfortable with what ha-manager status looks like in both states, and build the disarm and re-arm steps into your actual maintenance documentation rather than treating them as something you'll remember to do live.

Vormox is an independent virtualization management platform built on Proxmox VE. Proxmox® and Proxmox VE® are trademarks of Proxmox Server Solutions GmbH. Vormox is not affiliated with, endorsed by, or sponsored by Proxmox.

Disarming HA in Proxmox VE 9.2: Safe Cluster-Wide Maintenance Without Fencing