Every Proxmox VE cluster eventually reaches the same milestone: someone asks what happens if a node just dies. If the honest answer is "someone gets paged and migrates things by hand," you don't have high availability yet—you have a cluster that merely shares storage and a web UI. Proxmox VE's HA Manager exists to close that gap: it watches your nodes, and when one disappears, it fences it and restarts its VMs and containers somewhere else, automatically, without anyone touching a keyboard.

This guide walks through actually configuring HA for VMs and LXC containers in Proxmox VE—enabling it on resources, controlling where they're allowed to run with node affinity rules, keeping related services together or apart with resource affinity rules, tuning restart and relocation behavior, and testing the whole thing before you trust it with anything that matters. This is about building HA from scratch, not about temporarily suspending it for maintenance—if you're looking for how to safely disarm HA during planned network work, that's a different task with a different set of commands.

Prerequisites

HA in Proxmox VE is a cluster feature, not a single-node one. Before any of this works, you need:

  • A working Proxmox VE cluster with at least three nodes and quorum. Two-node clusters can run HA but need a QDevice to avoid split-brain during a node loss—without one, losing either node can leave the survivor without quorum, which means no fencing and no automatic recovery.
  • Shared storage reachable from every node that might run a given guest—Ceph, ZFS over iSCSI, NFS, or similar. HA can restart a VM on another node only if that node can actually access the VM's disk. A guest sitting on node-local storage with no replication configured is not a real HA candidate no matter what state you set on it.
  • A watchdog device on every node. Proxmox VE uses the kernel's softdog module by default, which is sufficient for most setups and requires no extra hardware. If your servers have IPMI or a dedicated watchdog chip, you can point pve-ha-manager at it instead for more reliable fencing.
  • Reliable corosync links. HA's failure detection rides entirely on corosync's view of cluster membership. A flaky or single-homed corosync network will produce false-positive fencing events long before it produces a real one.

If any of these aren't in place yet, fix them first. HA amplifies whatever state your cluster is already in—a solid cluster gets automatic recovery, a shaky one gets automatic chaos.

How the HA Manager Actually Works

Two daemons run on every node and do all the work. The Local Resource Manager (LRM), pve-ha-lrm, is responsible for actually starting, stopping, and migrating the resources assigned to its node, and for holding a watchdog lease open as proof of life. The Cluster Resource Manager (CRM), pve-ha-crm, runs an election among all nodes and elects one as master; the master is the one that decides where resources should run and issues fencing when a node goes silent.

The mechanism that makes fencing safe is the watchdog. As long as an LRM is in contact with the CRM master and the node has quorum, it keeps refreshing its watchdog lease. If a node hangs, crashes, or gets cut off from the rest of the cluster, its LRM stops refreshing the watchdog, the timer elapses, and the node reboots itself—before the CRM ever assumes control of its resources on another node. That ordering is the whole point: it guarantees a resource can never run on two nodes at once, because the node that used to have it is guaranteed to be rebooting by the time anything else tries to start it.

Everything else in this guide—affinity rules, restart limits, shutdown policy—is really just configuring the inputs to those two daemons.

Step 1: Enable HA on a VM or Container

HA resources are identified by a service ID combining the resource type and its numeric ID: vm:100 for a virtual machine, ct:102 for a container. Adding a guest to HA management is a single command:

# ha-manager add vm:100
# ha-manager set vm:100 --state started

Or from the GUI: Datacenter → HA → Add, pick the VM or CT, and confirm. Either path writes an entry into /etc/pve/ha/resources.cfg, which is the file the CRM actually reads:

vm: 100
    state started
    max_relocate 2
    max_restart 1

ct: 102
    state started

The state field is what tells the CRM whether it should be managing this resource at all, and it accepts more than just started:

  • started — the normal state; the CRM keeps it running and will restart or relocate it on failure.
  • stopped — the CRM keeps the resource stopped and won't react to it being manually started outside of HA.
  • disabled — HA stops managing it entirely without removing the configuration; useful for temporarily opting a single guest out.
  • ignored — similar to disabled, used when you want to manage the resource by hand for a while without deleting its HA config.

Don't add every guest on the cluster to HA reflexively. A guest with no replicated storage, or one whose downtime genuinely doesn't matter (a scratch dev VM, a disposable test container), just adds noise to your HA status output and a false sense that it's protected when it isn't.

Step 2: Control Placement with Node Affinity Rules

Older Proxmox VE releases used HA groups for this—a named list of nodes with priorities that a resource was pinned to. Groups still work for backward compatibility, but current Proxmox VE versions replace them with node affinity rules, defined in /etc/pve/ha/rules.cfg, which do the same job with a cleaner syntax and support layering multiple rules together.

Create a basic rule pinning a resource to a preferred node:

# ha-manager rules add node-affinity ha-rule-vm100 \
    --resources vm:100 --nodes node1

Which produces:

node-affinity: ha-rule-vm100
        resources vm:100
        nodes node1

By default this is a preference, not a hard requirement—if node1 is unavailable, the resource can still start on another cluster member. If you need a resource to run only on a specific subset of nodes and stay stopped otherwise (because of licensing, local hardware passthrough, or a compliance requirement), make the rule strict:

# ha-manager rules set node-affinity ha-rule-vm100 --strict 1

For cascading failover across more than two nodes, assign priorities—higher numbers are preferred:

# ha-manager rules add node-affinity priority-cascade \
    --resources vm:200,ct:300 --nodes "node1:2,node2:1,node3:1,node4"

This tells the CRM to run vm:200 and ct:300 on node1 whenever it's available, fail over to node2 or node3 if it isn't, and only fall back to node4 as a last resort. This is the same shape of behavior the old HA groups gave you with nodes node1:2,node2:1,node3:1,node4, just expressed as a rule instead of a standalone group object, and it composes with resource affinity rules in a way groups never could.

Step 3: Keep Related Resources Together or Apart

Node affinity controls where a single resource can run. Resource affinity rules control the relationship between resources—whether they should always land on the same node or must never share one.

Positive affinity (colocation) keeps resources together, which matters for guests that talk to each other constantly over what would otherwise be cross-node traffic—an application server and its cache, for instance:

# ha-manager rules add resource-affinity keep-together \
    --affinity positive --resources vm:100,vm:200

Negative affinity (anti-affinity) does the opposite—forcing resources apart so a single node failure can't take out both replicas of something at once:

# ha-manager rules add resource-affinity keep-separate \
    --affinity negative --resources vm:200,ct:300

A negative affinity rule can only list as many resources as you have nodes to spread them across—three resources need at least three eligible nodes, or the rule is unsatisfiable. The CRM also won't let you contradict yourself: if two resources are already in a positive affinity rule together, you can't separately place one of them in a negative rule with the other, and any rule set that would keep two resources both together and apart at once gets flagged and disabled rather than silently applied. Resources in a positive affinity group also inherit each other's node affinity restrictions—if one member is pinned to node1, the whole group effectively is.

Step 4: Tune Restart and Relocation Behavior

Not every failure is a dead node. Sometimes a guest crashes on a perfectly healthy host, and the right response is a restart in place, not a fencing event. Two properties on the resource itself control this:

  • max_restart (default 1) — how many times the LRM will try to restart the resource on its current node before giving up on that node and asking the CRM to relocate it.
  • max_relocate (default 1) — how many other nodes the CRM will try before marking the resource as failed and stopping automatic recovery entirely.
vm: 100
    state started
    group priority-group
    max_restart 2
    max_relocate 3
    failback 1

The relocate counter only resets to zero after the resource actually starts successfully somewhere—so a resource that keeps failing everywhere it lands will eventually stop being retried rather than bouncing around the cluster forever. Set these on a per-resource basis based on how disruptive a failed start actually is: a stateless web frontend can tolerate a higher max_relocate than a guest with a large in-memory dataset that takes minutes to reload.

failback controls whether a resource automatically moves back to its higher-priority node once that node returns and rejoins the cluster. Leave it enabled if you've deliberately built an asymmetric cluster (a beefy primary node and a leaner standby) and want workloads to return home once the primary is healthy again; disable it if you'd rather avoid the extra migration and just let things stay wherever they ended up.

Step 5: Set the Node Shutdown Policy

What should happen to HA resources when you deliberately shut down or reboot a node—for a kernel update, say—is a separate decision from what happens when a node fails unexpectedly. Configure it under Datacenter → Options → HA Settings in the GUI, or directly in /etc/default/pve-ha-manager:

# shutdown policy: migrate | freeze | failover | conditional
shutdown_policy=migrate
  • migrate — the LRM marks itself unavailable and the CRM proactively migrates its resources to other nodes before the shutdown completes, then moves them back when the node returns (subject to failback). This is the right default for planned reboots on a cluster with spare capacity.
  • freeze — resources are left exactly where they are and simply paused from HA's perspective until the node comes back; use this when you expect the node to be down only briefly and don't want the churn of migrating everything out and back.
  • failover — treats the shutdown like a node failure and fences immediately, restarting resources elsewhere without waiting for a graceful migration. This is rarely what you want for planned maintenance.
  • conditional — the legacy default; behavior depends on the individual resource and affinity configuration rather than one blanket rule.

If you find yourself doing rolling reboots across a cluster regularly, migrate is almost always the better choice—it turns a reboot into a clean, watched migration instead of a fencing event that just happens to be intentional.

Step 6: Watch the Cluster's HA State

ha-manager status is the single command you'll run more than any other once HA is live:

# ha-manager status
quorum OK
master pve01 (active, Thu Jul 16 09:12:44 2026)
lrm pve01 (active, Thu Jul 16 09:12:41 2026)
lrm pve02 (active, Thu Jul 16 09:12:39 2026)
lrm pve03 (active, Thu Jul 16 09:12:42 2026)
service vm:100 (pve01, started)
service vm:200 (pve02, started)
service ct:300 (pve03, started)

Each service line shows both where a resource currently lives and its state. Beyond the usual started/stopped, you'll see transitional states worth recognizing when something's actively happening: fence (a node is being fenced and its services are queued for recovery), recovery (the CRM is actively placing a resource on a new node), migrate (a live migration is in flight), and error (automatic recovery has given up after exhausting max_restart/max_relocate and needs a human). If you ever see a service sitting in error, resolve whatever's actually broken first, then clear it with ha-manager set <sid> --state started rather than just re-adding the resource.

To see the affinity rules and resource definitions currently in force without opening the config files directly:

# ha-manager config
# ha-manager rules config

Testing Failover Before You Trust It

Don't wait for a real outage to find out whether your HA configuration actually does what you think it does. There are two reasonable ways to test it, and you should really do both.

On a lab cluster, test a real failure. Pick a node running a non-critical HA-managed guest and pull its power, or block corosync traffic on it with a firewall rule, and watch ha-manager status and journalctl -u pve-ha-crm -f on another node. You should see the node marked unknown, then fenced, then its resource restarted elsewhere within roughly the corosync token timeout plus a few seconds. If that doesn't happen the way you expect—if the resource sits in fence for minutes, or never recovers—you want to find that out on a lab node, not on your production database VM.

Without touching real hardware, use the HA simulator. Proxmox ships a standalone simulator specifically for modeling CRM/LRM behavior without a live cluster:

apt install pve-ha-simulator
mkdir ha-test
pve-ha-simulator ha-test/

It lets you define a virtual cluster topology, assign simulated resources with the same affinity rules and restart limits you'd use in production, then "kill" nodes and step through exactly how the CRM would react—which node it elects as master, which resource gets relocated where, and in what order. It's the fastest way to sanity-check a complex affinity rule set (three node-affinity rules plus a couple of anti-affinity pairs, say) before you commit it to a real cluster where a mistake means an actual outage during a test.

Common Mistakes

Adding HA to a guest without shared, replicated storage underneath it is the most common one. HA can restart a VM on another node, but only if that node can read the VM's disk. If the disk only exists on the node that just died, "recovery" just means the CRM trying to start a VM against storage that isn't there.

Running a two-node HA cluster without a QDevice is the second. Two nodes with plain corosync quorum means losing either one leaves the other without quorum—and without quorum, the CRM won't fence or recover anything. A QDevice is a small, cheap fix for what would otherwise be a cluster that can't actually do the one thing you built it to do.

Setting max_restart and max_relocate too high for a resource that's actually broken is a subtler one. If a guest is crashing because of a genuine configuration problem inside it, generous retry limits just mean the CRM spends longer bouncing it around the cluster before giving up, generating noise and consuming migration bandwidth for a problem that a restart was never going to fix.

Forgetting that resource affinity constraints compound is another. A positive affinity rule between two VMs plus a strict node affinity rule on one of them effectively pins both to that node—which is easy to lose track of once you have a handful of overlapping rules, and can quietly defeat the failover behavior you thought you had configured elsewhere.

Finally, treating ha-manager status showing "quorum OK" as proof that HA itself is healthy. Quorum is a corosync-level fact; it says nothing about whether the CRM master is actually up, whether LRMs are responding, or whether a resource has been silently sitting in error for a week. Check the actual service lines, not just the quorum line.

Best Practices

  • Start with a small, well-understood set of HA-managed resources and expand gradually, rather than adding every guest on the cluster on day one. It's much easier to reason about failover behavior for five resources than for fifty.
  • Prefer node affinity rules with priorities over strict pinning wherever you can. Strict rules are sometimes necessary, but every strict rule is a resource that simply stops if its preferred nodes are all down, instead of failing over somewhere less ideal but still running.
  • Document your affinity rules and why they exist somewhere outside rules.cfg itself—a comment field helps, but a short runbook entry explaining "vm:100 and vm:200 are pinned together because they share a database connection pool" saves the next person from guessing.
  • Run the HA simulator against any nontrivial rule set change before applying it live, especially once you have more than two or three interacting affinity rules.
  • Alert on services sitting in error or fence for longer than expected, and alert separately on quorum loss. Both matter, and neither implies the other.
  • Revisit max_restart/max_relocate and shutdown policy periodically as the cluster grows—defaults tuned for a three-node cluster with spare capacity may not fit once you're running closer to capacity across more nodes.

FAQ

Do I need shared storage for HA to work at all?

Yes, for any meaningful recovery. HA can technically manage a resource on local storage, but a node failure means the disk is gone along with the node, so there's nothing for the CRM to restart elsewhere. Use Ceph, ZFS with storage replication, or shared NFS/iSCSI-backed storage for anything you actually want HA to protect.

What's the difference between node affinity rules and the older HA groups?

They solve the same problem—constraining which nodes a resource can run on—but node affinity rules are the current mechanism and compose with resource affinity rules (positive/negative) in ways the older, standalone group objects didn't. Existing groups keep working for backward compatibility, but new configurations should use rules.

Can a resource be in more than one affinity rule at once?

Yes. A resource can have a node affinity rule constraining where it runs and be part of a resource affinity rule governing its relationship to other resources at the same time. Just be aware the constraints combine—a positive affinity partner inherits your resource's node restrictions too.

How quickly does HA actually react to a failed node?

Roughly the corosync token timeout (3000 ms by default on new clusters) plus the time for the CRM to confirm the node is gone and issue fencing, plus however long the guest itself takes to boot on its new node. For most clusters that's well under a minute for the CRM's decision, though the guest's own boot time usually dominates the total.

Will HA fight me if I manually migrate an HA-managed guest?

A manual live migration of an HA-managed resource is tracked by the CRM as a normal migration, not treated as a failure, so it won't trigger fencing or recovery logic. It's only unplanned disappearance—a node going silent past the watchdog timeout—that the CRM interprets as a failure needing fencing.

Do I need a hardware watchdog, or is softdog good enough?

Softdog is adequate for most environments and is what Proxmox VE uses by default with no extra configuration. A hardware watchdog (IPMI-based, or a dedicated chip) is more reliable in the specific case where the kernel itself is too wedged to even run the software watchdog's reset logic, which is a narrow but real failure mode worth covering on production clusters if the hardware supports it.

Conclusion

High availability in Proxmox VE isn't a single checkbox—it's a small stack of decisions: which resources actually deserve HA, where they're allowed to run, which ones need to stay together or apart, how many times to retry before giving up, and what happens when you shut a node down on purpose versus when it dies on its own. Node and resource affinity rules give you real control over all of that without hand-editing config files from scratch, and the HA simulator means you can validate a rule set before it ever runs against production traffic.

The payoff is real: a node failure that used to mean a 2 a.m. page and a manual migration instead becomes something you read about the next morning in the logs, because the cluster already handled it. Get there by building HA up gradually—shared storage first, a handful of well-chosen resources next, affinity rules layered in as your topology actually needs them—and by testing failover deliberately rather than finding out how it behaves the first time it matters.