How to Remove a Node From a Proxmox VE Cluster (Safely, Without Breaking Quorum)

Why You'd Need to Remove a Node From a Proxmox VE Cluster

Clusters aren't static. Hardware gets decommissioned, nodes get reinstalled from scratch after a failed disk, budgets shrink a lab from five nodes to three, or a node dies outright and needs to be scrubbed out of the configuration before it starts causing quorum problems. Whatever the reason, removing a node from a Proxmox VE cluster is a routine operation — but it's one with a specific order of steps, and doing it out of order is the most common way people end up with orphaned Ceph OSDs, stuck replication jobs, or a cluster that won't reach quorum.

This guide walks through the full process end to end: draining a healthy node before removal, forcibly removing a node that's already dead, cleaning up everything pvecm delnode doesn't touch on its own, and recovering if quorum drops out mid-removal. It assumes a standard Proxmox VE cluster (version 7 through 9) using corosync for cluster communication.

Before You Start: A Pre-Removal Checklist

Run through this list before you touch pvecm. Skipping any of these is the usual cause of a "clean" removal turning into a troubleshooting session.

• Migrate or destroy every VM and LXC container on the node you're removing. A node with running guests cannot be cleanly removed — you either live-migrate them elsewhere or shut them down and decide whether to keep or discard them.
• Delete any storage replication jobs that involve the node, in both directions. A stale replication job referencing a node that no longer exists is one of the most common post-removal support tickets — it will sit in the job list erroring out indefinitely until you remove it manually via the API.
• Tear down Ceph roles on the node if you're running a hyper-converged Ceph cluster: OSDs, monitors, managers, and metadata servers all need to be removed from Ceph's own configuration before the node leaves the Proxmox cluster. Proxmox's cluster removal does not touch Ceph — that's a separate cleanup pass.
• Check High Availability (HA) — remove the node from any HA groups, and confirm no HA-managed resources are pinned to it.
• Back up anything local to that node's storage (local-lvm, local-zfs, directory storage) that isn't already replicated or backed up elsewhere. Once the node is powered off and removed, that data is gone unless you copied it out first.
• Decide the node's fate. Is it being wiped and reinstalled, repurposed as a standalone host, or scrapped entirely? This affects what you do with it after removal (see the "Reusing the Removed Node" section below).

If you skip the migration and replication cleanup and just power off the node and run pvecm delnode, the command usually still succeeds — but you're left with dangling references and jobs that fail silently until someone notices.

Step 1: Migrate Workloads Off the Node

For VMs, use live migration if the node is still online and the cluster has shared storage (Ceph, NFS, iSCSI, ZFS over iSCSI, etc.):

qm migrate <vmid> <target-node> --online

If storage is purely local to the node (local-lvm or local-zfs with no replication), you'll need offline migration with storage relocation instead, which copies the disk to the target node:

qm migrate <vmid> <target-node> --with-local-disks

For LXC containers, the equivalent is:

pct migrate <ctid> <target-node> --restart

The --restart flag stops and restarts the container automatically around the migration, since LXC doesn't support true live migration the way QEMU does. You can also do all of this from the GUI under Datacenter → node → VM/CT → Migrate, which is often easier when you're moving a batch of guests and want to watch progress in the task log.

If a guest doesn't need to survive — a throwaway test VM, for instance — it's faster to just delete it than migrate it. Don't migrate things you're going to delete anyway.

Step 2: Remove Replication Jobs Involving the Node

Storage replication jobs are defined per-VM and reference both a source and target node. If the node you're removing is either endpoint of a replication job, delete the job first — pvecm delnode does not clean these up, and Proxmox VE will otherwise keep trying to run a replication task against a node that no longer answers.

Check existing jobs from the GUI under Datacenter → Replication, or from the CLI:

pvesr list

Remove a specific job with:

pvesr delete <vmid>-<jobid>

Do this for every job touching the node before proceeding — trying to clean these up after the node is already gone is significantly more annoying, since some of the tooling expects the referenced node to still exist.

Step 3: Remove Ceph Roles (Hyper-Converged Clusters Only)

If the node runs Ceph services, Proxmox's cluster membership and Ceph's cluster membership are two separate things, and you need to unwind Ceph first. Skip this section entirely if you're not running Ceph.

From the GUI, go to node → Ceph → OSD, select each OSD on the node, click Stop, then Out, then Destroy once data has rebalanced off it. From the CLI, the equivalent sequence per OSD is:

ceph osd out osd.<id>
# wait for rebalance to complete — watch with:
ceph -w
pveceph osd destroy <id>

Once all OSDs are gone, remove the monitor and manager if the node runs them:

pveceph mon destroy <nodename>
pveceph mgr destroy <nodename>

And the metadata server if the node runs CephFS MDS:

pveceph mds destroy <nodename>

Confirm with ceph -s that the node no longer appears anywhere in the Ceph topology before moving on. Removing a node from the Proxmox cluster while it still holds Ceph monitors or OSDs is a reliable way to degrade your storage cluster's redundancy unexpectedly.

Step 4: Clean Up HA References

If the node is a member of an HA group, or if the datacenter-wide HA rules reference it by name, update those first. Go to Datacenter → HA → Groups and remove the node from any group member lists, and check Datacenter → HA → Resources for anything explicitly pinned to it. Leaving stale HA references around doesn't usually block the node removal itself, but it clutters the HA manager's state and can cause confusing "node not found" warnings later.

Step 5: Power Off the Node

Before running the actual removal command, shut the node down and make sure it will not rejoin the cluster on its own — for example, don't leave it configured to auto-start and reconnect via corosync after a reboot. The official guidance is explicit here: powering off the node before removal is what prevents a "split-brain"-style situation where the node reconnects mid-removal and corosync gets two conflicting views of cluster membership.

ssh root@<node-to-remove> "shutdown -h now"

If the node is already dead — hardware failure, disk failure, whatever — this step is already done for you, and you'll go through the same removal command below; the only difference is you can expect a specific harmless error message, covered in the troubleshooting section.

Step 6: Remove the Node From the Cluster

This is the actual removal. Run it from any remaining node in the cluster — never from the node you're removing:

pvecm delnode <nodename>

Use the short hostname exactly as it appears in pvecm status or pvecm nodes, not the FQDN or an IP address. On success, corosync's configuration is updated cluster-wide and the node is dropped from the corosync membership list. You can confirm this by checking /etc/pve/corosync.conf on a remaining node — the node's entry should be gone from the nodelist block.

If the node was already offline, you'll typically see output like:

Killing node 4
copy corosync auth key
stopping pve-cluster service
backup old database to '/var/lib/pve-cluster/backup/config-.sql.gz'
Purged local memberlist entry
Removed node 4 from corosync.conf

You may also see a message stating the node could not be contacted for a graceful shutdown — that's expected for an offline node and can be ignored; the important part is that it's removed from corosync.conf.

Handling a Quorum Loss Mid-Removal

In a small cluster (three nodes especially), removing a node can momentarily push the remaining nodes below quorum, particularly if you're removing more than one node in a short window, or if another node was already flaky. If pvecm delnode fails with a quorum-related error, temporarily lower the expected vote count on a remaining node:

pvecm expected 1

This tells corosync to consider quorum reached with just the votes currently present, letting you push the removal through. Re-run pvecm delnode <nodename> immediately after. Once the removal completes and the cluster's actual node count reflects the change, expected votes recalculate automatically on the next corosync restart — you don't need to manually revert this setting, but don't leave a cluster running indefinitely with an artificially lowered expected-votes value, since it weakens your protection against real split-brain scenarios.

Never use pvecm expected 1 as a routine workaround to avoid fixing an actually broken cluster — it's a scoped tool for pushing through this one operation, not a general quorum bypass.

Removing a Node That's Already Dead

If the node suffered a hardware failure and will never come back, you don't need to migrate anything from it (there's nothing to migrate to — it's gone) or gracefully power it off. Instead:

1. From a remaining node with quorum, check whether any VMs or CTs are still configured as belonging to the dead node — their configs live under /etc/pve/nodes/<dead-node>/qemu-server/ and /etc/pve/nodes/<dead-node>/lxc/. If you have backups and don't need to recover the running state, you can restore those guests onto a healthy node from your most recent backups.
2. Run pvecm delnode <dead-node> from a healthy node. Expect the "could not kill node" message mentioned above — it's harmless since the node isn't reachable anyway.
3. If quorum is already lost because the dead node's votes are missing, use pvecm expected 1 (or the appropriate lower number for larger clusters) as described above before retrying.
4. Proceed to the cleanup steps below — this matters more for a dead node than a gracefully removed one, since nothing about the dead node's shutdown was under your control.

Post-Removal Cleanup

pvecm delnode only handles corosync membership. It does not clean up the cluster filesystem or authentication artifacts left behind by the node. Do these manually on a remaining node:

Remove the node's directory from the cluster filesystem

rm -r /etc/pve/nodes/<nodename>

Double- and triple-check the node name here. This command is recursive, and running it against the wrong directory — or against a parent path — will delete VM and container configuration for guests that are still in use. If any VM or CT configs still live under that node's directory and you need them, move them to another node's directory (or restore from backup) before deleting.

Remove SSH host key entries

Every remaining node has an SSH known-hosts entry for the removed node, used for internal cluster SSH operations (migrations, etc.). Clean it up on each remaining node:

ssh-keygen -R <nodename>
ssh-keygen -R <node-ip>

Remove the node's public key from authorized_keys

Proxmox VE maintains a shared authorized_keys file for inter-node SSH at /etc/pve/priv/authorized_keys. Open it and remove the line corresponding to the removed node's key (it's typically commented or identifiable by the node's hostname in the key comment field).

Verify corosync.conf is clean

cat /etc/pve/corosync.conf

Confirm the removed node no longer appears in the nodelist block and that quorum settings (particularly expected_votes, if set explicitly) reflect the new node count.

Verifying Cluster Health After Removal

Once cleanup is done, confirm the cluster is in a healthy state:

pvecm status
pvecm nodes

pvecm status should show the correct total node count, all remaining nodes online, and Quorate: Yes. pvecm nodes lists current membership — the removed node should be completely absent, not shown as offline. If it still shows up as an offline member, the delnode step didn't fully propagate; re-check corosync.conf on all remaining nodes for consistency, since a node with a stale local copy of the config can reintroduce a phantom entry.

Also check the GUI — log into any remaining node's web interface and confirm the removed node no longer appears in the Datacenter tree. It may take a few seconds for the interface to refresh after removal.

Reusing the Removed Node

If you plan to repurpose the hardware — either as a standalone Proxmox VE host or to join a different cluster — the node itself still has leftover cluster state that needs clearing, since it was never told it was removed (it was powered off). Boot it back up, then on that node:

systemctl stop pve-cluster
systemctl stop corosync
pmxcfs -l
rm /etc/pve/corosync.conf
rm -rf /etc/corosync/*
rm -rf /var/lib/corosync/*
rm -rf /etc/pve/nodes/*
killall pmxcfs
systemctl start pve-cluster

This resets the node's cluster filesystem to a standalone state. After this, it behaves like a fresh Proxmox VE install as far as clustering is concerned, and you can either use it standalone or join it to a different cluster with pvecm add <cluster-ip>. If you have any doubt about the node's state, a clean reinstall of Proxmox VE on it is often faster and safer than trying to fully unwind cluster state by hand — especially if it was also running Ceph, since Ceph leaves its own disk signatures on OSD devices that need wiping separately with ceph-volume lvm zap or a full disk wipe.

Common Errors and What They Mean

Message	Meaning	What to do
"could not kill node (error = CS_ERR_NOT_EXIST)"	The node was already offline when you ran delnode.	Harmless — the node is removed from corosync.conf regardless. Ignore it.
"Unable to get corosync config" / quorum errors	Remaining nodes lost quorum, often because removal briefly changed vote counts.	Run pvecm expected 1 on a remaining node, then retry delnode.
Node still listed in pvecm nodes as offline	corosync.conf wasn't fully propagated to all remaining nodes.	Compare /etc/pve/corosync.conf across nodes; restart corosync and pve-cluster on any node with a stale copy.
Replication job stuck referencing the removed node	The job wasn't deleted before removal.	Delete it via pvesr delete <vmid>-<jobid>, or through the API if the CLI refuses due to the missing node.
Ceph reports degraded/undersized PGs after removal	OSDs on the removed node weren't drained and destroyed in Ceph before the Proxmox-level removal.	This is a Ceph-side problem, not a Proxmox one — check ceph osd tree and remove any remaining down/out OSDs referencing the dead node.

FAQ

Can I remove a node without powering it off first?

Technically pvecm delnode will still run against an online node, but the official guidance is to power it off first specifically to prevent it from reconnecting mid-removal and creating a split-brain state in corosync. If the node is going to be wiped or repurposed anyway, shutting it down first costs nothing and removes an entire class of edge cases.

Does removing a node delete its VMs and containers?

No — not automatically. Guest configuration files live under /etc/pve/nodes/<nodename>/ and are only deleted if you explicitly run rm -r on that directory during cleanup. If you haven't migrated the guests elsewhere first, their configs (and any local-only disk data) are effectively orphaned once you remove that directory, so migrate or back up anything you need before cleanup.

What's the minimum cluster size where node removal is risky?

Two- and three-node clusters are the most sensitive, since losing a single node's vote can push you below quorum immediately. This is exactly the scenario a QDevice is designed to prevent — if you're regularly adding and removing nodes in a small cluster, running a QDevice for tie-breaking quorum votes is worth setting up separately.

Do I need to remove the node from Ceph even in a non-hyper-converged setup?

Only if the node runs any Ceph daemons at all (OSD, MON, MGR, MDS). A node that only consumes Ceph storage as a client (mounted via the Ceph storage plugin, running no local Ceph services) has nothing to remove on the Ceph side — you only need to worry about Proxmox's own cluster membership.

Can I remove multiple nodes at once?

You can run pvecm delnode multiple times in sequence, but do it one node at a time and confirm cluster quorum and health between each removal, especially in smaller clusters. Removing several nodes back-to-back without checking state in between is a common way to accidentally strand the cluster below quorum with no easy path back.

Conclusion

Node removal in Proxmox VE is a one-line command at its core — pvecm delnode <nodename> — but the safety of the operation lives entirely in what you do before and after that command. Migrate or back up workloads, tear down replication jobs and Ceph roles first, power the node off, then run the removal and follow through with cleanup of the cluster filesystem, SSH keys, and authorized_keys. Skipping the prep work rarely breaks the removal itself; it just leaves debris that surfaces as confusing errors days or weeks later. Do the checklist in order, verify cluster health with pvecm status when you're done, and the whole process is uneventful even on a cluster that's actively serving production workloads.