How to Set Up Storage Replication Between Proxmox VE Nodes (ZFS-Based, No Shared Storage Required)

Introduction

If you've read anything about Proxmox VE clustering, you've probably run into two very different pieces of advice: "just use Ceph" and "just use shared storage." Both are good advice if you already have a SAN, a Ceph cluster, or the hardware budget for either. A lot of homelabs and small production clusters don't. What most of those setups do have is two or three nodes, each with its own local ZFS pool, and a VM that needs to survive a node dying without anyone driving to the datacenter at 2 a.m.

That's exactly the gap storage replication fills. Instead of one shared copy of a disk that every node can reach, replication keeps a full local copy of the guest's disks on one or more other nodes, and quietly re-syncs the differences on a schedule using ZFS snapshots. No SAN, no Ceph, no shared filesystem — just periodic zfs send under the hood, orchestrated by a tool called pvesr.

This guide walks through setting up storage replication from a clean two-node cluster: matching up storage on both sides, creating your first replication job from the GUI and the CLI, watching the initial sync happen, tuning the schedule and bandwidth, and wiring replication into HA so a guest actually fails over automatically when a node goes down. We're using Proxmox VE 9.x for the screenshots and commands below, but everything here has worked unchanged since replication was introduced back in PVE 5.0.

What You Will Learn

• What Proxmox VE storage replication actually does, and how it's different from a backup or a Ceph pool
• The exact storage requirements both nodes need before a replication job will even save
• How to create a replication job from the Proxmox VE web interface
• How to create and manage the same job entirely from the command line with pvesr
• How to set a schedule and a bandwidth rate limit so replication doesn't compete with production traffic
• How replication pairs with HA to give you automatic failover
• How to read pvesr status output and fix the replication failures you're most likely to hit

What Is Storage Replication?

Storage replication is a Proxmox VE feature that copies a VM's or LXC container's disks from one node to another node's local storage on a recurring schedule, so a near-current copy of the guest already exists elsewhere in the cluster before anything goes wrong. It's implemented as a thin, cluster-aware wrapper — the pvesr command and its systemd timer — around ZFS's native snapshot and send/receive mechanism.

The first replication run does a full zfs send of the guest's disk to the target node — this can take a while depending on disk size and network speed. Every run after that only sends the blocks that changed since the last successful sync, by diffing two ZFS snapshots and sending just the delta with zfs send -i. That's why a guest with a 200 GB disk and a 1% daily change rate can replicate every 15 minutes without saturating a gigabit link: after the first sync, there usually just isn't that much new data to move.

It's worth being clear about what replication is not. It is not a backup — there's no retention of independent restore points beyond the last couple of synced snapshots, and a mistake made on the source (an rm -rf inside the guest, a bad partition table) replicates straight to the target on the next run. It's also not Ceph or shared storage — the target copy only becomes the "live" copy when you deliberately migrate to it or when HA fails the guest over to it; the two nodes don't serve reads and writes from the same disk simultaneously the way a real shared storage backend does.

Why Would You Use It?

The honest answer is: because you don't have shared storage, but you still want fast recovery if a node dies. A few concrete scenarios where replication earns its keep:

• Two or three-node clusters without a SAN. Ceph wants at least three nodes and real disks dedicated to OSDs to be reliable. Replication works fine on two nodes with plain local ZFS.
• HA without shared storage. Proxmox VE's HA manager can restart a failed guest on another node, but without shared or replicated storage, that guest's disk simply isn't there to restart from. Replication is what makes HA meaningful on a local-storage cluster.
• Fast live migration. Migrating a VM to a node it's already replicated to only has to send the delta since the last sync, not the entire disk — a multi-hundred-gigabyte VM that would take twenty minutes to migrate cold can move in seconds.
• Cheap disaster recovery for a subset of guests. You don't have to replicate everything. Pick the VMs that actually matter and leave test boxes and scratch containers off the schedule entirely.

Prerequisites

• At least two nodes in the same Proxmox VE cluster (replication is a cluster feature — it doesn't work between two standalone hosts).
• A ZFS pool on each node you want to replicate to or from. Replication only works with the zfspool storage plugin — it does not support LVM, directory storage, NFS, Ceph, or any other backend, because it relies on ZFS snapshots and zfs send/zfs receive specifically.
• The storage entry on both nodes must use the exact same storage ID (the name you gave it in Datacenter → Storage). If your ZFS storage is called local-zfs on node A, it needs to be called local-zfs on node B too — replication matches jobs to storage by ID, not by pool name or mount point.
• The guest's disks must all live on replicable (ZFS) storage. If a VM has one disk on local-zfs and a second disk on plain directory storage, replication will sync the first disk and simply skip the second — worth knowing before you assume the whole VM is covered.
• Root or a user with the PVEAuditor role on the guest and PVEDatastoreAdmin-equivalent rights on the storage, if you're not using root directly.
• A reasonably fast link between nodes for the first sync — after that, ongoing syncs are small, but the initial full copy moves as much data as the guest's used disk space.

Step-by-Step Tutorial

Step 1: Confirm both nodes have matching ZFS storage

Before you touch replication at all, check Datacenter → Storage and confirm the ZFS storage you want to replicate to is listed with Nodes set to include both the source and target — or "All" — and that it's actually type ZFS, not a directory storage sitting on top of a ZFS mountpoint (those look similar in the GUI but only the former works with replication).

From the shell, you can confirm the pool itself is present and healthy on the target node:

zpool list
zpool status rpool

If the target node doesn't have a ZFS pool with the same storage ID yet, add one first via Datacenter → Storage → Add → ZFS, pointing it at a pool that already exists on that node (or create the pool first with zpool create if you're starting from bare disks).

Step 2: Create a replication job from the GUI

Click the VM or container you want to protect in the resource tree, then open its Replication tab.

1. Click Add.
2. Set Target to the node you want the copy to live on.
3. Set a Schedule — the default is every 15 minutes, written as a calendar event string. You can leave the default or set something coarser like */30 for every 30 minutes, or 02:00 for once a night.
4. Optionally set a Rate Limit in MB/s so an initial full sync of a huge disk doesn't flood the link between nodes.
5. Click Create.

The job appears immediately with a status of "waiting" or "syncing" once the schedule triggers. You'll also find a datacenter-wide view of every replication job, its last run time, and its status under Datacenter → Replication — that's the fastest place to check on everything at once instead of clicking into each guest individually.

Step 3: Create the same job from the command line

Everything the GUI does maps directly onto pvesr. To replicate guest 100 to node pve2 every 5 minutes with a 10 MB/s rate cap:

pvesr create-local-job 100-0 pve2 --schedule "*/5" --rate 10 --comment "replica for pve2 failover"

The job ID, 100-0, follows the pattern <vmid>-<jobnum> — the -0 suffix lets a single guest have more than one replication job (for example, replicating the same VM to two different target nodes) without the IDs colliding.

List every job in the cluster and see when each one last ran:

pvesr list

Or check the detailed status for one guest specifically:

pvesr status --guest 100

Step 4: Watch the first sync happen

The first run of a new job is always a full sync — there's no earlier snapshot to diff against yet, so pvesr takes a fresh snapshot and streams the entire disk over with zfs send. You can force it to start immediately instead of waiting for the schedule:

pvesr schedule-now 100-0

Watch it in real time from either node:

journalctl -u pvesr.service -f

Once it finishes, a quick sanity check on the target node's pool will show the replicated dataset sitting there under a name like rpool/data/vm-100-disk-0, plus a snapshot named something like __replicate_100-0_<timestamp>__ that pvesr uses as the sync point for the next incremental run.

Step 5: Tune the schedule and bandwidth for ongoing syncs

A 5 or 15-minute schedule is a reasonable default for anything you'd actually fail over in an emergency. For less critical guests, stretching it out to hourly reduces both I/O load and the number of intermediate snapshots ZFS has to manage. You can change an existing job without recreating it:

pvesr update 100-0 --schedule "*/15" --rate 20

The schedule field uses systemd's calendar event syntax, the same format used elsewhere in Proxmox VE (backup jobs, timers). */15 means every 15 minutes; you can also use explicit times like Mon..Fri 22:00 for a nightly-only schedule on weekdays. The shortest supported interval is one minute; the longest gap between syncs is one week.

Step 6: Pair replication with HA for automatic failover

Replication by itself only keeps a copy up to date — it doesn't start the guest anywhere automatically. To get real failover, add the guest to an HA group that includes both the source and target nodes, via Datacenter → HA → Add, and set its state to Started.

With both replication and HA configured for the same guest, here's what actually happens if the source node dies: the HA manager on a surviving node detects the failure through the usual fencing and quorum checks, and starts the guest on the replication target using the most recent locally-available replicated data. You lose whatever changes happened between the last successful sync and the moment the node died — with a 5-minute schedule, that's a worst case of five minutes of writes, which is a very different failure mode than losing the guest entirely.

One detail that surprises people the first time they see it: after a failover (or a manual migration) to the replica target, Proxmox VE automatically reverses the replication direction. The node that used to be the target starts replicating back to whatever node the guest just came from, so you're covered again if it fails back later.

Step 7: Test a manual failover before you need a real one

Don't wait for an actual outage to find out replication was silently broken. With the guest still running normally on the source node, do a live migration to the replication target:

qm migrate 100 pve2 --online

Because the target already has almost all the data from the last replication sync, this migration should be noticeably faster than migrating to a node with no copy at all — that speed difference is a good sign replication has actually been working. Once you're satisfied, migrate it back the same way.

Commands Explained

Common Errors

• "target storage 'local-zfs' does not exist on node 'pve2'" — the storage ID on the target node doesn't match the source, or the storage isn't enabled for that node in Datacenter → Storage. Renaming storage is not supported; you have to add a new one with the matching ID.
• "volume ... is not a in a directly usable state (importing)" — usually appears mid-sync if a previous run was interrupted (node reboot, network blip). Let the next scheduled run retry; if it keeps failing, see the broken-snapshot-chain fix in the troubleshooting section below.
• "cannot receive incremental stream: destination ... has been modified since most recent snapshot" — something wrote to the replicated dataset on the target directly (a manual zfs rollback, someone booting the guest there without migrating it first). Replication assumes it's the only thing touching that dataset on the target.
• Job silently stuck at 0% with no progress for a long time — almost always a network or firewall problem between the two nodes rather than a ZFS issue. Replication traffic uses the migration network if one is configured, or the cluster's management network otherwise.
• "guest has (non-replicable) volume on storage 'local'" — one of the guest's disks lives on non-ZFS storage. Replication will still sync the disks it can and warn about the ones it can't; move the offending disk to ZFS storage if you need it fully covered.

Troubleshooting

Start every investigation with pvesr status --guest <vmid> — it tells you the job's fail count and the timestamp of the last successful sync, which immediately tells you whether you're looking at a one-off blip or a job that's been broken for days.

If the fail count is climbing and errors mention a broken snapshot chain, the cleanest fix is usually to let replication rebuild from scratch rather than trying to hand-patch ZFS snapshots:

1. Disable the job so it stops retrying on its schedule while you work: pvesr disable 100-0.
2. On the target node, remove the orphaned dataset so the next sync starts clean — double-check the dataset name first, this destroys whatever's currently there: zfs destroy -r rpool/data/vm-100-disk-0.
3. Re-enable the job: pvesr enable 100-0.
4. Force an immediate full resync: pvesr schedule-now 100-0.
5. Watch it complete: journalctl -u pvesr.service -f.
6. Confirm the fail count reset to zero: pvesr status --guest 100.

If replication seems to be running but taking far longer than it should, check whether a rate limit is set lower than you remember (pvesr list shows the configured rate for each job) and confirm the network path between nodes isn't accidentally routing through a slower link — clusters with a dedicated migration network sometimes fall back to the management network if that link goes down, and nobody notices until replication is crawling.

If a guest with an HA resource won't let you delete its replication job, that's expected — remove the HA resource first, or pass --force to pvesr delete if you're certain you want to break that link.

Best Practices

• Only replicate what you'd actually fail over. Every job adds ongoing I/O and snapshot churn on both ends; scratch VMs and disposable test containers don't need it.
• Match the schedule to how much data loss you can tolerate, not to what feels aggressive. A 5-minute schedule on a database VM with a slow disk can end up backed up behind its own previous run — watch the fail count after tightening a schedule.
• Set a rate limit on the first job you create for a large guest before it saturates the link between nodes during the initial full sync; you can loosen it once you've confirmed the network handles it fine.
• Replication is not a backup strategy on its own. Keep it paired with real Proxmox Backup Server or vzdump backups for anything you'd need to restore from a point further back than the last sync.
• Test failover on purpose, periodically — a replication job that's been silently failing for a month is worse than no replication at all, because it gives you false confidence.
• Keep storage IDs identical across every node in the cluster from day one, even nodes you haven't set up replication for yet — it saves a rename-that-isn't-really-possible headache later.

Frequently Asked Questions

Does storage replication work with Ceph or other shared storage?

No, and it doesn't need to. Replication exists specifically to solve the "no shared storage" problem. Ceph, NFS, and iSCSI-backed shared storage are already reachable from every node, so there's nothing to replicate — any node can already read and write the same disk.

Can I replicate one VM to more than one other node?

Yes. Create a separate job for each target — the guest ID's job number (the -0, -1 suffix) is exactly what lets multiple jobs coexist for the same guest, each pointed at a different node.

What happens to replication when I live-migrate a guest to a node it's already replicating to?

Migration only has to send the delta since the last replication sync instead of the whole disk, so it's much faster than a cold migration. Afterward, replication direction flips automatically — the old source becomes the new target.

How much data can I lose if a node dies right before a sync?

Up to whatever your schedule interval is, worst case. If your schedule is every 5 minutes, you can lose up to roughly 5 minutes of writes. That's why the schedule should reflect how much loss is actually acceptable for that specific guest, not just be left at the default everywhere.

Do I need Proxmox Backup Server if I already have replication set up?

Yes, for anything you care about. Replication protects you against a node failing; it does nothing for a bad update inside the guest, a ransomware event, or an accidental deletion — those mistakes replicate to the target just as faithfully as the good data does. Backups give you independent restore points further back in time; replication gives you fast recovery from hardware failure. They solve different problems.

Can I use replication on a two-node cluster without a QDevice?

Replication itself doesn't require quorum tooling. But if you're pairing it with HA for automatic failover, a two-node cluster needs a QDevice (or a third voting node) so the surviving node can reach quorum and actually take action when the other one disappears — otherwise it just sits there unable to confirm it's safe to start the guest.

Conclusion

Storage replication is one of those Proxmox VE features that quietly closes the gap between "single point of failure" and "full shared-storage cluster" without asking for a SAN or a Ceph deployment. Once matching ZFS storage exists on both nodes, creating a job takes one GUI dialog or one pvesr create-local-job command, and from then on it just runs in the background, snapshot by snapshot. Pair it with HA if you want failover to be automatic, keep it paired with real backups for anything further back than "a few minutes ago," and test a manual migration to the replica target every so often so you actually know it's working before a node failure forces the question.