How to Replace a Failed Disk in a Proxmox VE ZFS Pool (Step-by-Step Guide)

You're staring at a red icon in the Proxmox web interface, and your ZFS pool says DEGRADED instead of ONLINE. Maybe you got an email at 3 a.m. from your node. Maybe you just noticed it during a routine check. Either way, one of your disks has failed, and now you need to swap it out without losing your VMs.

The good news: this is exactly the situation ZFS mirrors and RAIDZ arrays are built for. A single failed disk shouldn't mean a rebuild from backup. It should mean twenty minutes of careful work and a resilver that runs in the background while your VMs keep running.

This guide walks through the whole process on Proxmox VE 9.x, from figuring out which physical disk actually failed to watching the resilver finish and clearing the old error counts. We'll cover both a plain data pool and the trickier case of replacing a disk in your boot pool (rpool).

What You Will Learn

• How to read zpool status and tell which disk actually failed
• How to identify a physical drive by serial number so you don't pull the wrong one
• The exact commands to offline, physically swap, and replace a failed disk
• What's different when the failed disk is part of your Proxmox boot pool
• How to watch a resilver and confirm the pool is healthy again
• Common error messages during replacement and what they actually mean

What Is This Feature?

ZFS is the storage system built into Proxmox VE, and it does something most other filesystems don't: it manages both the filesystem and the underlying disks together, instead of relying on a separate RAID controller. A ZFS pool is a group of physical disks organized into one or more vdevs (virtual devices) — usually mirrors (like RAID 1) or RAIDZ (similar in spirit to RAID 5/6, but with checksums for every block).

When a disk in a mirror or RAIDZ vdev dies, the pool doesn't go down. It keeps serving data from the surviving disks, just in a DEGRADED state instead of ONLINE. Once you swap in a replacement disk, ZFS runs a process called resilvering — it reads the data that should exist on the new disk from the healthy disks and writes it across, block by block, until the new disk is fully in sync.

This is different from a traditional RAID rebuild in one useful way: ZFS only resilvers data that's actually in use, not every sector of the disk. On a pool that's half empty, that can mean the difference between a two-hour rebuild and an eight-hour one.

Why Would You Use It?

You don't get a choice about needing this — disks fail. Spinning hard drives fail more often as they age, but SATA and NVMe SSDs die too, usually more abruptly. If you're running Proxmox with any kind of ZFS mirror or RAIDZ, at some point a drive in that array is going to drop out, and you'll need to replace it without tearing the whole pool down.

The alternative — restoring every VM from your last Proxmox Backup Server snapshot — works, but it means downtime and you lose anything written since the last backup ran. Replacing the failed disk in place keeps your VMs running (or gets them back up in minutes if the pool briefly went offline) and preserves everything, including changes from an hour ago.

Honestly, if you're running ZFS in anything other than a mirror or RAIDZ — a single disk, "stripe," whatever you want to call it — none of this applies to you. A single failed disk in a striped or single-disk pool just means the pool is gone. That's the whole reason redundant vdevs exist.

Prerequisites

• A Proxmox VE 9.x host (the commands below also work on 8.x — ZFS replacement hasn't changed) with a ZFS pool using a mirror or RAIDZ vdev
• Root shell access, either through the Proxmox web console (Node → Shell) or SSH
• A replacement disk of the same size or larger than the failed one — smaller won't be accepted, even by a few megabytes
• Physical access to the server, or remote hands if it's colocated, since someone has to actually pull and insert a drive
• A recent backup (you should already have this, but if a pool goes fully degraded and you're not sure your backups are current, this is the moment to check before you touch anything)

If you've never looked at zpool status before, it's worth reading through the What Is ZFS in Proxmox VE? article first for the basics on pools and vdevs. This guide assumes you already know what a mirror is and jumps straight into the repair.

Step-by-Step Tutorial

Step 1: Confirm the pool is actually degraded

Open a shell on the node and run:

zpool status -v

Look for a pool state of DEGRADED and a device listed as FAULTED, UNAVAIL, or REMOVED. A healthy pool shows every device as ONLINE with zero in the READ, WRITE, and CKSUM columns. A few CKSUM errors on an otherwise ONLINE disk isn't the same problem — that can be a bad cable, not a dying drive. This guide is specifically for a disk that's actually dropped out of the pool.

Step 2: Identify the physical disk by serial number

This is the step people rush and regret. The device names ZFS shows you, like sda or sdc, can shift between reboots. Pulling the wrong drive out of an 8-bay chassis because you guessed based on sdb is a genuinely common mistake, and it turns a one-disk problem into a two-disk problem.

Instead, match the pool's device path to a serial number:

ls -la /dev/disk/by-id/ | grep sdc

This shows you the by-id name (which usually includes the model and serial number) that maps to whatever /dev/sdX device ZFS listed. Write that serial number down, then check it against a label on the physical drive tray or use lsblk -o NAME,SERIAL,SIZE to double check. If your chassis has per-drive activity LEDs, you can also run zpool offline on the disk (next step) and watch which LED goes dark.

Step 3: Offline the failed disk (if it isn't already)

If ZFS already shows the disk as FAULTED or REMOVED, you can usually skip straight to swapping it. If the disk is still technically responding but throwing errors, take it offline cleanly first:

zpool offline tank /dev/disk/by-id/ata-WDC_WD40EFRX-xxxxx

Replace tank with your pool's actual name (check it with zpool status if you're not sure — Proxmox's installer defaults to rpool for the boot pool).

Step 4: Physically swap the drive

Most consumer SATA setups without a proper hot-swap backplane aren't safe to pull live, even though SATA is technically hot-pluggable in theory. If you've got an enterprise hot-swap chassis with drive trays and the OS already sees the disk as offline or removed, you can usually pull and replace without shutting down. If you're on a desktop tower with SATA cables run straight to the motherboard, power the machine down first. It's not worth the risk of yanking the wrong signal timing and corrupting a different disk's connection mid-write.

Insert the new disk of equal or larger capacity, boot back up if you powered off, and confirm the system sees it with lsblk.

Step 5: If this is your boot pool, prepare the new disk's partition table

Skip this step for a plain data pool — ZFS will happily consume a blank disk directly. But if you're replacing a disk in rpool (the pool Proxmox itself boots from), the disk needs a small EFI System Partition alongside the ZFS partition, or the server won't boot from it later. Clone the partition layout from a surviving healthy disk in the pool:

sgdisk /dev/disk/by-id/ata-healthy-disk -R /dev/disk/by-id/ata-new-disk
sgdisk -G /dev/disk/by-id/ata-new-disk

The first command copies the partition table from the healthy disk onto the new one. The second regenerates the new disk's GUIDs so it doesn't collide with the disk you copied from.

Step 6: Run the replace

For a data pool:

zpool replace -f tank /dev/disk/by-id/ata-old-disk /dev/disk/by-id/ata-new-disk

For the boot pool, point it at the ZFS partition specifically (usually partition 3 on a Proxmox-installed disk):

zpool replace -f rpool /dev/disk/by-id/ata-old-disk-part3 /dev/disk/by-id/ata-new-disk-part3

The -f flag forces the replace even if ZFS thinks the new disk might already contain a filesystem — which it often will, since brand new drives sometimes ship pre-formatted, or you're reusing a disk that had a partition table on it before.

Step 7: Watch the resilver

zpool status -v tank

You'll see a line like resilver in progress since ... along with a percentage and an estimated time remaining. On a 4 TB drive that's maybe 60% full, expect somewhere around two to six hours depending on disk speed and how busy the pool is with other work — spinning rust mirrors resilver noticeably slower than SSDs. The pool stays usable the entire time; your VMs keep running, just with slightly higher disk latency than usual.

Step 8: If this was the boot pool, reinitialize the bootloader

This step trips people up constantly, and it's the reason a boot pool replacement isn't quite as simple as a data pool one. ZFS resilvering the data doesn't put a bootloader on the new disk — you have to do that separately, or the server won't boot if the surviving old disk ever fails too:

proxmox-boot-tool status
proxmox-boot-tool format /dev/disk/by-id/ata-new-disk-part2
proxmox-boot-tool init /dev/disk/by-id/ata-new-disk-part2

proxmox-boot-tool status lists which ESPs (EFI System Partitions) Proxmox currently knows about. format creates a FAT filesystem on the new disk's ESP, and init registers it and installs the bootloader. Run status again afterward and confirm the new disk shows up.

Step 9: Confirm the pool is healthy and clear old errors

Once the resilver finishes, run zpool status -v one more time. You want to see ONLINE across the board. If the old error counts from before the replacement are still showing on other disks, clear them so future problems are easy to spot:

zpool clear tank

Commands Explained

Command	What it does
zpool status -v	Shows the health of every pool, vdev, and disk, plus any read/write/checksum errors
zpool offline <pool> <disk>	Marks a disk as offline so ZFS stops trying to use it, without removing it from the pool config
zpool replace -f <pool> <old> <new>	Swaps a disk out of a vdev and starts resilvering the new one
zpool clear <pool>	Resets the error counters shown in zpool status
lsblk -o NAME,SERIAL,SIZE	Lists block devices with their serial numbers, useful for matching a drive bay to a pool device
sgdisk -R / sgdisk -G	Clones a partition table onto a new disk, then regenerates its unique GUIDs
proxmox-boot-tool status / format / init	Manages the EFI boot partitions Proxmox uses to boot from a ZFS mirror

Common Errors

"invalid vdev specification: use '-f' to override the following errors: /dev/sdc1 contains a filesystem" — ZFS found an existing partition table or filesystem signature on the new disk and is refusing to overwrite it silently. This is normal on a reused or pre-formatted drive. Add the -f flag as shown above, after confirming you've got the right disk.

"cannot replace X with Y: devices have different sector alignment" — usually means the new disk is smaller, or reports a different logical block size (512 vs 4Kn), than the one you're replacing. You need an equal-size or larger disk with a matching sector size.

"one or more devices is currently unavailable" during replace — the old disk hasn't actually been marked offline or removed yet. Run zpool offline first, or check that the drive is properly seated.

System won't boot after a boot pool disk replacement — almost always means Step 8 got skipped. The ZFS data resilvered fine, but the new disk never got a bootloader installed on it. If the other original disk is still healthy, boot from that and run the proxmox-boot-tool commands afterward.

Troubleshooting

If the resilver seems stuck at the same percentage for a long time, check dmesg for disk I/O errors — a "failing" replacement disk isn't unheard of, especially if it's been sitting on a shelf for years. Reseating the cable or trying a different SATA port rules out a connection issue before you assume the new drive itself is bad.

If the new disk shows up as UNAVAIL right after you insert it, that's almost always power or cabling, not ZFS. Check both ends of the SATA and power cables, and confirm the drive spins up (you should be able to feel or hear it).

If zpool status still lists the old, physically-removed disk after you've inserted the new one and it hasn't automatically taken its place, that's expected — Proxmox doesn't enable ZFS's autoreplace property by default. You have to run zpool replace manually, which is exactly what this guide walks through.

If checksum errors keep climbing on a disk you already replaced, suspect the controller, backplane, or cable rather than the drive itself. A brand new disk showing errors within minutes is a red flag for the connection, not the hardware you just installed.

Best Practices

• Always reference disks by their /dev/disk/by-id/ path in ZFS commands, not /dev/sdX — device letters can and do change across reboots
• Keep at least one spare disk of the right size on hand if you're running anything you actually care about — waiting three days for a drive to ship while running degraded is asking for a second failure
• Check zpool status periodically, or set up email alerts for pool state changes, so you find out about a degraded pool from Proxmox and not from a VM crashing
• Don't run a pool in DEGRADED state longer than you have to — the surviving disks are doing extra work, and a second failure during that window means real data loss on RAIDZ1 or a two-disk mirror
• Label your physical drive bays with their by-id serial numbers ahead of time, before anything fails — figuring this out during an actual outage is the worst time to do it

Frequently Asked Questions

Do I need to shut down Proxmox to replace a failed ZFS disk?

Only if your hardware doesn't support hot-swapping SATA or SAS drives. A proper hot-swap backplane lets you pull and insert drives live; a desktop tower with cables run to the motherboard doesn't.

How long does a resilver take?

It depends on how much data is actually stored and how fast the disks are — a mostly-empty pool resilvers in minutes, while a nearly-full multi-terabyte mirror on spinning disks can take most of a day.

Can I replace a failed disk with a smaller one?

No. ZFS requires the replacement to be at least as large as the disk it's replacing, even if the difference is a few megabytes.

What happens if I have no redundancy and a disk fails?

If the pool is a single disk or a plain stripe with no mirror or RAIDZ, a failed disk means the pool is gone. There's nothing to resilver from. This is why single-disk ZFS pools aren't a substitute for backups or redundancy.

Does Proxmox replace a failed disk automatically?

No. The autoreplace ZFS property is off by default, so you'll always need to run zpool replace yourself after swapping the drive.

Will my VMs slow down during the resilver?

Somewhat, yes. The pool stays fully usable, but the extra read/write load from resilvering competes with normal VM disk activity, so expect higher latency until it finishes.

Conclusion

A degraded ZFS pool looks alarming the first time you see it, but the actual fix is mechanical once you know the steps: confirm which disk failed, match it to a physical drive by serial number, swap it, run zpool replace, and watch the resilver finish. The only part that regularly catches people out is the boot pool case, where you also need to reinitialize the bootloader — so if you're replacing a disk in rpool, don't skip that step.

This is also a good moment to double check your setup going forward. If you're not already labeling drive bays with their serial numbers, or you don't have a spare disk sitting on a shelf, this is the cheapest time to fix that — before the next disk decides to fail at 3 a.m. on a Friday.