Speeding Up HDD-Backed ZFS Pools in Proxmox VE with a Special Device

Introduction

If you're running Proxmox VE on a pool of spinning disks, you already know the pain: zfs list -t snapshot takes forever, du -sh on a dataset with a few million files crawls, and Proxmox Backup Server garbage collection jobs that should take twenty minutes stretch into three hours. None of that is a bug. It's what happens when metadata and small files share the same slow rotational vdevs as your bulk data. A ZFS special device fixes this by giving metadata (and optionally small blocks) a dedicated, fast home — usually a mirrored pair of SSDs sitting next to your HDD vdevs.

I've added special vdevs to a handful of production PVE and PBS pools over the past couple of years, and the difference is not subtle. A PBS datastore that used to take four hours to garbage-collect dropped to under thirty minutes after moving metadata off spinning rust. This article walks through why that happens, how to diagnose whether your pool would benefit, and exactly how to add and tune a special device without breaking anything.

Problem Overview

ZFS stores two broad categories of data in a pool: the actual file/block contents, and metadata — directory entries, block pointers, dnodes, the dedup table if you have dedup enabled, and (if you configure it) small file blocks below a size threshold. On a pool built entirely from HDD vdevs, all of that metadata lives on the same disks as your large sequential writes. Metadata access is inherently random: opening a directory, stat'ing a file, walking a snapshot tree — these generate a storm of small, scattered IOPS that mechanical disks handle poorly.

This matters more on Proxmox than on a generic file server for one specific reason: Proxmox Backup Server. A PBS datastore's chunk store is, structurally, millions of small files spread across a two-level directory hash. Garbage collection has to stat every chunk in the pool to check its atime, and verify jobs read chunk headers across the whole tree. On HDD-backed ZFS, that workload is close to a worst case. VM and container storage on ZFS is a smaller version of the same problem — every snapshot, clone, and directory listing on your local-zfs pool pays the same random-IO tax.

Symptoms

You'll usually notice this before you diagnose it. The tells are pretty consistent:

• PBS garbage collection or verify jobs that take hours on a datastore that isn't even that full.
• zfs list -t snapshot or zfs list -r pausing for several seconds per pool.
• pvesm status hanging briefly when it queries a ZFS-backed storage.
• High %util and await on the HDDs in iostat -x 1 during operations that shouldn't be data-heavy — deletes, directory walks, snapshot creation.
• The GUI's storage view or the Datacenter summary feeling sluggish specifically when ZFS pools are involved, while CPU and network stay idle.

None of these alone proves you have a metadata bottleneck — a failing disk or a full pool can cause similar symptoms — but the pattern of "small/random operations are slow, large sequential ones are fine" is the signature to look for.

Root Cause

The root cause is architectural, not a misconfiguration: metadata and data compete for the same physical IOPS on your HDD vdevs. Every stat, every directory read, every block pointer traversal is a random 4K-ish read or write on a device that's good at maybe 100-150 random IOPS. Compare that to even a mid-range SATA SSD doing tens of thousands. ARC (ZFS's in-memory cache) absorbs a lot of read-side metadata traffic if you have enough RAM, but it doesn't help with writes, and it doesn't help once your working set of metadata exceeds ARC size — which happens fast on a PBS datastore with a few TB of chunks.

L2ARC doesn't really solve this either. It's a read cache, it warms up slowly, and by default it doesn't even cache metadata unless you explicitly set secondarycache=metadata on the pool or datasets. A special vdev is different: it's not a cache, it's a persistent, mirrored tier that stores metadata (and optionally small blocks) as primary data, all the time, for both reads and writes.

There's a secondary effect worth understanding too. If you have deduplication enabled anywhere on the pool, the dedup table (DDT) is itself metadata, and a poorly performing DDT lookup on spinning disk is one of the classic ways a ZFS pool grinds to a halt during writes. Moving the DDT onto a special vdev doesn't make dedup free, but it removes the single biggest reason dedup on HDD pools gets a bad reputation. Even without dedup, every ARC eviction of a metadata block means the next access to that directory entry or block pointer has to come from disk — and on a pool with more files than your RAM can comfortably index, that eviction happens constantly.

Diagnosis

Before adding hardware, confirm the bottleneck is actually metadata IO and not something else — a resilver in progress, a dying disk, or a genuinely undersized pool. Start with:

zpool status -v
zpool iostat -v 1

Watch the per-vdev IOPS during a slow operation (kick off a zfs list -t snapshot -r rpool or a PBS GC run in another terminal). If you see a wall of small read/write ops hitting your HDD vdevs with low throughput but high op counts, that's your metadata traffic. Compare against a large sequential copy — if that runs at expected throughput, the disks themselves are fine; it's the access pattern that's the problem.

Check ARC pressure with arc_summary (installed via the zfsutils-linux package, which ships on every PVE node):

arc_summary | grep -A5 "ARC size"

A low hit ratio combined with a metadata-heavy workload is a strong signal. Also check whether the pool already has secondarycache set to metadata-only anywhere — sometimes people set this hoping L2ARC will fix the problem, and it helps a little but not nearly as much as a special vdev.

Finally, confirm you actually have (or can get) spare SATA/NVMe ports or a PCIe slot for two identical fast drives. Skip ahead if you don't — there's no way to add a special vdev without new hardware, and there's no safe way to do it with a single, unmirrored device.

A quick reference for what to look at during zpool iostat -v 1, and what each pattern tends to mean:

Step-by-Step Solution

The core operation is a zpool add with the special vdev type. It can be done live, on an existing production pool, without downtime — but the special vdev has to be a mirror. This is not optional the way it might be for a plain data vdev: if an unmirrored special device dies, every dataset that references metadata on it becomes unreadable, and the whole pool goes with it.

1. Pick two identical, enterprise-grade SSDs (or NVMe drives) with power-loss protection. Consumer drives without PLP can silently corrupt metadata on an unclean shutdown — this is the one place I wouldn't cut corners.
2. Identify stable device paths under /dev/disk/by-id/ rather than /dev/sdX, since device names can shift across reboots.
3. Add the mirrored special vdev to the existing pool:

   zpool add rpool special mirror /dev/disk/by-id/ata-SSD1_serial /dev/disk/by-id/ata-SSD2_serial

4. Confirm it landed correctly:

   zpool status rpool

5. Decide whether small file blocks (not just metadata) should also go to the special device, and set the threshold per dataset:

   zfs set special_small_blocks=32K rpool/data

6. Give it time. New metadata writes go to the special vdev immediately, but existing metadata on the HDD vdevs only migrates as blocks get rewritten — a rewrite-heavy dataset (like an active PBS datastore doing daily GC) will "warm up" within days; a mostly static archive dataset may take much longer, or you can force it by rewriting data (e.g. zfs send | zfs receive into a fresh dataset).

For a brand-new pool, you'd fold the special vdev into the create command instead of adding it after the fact — see the configuration examples below.

Commands

The commands you'll actually use, in the order you'll typically run them:

# List disk IDs to find stable paths
ls -l /dev/disk/by-id/ | grep -v part

# Check current pool layout
zpool status rpool

# Add a mirrored special vdev to an existing pool
zpool add rpool special mirror /dev/disk/by-id/nvme-Model_S1 /dev/disk/by-id/nvme-Model_S2

# Verify the new vdev is online
zpool status -v rpool

# Check special vdev capacity usage separately from the pool total
zpool list -v rpool

# Set the small-block threshold on a dataset (0 disables, default)
zfs get special_small_blocks rpool/data
zfs set special_small_blocks=32K rpool/data

# Watch metadata migrate onto the special vdev over time
zpool iostat -v rpool 5

# Confirm ashift matches across the special mirror and the main vdevs
zdb -C rpool | grep ashift

If you're building this into a fresh install rather than retrofitting an existing pool, do it at pool creation time so there's no migration wait:

zpool create -o ashift=12 rpool \
  mirror /dev/disk/by-id/hdd1 /dev/disk/by-id/hdd2 \
  special mirror /dev/disk/by-id/ssd1 /dev/disk/by-id/ssd2

Configuration Examples

A pool with a special vdev shows up in zpool status as its own top-level entry, distinct from the regular mirror/raidz vdevs:

  pool: rpool
 state: ONLINE
config:

	NAME                        STATE     READ WRITE CKSUM
	rpool                       ONLINE       0     0     0
	  mirror-0                  ONLINE       0     0     0
	    ata-HDD1_Z1D2AB3C       ONLINE       0     0     0
	    ata-HDD2_Z1D2AB4D       ONLINE       0     0     0
	special
	  mirror-1                  ONLINE       0     0     0
	    nvme-SSD1_S4EWNX0N1234  ONLINE       0     0     0
	    nvme-SSD2_S4EWNX0N5678  ONLINE       0     0     0

errors: No known data errors

For a Proxmox Backup Server datastore sitting on this pool, the /etc/proxmox-backup/datastore.cfg entry doesn't need to reference the special vdev at all — it's transparent to PBS, since it's a ZFS-level construct beneath the filesystem PBS writes to:

datastore: hdd-pool-ds
	path /rpool/pbs-datastore
	gc-schedule daily
	prune-schedule daily
	comment PBS datastore on HDD pool with SSD special vdev

If you want small chunk files (PBS chunks smaller than a threshold, or VM disk blocks under a certain size) to land on the special device too, size the threshold to your recordsize. For a PBS chunk store where most chunks are near the 4 MB default, small_blocks won't do much — the win there is almost entirely from metadata. For a general-purpose VM storage dataset with a smaller recordsize, something like:

zfs set recordsize=128K rpool/data/vm-storage
zfs set special_small_blocks=16K rpool/data/vm-storage

keeps small, scattered blocks (config files, database index writes) on fast media while bulk sequential VM disk IO stays on the HDDs where capacity is cheap.

Common Mistakes

The most common and most damaging mistake is adding an unmirrored special device. A single SSD as a special vdev works — ZFS won't stop you — right up until that SSD fails, at which point the pool is gone, not degraded. Always mirror it, no exceptions.

Undersizing the special device is the second issue I see a lot. Metadata usage scales with file count, not pool capacity — a PBS datastore with millions of small chunks needs proportionally more special vdev space than a VM storage pool with a handful of large zvols, even at the same total size in TB. As a starting point, 0.3% to 1% of pool capacity per TB is a reasonable floor for pure metadata, but push closer to 2-3% if you're also using special_small_blocks to offload small file data. Run out of space on the special vdev and ZFS quietly spills new metadata back onto the HDD vdevs — you don't lose data, but you silently lose the performance benefit you paid for.

People also forget that special_small_blocks can't exceed the dataset's recordsize — ZFS will reject or clamp it. And it's easy to assume adding the special vdev retroactively speeds things up instantly; it doesn't touch existing metadata until those blocks get rewritten, so a freshly-added special vdev on a mostly idle pool can look disappointing for weeks.

Finally, watch ashift. If your HDD vdevs were created with ashift=12 and you add a special mirror with drives that report a different sector size, you can end up with a mismatch that wastes space or hurts performance on the special vdev. Set ashift explicitly with -o ashift=12 when adding the vdev rather than trusting autodetection.

Best Practices

Use enterprise SSDs or NVMe drives with power-loss protection for the special mirror — this is where I'd never compromise, because metadata corruption from a bad write on power loss can be far more damaging than losing a chunk of bulk data. A three-way mirror is worth considering if the special vdev is protecting a datastore you genuinely can't afford to lose, since resilvering a two-way mirror after one drive fails leaves you running on a single point of failure until it completes.

Size generously rather than exactly. Special vdev capacity is cheap relative to what it protects against — a full special vdev degrades silently back to HDD-speed metadata, and you probably won't notice until GC times creep back up. Monitor usage the same way you'd monitor any capacity-constrained resource:

zpool list -v rpool

Set special_small_blocks deliberately per dataset rather than pool-wide. A PBS chunk-store dataset gets little benefit from it since chunks are large; a general VM storage dataset with lots of small random writes benefits more. Test with a conservative value (16K-32K) before going higher — pushing too much small data onto the special vdev defeats the point of having cheap HDD capacity for bulk storage in the first place.

Don't skip the migration window. If you need the performance win immediately — say, right before a big backup run — consider rewriting the dataset's existing data (a zfs send to a new dataset and back, or simply letting a full resync happen) rather than waiting for organic rewrites to slowly populate the special vdev.

And back up your pool configuration and zpool status output somewhere outside the pool itself before you touch vdev layout. Adding a special vdev is one of the few ZFS operations that, once done, can't be casually undone on older OpenZFS versions or if your remaining vdevs include raidz.

Wire capacity monitoring into whatever you're already using for node alerting rather than relying on someone remembering to run zpool list -v by hand. A special vdev at 85% full is still fine; one that hits 95% and starts spilling metadata back onto the HDD vdevs will quietly undo the entire point of the exercise, and the symptom you'll see — GC times creeping back up — looks identical to the original problem, which makes it confusing to re-diagnose months later if you've forgotten the vdev exists.

FAQ

Can I remove a special vdev once it's added?

Only if the rest of the pool's top-level vdevs are mirrors or single disks — not raidz. If any vdev in the pool is raidz, zpool remove won't work on the special mirror either, and you're committed to that layout.

Does a special vdev help an all-SSD or all-NVMe pool?

Not much. The whole benefit comes from moving random IO off slow media onto fast media. If your main vdevs are already NVMe, there's little headroom left to gain, and the extra hardware is better spent elsewhere.

What size should the special mirror be?

Start around 0.3-1% of total pool capacity for metadata-only use, and go higher, toward 2-3%, if you're also offloading small blocks with special_small_blocks. File count matters more than raw pool size — a PBS datastore with millions of chunks needs more headroom than a VM pool with a few large zvols of the same total capacity.

Will this help my PBS garbage collection times specifically?

Yes, usually the most dramatic improvement you'll see. GC has to stat every chunk in the datastore, which is almost pure metadata IO — exactly what a special vdev is built for.

Do I need to reboot the node after adding the special vdev?

No. zpool add is an online operation. Existing VMs, containers, and PBS jobs keep running throughout.

Conclusion

A mirrored special vdev is one of the highest-value upgrades you can make to an HDD-backed ZFS pool on Proxmox, and it's one of the few storage changes that doesn't require rebuilding the pool from scratch. Two enterprise SSDs and a single zpool add command turn a metadata-bound pool into something that behaves like flash for the operations that actually make PVE and PBS feel slow day to day — directory listings, snapshot management, and garbage collection. Get the mirroring and sizing right up front, give it time to migrate existing metadata, and check zpool list -v periodically so you notice before it fills up rather than after your GC times creep back to where they started.