Introduction
ZFS on Proxmox VE handles metadata the same way it handles everything else by default: striped or mirrored across whatever vdevs make up the pool, sitting on the same spinning disks or the same SSDs as your bulk data. That's fine until your workload does a lot of small-file operations, snapshot-heavy backups, or directory-heavy scans, and metadata lookups start competing with bulk I/O for the same disk heads. A special allocation class vdev — usually just called a "special vdev" — moves that metadata onto dedicated, faster storage, and it's one of the more effective ZFS tuning options that doesn't get much attention in PVE deployments.
This walks through what a special vdev actually does, when it's worth the added complexity, how to spot a workload that would benefit from one, and how to add one to an existing pool without rebuilding it from scratch.
Problem Overview
ZFS stores three broad categories of data: the actual file contents (data blocks), metadata describing the filesystem structure (indirect blocks, dnodes, directory entries), and optionally deduplication tables if dedup is enabled. On a standard pool, all three land on the same vdevs as your regular data. If that pool is built on spinning disks — a common setup for bulk Proxmox Backup Server storage or archival datastores — every metadata lookup competes with sequential data reads and writes for the same mechanical seek time.
A special vdev is a separate top-level vdev, built from faster media (SSD or NVMe), that ZFS automatically directs metadata to, along with small file blocks below a size threshold you configure. The rest of the pool keeps using its original vdevs for bulk data. Done correctly, directory listings, find operations, snapshot management, and small-file access all get dramatically faster, without needing to migrate the whole pool to SSD.
The tradeoff, and the reason this isn't a default recommendation for every pool, is redundancy risk. A special vdev holds metadata for the entire pool. If it's not mirrored and it fails, you don't lose part of the pool — you can lose the whole thing, because ZFS can't reconstruct pool metadata without it. This single detail is why special vdevs need to be treated with at least the same redundancy consideration as your main pool, not less.
Symptoms
The workload signature that points toward "this pool needs a special vdev" usually looks like disproportionately slow directory operations relative to raw throughput. A pool can report perfectly reasonable sequential read/write numbers in a straight dd test while ls on a directory with tens of thousands of entries takes multiple seconds, or a PBS garbage collection job runs far longer than the data volume would suggest.
Check zpool iostat with the -r flag during one of these operations and you'll often see a request size histogram dominated by small, random reads even though the workload conceptually seems sequential:
root@pve01:~# zpool iostat -r tank 1
sync_read sync_write async_read async_write scrub
req_size ind agg ind agg ind agg ind agg ind agg
---------------------------- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
512 0 0 0 0 412 0 0 0 0 0
1K 0 0 0 0 891 0 0 0 0 0
2K 0 0 0 0 340 0 0 0 0 0
4K 0 0 0 0 122 0 0 0 0 0
A heavy concentration of sub-4K async reads during what should be a bulk operation like a scrub or a large restore is often metadata traffic riding along with the data path, competing for the same spindles.
On the PBS side specifically, if your datastore lives on a ZFS pool without a special vdev, garbage collection and verify jobs (which walk chunk metadata extensively) tend to take noticeably longer as the datastore grows into the millions-of-chunks range, even when the underlying disks aren't otherwise busy.
Root Cause
Spinning disks are good at sequential throughput and bad at random small I/O, and metadata access is inherently random — directory entries, dnodes, and indirect blocks are scattered across the pool based on when they were written, not grouped by any access pattern a workload cares about. Every metadata lookup on an HDD-backed pool costs a seek, and seeks are the slowest thing a mechanical disk does.
When bulk data and metadata share the same vdevs, the disk heads are constantly switching between servicing large sequential requests and small random metadata lookups. Each switch costs a seek that neither operation actually needed on its own — the metadata lookup would've been instant on SSD, and the sequential read would've stayed sequential if it hadn't been interrupted.
A special vdev fixes this by physically separating the two workloads onto different media. ZFS uses the special_small_blocks property to decide, per dataset, whether small file blocks below a given size also get diverted to the special vdev alongside metadata — useful for datastores with lots of genuinely small files, since those benefit from the same random-I/O-friendly SSD storage that metadata does.
This matters more on Proxmox VE specifically than it might on a general-purpose NAS, because two of the most common ZFS use cases on PVE — a PBS datastore holding millions of deduplicated backup chunks, and a pool backing many small LXC container root filesystems — are both metadata-heavy by nature. Container filesystems in particular tend to have deep directory trees and lots of small files compared to a handful of large qcow2 or raw VM disk images, so a host running a mix of containers and VMs on the same spinning-disk pool often feels this contention more than a host running VMs alone.
Diagnosis
Before adding a special vdev, confirm the pool is actually metadata-bound rather than bandwidth-bound — the fix doesn't help if the real bottleneck is something else, like an undersized ARC or a genuinely saturated bulk data path.
Check current pool layout and look for whether a special vdev already exists:
root@pve01:~# zpool status tank
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
No special vdev listed — this pool is a plain raidz2 with everything, metadata included, on the same four spinning disks.
Check ARC hit rate for metadata specifically, since a special vdev only helps with what actually misses cache:
root@pve01:~# arc_summary | grep -A5 "Metadata"
Metadata: 41.2 % 3.8 GiB
Metadata hit %: 78.44 %
Metadata miss %: 21.56 %
A metadata miss rate over roughly 15-20% on a pool with a workload that's frequently touching directory structures or many small files is a reasonable signal that a special vdev will make a measurable difference — those misses currently fall through to slow disk seeks and would instead land on fast SSD reads.
Also check what's actually consuming I/O during the slow operation with zpool iostat -v tank 1 alongside iostat -x 1 at the OS level, and compare service times per disk. Disks showing high await relative to modest throughput usually confirms seek-bound behavior rather than a genuine bandwidth ceiling.
It also helps to separate the question of "is the pool slow" from "is this specifically a metadata problem." Run a straight sequential throughput test with fio or even dd against a large file on the pool while nothing else is running. If sequential numbers look reasonable and it's specifically directory-heavy or snapshot-heavy operations that crawl, that's a much stronger case for a special vdev than a pool that's uniformly slow across every access pattern — a uniformly slow pool usually points at something else, like undersized ARC, a failing disk dragging down the whole vdev, or a raidz configuration mismatched to the workload's I/O size.
Step-by-Step Solution
- Confirm the workload is genuinely metadata- or small-file-bound using the diagnosis steps above, not just assuming it based on subjective slowness.
- Choose special vdev media carefully. Use enterprise SSDs or NVMe with power-loss protection, the same standard you'd apply to a SLOG device, since this holds data the pool cannot function without.
- Mirror the special vdev at minimum — never add it as a single unmirrored device on anything beyond a throwaway lab pool, given that its failure can take the entire pool with it.
- Add the special vdev to the existing pool rather than rebuilding, since ZFS supports adding a special allocation class vdev to a live pool.
- Set
special_small_blockson the relevant datasets if you also want small file blocks redirected, not just metadata — this needs to be set explicitly, since the default is 0 (metadata only). - Confirm allocation is actually landing on the new vdev by checking pool status and space usage on the special device after some write activity.
On step 5, be conservative with special_small_blocks. Setting it too high (say, matching your average file size) can fill the special vdev with regular data rather than the metadata and genuinely small files it's meant for, and a full special vdev degrades pool performance rather than helping it.
Commands
zpool status tank
zpool iostat -v tank 1
zpool iostat -r tank 1
arc_summary | grep -A5 Metadata
zpool add tank special mirror /dev/disk/by-id/nvme-A /dev/disk/by-id/nvme-B
zfs set special_small_blocks=32K tank/pbs-datastore
zfs get special_small_blocks tank/pbs-datastore
zpool list -v tank
iostat -x 1Always reference disks by /dev/disk/by-id/ rather than /dev/sdX when adding vdevs — device node names aren't guaranteed to stay consistent across reboots, and a special vdev is the last place you want ambiguity about which physical device ZFS is actually using.
Configuration Examples
Adding a mirrored special vdev to an existing raidz2 pool used for a Proxmox Backup Server datastore:
root@pve01:~# zpool add tank special mirror \
/dev/disk/by-id/nvme-Samsung_PM9A3_S6BFNS0T500001 \
/dev/disk/by-id/nvme-Samsung_PM9A3_S6BFNS0T500002
root@pve01:~# zpool status tank
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
special
mirror-1 ONLINE 0 0 0
nvme-...1 ONLINE 0 0 0
nvme-...2 ONLINE 0 0 0Enabling small block redirection for a specific dataset holding a PBS chunk store, where many chunks are well under the typical filesystem record size:
root@pve01:~# zfs set special_small_blocks=32K tank/pbs-datastore
root@pve01:~# zfs get special_small_blocks tank/pbs-datastore
NAME PROPERTY VALUE SOURCE
tank/pbs-datastore special_small_blocks 32K localConfirming space is actually being consumed on the special vdev after new writes land:
root@pve01:~# zpool list -v tank
NAME SIZE ALLOC FREE
tank 14.5T 6.20T 8.30T
raidz2-0 14.5T 5.95T 8.55T
special 894G 256G 638GCommon Mistakes
Adding a single, unmirrored special vdev because it's "just metadata" and seems low-risk. It's the opposite — losing an unmirrored special vdev can take down the entire pool, not just the metadata that lived on it, because ZFS cannot reconstruct pool structure without it.
Setting special_small_blocks far too aggressively, redirecting most of the dataset's actual data onto what was meant to be a small, fast metadata tier, filling it up and defeating the purpose.
Using consumer-grade SSDs without power-loss protection for a special vdev. The same reasoning that applies to SLOG devices applies here — this is small, latency-sensitive, sync-heavy I/O, and consumer drives tend to perform far worse than their spec sheet suggests under that specific pattern, and some don't handle power loss safely at all.
Assuming a special vdev can be removed later like a cache or log device. Depending on your ZFS version and pool configuration, removing a special vdev isn't always straightforward or even possible — plan the sizing and redundancy as a permanent structural decision, not something you'll casually revisit.
Adding the special vdev without first confirming the workload is actually metadata-bound. If the real bottleneck is bulk sequential throughput or an undersized ARC, a special vdev adds cost and complexity without meaningfully moving the needle.
Best Practices
Mirror the special vdev, and mirror it with the same or better redundancy than you'd accept for the main pool's most important data — a two-way mirror at minimum, three-way if the pool backs something you genuinely can't lose or easily rebuild.
Size the special vdev generously relative to expected metadata volume, not just current usage. Metadata grows with file count and snapshot count, not just data volume, and a PBS datastore in particular accumulates enormous chunk metadata over time as retention policies keep years of incrementals around.
Set special_small_blocks deliberately per dataset rather than globally at the pool level, since different datasets on the same pool can have very different file size distributions — a PBS chunk store and a general-purpose ISO share don't belong on the same threshold.
Monitor special vdev fill percentage the same way you'd monitor any other critical capacity metric, since a full special vdev doesn't just stop helping — it can degrade write performance across the whole pool as ZFS falls back to placing metadata wherever it can.
Test the addition on a non-production pool or a snapshot-restored copy first if at all possible. Adding a special vdev is straightforward technically, but it's also permanent enough in practice that I'd rather validate the sizing and small-block threshold on something disposable before committing a production backup pool to it.
Keep an eye on write amplification too. Because the special vdev now absorbs a disproportionate share of the pool's IOPS-heavy traffic, its endurance rating matters more than its raw capacity in most sizing decisions — an undersized consumer NVMe drive rated for light desktop use will wear out far faster here than the same capacity would in a general-purpose role, simply because of how concentrated the write pattern becomes.
FAQ
Can I remove a special vdev after adding it?
Not reliably in all configurations — treat it as a permanent structural addition to the pool rather than something you can undo like a cache device.
What happens if the special vdev fails and isn't mirrored?
You risk losing the entire pool, not just the metadata. ZFS needs the special vdev's contents to make sense of the rest of the pool's structure.
Does a special vdev help with random reads on regular data too?
Only for data blocks below the special_small_blocks threshold you configure. Everything above that threshold still lives on the pool's regular vdevs.
How big should a special vdev be?
Size it against expected metadata volume for the life of the pool, factoring in file count growth and snapshot retention, not just today's usage. Metadata-heavy workloads like PBS datastores can consume more than people expect.
Is this the same thing as an L2ARC cache device?
No. L2ARC is a volatile read cache that can disappear without data loss. A special vdev is a permanent part of the pool's structure holding data that doesn't exist anywhere else.
Conclusion
A special allocation class vdev is a narrow, deliberate tool: it solves the specific problem of metadata and small-file I/O contending with bulk data on the same slow disks, and it does that well when sized and mirrored properly. It's not a general performance dial to turn up on every pool, and the redundancy requirement isn't optional the way it might feel for "just metadata." Confirm the workload actually needs it, mirror it properly, and it quietly fixes exactly the kind of directory-crawl and small-file slowness that's otherwise hard to explain away with disk throughput numbers alone.