You just built a Proxmox VE box with a shiny NVMe drive, or maybe you're trying to decide between ZFS and plain LVM-Thin for your VM storage. Either way, you've probably asked the same question everyone asks eventually: is this thing actually fast, or does it just feel fast because nothing's using it yet?
The Summary tab graphs in Proxmox will show you disk activity, but they won't tell you how many IOPS your storage can sustain under real load, or whether switching your VM's disk cache mode from none to writeback actually changed anything measurable. For that you need to generate load on purpose and watch what happens. That's what fio is for.
This guide walks through installing fio, running two test patterns that actually mean something (random small-block IOPS and sequential large-block throughput), and reading the output without getting lost in the wall of numbers it prints. We'll also cover the mistakes that quietly invalidate a benchmark — testing through the host page cache by accident being the big one.
What You Will Learn
- What fio is and why it's the standard tool for storage benchmarking on Linux
- Where to run it — inside a guest VM versus directly on the Proxmox host — and why that choice changes your numbers
- How to install fio and run a random-IOPS test and a sequential-throughput test
- What every flag in the command actually does, so you're not copy-pasting blind
- How to read fio's output: IOPS, bandwidth, and latency percentiles
- Why cache mode and the iothread setting can make the same disk look completely different in two runs
- Common errors, and the mistakes that produce numbers you shouldn't trust
What Is This Feature?
fio, short for Flexible I/O Tester, is a benchmarking tool originally written by Jens Axboe, who maintains the Linux kernel's block I/O layer. It doesn't ship as part of Proxmox VE — it's a regular Debian package — but it's become the de facto standard anywhere someone needs a real answer to "how fast is this storage," from single-disk homelabs to Ceph clusters holding thousands of VMs.
What makes fio different from just copying a big file and timing it is that it can simulate almost any I/O pattern you care about: pure sequential reads, pure random writes, a realistic mix of both, at whatever block size and queue depth you specify. A single large sequential copy tells you almost nothing about how your storage will behave once fifteen VMs are all doing small random reads and writes to a database or a mail spool at the same time. fio can simulate that specific, uglier workload directly.
In a Proxmox VE context, you'll typically run fio in one of two places: inside a guest VM, against its virtual disk, to see what your actual workloads will experience — or directly on the Proxmox host, against the underlying storage (a ZFS pool, an LVM-Thin volume, a directory), to measure the hardware itself with none of the virtualization layer in the way. Both are useful, and they will not give you the same numbers.
Why Would You Use It?
Most people reach for fio at one of a few specific moments. Before committing to a storage layout — deciding between ZFS mirror, ZFS RAIDZ, or plain LVM on the same disks — a quick benchmark tells you the real cost of ZFS's checksumming and copy-on-write overhead on your specific hardware, instead of trusting a forum post about someone else's SSDs.
It's also the right tool when a VM feels sluggish and you suspect storage rather than CPU or RAM. Running the same test on the host and inside the guest isolates whether the bottleneck is the physical disk, the storage backend, or something in the virtual disk configuration — cache mode, controller type, or a missing iothread.
And it's genuinely useful after any change you're not sure actually helped: switching disk cache mode, enabling iothread on a SCSI disk, moving from IDE to VirtIO SCSI, adding a ZFS special device for metadata. "Feels faster" isn't a number. fio gives you one, and lets you compare it against the number from before the change.
Prerequisites
- A Proxmox VE 8.x or 9.x host, and either shell access to the host itself or to a Linux guest VM running on it
- Root or sudo access, since fio with
direct=1needs to bypass normal caching, and installing packages requires it anyway - A test target that isn't holding data you care about — a scratch VM disk, a spare partition, or a plain file on a filesystem with a few gigabytes free. Never point fio directly at a raw block device (like
/dev/sdb) that has real data on it; writing to a device path instead of a file will overwrite whatever's there - At least 10–15 minutes of uninterrupted time per test run — cutting a benchmark short gives you numbers that reflect a burst, not sustained performance
- Ideally, a quiet system. Running fio while three other VMs are doing their own disk-heavy work will give you a number, just not the number you're trying to measure
Step-by-Step Tutorial
Step 1: Decide Where You're Testing
If you want to know what your VMs will actually experience, test from inside a guest, against its virtual disk. If you want to know what the raw storage underneath Proxmox is capable of — useful when comparing ZFS to LVM-Thin, or evaluating new drives — test on the Proxmox host itself, against a scratch file on that storage.
Both are valid. Just be clear about which one you're doing, because the numbers from each answer a different question.
Step 2: Install fio
On a Debian or Ubuntu-based guest, or on the Proxmox host itself:
apt update
apt install fioThis pulls fio from the standard Debian repositories — no third-party sources needed. Confirm it installed with fio --version; anything in the 3.x range is fine for what we're doing here.
Step 3: Run a Random 4K IOPS Test
This pattern simulates the kind of I/O a database, a mail server, or a busy VM boot disk generates: lots of small, scattered reads and writes rather than one big sequential stream.
fio --name=randwrite-test \
--directory=/root \
--rw=randwrite \
--bs=4k \
--ioengine=libaio \
--iodepth=32 \
--numjobs=4 \
--size=1G \
--runtime=60 \
--time_based \
--direct=1 \
--group_reportingLet this run the full 60 seconds. It creates four 1 GB test files under /root and hammers them with random 4K writes for a minute, then deletes them unless you add --unlink=0. Swap --rw=randwrite for --rw=randread to test reads instead, or --rw=randrw for a mixed 50/50 pattern that's closer to a real application.
Step 4: Run a Sequential Throughput Test
This pattern simulates large transfers — a backup job, restoring a VM, copying a big ISO or media file.
fio --name=seqwrite-test \
--directory=/root \
--rw=write \
--bs=1M \
--ioengine=libaio \
--iodepth=16 \
--numjobs=1 \
--size=4G \
--runtime=60 \
--time_based \
--direct=1 \
--group_reportingNotice the block size jumped from 4k to 1M and numjobs dropped to 1 — sequential throughput tests don't benefit from the same massive parallelism random-IOPS tests do, and a single well-queued stream usually gets closer to a drive's real sequential ceiling.
Step 5: Read the Output
At the end of a run, fio prints a summary block. The lines that matter most:
write: IOPS=48.2k, BW=188MiB/s (197MB/s)(11.0GiB/60002msec)
clat (usec): min=12, max=8421, avg=331.94, stdev=210.55
lat (usec): min=13, max=8433, avg=333.10, stdev=210.61
clat percentiles (usec):
| 1.00th=[ 109], 50.00th=[ 289], 99.00th=[ 1156]IOPS is operations per second — the number that matters most for random small-block workloads. BW is bandwidth, the number that matters most for sequential large-block workloads. clat is completion latency: how long a request took from submission to completion. The average is useful, but the percentiles tell the real story — a 50th percentile of 289 microseconds with a 99th percentile of 1.16 milliseconds means most requests are fast, but roughly one in a hundred takes four times as long. On spinning disks or heavily loaded ZFS pools, that tail can stretch a lot further than the average suggests.
Step 6: Compare, Don't Just Read Once
A single number tells you almost nothing on its own. Run the same test before and after a change — cache mode, iothread, storage backend — and compare. Run it two or three times and check whether the numbers are stable; if they swing by 30% between identical runs, something else on the system is interfering, or your test size is too small to escape caching effects.
Commands Explained
| Flag | What it controls |
|---|---|
--rw | The I/O pattern: randwrite, randread, randrw, write, read. Random patterns stress IOPS; sequential patterns stress bandwidth. |
--bs | Block size per operation. 4k mimics database and metadata-heavy workloads; 1M mimics large file transfers. |
--ioengine | How fio submits I/O to the kernel. libaio is the safe, widely-supported default on Linux; newer kernels also support io_uring, which can show higher numbers on fast NVMe drives. |
--iodepth | How many I/O requests are kept in flight at once per job. Higher values simulate a busier system; depth 1 simulates a single-threaded application waiting on each request. |
--numjobs | How many parallel worker processes run the same test simultaneously. More jobs simulate multiple concurrent VMs or applications hitting the same storage. |
--direct | Set to 1 to bypass the OS page cache and talk to storage directly. Without this, you're partly benchmarking RAM, not disk. |
--time_based / --runtime | Runs the test for a fixed duration instead of stopping once the file size is written, which gives storage time to settle into steady-state behavior rather than measuring a burst. |
--group_reporting | Combines the results from all parallel jobs into one summary instead of printing a separate block per job. |
Common Errors
"Permission denied" or a job that exits immediately with an error code. Usually means you tried to run fio without root against a path or device that requires it, or you pointed --filename at a device node without the right permissions. Run as root, or double-check the target path exists and is writable.
"No space left on device" partway through. Your --size is bigger than the free space on the target filesystem. Drop the size, or point at a location with more room. On a thin-provisioned ZFS pool or LVM-Thin volume, remember that "free space" reported by df can be optimistic if other thin volumes are also growing.
Wildly inconsistent numbers between identical runs. This is almost always caching. If your test file is smaller than available RAM (or, on Proxmox, smaller than ZFS's ARC cache), repeated reads may be served from memory instead of disk, and your "storage" benchmark becomes a memory benchmark by accident. Use a test size larger than your RAM, or add --direct=1 if you somehow left it off.
Suspiciously high numbers you don't believe. If the VM's disk is set to writeback cache mode, writes can be acknowledged as soon as they hit the host's page cache, before they've actually reached the physical disk. That can make a benchmark look fantastic right up until an unexpected power loss, at which point that cached-but-unwritten data is gone. Test with none or writethrough cache mode if you want numbers that reflect actual persisted performance.
Troubleshooting
If IOPS numbers look far lower than you'd expect from the drive's spec sheet, check whether the VM's disk has iothread enabled (under the disk's Advanced options in the Proxmox web interface). Without it, all I/O for that disk shares a single QEMU thread with everything else the VM is doing, which caps random I/O performance well below what the hardware can actually do — this is one of the most common causes of "my NVMe drive benchmarks slow inside the VM but fast on the host."
If results differ a lot between the host and a guest VM on the same physical disk, that gap is virtualization overhead, and it's expected to some degree — the question is whether it's a small tax or a large one. A 10–20% gap is normal. A 5x gap usually points to cache mode, missing iothread, or an old disk controller (IDE or SATA emulation instead of VirtIO SCSI).
If a ZFS-backed test shows unexpectedly poor random write performance, remember that ZFS is copy-on-write and does synchronous write handling differently from ext4 or XFS — a separate ZIL/SLOG device on fast storage can meaningfully change small-write latency, and this is worth testing for specifically rather than assuming ZFS itself is "just slow."
If numbers seem to degrade the longer a test runs, especially on consumer-grade SSDs, you may be watching the drive's write cache fill up and the controller fall back to its slower native write speed — a known behavior on lower-end NVMe and SATA SSDs under sustained load, not a Proxmox or fio problem.
Best Practices
Change one variable at a time. If you're comparing cache modes, controller types, or storage backends, don't change two things between runs — you won't know which change actually mattered.
Run tests long enough to matter. Thirty seconds is barely enough to warm anything up; 60 seconds is a reasonable minimum, and longer is better for catching the kind of SSD throttling mentioned above.
Keep the system otherwise idle while testing. A benchmark run alongside a live backup job or another VM's disk-heavy workload measures contention, not the storage's actual ceiling — which is a legitimate thing to want to know, just not what you're usually trying to measure.
Delete your test files when you're done, especially if you skipped --unlink handling — a few gigabytes of leftover fio scratch files are an easy thing to forget about until your storage looks mysteriously fuller than expected.
Frequently Asked Questions
Is it safe to run fio on a Proxmox host in production?
It's safe as long as you point it at a scratch file, not a device holding real data, and you're mindful that it will generate real load — run it during a maintenance window if other VMs on the same storage are latency-sensitive.
What's a "good" IOPS number?
There isn't a universal target — a budget SATA SSD might do 20,000–40,000 random 4K IOPS, while a fast enterprise NVMe drive can exceed 500,000. The useful comparison is against your own hardware's baseline and against your own previous test, not against a number from someone else's setup.
Why do my host and guest numbers differ so much?
Virtualization overhead, disk cache mode, whether the disk uses an iothread, and the controller type (VirtIO SCSI versus emulated SATA or IDE) all affect this. A large gap is usually explainable by one of those, not a mystery.
Can fio test network performance too?
No — fio is a storage tool. For network throughput between VMs or nodes, the equivalent tool is iperf3, which is a separate topic.
Does this work the same way on Proxmox VE 8 and 9?
Yes. Neither fio nor the concepts here — cache modes, iothread, VirtIO SCSI — have changed between those releases.
Conclusion
fio won't tell you whether your storage is "good" in the abstract — it tells you exactly what your storage does under a specific, repeatable load, which is a lot more useful. Run the random-IOPS test to understand database-and-boot-disk behavior, run the sequential test to understand backup-and-large-file behavior, and change one setting at a time when you're chasing an improvement.
The numbers you get on your first run matter less than having a baseline to compare against later. Save your results somewhere before you start changing cache modes and controllers, so six months from now you can tell whether that new NVMe drive actually helped or just felt like it did.