A cluster node that was running fine for eighteen months suddenly starts throwing disk I/O errors on three VMs at once. Nothing changed in the last maintenance window. No firmware updates, no storage migrations, no snapshot jobs kicked off manually. And yet dmesg is full of thin-provisioning errors and two of the guests have gone read-only. This is what LVM-thin metadata exhaustion looks like in the wild, and it catches experienced Proxmox VE admins off guard because the pool itself still has plenty of free data space. The metadata volume is a separate, much smaller allocation, and almost nobody watches it until it's nearly full.

This one is worth understanding in detail because the fix is simple once you know what's happening, but the failure mode is confusing enough that a lot of admins reach for a reboot or a restore from backup before they've even looked at lvs output. A reboot won't help. In some cases it makes things worse, because the pool can fail to activate cleanly on the next boot if metadata is already at 100%.

Problem Overview

Proxmox VE's default local storage layout on non-ZFS installs uses LVM-thin: a thin pool (commonly pve/data) backing a shared logical volume that VM disks and container volumes are carved out of as thin logical volumes. Thin provisioning means blocks are only allocated when actually written, which is exactly why it's attractive for storage efficiency — you can overprovision a 500GB pool across a dozen VMs that each think they have 100GB.

What a lot of admins don't internalize is that the pool has two separate storage areas: the data volume, which holds actual block contents, and the metadata volume, which holds the mapping table — which blocks belong to which LV, plus every snapshot's delta chain. The metadata volume is tiny by comparison, usually well under 1% of the pool size, and it fills up based on the number of allocated blocks and snapshots, not on how much data you've written. A pool that's 40% full on the data side can still exhaust its metadata if you're running a lot of snapshots or have a large chunk count from frequent small writes.

When the metadata LV hits 100%, LVM can no longer track new block allocations. Any thin volume in that pool that tries to write a new block fails at the device-mapper layer. Existing allocated blocks are still readable, which is why some VMs keep running in a degraded state while others go read-only or crash outright depending on what they're trying to write at the moment metadata fills.

Symptoms

The failure signature is fairly distinct once you've seen it, but it doesn't announce itself as a storage problem right away. You'll typically see some combination of the following:

  • Guest OS inside the VM reports disk I/O errors — EXT4-fs error, Buffer I/O error on dev vda1, or the NTFS/XFS equivalent, depending on the guest.
  • The VM itself may pause, go into a paused/stopped state in the Proxmox UI, or simply hang with high CPU wait time.
  • Snapshot or clone operations that previously worked start failing with errors referencing the thin pool.
  • dmesg on the Proxmox host shows lines like:
device-mapper: thin: 253:6: reached low water mark for metadata device: sending event
device-mapper: thin: 253:6: no space for new metadata blocks
device-mapper: thin: 253:6: switching pool to read-only mode

Note that "low water mark" event fires well before you're actually out of space — usually around 80% by default — and if nothing responds to it, the pool keeps filling until it hits the hard limit and switches to read-only. That switch is the point where things visibly break for the guests.

  • journalctl -b on the host shows repeated dmeventd or lvm messages about the thin pool, sometimes interleaved with QEMU errors for the affected VMIDs.
  • pvesm status may still report the storage as active with plenty of free space — because it's reporting data usage, not metadata usage. This is the detail that sends people down the wrong troubleshooting path first.

Root Cause

Almost every case of this comes down to one of two things, often both at once.

First, the metadata volume was undersized from the start. When you create a thin pool with pvcreate/vgcreate/lvcreate without explicitly setting --poolmetadatasize, LVM auto-calculates a size based on the pool's initial size and chunk size at creation time. That calculation assumes a fairly modest number of distinct extents. If you later run heavy snapshot workflows — nightly PVE snapshots retained for a week, or a backup tool that layers snapshots — the number of tracked block ranges grows much faster than the data itself, and the auto-sized metadata volume runs out long before the data volume does.

Second, autoextend isn't working the way people assume it is. LVM does support automatic extension of both the data and metadata portions of a thin pool through lvm.conf, specifically thin_pool_autoextend_threshold and thin_pool_autoextend_percent under the activation section. But autoextend depends on dmeventd actually monitoring the LV, and monitoring has to be enabled at the time the pool was created or activated — check with lvs -a -o+seg_monitor. It's common to find "not monitored" on a pool that's been running for a year, either because monitoring was never turned on, or because a package upgrade restarted lvm2-monitor without re-establishing monitoring on that particular pool. No monitoring means no autoextend, and the low water mark event gets logged but nothing acts on it.

There's also a subtler variant worth calling out: even when autoextend is active for the pool, it can only extend into free space in the volume group. If the VG itself is full — no unallocated PEs left — autoextend has nothing to extend into, and you'll see the same failure even with monitoring correctly enabled. I've seen this exact case on a homelab box where the admin had carved every last gigabyte of the VG into VM disks, leaving zero headroom for either data or metadata autoextension.

Diagnosis

Start by confirming this is actually a metadata problem and not a data-space problem, because the recovery steps differ. Run:

lvs -a -o+lv_name,data_percent,metadata_percent,seg_monitor /dev/pve

You're looking at Meta% for the thin pool LV (something like data_tmeta or shown against data depending on your LVM version's display). If that number is at or near 100.00 while Data% is comfortably lower, you've confirmed it. The seg_monitor column tells you whether dmeventd is watching this LV — if it says "not monitored," that's your smoking gun for why autoextend never fired.

Cross-check with the device-mapper layer directly:

dmsetup status pve-data-tpool

The output line looks something like:

0 1048576000 thin-pool 1 24/1048576 1998372/1998372 - rw discard_passdown queue_if_no_space

The two ratios are transaction id / metadata blocks used/total, then data blocks used/total. In this example the metadata field 1998372/1998372 shows full utilization — that's the exhaustion, plain as day, even though the data blocks in this hypothetical are nowhere near full.

Check free space in the volume group, since it determines whether a simple extend is even possible:

vgs pve -o vg_name,vg_size,vg_free

If vg_free is close to zero, you'll need to free space before you can grow the metadata LV — either by removing old snapshots you no longer need, or by adding a physical volume to the VG if there's another disk or partition available.

Finally, look at how you got here. Count active snapshots against thin volumes in that pool:

lvs -a | grep -i snap

A long list of retained snapshots, especially ones tied to a backup or replication schedule, is almost always the thing that drove metadata consumption up faster than anyone expected.

Step-by-Step Solution

The fix depends on whether the VG has free space to extend into.

If the VG has free space (the common case), extend the metadata LV directly. This is a live operation on current LVM versions and doesn't require stopping VMs:

lvextend --poolmetadatasize +1G pve/data

Adjust the size to whatever you have room for — 1GB is often enough to get well clear of exhaustion for a typical pool, but if you're running dozens of snapshots, go with 2GB. Verify immediately afterward:

lvs -a -o+metadata_percent pve/data

You should see the metadata percentage drop substantially now that the denominator increased. VMs that were stuck mid-write should recover on their own once the pool comes off read-only mode, though a hung guest may still need a reset from inside the OS.

If the pool is already read-only and guests are erroring hard, you may need to explicitly bring the pool back to read-write after the extend:

dmsetup message pve-data-tpool 0 "release_metadata_snap"
lvchange --refresh pve/data

In practice a clean lvextend followed by a refresh is usually enough. Reboot the affected VMs (not the host) once storage is confirmed writable again, since the guest filesystem may have marked itself read-only in response to the earlier I/O errors and won't clear that state without a restart.

If the VG has no free space, you have to create room first. Options in order of preference: remove snapshots you don't need (lvremove on the specific snapshot LV, not the origin), migrate a VM off to another storage target to reclaim space, or extend the VG by adding another physical volume with vgextend pve /dev/sdX if you have an unused disk or partition. Only after the VG shows free extents can you run the lvextend --poolmetadatasize command above.

If the pool won't activate at all — this happens when metadata was already at 100% before a reboot and the pool failed to come up cleanly — boot into a rescue environment or use the Proxmox host's own recovery shell, install thin-provisioning-tools if it's not already present, and run a metadata check before attempting repair:

apt install thin-provisioning-tools
lvchange -ay --activationmode partial pve/data_tmeta
thin_check /dev/mapper/pve-data_tmeta

If thin_check reports errors, follow it with thin_repair against a new metadata LV before attempting to reactivate the pool normally. This path is rare — most people catch the problem while the pool is merely read-only, not fully wedged — but it's worth knowing it exists rather than reaching straight for a restore from Proxmox Backup Server.

Commands

# Check metadata and data usage plus monitoring state
lvs -a -o+lv_name,data_percent,metadata_percent,seg_monitor

# Confirm at the device-mapper level
dmsetup status pve-data-tpool

# Check volume group free space
vgs pve -o vg_name,vg_size,vg_free

# Extend metadata volume (most common fix)
lvextend --poolmetadatasize +1G pve/data

# Refresh the LV mapping after extension
lvchange --refresh pve/data

# List snapshots consuming metadata
lvs -a | grep -i snap

# Remove a specific stale snapshot
lvremove pve/vm-101-disk-0-snap

# Extend the volume group with an additional disk
vgextend pve /dev/sdb1

# Check dmeventd monitoring status for the pool
lvs -a -o+seg_monitor pve/data

# Re-enable monitoring if it dropped
lvchange --monitor y pve/data

Configuration Examples

To prevent this from recurring, tune autoextend behavior in /etc/lvm/lvm.conf under the activation section:

activation {
    thin_pool_autoextend_threshold = 80
    thin_pool_autoextend_percent = 20
    ...
}

This tells LVM to auto-extend the pool (both data and metadata portions, when monitored) once utilization crosses 80%, growing it by 20% of current size each time. After editing, no service restart is required for new activations, but confirm dmeventd is actually running:

systemctl status lvm2-monitor

If a given pool shows "not monitored" in lvs -a -o+seg_monitor despite the daemon running, force monitoring back on:

lvchange --monitor y pve/data

When you build a thin pool from scratch — say, adding a second local-lvm storage target on a fresh disk — size the metadata volume explicitly rather than trusting the default calculation, especially if you know you'll be running frequent snapshots:

vgcreate pve2 /dev/sdb1
lvcreate --type thin-pool \
  --size 900G \
  --poolmetadatasize 4G \
  --thinpool data \
  pve2

4GB of metadata for a 900GB pool is generous relative to the LVM default, and the cost is negligible against the size of the data volume. It buys you a lot of headroom against snapshot-heavy workloads before you ever need to think about this again.

Common Mistakes

The single biggest mistake is trusting pvesm status or the Proxmox UI's storage summary as a complete picture of pool health. Both report data utilization, and neither surfaces metadata percentage anywhere in the GUI. You have to check with lvs directly, and if you want ongoing visibility you need a monitoring check that specifically pulls metadata_percent, not just free space on the storage.

Another common one: rebooting the host as a first response. It doesn't fix metadata exhaustion, and if the pool was already at the hard limit, a reboot can leave it failing to activate cleanly, turning a read-only-but-recoverable situation into an activation failure that needs thin_check/thin_repair to sort out. Diagnose first, extend second, reboot only if a specific guest needs it after storage is confirmed healthy.

People also assume enabling autoextend once means it's protected forever. Package upgrades to lvm2 have, in the past, reset monitoring state on existing LVs without anyone noticing, since the config setting in lvm.conf stays correct but the per-LV monitored flag doesn't. Check seg_monitor periodically instead of assuming it's still active from when you set it up two years ago.

And a smaller but real mistake: setting thin_pool_autoextend_percent too low relative to your snapshot churn rate. A 20% growth increment sounds reasonable until you realize a burst of snapshot activity can consume that entire increment within a day, and if the VG doesn't have room for the next increment, you're right back to exhaustion with autoextend technically "working" the whole time.

Best Practices

Size metadata generously when you create a thin pool — don't let the LVM default decide for you, particularly on any storage target that will host frequent PVE snapshots or replication jobs. Going from the auto-calculated default to 2-4GB of metadata costs almost nothing relative to a multi-terabyte pool.

Keep 10-15% of the volume group unallocated rather than carving every last extent into VM disks. That headroom is what makes autoextend actually useful instead of theoretical — without free PEs in the VG, autoextend has nothing to grow into no matter how the thresholds are configured.

Add a monitoring check for thin pool metadata percentage specifically, separate from your general disk-space alerting. A simple cron-driven script parsing lvs --reportformat json -o+metadata_percent and alerting past 70% gives you days of lead time instead of finding out from a paused VM.

Review snapshot retention regularly. Snapshots you forgot about are the number one driver of unexpected metadata growth — a snapshot taken before every patch cycle and never cleaned up will quietly eat into your metadata headroom month after month. qm listsnapshot <vmid> across your fleet, run periodically, catches this before it becomes an incident.

If you're building new storage today and don't have a specific reason to stick with LVM-thin, weigh whether ZFS fits your hardware better — it doesn't have this particular metadata exhaustion failure mode, though it brings its own tuning concerns around ARC and special vdevs. That's not an argument for migrating existing pools; it's a consideration for new deployments where you're choosing a storage backend from scratch.

FAQ

Can I lose data when metadata fills up?

Not from the exhaustion itself. Existing block mappings stay intact and readable; the pool simply can't allocate new blocks. Data loss risk comes from what you do next — forcing removals or running repair tools without checking first.

Does this affect ZFS-backed storage too?

No. This is specific to the LVM-thin backend. ZFS pools have entirely different metadata handling and don't hit this particular failure mode, though they have their own capacity and fragmentation considerations.

How much metadata headroom do I actually need?

It depends on snapshot count and chunk size more than raw pool size. As a starting point, 2-4GB comfortably covers a few hundred gigabytes of pool with regular snapshot activity — check metadata_percent trend over a few weeks and adjust from there rather than guessing once and forgetting about it.

Will extending the metadata LV interrupt running VMs?

No, lvextend --poolmetadatasize is an online operation on current LVM versions and doesn't require pausing or stopping guests. If a VM already hit an I/O error before the extend, it may still need a restart afterward to clear its own error state.

Can Proxmox's built-in alerting catch this before it happens?

Not out of the box. The storage view in the GUI reports data usage, not metadata usage, so you need a separate check — a cron job or external monitoring agent parsing lvs output — to get early warning specifically on metadata percentage.

Conclusion

Metadata exhaustion is one of those failures that looks alarming from inside a guest OS and completely mundane once you're looking at the right LVM output. The fix, in the overwhelming majority of cases, is a single lvextend --poolmetadatasize command — the hard part is knowing to check for it instead of assuming a full data volume or a failing disk. Build the headroom in up front, keep autoextend actually monitored rather than just configured, and put a metadata-specific check into whatever you use for alerting. The next time a VM throws disk errors for no obvious reason, this is one of the first places worth looking, right alongside the usual suspects of failing hardware and network storage timeouts.