Introduction

If you have already built a Proxmox VE cluster, the next question is almost always the same: where does the storage live? Shared storage is what makes live migration, high availability, and painless node maintenance actually work, and most homelabs and small production clusters eventually outgrow a single NFS share or a pile of local ZFS pools that can't be moved between nodes without a full copy. Ceph solves this by turning the local disks in every node into one distributed, self-healing storage pool that any node in the cluster can read and write to at the same time.

Proxmox VE ships with native Ceph integration, meaning you do not need a separate storage cluster, a separate set of servers, or a third-party management layer. The same nodes that run your VMs and containers can also run Ceph's monitors, managers, and object storage daemons (OSDs) — this is what "hyper-converged" means. This guide walks through a complete, from-scratch deployment: installing Ceph, creating monitors and managers, turning raw disks into OSDs, building a pool, and wiring that pool up as VM/CT storage inside Proxmox VE 9.2, which defaults new deployments to Ceph Tentacle (20.2.1) while still fully supporting Ceph Squid (19.2) for existing clusters.

What You Will Learn

  • What Ceph actually does inside a Proxmox VE cluster, and when it's the right choice
  • Hardware and network requirements before you touch a single command
  • How to install Ceph and initialize the cluster configuration with pveceph
  • How to create monitors (MONs) and managers (MGRs) for quorum and cluster management
  • How to turn physical disks into OSDs, including separate DB/WAL devices for HDD+NVMe setups
  • How to create a replicated pool with sane defaults and enable the PG autoscaler
  • How to add the resulting pool as RBD storage so VMs and containers can actually use it
  • How to verify cluster health and read the warnings Ceph gives you before they become outages

What Is Ceph?

Ceph is a distributed storage system that spreads data redundantly across multiple disks on multiple nodes. Instead of one server owning one disk, Ceph pools every OSD (one daemon per physical disk) across the cluster into a single logical storage layer, then uses the CRUSH algorithm to decide where each object's replicas live. If a disk or an entire node dies, Ceph already has redundant copies elsewhere and rebuilds automatically, without you moving a single VM disk by hand.

In a Proxmox VE context, this shows up as an RBD (RADOS Block Device) storage pool that behaves like any other shared storage backend in the web UI. You can create VM disks on it, migrate VMs between nodes without downtime, and let Proxmox HA restart a VM on a different node after a hardware failure — because the VM's disk was never tied to that node's local storage in the first place.

Why Would You Use It?

Ceph is the right tool when you need storage that survives a node failure without a SAN, without a separate storage appliance, and without paying a dedicated storage vendor. It's what makes Proxmox VE's High Availability stack meaningful for VM disks, not just VM configuration. It also scales horizontally — need more capacity or throughput, add another node with disks, and the cluster rebalances into it.

It is not free, though: Ceph needs real network bandwidth, it wants at least three nodes to have any redundancy at all, and it is noticeably less forgiving of underpowered hardware than a single ZFS pool. If you're running a single node, or two nodes with a QDevice for quorum, Ceph is the wrong tool — look at ZFS with replication instead. Ceph earns its complexity once you have three or more nodes and actually need VM disks to survive a node going down.

Prerequisites

  • At least three Proxmox VE nodes, already joined into a cluster (pvecm status should show quorum)
  • Each node has at least one unused, unpartitioned physical disk to dedicate to Ceph — OSDs should never share a disk with the OS
  • A dedicated network for Ceph traffic, ideally 10 Gbps or faster, physically or logically separated from your corosync and VM traffic. Ceph is chatty and will compete with corosync for bandwidth if they share a link, which is a common cause of cluster instability
  • At least 8 GiB of RAM per node beyond what your VMs already need — the OSD daemon defaults to a 4 GiB memory target per OSD, and recovery operations use noticeably more
  • At least one CPU core reserved per Ceph daemon you plan to run on a node (MON, MGR, and each OSD all count)
  • Root or equivalent (Sys.Modify / Datastore.Allocate) access on every node, since some Ceph setup steps require CLI access outside the regular PVE permission model
  • Avoid hardware RAID controllers for the disks you plan to use as OSDs — pass disks through as JBOD/HBA so Ceph can manage redundancy itself

This guide uses three nodes named pve1, pve2, and pve3, each with one spare disk at /dev/sdb. Substitute your own node names and device paths throughout — always double check device names with lsblk before running anything destructive, since /dev/sdb can point to a different disk after a reboot.

Step-by-Step Tutorial

Step 1: Install the Ceph packages on every node

Run this on each of the three nodes, one at a time, starting with the first:

pveceph install

By default this installs the current default Ceph release for new deployments (Tentacle 20.2.1 on Proxmox VE 9.2). If you want to pin a specific release — for example to match an existing cluster still running Squid — specify it explicitly:

pveceph install --version squid

Wait for the install to finish on one node before moving to the next. You can also trigger this from the web UI: select a node, go to Ceph, and click Install Ceph if it isn't already installed — the wizard calls the same pveceph install command underneath.

Step 2: Initialize the Ceph configuration

This step only needs to run once, on the first node. It creates /etc/pve/ceph.conf, which Proxmox VE's clustered filesystem (pmxcfs) automatically replicates to every other node in the cluster:

pveceph init --network 10.10.10.0/24

Replace the subnet with your dedicated Ceph network. If you have separate public and cluster (replication) networks, set both:

pveceph init --network 10.10.10.0/24 --cluster-network 10.10.20.0/24

The public network is what clients (your VMs, via the hypervisor) use to reach storage. The cluster network is what OSDs use to replicate and rebalance data between themselves. Splitting them keeps a rebalance storm from starving your VM I/O, but a single shared 10G+ network is a reasonable starting point for smaller clusters.

Step 3: Create monitors

A monitor (MON) holds the cluster map and is how every other component agrees on the current state of the cluster. Ceph needs an odd number of monitors — at minimum three — to maintain quorum if one goes offline. The first monitor is created automatically by pveceph init. Create the remaining two, one per node, from each respective node:

# on pve2
pveceph mon create

# on pve3
pveceph mon create

After this, check the Ceph panel in the web UI or run ceph -s — you should see three monitors listed and the cluster reporting quorum.

Step 4: Create managers

The manager (MGR) handles cluster metrics, the dashboard, and several background modules. Only one MGR is active at a time, but you should run at least two for failover. Create one on each of your monitor nodes:

# on pve1
pveceph mgr create

# on pve2
pveceph mgr create

A third manager on pve3 is fine too — extra standby managers don't cost much and give you one more layer of redundancy if a node goes down during maintenance.

Step 5: Create OSDs

This is where your physical disks actually become usable storage. On each node, identify the spare disk (never run this against a disk that's currently mounted or holding data you need):

lsblk

Then create the OSD:

pveceph osd create /dev/sdb

Repeat this on every node for every disk you're dedicating to Ceph. If a disk was previously used by Ceph or another filesystem and the command refuses to proceed, wipe it first:

ceph-volume lvm zap /dev/sdb --destroy

If you're mixing spinning HDDs with a fast NVMe drive, put the HDDs' write-ahead log (WAL) and metadata database (DB) on the NVMe device instead of the slow disk — this is the single biggest performance win available for HDD-backed OSDs:

pveceph osd create /dev/sdb -db_dev /dev/nvme0n1 -wal_dev /dev/nvme0n1

If you don't specify sizes, Proxmox VE allocates roughly 10% of the OSD's size to the DB partition and 1% to the WAL by default. You can override this explicitly with -db_size and -wal_size if you're carving one NVMe drive up for several HDD OSDs and need to plan capacity carefully.

Official Proxmox guidance recommends at least 12 OSDs, evenly spread across nodes, for a cluster that's meant to carry real production load — three nodes with one OSD each will work for testing and small homelabs, but recovery from a single disk failure has much less headroom to work with.

Step 6: Create a pool

A pool is the logical container VMs actually store their disks in. Create one with the CLI, letting Proxmox VE register it as storage automatically:

pveceph pool create vm-storage --add_storages

This creates a replicated pool with Ceph's default settings — size 3 (three copies of every object) and min_size 2 (writes are only acknowledged once at least two copies exist). Do not reduce min_size to 1: a pool with min_size 1 will accept writes with only a single surviving replica, and a second failure during that window can permanently lose data. It's a common shortcut people take to "fix" a HEALTH_WARN state, and it defeats the entire point of running Ceph.

The default pool starts with 128 placement groups (PGs), which controls how objects are distributed across OSDs. Rather than calculating this by hand, enable the PG autoscaler so Ceph adjusts it automatically as the pool grows:

ceph osd pool set vm-storage pg_autoscale_mode on

Step 7: Verify the pool shows up as Proxmox VE storage

Because you passed --add_storages in Step 6, the pool should already appear under Datacenter → Storage as an RBD storage entry available to every node in the cluster. If you created the pool without that flag, or you're adding an existing pool later, register it manually:

pvesm add rbd vm-storage --pool vm-storage --content images,rootdir

You can now select this storage when creating a new VM or container disk, exactly like local-lvm or any NFS mount — the difference is that the disk now lives on every node's OSDs simultaneously, not on one node's local storage.

Step 8 (Optional): Add CephFS for ISO images and templates

RBD storage only holds block devices (VM/CT disks) — it can't store ISO files, container templates, or backup files directly. For a shared filesystem that every node can browse, deploy CephFS on top of the same cluster:

pveceph fs create --pg_num 32 --add-storage

This creates the required metadata pool, a metadata server (MDS), and registers the filesystem as a second storage entry you can use for ISOs, templates, and snippets across every node — no more copying an ISO to each node by hand.

Step 9: Confirm overall cluster health

Before you trust this cluster with production VM disks, check its actual health, not just that the commands didn't error out:

ceph -s

You're looking for HEALTH_OK. A fresh cluster sometimes sits in HEALTH_WARN for a few minutes while PGs finish activating — that's expected and resolves on its own. A warning that persists after ten minutes, or a status of HEALTH_ERR, means something needs attention before you go further; see the Troubleshooting section below.

Commands Explained

  • pveceph install — installs the Ceph packages matched to your Proxmox VE version, either at the cluster's current pinned release or, on the first node, the default for new deployments
  • pveceph init --network <cidr> — writes the initial cluster-wide Ceph configuration to /etc/pve/ceph.conf, replicated automatically to all nodes via pmxcfs; run once
  • pveceph mon create — deploys a monitor daemon on the local node and adds it to the monitor quorum
  • pveceph mgr create — deploys a manager daemon on the local node
  • pveceph osd create <device> — wipes and formats the given block device as a BlueStore OSD and joins it to the cluster
  • pveceph pool create <name> --add_storages — creates a replicated RADOS pool and simultaneously registers it as Proxmox VE storage on every node
  • pveceph fs create --add-storage — deploys CephFS (an MDS plus supporting pools) and registers it as file-based storage
  • ceph -s / ceph status — prints a live summary of cluster health, monitor quorum, OSD count, and data usage
  • ceph osd pool set <pool> pg_autoscale_mode on — hands placement group sizing over to Ceph's autoscaler instead of a fixed number

Common Errors

"Cluster is not quorate" or MON creation fails

This almost always means corosync itself isn't healthy yet. Ceph MONs depend on the underlying Proxmox VE cluster having quorum — fix pvecm status issues first before touching Ceph.

OSD creation fails with "device already in use" or refuses to proceed

The disk has an existing filesystem, LVM signature, or partition table. Zap it first with ceph-volume lvm zap /dev/sdX --destroy, and confirm with lsblk that you're targeting the correct device before retrying — device letters can shift after a reboot, especially on nodes with hot-swap bays.

Pool creation warns about "too few PGs" or "too many PGs per OSD"

This shows up on very small clusters (few OSDs) with a pool count that doesn't match the OSD count. Enabling the PG autoscaler (Step 6) resolves this automatically in most cases; if it persists, check ceph osd pool autoscale-status for a recommended target.

VM disk creation on the new RBD storage fails with a permission or authentication error

Confirm the storage's keyring exists at /etc/pve/priv/ceph/<storage-id>.keyring and that the pool name in Datacenter → Storage exactly matches the pool you created — a typo here is the most common cause.

Troubleshooting

If ceph -s reports HEALTH_WARN with a message like Degraded data redundancy shortly after creating OSDs, this is normal — Ceph is actively rebalancing placement groups across the new disks and will clear on its own; watch progress with ceph -w.

If it instead reports clock skew between nodes, install and enable chrony or another NTP client on every node — Ceph's monitors are sensitive to time drift and will refuse to reach quorum cleanly if nodes disagree on the time by more than a small threshold.

If OSDs repeatedly flap between up and down after creation, check the dedicated Ceph network for saturation or a misconfigured MTU (jumbo frames need to be consistent end-to-end across every switch port involved) before assuming a disk is failing.

If the cluster sits below your expected usable capacity, remember that a size-3 pool only gives you roughly one-third of raw OSD capacity as usable space — that is the cost of the redundancy Ceph provides, not a misconfiguration.

Best Practices

  • Use identical, or as close to identical as possible, hardware across nodes — mismatched disk sizes and CPU speeds make CRUSH balancing and recovery time harder to predict
  • Keep the Ceph network physically or VLAN-separated from corosync — a saturated Ceph network is one of the most common causes of a cluster losing quorum
  • Never set pool min_size to 1, even temporarily, unless you fully understand and accept the data-loss risk during that window
  • Leave real memory headroom — Ceph OSDs use significantly more RAM during recovery than during steady state, and an out-of-memory OSD during a rebuild compounds the original failure
  • Enable the PG autoscaler rather than hand-tuning placement groups unless you have a specific, measured reason not to
  • Start monitoring cluster health early with ceph -s and the Ceph dashboard in the Proxmox VE UI, not just after something breaks
  • Plan capacity assuming a full node can go offline — a healthy Ceph cluster should be able to lose one node and still have enough free space to re-replicate the data it was holding

Frequently Asked Questions

How many nodes do I actually need to run Ceph in Proxmox VE?

Three is the practical minimum for real redundancy — it lets Ceph maintain monitor quorum and satisfy a default replicated pool's size-3/min_size-2 settings even if one node fails. Two-node setups can technically run Ceph but lose most of its resilience benefits.

Can I mix Ceph storage with local ZFS or LVM storage on the same cluster?

Yes. Ceph, ZFS, and LVM storages can all coexist in the same Proxmox VE cluster, and you choose per-VM-disk which storage backend to use. Many clusters keep fast local NVMe storage for latency-sensitive VMs and Ceph for everything that needs to survive a node failure.

Do I need a separate physical network switch just for Ceph?

Not strictly, but you do need guaranteed, isolated bandwidth — a dedicated VLAN on a shared switch with proper QoS can work if it's genuinely isolated from corosync traffic. A fully separate switch and NICs remove an entire class of "why did my cluster lose quorum during a backup" problems.

What happens to my VMs if an OSD or an entire node fails?

With a healthy size-3 pool, losing one OSD or one full node does not interrupt running VMs — Ceph continues serving reads and writes from the remaining replicas and starts rebuilding redundancy onto other OSDs automatically. Combined with Proxmox VE HA, a VM on a failed node can also be restarted automatically on a surviving node, since its disk was never local to begin with.

Is Ceph Tentacle or Ceph Squid the right choice for a new cluster on Proxmox VE 9.2?

Fresh deployments default to Tentacle (20.2.1), which is the actively developed release. Squid (19.2) remains fully supported and is a reasonable choice if you need to match an existing cluster or prefer to wait for a release with a longer track record — Proxmox VE will not silently upgrade an existing cluster's Ceph release out from under you.

How much of my raw disk capacity will I actually get to use?

With the default replicated pool (size 3), expect roughly one-third of your total raw OSD capacity as usable space, before accounting for the free-space headroom Ceph needs to handle a node failure gracefully. Erasure-coded pools trade some performance for better space efficiency if raw capacity is a bigger constraint than IOPS.

Conclusion

Deploying Ceph turns a set of individually useful Proxmox VE nodes into a cluster that can actually survive hardware failure without you scrambling to restore VMs from backup. The process is more involved than pointing at an NFS share, but every step maps to a real design decision: how many replicas you can afford, how much network bandwidth you're dedicating, and how much headroom you're leaving for recovery. Start with three nodes, dedicated network bandwidth, and the defaults Proxmox VE ships with — size 3, min_size 2, PG autoscaler on — and only diverge from them once you have a specific, measured reason to.