Ceph calculator

How much usable space Ceph gives with 3-way replication

Dividing the raw capacity by three gives the theoretical space. The number that matters is a different one: how much data you can store so that, after the worst failure you want to survive and the recovery, the fullest OSD stays below the nearfull threshold (85% by default). The calculator calls it the recommended capacity and estimates it with ideal placement: data spreads across domains in proportion to capacity, with one copy or chunk per domain. The imbalance between OSDs is an explicit assumption, 5% by default, that you can change.

L   = (threshold - block.db share) / (1 + imbalance)
a_d = capacity of domain d × L, in decreasing order
X*  = min over j from 0 to w-1 of S_j / (1 - j/w)
logical capacity = X* / amplification

L is the maximum fill allowed for an OSD, net of block.db and imbalance; a_d is the usable space of domain d; w is the number of copies with replication, or k+m with erasure coding; S_j is the sum of the a_d excluding the j largest domains; the amplification is size with replication and (k+m)/k with erasure coding. A domain cannot hold more than X*/w, because every object has at most one copy or chunk per domain: this constraint is where both the space stranded by domains of different capacity and the failure reserve come from.

Worked example: 3 nodes with 4 OSDs of 3.84 TB, 3-way replication, host failure domain, block.db on a dedicated device. Raw capacity is 46.08 TB, or 41.91 TiB, and the theoretical capacity 13.97 TiB. With three nodes, the copies lost with a node cannot be rebuilt, because a fourth domain would be needed: the failure to survive therefore becomes the loss of one OSD. That OSD's data is rebuilt on the other three OSDs of the same node, whose usage grows by a third; to keep them below 85% you can store 8.91 TiB with no imbalance, 8.48 TiB with 5%.

Next to the recommended capacity the calculator shows two more values. The rebuild limit, based on the backfillfull threshold (90%): above it, backfill after the failure stops and the rebuild does not complete. The write limit, based on the full threshold (95%): above it, Ceph blocks writes.

Replication or erasure coding

3-way replication keeps three full copies of every object: amplification 3, min_size 2, reads from a single copy. Erasure coding k+m splits every object into k data chunks and m parity chunks: the amplification drops to (k+m)/k, that is 1.5 for a 4+2, at the cost of more CPU and more latency. To use an EC pool with RBD or CephFS you need allow_ec_overwrites, which enables partial writes.

You need at least k+m domains, and the Ceph documentation advises one more so that the pool can be rebuilt after a failure. With EC 4+2 on 6 hosts, losing a host leaves the data readable, but the lost chunks cannot be rebuilt; on 7 hosts they can.

Worked example: 7 nodes with 4 HDDs of 16 TB, the loss of one node to survive, no imbalance between OSDs and block.db on the same drives at 2% (the default). With 3-way replication you can store 96.62 TiB, with EC 4+2 193.25 TiB: twice as much, on the same drives.

What happens when a node fails

For every failure the calculator answers three questions. Is the data safe? Yes if the failure hits fewer domains than there are copies (with replication) or at most m domains (with erasure coding); beyond that it depends on how the PGs are placed, and the calculator says ‘not guaranteed’. Does I/O continue? Yes if every PG keeps at least min_size copies or chunks, the MONs stay in quorum and peering completes. Is full redundancy restored? Only if at least w domains remain and, after recovery, the fullest OSD stays below backfillfull, so that backfill can complete.

The worst failure, the one that sets the recommended capacity, is found by searching every combination of the failures to survive, not just by removing the largest node: with domains of different capacity the worst case can be a different one.

With 3 nodes and 3-way replication, after losing a node the data keeps two copies and I/O continues thanks to min_size 2, but the third copy is not rebuilt until the node comes back. In any case Ceph does not rebuild straight away: with the default settings it notices that an OSD is down in about 20 seconds (osd_heartbeat_grace) and marks it out after 10 minutes (mon_osd_down_out_interval). A whole rack is not marked out automatically (mon_osd_down_out_subtree_limit defaults to rack): in that case you start the recovery yourself.

How big block.db and WAL should be

With OSDs on HDDs, block.db on NVMe speeds up metadata and small writes. The published rules do not agree:

The pveceph 10% is what it allocates when you give no size and Ceph has none configured: it does not measure the need. The calculator recommends 2% for RBD and 4% for CephFS, RGW and mixed workloads, the top of the official Squid and Tentacle recommendation for each workload. It is the calculator's own choice: the development documentation asks for at least 2.5%, and to follow it you only need to add a margin in the DB/WAL tab.

Worked example: with 12 OSDs of 16 TB per node and an RBD workload, 2% gives 320 GB of block.db per OSD, 3.84 TB per node before any margin. A 3.84 TB NVMe drive would be enough only if all of its capacity were allocatable. The Ceph hardware recommendations suggest up to 10 OSDs per NVMe drive, 15 with a PCIe Gen4 or newer NVMe: 12 OSDs fit on one device only if it is Gen4. And if a device fails, every OSD it serves stops with it: with one device that is all 12 OSDs of the node, with two devices 6. The DB/WAL tab of the calculator simulates exactly that failure.

The WAL sits on the DB device and uses its space, so it must not be counted twice; a separate WAL only makes sense on a device faster than the DB one. The RocksDB levels (1, 8, 64 and 512 GiB, with use_some_extra) explain how the DB grows but are not sizing steps: the old 3, 30 and 300 GB values no longer hold as a rule. When block.db fills up, the excess metadata spills over to the slow drive and Ceph raises BLUEFS_SPILLOVER.

SSD and NVMe: what actually speeds things up

• DB and WAL on NVMe: they speed up metadata and small writes; they are not a cache for data.
• Pools per device class: an NVMe pool for VM disks and an HDD pool for backups and cold data; each pool uses only its own class, and the calculator treats them separately.
• Primary affinity: among the OSDs of a PG it makes the SSD one the primary, if the PG has one; reads go there, but writes still wait for every copy.
• Cache tiering: deprecated since Ceph Reef, because it has long had no maintainer and may be removed; better not to use it in a new cluster.

RAM, CPU and network per node

The starting point for RAM is osd_memory_target, 4 GiB per OSD. It is a target, not a cap: Ceph frees memory on a best-effort basis and an OSD can go over it, so the calculator adds 20%. The Proxmox VE documentation suggests 8 GiB per OSD instead. With 12 OSDs per node that is 57.6 GiB with the first rule and 96 GiB with the second, plus MON (4 GiB), MGR (2 GiB), MDS if you run CephFS (5 GiB), the operating system and the VMs.

For CPU the calculator takes 1 to 3 threads per OSD on HDD, 2 to 4 on SATA SSD and 4 to 6 on NVMe as a starting point, plus one thread for each service (MON, MGR, MDS). These are threads, not physical cores.

Proxmox VE recommends at least 10 Gb/s dedicated to Ceph, 25 Gb/s or more with NVMe drives, and 40 or 100 Gb/s to make the most of fast NVMe, with Corosync on a separate network. Every GB a client writes generates more GB of traffic between OSDs: size minus 1 with replication, (k+m-1)/k with erasure coding for full writes, that is 2 GB with 3-way replication and 1.25 GB with EC 4+2. Partial erasure-coded writes can generate more.

How long recovery takes

The transfer time is the longest of four times, one for each possible bottleneck: the drives receiving the data, the network of the nodes receiving it, the surviving drives (which read and write at the same time) and the cluster network as a whole.

time = max( received by the busiest OSD / OSD bandwidth,
            received by the busiest node / network bandwidth,
            V × (read factor + 1) / (surviving OSDs × OSD bandwidth),
            V × network factor / (surviving nodes × network bandwidth) )

V is the volume to rebuild; the factors are 1 with replication and k with erasure coding, because the primary reads k chunks to rebuild one. As the bandwidth of an OSD the calculator uses the mClock defaults (osd_mclock_max_sequential_bandwidth_hdd and _ssd): 150 MiB/s for HDDs, 1,200 MiB/s for SSDs and NVMe. They are configuration settings, not drive measurements: you can enter your own measurement in the advanced settings. The conservative scenario uses half of that bandwidth with the network at 50%, the optimistic one all of it with the network at 90%. It is an estimate, not a guaranteed speed: your own measurements will change the number.

Worked example: 5 nodes with 6 NVMe drives of 7.68 TB and a 25 Gb/s network, the cluster filled up to the recommended capacity, no imbalance between OSDs, block.db on the same drives at 2% and the loss of one node. There are 30.6 TB to rebuild: the transfer takes between 45 and 82 minutes and the bottleneck is the network. Failure detection and, with the default settings, the 10 minutes before the OSDs are marked out come on top.

How we calculate: formulas and sources

Model 1.2.0, dated 19 October 2026, for Ceph Squid 19.2 and Tentacle 20.2; Proxmox VE 9.2 installs Tentacle 20.2.1 by default. The calculator engine is checked against an independent reference implementation on 22 capacity cases from 3 to 128 nodes, failure scenarios, recovery estimates, PGs and block.db, and the worst-failure search against an enumeration of every combination.

Left out of version 1: several protection schemes on the same drive class, pools spanning classes, custom CRUSH rules beyond OSD, host and rack, stretch clusters, compression, small-object amplification and client performance.

• Ceph, pools, size and min_size
• Ceph, erasure coding
• Ceph, nearfull, backfillfull and full thresholds
• Ceph, when an OSD is marked out
• Ceph, BlueStore and block.db (Tentacle)
• Red Hat Ceph Storage 8, BlueStore
• Ceph, hardware recommendations
• Ceph, mClock and sequential bandwidth
• Ceph, cache tiering deprecation
• Proxmox VE, hyper-converged Ceph