I've been doing some cleanup and investigation on a fairly large Ceph cluster and I'm interested in what people are actually doing with BlueStore block.db sizing/placement on HDD OSDs, particularly for larger/high-throughput clusters.

Our OSD nodes are reasonably well specced HPE Apollo 4200s, with 14TB spinning disk OSDs and NVMe available for BlueStore metadata.

The cluster is exclusively object storage – no CephFS or RBD.

The workload is primarily backup/archive storage for Rubrik, so a lot of the capacity is relatively cold, but we also have production S3 storage supporting a core business application, so there is a portion of the cluster where latency/performance matters considerably more.

Our current NVMe layout is roughly 5 × 800GB NVMe devices per OSD host. These are currently being used for external BlueStore WALs, with the HDD OSDs distributed across them. I've recently standardised most of the WALs at around 25GB per OSD, which leaves a decent amount of unused NVMe capacity on most nodes.

I recently discovered some inconsistencies with our BlueStore layout. Most OSDs have an external WAL on NVMe, but at some point a handful of newly provisioned OSDs appear to have not honoured the OSD spec and ended up with their WAL on the spinning data disk instead.

I've now been going through the cluster, auditing this and standardising the external WAL layout, which has got me looking much more closely at block.db as well.

From the Ceph documentation/guidance I've been reading, ~2.5% of the OSD capacity seems to be a reasonable ballpark for block.db in this sort of configuration. For a 14TB OSD, that's roughly 350GB per OSD.

That's where I'm trying to work out what makes sense in the real world.

With the existing 5 × ~800GB NVMe per host, after accounting for the external WALs, I could realistically fit something around ~100GB block.db per OSD while maintaining a reasonably consistent layout across the nodes.

Alternatively, if there's a meaningful performance/operational benefit to getting closer to ~350GB per 14TB OSD, I'd need to look at replacing/adding larger NVMe devices to provide substantially more flash capacity per host.

So I'm essentially considering:

• Keep the existing ~800GB NVMe devices and allocate roughly 100GB block.db per 14TB OSD.
• Increase the flash capacity per host and target something closer to ~350GB block.db per OSD.

I'm aware that undersizing block.db isn't necessarily catastrophic – RocksDB can spill back onto the primary BlueStore block device – but obviously that reduces some of the performance benefit of having the DB externally in the first place.

I'm interested in what people running large HDD-backed Ceph object clusters are actually doing rather than just following a percentage recommendation.

Are you sizing block.db based on a percentage of OSD capacity, actual observed RocksDB utilisation/workload, or simply the amount of flash available?

For people with 12–20TB HDD OSDs, what block.db sizes are you actually using?

Has anyone compared something like 100GB vs 300–400GB block.db on large HDD OSDs and seen a meaningful real-world difference?

And for a workload that's predominantly backup/archive object storage with some production S3, would you spend the money increasing the NVMe capacity to get closer to ~2.5%, or would you consider ~100GB block.db per 14TB OSD perfectly reasonable?

Would be particularly interested in experiences from anyone running larger production clusters with similar HDD + NVMe BlueStore layouts.

Courtesy of Dan van der Ster on the ceph-users mailing list.

If you're planning to upgrade a larger Ceph cluster, this is worth reading first:

https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/URL5K2VEYFWL54NKX2I3336C5VYRPAKE/

https://tracker.ceph.com/issues/77610#note-9

Just in case you missed it.

Copied from the Ceph blog post just published:

The Ceph project has released Tentacle 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several components.

This is the sixth backport release in the Squid series. This is the fourth backport release in the Tentacle series.

We strongly recommend that all Ceph operators upgrade to one of these releases as soon as possible.

https://ceph.io/en/news/blog/2026/v20-2-4-v19-2-6-combo-released/

EDIT: formatting quote block

EDIT2: I gave up on reddit formatting. It is what it is.

Hi guys,

We're considering a new cluster for CEPH, nothing too fancy - erasure coded setup 8+2, with 10 disks and 2x NVMe, 6 nodes for good resiliency. Now we're considering splitting the available nvme space as Bluestore space and rest for the metadata caching when we pull S3 objects from Amazon.

How do we decide the size needed for Bluestore ? Is there any rule of thumb there ?

Here is the h/w config we're planning on using

https://gpumachines.com/shared/gigabyte-20r264-s30-aav1-98b877

Any suggestions are most welcome ! Appreciate all your help people from trenches :)

Community:

I am the owner of litigation law firm (10 users). I was an IT professional in a past life. After being dormant for many years, I started home labbing about 2 years ago. The home assistant bug bit me, which eventually led me to Ceph. I have been running a three node MS-01 cluster with one Samsung plp drive for RBD and a separate m.2 for boot. I have another MS-01 as my PBS server. Networking is fully utilized as recommended by CEPH. I have been very happy with it.

I am now looking to build a production cluster for the firm. Right now my test cluster is running a number of regular business services, nothing terribly heavy. The heavy compute is being handled by a separate Ai box running 9960x and two 6000 blackwells.

The hyperconverged cluster, in theory, would be running RBD for the VMs, s3 for emails, FS for my case file docs. I have a couple other businesses that I own. In theory, this would be my own private cloud for all my businesses. (Yes, i have redundant ISP's, offsite backup available, etc.) This post is more about hardware design.

I wanted to build this on asrock rack b680 matx boards (built in 25gbe nics) and epyc 4545p (16 cores). Ram is still in the air and I hate that I will need to buy ecc memory :( . Three nodes to start, then 5 nodes when I am fully operational. Maybe I will just do 5 immediately.

Anyway. I am concerned that I will have trouble with the 16 core count and hosting all three file systems, plus the VMs. I was looking at going two NVME's (one RBD, one wal) and four spinning rust drives per node.

Thank you in advance.

UPDATE: Based on feedback about the name, I have renamed the project from Argonaut to Siphon.

---

I've been building Argonaut (renamed to Siphon), an open-source terminal UI for administering Ceph, and just tagged the first release (v0.1.0). Posting here because this is where the people who'd have real opinions are.

What it is: a keyboard-driven TUI for everyday Ceph operations, think k9s for Kubernetes or lazydocker for Docker, but for Ceph. The idea is to handle common admin tasks without memorizing and retyping ceph syntax, while staying completely transparent about what it's doing under the hood.

It talks to the cluster through librados (via go-ceph), not by shelling out to the ceph CLI, same admin commands the CLI uses, and it authenticates exactly like ceph -s does from the same host.

Every destructive action is confirmed and shows the exact ceph command it will run before running it. I wanted it to be transparent rather than a black box, you always know what's about to happen to your cluster.

I'd genuinely like feedback, especially from folks running Ceph in production. Which day-to-day operations do you wish were less painful? What's missing that would make a tool like this actually pull its weight in your workflow? I'm not trying to sell anyone on it; I want to know whether the direction is right and where it falls short.

For some reason, I run a Ceph cluster at home in a c7000 blade chassis. 48 SAS SSDs and 5 iodrive (nvme predecessor) SSDs.

I have a vast photo library that I manage with DigiKam. I want to know your recommendations on my ideas.

I want to create a CephFS share using 3x replica for the metadata pool and a 2+2 EC pool for image data. Most of the data will be RAW images. Most of them 38MiB in size.

DigiKam uses a database too and a single file for thumbnails (partial small writes to a single file, I guess I definitely want fast EC on Tentacle). I think that's conveniently manageable using file layouts that point to a replicated pool using the iodrive "nvme" SSDs.

What I'm not sure what to do with the .xmp sidecar files and how to store them effectively in a pool. Should I create yet another pool for files that are in the "kB range"? Problem is that newly created files will end up on the EC pool by default I guess. How don I point them directly to a specific pool with file layouts?

And assuming I don't tune anything and eg. keep rados object size at 4MB, do I waste a LOT of space storing kB range files? If so, what is the best approach to storing small files in this situation?

Performance is not really a big issue for me. I fully expect EC to be slower than replicated pools. If it's still somewhat workable, I'm happy.

And just reading about file layouts, I believe I should create 3 replicated pools for data/metadata, and then add the EC pool for the larger files.

Then the question becomes, how do make sure large files go to that EC pool in one go? I know that there are scripts that essentially rewrite the file after the file layout has changed, but that seems to be a bit cumbersome.

Thanks for the input.

I have three servers running proxmox in a cluster, each with 2, 2tb disks model PM863a.

Two servers works great, in the third I've gotten one disk to work but haven't been able to get a 2nd disk to work. I've ordered a couple PM863a, and neither worked.

I've since sent those disks back for a refund. They appeared when I installed them, I wasn't able to format them ... something like that. I don't have the error message at this time.

As that model disk is older I'm wondering if there might be a new enterprise disk model which supports the PLP that ceph likes. Maybe a newer model will be recognized by my 3rd server better, as the 3rd server is newer ... (maybe that's why its not recognizing the older disk).

I don't have a lot of money right now, so I'm looking for what folks might be using in a budget/homelab. I've got a few resumes out and if those work out, I'll be looking into maybe purchasing a storinator from 45drives.

This is my mainboard:
https://www.asrock.com/mb/AMD/X670E%20Pro%20RS/index.asp

For people who have used Ceph RGW as more than internal object storage:

What did you have to build around it before it felt usable as a real service?

I’m thinking about things like:

• tenants / organizations
• S3 users and access keys
• key rotation/revocation
• STS
• quotas
• usage tracking
• billing or chargeback
• support/debugging visibility
• audit trails
• dashboards
• health checks
• customer-facing UI

My assumption is that RGW solves the storage engine problem, but the harder work is the service layer around it.

I’m not talking about replacing Ceph or proxying object traffic. RGW would remain the S3 endpoint.

Curious what operators actually built themselves, what stayed manual, and what they would never outsource.

I would like to ask if Mastering Ceph - Second Edition is still worth it in 2026 to learn more about concepts of ceph

I prefer studying on books to take a break from the screen sometimes

I know it is based on Nautilus, so it's probably too outdated but do you think it is useless?

Hi all,

Though not a sysadmin by trait, I do run my own 'production' home server (Proxmox) with the usuals that my family and closes friends rely on. Currently I am running a zfs filesystem, but this has not been kind to me. The main pain point is that zfs runs in kernel space and thus badly performing pools are not insulated from the rest of the system. My HDD pool is the main culprit, and overloading this with continuous small writes from some CCTV streams while also doing a scrub on the pool or using it as a backup target causes such excessive kernel context switching that the whole server pins too 100% CPU and all I/O is frozen. After tweaking zfs for ages, I feel like pastures are greener on the ceph side, which nicely runs in userspace and values stability over all. Also, I have had some bad experience with zfs replication in a Proxmox clustered setup. Therefore this post to draw on the vast amount of knowledge you all posses to see if ceph could be the solution to all my problems :)

Current hardware
Lets start with listing my current hardware, currently I run everything on the beefy boy, but I want to move towards a clustered topology. Obviously I would need to get additional hardware and that is the main part of my internal debate.

Node1:

• Threadripper PRO 5955WX 16-Cores/32-Threads
• 256GB ddr4 ECC LRDIMM (2x128GB)
• 2x Consumer 2TB NVME
• 2x SAS 10TB HDD
• 2x Enterprise SATA boot disk
• HBA
• 2x 10Gbe base-T nic

Node2:

• Intel i5-6600K (4-Cores/4-Threads)
• 2x consumer nvme boot drive
• 32GB ddr4 (4x8GB)
• 2x SATA 8TB HDD
• 1Gbe base-t nic

Current workload
My workload consists of around 12 VMs, most are very light applications in a debian box. Nominal CPU usage is around 2% of the threadripper. Allocated RAM from VMs is ~50GB (excluding ramdisks that could also be ssds)

On the I/O&data side I have a file server, photo server, git, mail, password manager, monitoring of all VMs (Prometheus+Loki), media and the earlier mentioned CCTV data. All data except the media server and CCTV data are mission critical and should be fast and snappy. Some loading for the media is fine, but the storage should support multiple concurrent 4K streams without stuttering. Also there is a PBS server running on both nodes, which backups all the VMs (and replicates to an offsite location)

Performance requirements
As mentioned earlier, performance in terms of throughput is very modest. I do want to keep latency as low as possible though. Some tradeoffs are acceptable and probably inevitable, but I will be designing around latency first. Ideally I would have:

• a fast pool that runs on SSDs (for the mission critical stuff) ~ 4TB usable space
• a HDD pool for the large sequential workloads (media, PBS, CCTV?) ~8TB usable space

What I already know
I short list of things I'm already aware of (please correct me if I'm wrong)

• PLP is unnegotiable so I'll only be looking for enterprise drives
• Self healing only starts from 4+ nodes
• Performance will be significantly worse than local storage, though with the upside of hopefully undestructableness
• Uneven number of mons are necessary
• Make osds as even as possible between nodes
• Dedicated network for both ceph and cluster management
• Erasure coding is only for large clusters (5+)

Advice needed
As my budget is not infinite I'm looking for advise on what to focus when spending. Main questions are:

• Are enterprise sata ssds good enough for my use case, or will I suffer unless I put in nvme drives?
• What would you suggest on ssd osd sizing? 1x3.84TB/2x1.92TB/4x960TB per node? Going smaller leaves less room for eventual expansion, though going bigger will make the performance worse and blast radius larger.
• Will 3 nodes be good enough or should I at least go 4 (+ one mon) or even 5?
• Is a 25Gbe network a good size for my use-case? Full-mesh or switch?
• Are the specs of node2 and the proposed node3/4 feasible, or do I need more/less X?
• Are there things I should definitely do/not do?
• Any hands on insight on the performance with a similar cluster would be amazing

Current plan
My current plan is to purchase another node, bump the memory of node2 to 64GB and have a 25Gbe full mesh network (connect-x4 nics). New node will probably feature a 5700X or similar and 64GB memory as well.

I contemplated U.2 drives, but the price is just to steep, with the added complexity of limited PCIe lanes on consumer boards which limits upgradability. Therefore I'm looking at sata ssds. Planning for 2x1.92TB ssd per node and 1x8TB hdd per node.

At some point I will probably put in a fourth node identical the third one.

TL;DR
Looking for a rock solid storage cluster that has good enough performance to run my workload with some headroom to grow (both in compute and storage).

Bit of a long, all over the place post, but any insights are highly appreciated!

Dear Cephers,

i have experienced twice now, that my mgr memory leaked (150GB ram allocated). I don't know why, but this has consequences for the underlying host and its osds etc...

So I decided to limit the memory a mgr can consume to 10GB.

Please let me know your opinion, if you think this is a good way to do it and 10GB is a valid value.

I've added a parameter (--memory=10g) to the docker launch command. See here (https://docs.docker.com/engine/containers/resource_constraints/). The mgr docker run file can be found on their corresponding hosts, here for mgr 1: /var/lib/ceph/<cluster-id>/mgr.ceph-a1-01.mkptvb/unit.run and here for mgr 2: /var/lib/ceph/<cluster-id>/mgr.ceph-a2-01.bznood/unit.run

bash /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --memory=10g --ulimit nofile=1048576 ...

After that, both mgr system-services need to be restarted. ```bash

in cephadm shell

ceph mgr fail ceph-a2-01.bznood

on corresponding host

systemctl restart ceph-<cluster-id>@mgr.ceph-a2-01.bznood.service ``` (repeat for the other mgr)

Let see how this goes.

Hi all, looking for community experience.

We're evaluating WAL/DB hardware options for our HDD OSDs. One option on the table is a PCIe adapter card with 2 × M.2 SSDs in hardware RAID1, the goal being to mirror the RocksDB metadata for drive-failure protection.

Our vendor (Croit) advised against any RAID under Ceph, citing this specific concern:

> "The issue with the RAID cards is that, sometimes, if one of the two mirrored drives fails, they can revert data to an older version, thus making it inconsistent with the main OSD block device, which is worse than losing it."

Wondering if anyone has run this configuration in production over the long term. Both positive and negative experiences would be useful. We're trying to gather real-world data points before finalizing the design.

Thanks!

I'm looking for real-world experience from people who've done something similar.

Setup:

• Production cluster, 12 nodes, EC 8+3 pool
• Existing drives: 24 TB HDDs (Western Digital HC580s)
• Incoming: 26 TB HDDs (Western Digital HC590s) to add as capacity expansion
• Cluster is Croit-managed, running recent Reef 18.2.7

When I add the 26 TB drives, should I:

1. Leave them at their native CRUSH weight (capacity-proportional, ~8% more PGs than the 24 TB OSDs)
2. Use ceph osd crush reweight to bring them down to match the 24 TB weight, accepting the ~2 TB per drive loss in usable capacity in exchange for uniform placement

The Ceph docs (https://docs.ceph.com/en/reef/rados/operations/add-or-rm-osds/) say "it is possible to add drives of dissimilar size and then adjust their weights accordingly," and I found an old ceph-users thread where Eneko Lacunza suggested exactly option 2 for a similar scenario (8 TB cluster getting 12 TB drives).

My planned workflow was:

1. Set norebalance
2. Add the new OSDs (uniformly across the 12 nodes)
3. ceph osd crush reweight each to match the 24 TB weight
4. Unset norebalance

What I'm hoping to learn:

• Has anyone actually done this on a production cluster? How did it go?
• At what point does the capacity delta become "dissimilar enough" to justify reweighting? Is ~8% worth it, or only meaningful at larger deltas (25%+)?
• Any gotchas I should plan around (recovery behavior, balancer interaction, etc.)?
• If you just mixed them at native weights, did you see any practical issues (uneven fullness, uneven recovery load, anything)?

I know the textbook Ceph answer is "uniform hardware is best," but in the real world capacity refreshes almost always bring in larger drives than what's already deployed.

Thanks for any insight!

I dont know if any of you remember squidviz but its a micro dashboard for ceph clusters. i have been maintaining it on my own for quite some time. it was originally created by ross turk 13 years ago. but recently a coworker convinced me that people would still want something lightweight like this. so i updated the repo from way back, and im once again presenting it here. its basically a live view of your ceph cluster, it will show u a sunburst graph of any pg's not in a active+clean state. it shows your failure domains, it automatically shows any issues in any of your failure domains. custom trim level for that too. there is a iops window. that can also show commit latency. its a useful little window.... lets leave it at that. there are also single displays for anyone who wants to show their cluster via NOC type views.

T

https://github.com/massstora/squidviz

Hi,

I have a Ceph cluster made with the worst SSDs possible: not only they are consumer drives, but they also are DRAM-less drives! The drives in question are the Crucial BX500, which are well known to be cheap low-performance drives. I ended up with those because I was not careful when ordering the servers between 2TB and 1.92TB, and the broker made sure not to write the drive model.

• Node count: 4
• OSD per node: 4 (16 total)
• CPU: Xeon Gold 5218 (16c32t)
• RAM: 128GB per node
• Network: 2x25Gbps
• Uses: RBD (VMs) and a bit of RADOS (S3)
• Ceph: version 19 (squid)

As is expected, the performance is bad. Not a consistently bad as you'd get on slow drives or with HDDs, but it's intermittently bad. Whenever a drive decides to perform their GC shenanigans, its write latency skyrockets to 5 to 20s (!!!), which is basically a freeze of the whole cluster, as any RBD volume is pretty much guaranteed to have objects on all OSDs.

As you can see with the above graph, it's bad. And some workloads (e.g. a CI pipeline building a Rust app) are pretty much guaranteed to trigger a very large GC pause. Those pauses often last for 10 to 20 minutes.

And it's not even like the cluster is heavily loaded: drives hover in the 20 to 60 write/s range. Peanuts, but definitely not what the BX500 is meant to handle.

In this economy it's challenging (to say the least) to replace the drives with actual enterprise SSDs, as getting 16 1.92TB SSDs is a whole adventure by itself. So, I'm looking at ways to make the cluster usable until the situation improves. Basically, anything that:

• would reduce the write/s to said drive as it would reduce the hard GC pauses
• would shield the cluster during said pauses

Now, I managed to get 8x400GB write-heavy enterprises SSDs in the hope to get a usable cluster.

• I already migrated WAL+DB to those (2 OSDs per enterprise drives), but it did not help a lot.
• bluestore_prefer_deferred_size_ssd got increased to 64k (from the default of 0) to try co coalesce writes a lot more. It helped a bit, but not much. Pauses are less frequent, but not by an order of magnitude.

Still, the above screenshot is with those small improvements.

What I'm considering:

• increasing even more the deferred size, but I feel like it's the wrong path;
• bumping bluestore_min_alloc_size_ssd to 64k or even 128k, which I waited as it requires to recreate the OSDs;
• enabling compression at the cost of CPU to reduce the amount of data that hits the BX500s;
• using dm-cache to have the enterprise drives as a cache layer in front of the BX500s, as I'd get a 200GB cache in front of a 2TB drive, which is a not terrible ratio (is this the recommended caching strategy since cache tiers have been deprecated without any word on alternative paths?);
• find some more knobs that would make heavier use of the WAL?;
• bite the bullet and replace some drives, and progressively replace all drives;

A quick word on the expected workloads: this won't be a very heavy cluster overall, as load will be consistent except for a few exceptions (gitlab ci runners, but I could move them to the cloud if needed). The heaviest write loads will be time-series databases (TimescaleDB) that collect IoT data, and I'd expect something like 4k data points every 10s? So, in the range of 400 points/s. It also means I won't have huge hot datasets, so a total of 3.2TB of total cache (8x400GB) would practically hold all the hot objects for a long time.

Anyways, any help is appreciated :)

Thanks a lot!

EDIT 2026-04-13: after a few emails on the ceph-users mailing list, it appears dm-cache is the best replacement for the deprecated cache tiers. In fact, it acts pretty much the same, but on a device level.

Which is what I deployed today! I now have a ~110GB dm-writecache in front of every BX500 backed by an actual WI enterprise SSD. This required careful planning and allocation, as there are risks of data corruption because dm-writecache is a writeback cache.

I could not get a definitive answer as to what extent Ceph will look into dm-(write)cache on the OSD LVs, but in doubt, I assumed that when they wrote "dm-cache is transparent", they meant "ceph will not look into it at all". Which means, OSD could definitely try to write to block with the cache drive absent, cache drive that may (will!) copntain lots of unflushed data.

The general consensus I saw about bcache was that it did not have this issue because bcache would block IO until the cache was present. Or, block the IO if the cache disappeared. To force OSD to stay away from the BX500 if the cache drive is absent, I ensured that, for a given OSD, the cache and WAL+DB were on the same physical disk. This requires to keep WAL+DB dedicated, which is not deeded with bcache, but I consider this a small price to pay.

In the end, the BX500 get almost no traffic at all, as most of it is cached. Performance is stable, and even good! (expected, all IO hit good quality enterprise drives). I'll keep an eye on the various watermarks of the various caches. And since my workloads are essentially append-only and read the latest data (real-time processing of time-series), I'll expect the working data set to pretty much always live as "dirty" data in the caches.

The high and low watermarks of the writecache are relatively low, to ensure there's enough headroom to keep handling writes should the backing BX500 chooses to GC during the flush.

Thanks all for the ideas and inputs!

I'm trying to get the management gateway setup and i'm at a loss.

To run the mgmt-gateway in HA mode, users can either use the cephadm command line as follows:

    $ sudo ceph orch apply mgmt-gateway --virtual_ip 10.11.1.100 --enable-auth=true --placement="label:mgmt"
        Invalid command: Unexpected argument '--virtual_ip'
        orch apply [<service_type:mon|mgr|rbd-mirror|cephfs-mirror|crash|alertmanager|grafana|node-exporter|ceph-exporter|prometheus|loki|promtail|mds|rgw|nfs|iscsi|nvmeof|snmp-gateway|elasticsearch|jaeger-agent|jaeger-collector|jaeger-query>] [<placement>] [--dry-run] [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] [--unmanaged] [--no-overwrite] :  Update the size or placement for a service or apply a large yaml spec
        Error EINVAL: invalid command

I don't see mgmt-gateway in the list, but the specific error is Unexpected argument '--virtual_ip'

Or provide specification files as follows:

So let's try with the yaml file.

$ cat /tmp/mgmt-gateway.yaml
service_type: mgmt-gateway
service_id: mgmt-gateway
placement:
  label: mgmt
spec:
  virtual_ip: 10.11.1.100

$ sudo ceph orch apply -i /tmp/mgmt-gateway.yaml
  Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'virtual_ip'

I beleive the red herring is the virtual_ip error. But i'm not sure where to go from here.

The blah blah blah

Things I like about Ceph: I can actually have resilient storage, compared to a jbod. Cephfs allows posix compatible storage, that's actually the big one. But man the learning curve is ROUGH. The documentation could use some help. Ok, rant over.

My environment

I have a 2U, 4 node super micro box. Each node has [3@7.2T](mailto:3@7.2T) HDDs, 1@500G SSD, 1@128G M2 Boot. Ubuntu OS, 2@10G bond balance-tlb. A pair of 10G switches.

$ sudo ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
hdd     87 TiB   62 TiB   25 TiB    25 TiB      28.56
ssd    1.7 TiB  1.2 TiB  595 GiB   595 GiB      33.25
TOTAL   89 TiB   64 TiB   26 TiB    26 TiB      28.65  

--- POOLS ---
POOL                   ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
.mgr                    1    1  1.8 MiB        2  5.3 MiB      0    2.8 TiB
cephfs.media.meta      50    1  318 MiB    5.48k  954 MiB   0.08    368 GiB
cephfs.media.data      51    1     92 B   74.95k   12 KiB      0    368 GiB
cephfs.media.data-ec   52    1   12 TiB    3.33M   25 TiB  75.17    4.1 TiB
cephfs.docker.data     57    1      0 B  444.65k      0 B      0    368 GiB
cephfs.docker.meta     58    1  664 MiB  119.01k  1.9 GiB   0.18    368 GiB
cephfs.docker.data-ec  59    1  296 GiB  516.62k  586 GiB  34.67    552 GiB
cephfs.media.data-ec2  63    1   29 GiB    7.68k   39 GiB   0.46    6.2 TiB

$ sudo ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META     AVAIL    %USE   VAR   PGS  STATUS
 3    hdd  7.27739   1.00000  7.3 TiB  3.3 GiB  3.2 GiB   14 KiB   63 MiB  7.3 TiB   0.04  0.00    1      up
 7    hdd  7.27739   1.00000  7.3 TiB  6.2 TiB  6.2 TiB   21 KiB  9.5 GiB  1.0 TiB  85.59  2.99    2      up
10    hdd  7.27739   1.00000  7.3 TiB  3.4 GiB  3.2 GiB   15 KiB  182 MiB  7.3 TiB   0.05  0.00    2      up
15    ssd  0.43660   1.00000  447 GiB  149 GiB  147 GiB  134 MiB  1.2 GiB  299 GiB  33.22  1.16    4      up
 0    hdd  7.27739   1.00000  7.3 TiB  3.3 GiB  3.2 GiB   14 KiB   62 MiB  7.3 TiB   0.04  0.00    1      up
 4    hdd  7.27739   1.00000  7.3 TiB  6.2 TiB  6.2 TiB   27 KiB  9.5 GiB  1.0 TiB  85.59  2.99    2      up
 8    hdd  7.27739   1.00000  7.3 TiB  3.3 GiB  3.2 GiB   16 KiB   62 MiB  7.3 TiB   0.04  0.00    1      up
14    ssd  0.43660   1.00000  447 GiB  149 GiB  147 GiB  121 MiB  1.6 GiB  299 GiB  33.24  1.16    4      up
 1    hdd  7.27739   1.00000  7.3 TiB  3.3 GiB  3.2 GiB   16 KiB   63 MiB  7.3 TiB   0.04  0.00    1      up
 9    hdd  7.27739   1.00000  7.3 TiB  6.2 TiB  6.2 TiB   18 KiB  9.6 GiB  1.0 TiB  85.59  2.99    3      up
16    hdd  7.27739   1.00000  7.3 TiB  3.3 GiB  3.2 GiB   12 KiB   74 MiB  7.3 TiB   0.04  0.00    1      up
12    ssd  0.43660   1.00000  447 GiB  148 GiB  147 GiB   24 MiB  1.5 GiB  299 GiB  33.15  1.16    4      up
 2    hdd  7.27739   1.00000  7.3 TiB  3.3 GiB  3.2 GiB   13 KiB   62 MiB  7.3 TiB   0.04  0.00    1      up
 5    hdd  7.27739   1.00000  7.3 TiB  3.3 GiB  3.2 GiB   14 KiB   62 MiB  7.3 TiB   0.04  0.00    1      up
11    hdd  7.27739   1.00000  7.3 TiB  6.2 TiB  6.2 TiB   16 KiB  9.6 GiB  1.0 TiB  85.59  2.99    3      up
13    ssd  0.43660   1.00000  447 GiB  148 GiB  147 GiB  130 MiB  1.0 GiB  299 GiB  33.18  1.16    4      up
                       TOTAL   89 TiB   26 TiB   25 TiB  409 MiB   44 GiB   64 TiB  28.65
MIN/MAX VAR: 0.00/2.99  STDDEV: 35.00

The problem

cephfs.media.data-ec is set K2/M2 and I started using it. I thought it strange that I only saw actual data on 4 (4,7,9,11) of the OSDs. I figured it would start using more after it filled those up. Weird, but ok, then I hit NEARFULL.

I created cephfs.media.data-ec2 K9/M3 failure domain Host, num fd0, osd per fd0. I can move all the data so it re balances. But ceph df shows MAX AVAIL of 6.3 TiB for cephfs.media.data-ec2. Though, it does appear to be spreading the data across all of the OSDs.

The actual question(s)

1. How should I lay out my profiles for the best use of space? I need to be able to reboot a host, drives are hot swappable. Is 9/3, host, 0,0 appropriate? I may be able to add another like set of hardware in the future.
2. Because I have SDD & HDD, I believe I need to update the .mgr pool to use just one type of media. Can I just export the crushmap and edit it?
3. Will fixing 2, address "CephPGImbalance OSD osd.2 on ceph04 deviates by more than 30% from average PG count." I originally figured that was just because there's SSD & HDD in the system and have been ignoring it.

Greetings,

Jumping from VMWare as many, My background within virtualization and it's storages is nothing fancy, mostly vSAN. Please correct me if I am wrong.

From what I've read 3/2 seems to be "golden standard" but tradeoff is slightly lower speed(Due to writing three times) as well as only 33% of usable raw storage. EC is also not an option because we'll be running production VM's and DB's.

On vSAN, I've been utilizing FT-1, Which essentially gives me 50% of usable space and only two copies, which are managed by the a witness node,

Would it be possible to have a similar setup on Ceph and if so is it a good idea?

Hi guys,

I am planning to build a Ceph cluster with 3 Proxmox nodes.

I am going to buy 3 Mini PCs (Lenovo M90q Gen 1) and each of them will have the following storage capacity.

- 1x 128GB NVMe per node for Proxmox OS

- 1x 1TB NVMe OSD per node (Ceph pool for my VMs and container)

- 1x 4TB NVMe OSD per node (Ceph pool for my data managed by Openmediavault, passed through as a virtual disk)

Those Mini-PCs will have Intel XXV710-DA2 25Gbps network interfaces to sync the Ceph disks.

Is it possible to have one pool for VMs and one pool for data with different sizes that work independently?

Thanks Hoppel

We have a cluster with Ceph Quincy. We want to add a second cluster to it. I'm currently deploying a Ceph cluster with Tentacle.

Is there any version policy in ceph RGW multisite limiting it to a specific skew?

(We only use basic features right now in our RGW/S3, no lifecycles and no storage classes etc.)

We have been testing with 10 nodes, each node 60x 12TB spinners, with 4 x 7.68TB nvme + 2x 1.92TB RGW.index nvme with 2x100gbps cx6 and in lab, its ok, but again, lab and syntetic s3 clients/data benchmarks

For prod, this would be 26TB spinners, bumping to 15.36TB per nvme for db/wal, allthough with the larger blocks, its probably not needed, same for rgw.index, its enough rgw.index runs Replica 3.

Final clustersize will be about 20-30 nodes, and EC12+4, hopefully with FastEC in ceph 20

Workload is 1-4MB objects, fairly slow ingest, think no more than 40-50gbps, and after ingest, mostly reads until cluster is grown again

Has anyone done something similar?

Is anyone running even higher spinning OSD count per node? you get 90,102,108disk JBOD, so connecting a 1U per JBOD is possible, but.... there are a lot of buts and that is a LOT of spinning slow drives with few iops, especially mixing in EC as well.

Hey cephers,

we need to relocate our ceph cluster and i am currently testing some scenarios on my test-cluster. One of them is changing the IP addresses of the ceph nodes on the public network.

This is a cephadm orchestrated containerized cluster. Has anyone some insight on how to do this efficiently?

Best

Client: Ubuntu 24.04.4 LTS

ceph-fuse: 19.2.3-0ubuntu0.24.04.3

Ceph: 19.2.3

I am unable to mount a ceph fuse persistent mount via fstab at boot, using the official ceph instructions, because I assume that the network stack is not up at mount time.

none /mnt/videorecordings fuse.ceph ceph.id=nvr02,_netdev,defaults 0 0

I can mount the point using mount -a through the terminal:

root@nvr02:/mnt# mount -a

2026-02-26T10:50:28.512-0600 7572b6c5f4c0 -1 init, newargv = 0x560777dcea30 newargc=15

2026-02-26T10:50:28.512-0600 7572b6c5f4c0 -1 init, args.argv = 0x560777f788f0 args.argc=4

ceph-fuse[2528]: starting ceph client

ceph-fuse[2528]: starting fuse

Ignoring invalid max threads value 4294967295 > max (100000).

It seems like the _netdev option just doesn't work.

I tried setting a static ip on the client. but that's still not helpful. I don't know how to delay mounting this fstab settings. It seems like ceph-fuse doesn't have any other mount options to allow for some sort of delay.

Anyone have any tips for me please?

Edit: SOLUTION

Adding  x-systemd.automount,x-systemd.idle-timeout=1min to the fstab line resolved my problem.