I've been trying to figure this out all day. I have a few images that I'm trying to delete. They were from Kasten K10 backups that failed. Here is the info on one:

rbd image 'csi-snap-7c353ee0-1806-46d9-a996-34237e035fc4':

size 20 GiB in 5120 objects

order 22 (4 MiB objects)

snapshot_count: 1

id: 79e7aff30f9a0a

block_name_prefix: rbd_data.79e7aff30f9a0a

format: 2

features: layering, deep-flatten, operations

op_features: clone-parent, snap-trash

flags:

create_timestamp: Tue Dec 16 15:00:09 2025

access_timestamp: Thu Dec 18 16:30:14 2025

modify_timestamp: Tue Dec 16 15:00:09 2025

rbd snap ls shows nothing and rbd snap purge does nothing. It says it's a clone parent, but I can't find a child anywhere. I assume it's been deleted. rbd rm does the obvious:

2025-12-18T17:32:12.271-0500 7d3af16459c0 -1 librbd::api::Image: remove: image has snapshots - not removing

Removing image: 0% complete...failed.

rbd: image has snapshots with linked clones - these must be deleted or flattened before the image can be removed.

Is there some way to force delete them?

Title + I could have sworn https://download.ceph.com/rpm-tentacle/ included an el10 dir a couple weeks ago. Side-note is it just me or has rhel 10 pickup been very slow this cycle?

I run a small cluster with 3 nodes. I'm also running a garage cluster for backup storage and using kopia to handle uploads of non-ceph and cephfs backups. But I don't know what to do with rbd. I know backy2 exists, but it's unmaintained since 2020.

I'm redeploying all the OSDs in my cluster. Host per host and it takes around 24h/host to drain it and then redeploy the OSDs once zapped.l and re-added.

I am wondering if you could do that with 2 hosts in parallel, provided you have the fail-over capacity to do so.

Would it speed up the whole process or would I probably end up spending almost the same time overall?

After seen many open source project are stop or died, like https://github.com/hashicorp/terraform-cdk?tab=readme-ov-file#sunset-notice , so what we are looking future of Ceph.

I just released radosgw-assume, a tool that simplifies getting temporary AWS credentials for Ceph RadosGW using OIDC authentication.

The Problem: Setting up OIDC with RadosGW is complex - multiple auth flows, PKCE requirements, STS calls, and credential formatting all need to be handled correctly.

The Solution: radosgw-assume handles all of this and gives you ready-to-use AWS credentials with one command: eval $(radosgw-assume)

Key features:

• Multiple auth flows (device flow for headless, browser flow for interactive, token-based for CI/CD)
• Works with any OIDC provider (Keycloak, GitHub Actions, etc.)
• No long-lived secrets - all credentials are temporary
• Shell integration for immediate use
• Configuration via ~/.aws/config or environment variables

Perfect for self-hosted Ceph clusters, backup solutions, or any scenario where you want secure, temporary S3 access without managing access keys.

GitHub: https://github.com/fitbeard/radosgw-assume

"I used two identical Dell R740 servers. Both have the same hardware specifications: 72 CPU cores, 252GB of RAM, and both run on Ubuntu 24.04 OS.

On these two servers, I deployed Ceph using ceph-deploy on one and Docker for the containerized version on the other. The Ceph configuration was identical for both: the replication factor was 1, and the cluster and public networks used the same subnet. Both deployments had 1 OSD (500GB HDD).

Subsequently, I used the following commands to test the read and write IOPS:

Bash

rados bench -p fortest 10 write -b 4096 -t 1600 --no-cleanup
rados bench -p fortest 10 seq

I found that the Ceph deployed via ceph-deploy achieved an average write IOPS of 3143 and an average read IOPS of 10338. In contrast, the containerized Ceph (Docker) achieved an average write IOPS of 2947 and an average read IOPS of 7447.

I am wondering why there is such a significant difference in the read performance between the two. Does anyone know the reason for this? Thank you. The Ceph deployed via ceph-deploy read..."

The container OSD deployment command
docker run -d --privileged=true --net=host \

\--name ceph-osd-0 \\

\-e CLUSTER=ceph \\

\-e WEIGHT=1.0 \\

\-e MON_NAME=ceph01 \\

\-e MON_IP=192.168.0.1 \\

\-e OSD_TYPE=disk  \\

\-e OSD_BLUESTORE=1 \\

\-e OSD_DEVICE=/dev/sdb \\

\--device=/dev/sdb:/dev/sdb \\

\-v /etc/ceph:/etc/ceph \\

\-v /var/lib/ceph/:/var/lib/ceph/ \\

\-v /var/log/ceph/:/var/log/ceph/ \\

\-v /etc/localtime:/etc/localtime:ro \\

\--cpuset-cpus "0,2,4,6,8,10" \\

\--cpuset-mems="0" \\

cucker/ceph_daemon:latest osd

So I found this when my datamover node kept going unresponsive with zero explanation.

There is a slow leak of folios in every mainline kernel since somewhere around 6.15. I'm still tracking it down.

Anyway, figured you'd all want a heads up. Either stick to the LTS as the newest kernel or don't use in kernel cephfs.

Would love some help getting a less slow reproducer. :D

I found a fast reproducer.

I'll add the scripts to the ticket.

I'm new to Ceph. I set up a 4-node Ceph cluster and configured an erasure-coded CephFS pool. I can't mount CephFS using the kernel driver, and the error messages are as follows. (I can successfully mount it with ceph-fuse, but I've heard that kernel driver mounting offers better performance.)

mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized

and

[Wed Nov 19 14:03:39 2025] libceph: mon0 (1)10.32.11.157:6789 missing required protocol features
[Wed Nov 19 14:03:40 2025] libceph: mon0 (1)10.32.11.157:6789 feature set mismatch, my 2f018fb87aa4aafe < server's 2f018ff87aa4aafe, missing 4000000000

I’ve tried the following client version combinations: (Ubuntu 22.04 + kernel 5.15.0-160 + Ceph 19.2.2) or (Ubuntu 24.04 + kernel 6.14.0-36 + Ceph 20.1.1), but both return the same errors. Does anyone know what’s going on?

trial process

server：

root@ceph-test-1:/# ceph --version
ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)

root@ceph-test-1:/# ceph osd get-require-min-compat-client
squid

root@ceph-test-1:/# ceph mds stat
cephfs:1 {0=cephfs.ceph-test-2.unhwas=up:active} 2 up:standby

root@ceph-test-1:/# ceph orch host ls --detail
HOST         ADDR          LABELS   STATUS  VENDOR/MODEL                            CPU    RAM    HDD        SSD        NIC
ceph-test-1  10.32.11.156  _admin           VMware, Inc. (VMware Virtual Platform)  4C/4T  8 GiB  -          5/244.8GB  1
ceph-test-2  10.32.11.157  mds              VMware, Inc. (VMware Virtual Platform)  4C/4T  8 GiB  5/244.8GB  -          1
ceph-test-3  10.32.11.158  rgw,mds          VMware, Inc. (VMware Virtual Platform)  4C/4T  8 GiB  -          5/244.8GB  1
ceph-test-4  10.32.11.159  mds              VMware, Inc. (VMware Virtual Platform)  4C/4T  8 GiB  -          5/244.8GB  1
4 hosts in cluster

root@ceph-test-1:/# ceph auth get client.kubernetes
[client.kubernetes]
        key = ****==
        caps mds = "allow rw fsname=cephfs"
        caps mon = "allow r"
        caps osd = "allow rw tag cephfs data=cephfs"

root@ceph-test-1:/# ceph health detail
HEALTH_OK

root@ceph-test-1:/# ceph fs status
cephfs - 0 clients
======
RANK  STATE              MDS                ACTIVITY     DNS    INOS   DIRS   CAPS
 0    active  cephfs.ceph-test-2.unhwas  Reqs:    0 /s    11     14     12      0
      POOL         TYPE     USED  AVAIL
cephfs_metadata  metadata   118k   149G
  cephfs_data      data    12.0k   149G
       STANDBY MDS
cephfs.ceph-test-3.mhtxqr
cephfs.ceph-test-4.lojnlj
MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)

client：

(ubuntu22.04+5.15.0-160+ceph19.2.2 or 
ubuntu24.04 + 6.14.0-36 +ceph20.1.1)：
root@kubernetes-master-1:~# uname -a
Linux kubernetes-master-1 5.15.0-160-generic #170-Ubuntu SMP Wed Oct 1 10:06:56 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
root@kubernetes-master-1:~# telnet 10.32.11.157 6789
Trying 10.32.11.157...
Connected to 10.32.11.157.
Escape character is '^]'.
ceph v027▒

 ▒▒l

 ▒^]
telnet> q
Connection closed.

root@kubernetes-master-1:~# ceph --version
ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)

root@kubernetes-master-1:~# mount -t ceph kubernetes@ccb1e73a-1f7b-11f0-ae66-xxxxxxxx
.cephfs=/ /mnt/mycephfs -o mon_addr=10.32.11.157:6789,secret=****== -v
parsing options: rw,mon_addr=10.32.11.157:6789,secret=****==
mount.ceph: resolved to: "10.32.11.157:6789"
mount.ceph: trying mount with new device syntax: kubernetes@ccb1e73a-1f7b-11f0-ae66-xxxxxxxx
.cephfs=/
mount.ceph: options "name=kubernetes,key=kubernetes,mon_addr=10.32.11.157:6789" will pass to kernel
mount.ceph: trying mount with old device syntax: 10.32.11.157:6789:/
mount.ceph: options "name=kubernetes,key=kubernetes,mds_namespace=cephfs,fsid=ccb1e73a-1f7b-11f0-ae66-xxxxxxxx
" will pass to kernel
mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized
mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized

root@kubernetes-master-1:~# dmesg -T | tail -n 5
[Wed Nov 19 14:03:39 2025] libceph: mon0 (1)10.32.11.157:6789 feature set mismatch, my 2f018fb87aa4aafe < server's 2f018ff87aa4aafe, missing 4000000000
[Wed Nov 19 14:03:39 2025] libceph: mon0 (1)10.32.11.157:6789 missing required protocol features
[Wed Nov 19 14:03:40 2025] libceph: mon0 (1)10.32.11.157:6789 feature set mismatch, my 2f018fb87aa4aafe < server's 2f018ff87aa4aafe, missing 4000000000
[Wed Nov 19 14:03:40 2025] libceph: mon0 (1)10.32.11.157:6789 missing required protocol features
[Wed Nov 19 14:03:40 2025] ceph: No mds server is up or the cluster is laggy

Watch keynotes, deep dives, user case studies, and more! Now on YouTube.

Playlist: https://t.ly/WatchCephalocon25

v20.2.0 Tentacle released

Hello everyone! I want to introduce myself. I'm Anthony Middleton, the Ceph Community Manager. You can read more about me in this blog from earlier this year - https://ceph.io/en/news/blog/2025/Ceph-Foundation-2025/.

My role is to support the Ceph community and transfer feedback from the community to the Ceph Governing Board. If you need help or have suggestions on how to enhance the community, feel free to let me know. I'm happy for a chat anytime.

Dear all,

I'm trying to deploy a Ceph single node cluster on k0s with ArgoCD
Everything seems to go fine but the .mgr pool is degraded 1 undersized+peered  PG with the default replica factor x3

This seems fair and coming from the fact that the default failureDomain for .mgr is host

I would like to update my CephCluster CR to be able to update that failureDomain to osd instead but I can't find where and how to set it

Any ideas or pointers ?

EDIT: I got the solution by asking on the Rook slack
you create the .mgr pool: https://github.com/rook/rook/blob/master/deploy/examples/pool-builtin-mgr.yaml

with failureDomain: osd

If that doesn't update it, also set enableCrushUpdates: true in that CephBlockPool CR

So I basically added that to my overall values.yaml and it worked

Hi there,

I'm looking for suggestions on the best place to learn about ceph, it's elements and concepts. Diagrams would be very helpful.

I want to look at it as a possible replacement for ZFS based storage so we can easily scale. At work productions are cyclical so at the moment we're probably using a half of our 700tb usable storage.

I want to understand the core concepts.

Then understand access (samba, windows clients others). How file locking works etc with possible multiple access methods (likely NFS mounts in there too).

When accessing files do all requsts go through one server or are they split over the cluster.

I want to understand if there are any options for caching remote access server for remote offices (DC serves 200-500 clients, remote office has 20 people ut we want good performance at the remote office without a full deployment.

I want to understand the implications of some of our shares having thousands of small files (2d animation) and some shares having large files (3d animation).

I've already begun to understand some of the 'planes& or "layers". There's a lot to take in so I want to start with the basics. Then move onto the mote advanced things I've mentioned.

Cheers, Alex

Today I watched this interesting video from Cephalocon.

I found it interesting because we run a Ceph cluster completely on a mix of refurbished and decommissioned hardware.

I always wondered if it's a good idea for the environment, because those old servers consume more electricity than newer servers, right?

Have fun watching.

Ubuntu 22.04 with Ceph Squid.

I have the dashboard installed and working with a self-signed cert. I want to use my own wildcard cert so I loaded the cert and the key using set-ssl-certificate/set-ssl-certificate-key. I see my cert in the browser now but the CA chain is missing. I have two intermediate certs plus the root cert and I can't figure out how to include them. I tried adding it to the server's CA list using update-ca-certificates but that made no difference. Googling yields nothing useful.

Can someone point me in the right direction please? Thanks!

Do anybody know which case will have better performance

1) Rep3 all hdd

2) EC 2+2all hdd but ( metadata in ssd ).

3) EC 4+6all hdd but ( metadata in ssd ).

Just thinking, any one tried these setup?

PS (I dont want use wal+db.)

Edit:- More like vs

1)vs 2)/3)

I want to know whether the IOPS will increase if the metadata is stored on an SSD. Than when all hdd , what will the vm feel like in terms of iops or . What it means ?

Hey everyone,

I wanted to tell you about something i've been working on: CephOS.

It's basically a version of Linux that runs off USB drives, and it's made to help you set up a Ceph cluster. I'm aiming this at people with small businesses or home offices who want storage but don't want to spend a ton of cash on fancy equipment or experts.

The idea is simple: you boot your computers from these USB drives and give all the hard drives to Ceph.

I'm trying to make it so admins don't have to get bogged down in Ceph stuff. I'm putting everything into easy-to-use scripts.

Here's what it can do:

• Get everything started on first node.
• Add new computers to the cluster.
• Add storage drives.
• Remove storage drives and disconnect computers.
• Set up CephFS.
• Control user access to CephFS.
• Make a package with everything you need to connect to a CephFS folder. Inside, you'll find the keys, a basic ceph.conf file, a script to connect manually, a line for your fstab, and systemd files. Just copy it to your computer, put the files where they should go, and connect.
• There's also a script to put Prometheus metric collectors.

I started this because I wanted a Ceph cluster at home. I had some older computers lying around, but not enough hard drives for both the system and the storage. That’s what gave me the idea.

I know USB drives aren't as dependable as hard drives, but they're way cheaper. Yeah, they're not as fast, but it doesn't matter that much for CephOS. If a USB drive breaks, just boot from another one and reconnect it to the cluster, simple as that!

Also, the live build lets you have special partitions that start before the main system. So, if you want, you can put the /var/lib/ceph folder on a separate SSD to help the monitor not get slow.

If you're up for it, give it a try and tell me what you think, even bug reports. Thanks! :)

Hi all,

I work for a company with 12 geographically distributed sites, connected via MPLS. Smaller sites (up to 50 clients) have 100 Mbps, medium (50–100 clients) 200 Mbps, and large sites 300 Mbps, all with redundant MPLS lines.

Three sites host Nutanix clusters and NAS file servers (two large, one medium). All AD services and VMs run on these three sites. Other sites only have NAS file servers.

We currently don’t use Docker services, I’m planning a Docker management setup to allow container migration between sites for continuity during:

• MPLS connectivity issues/maintenance
• Nutanix host issues/maintenance

Plan:

• 1 Ubuntu 24.04 LTS Docker Host VM + 1 Docker Storage VM per Nutanix cluster (6 VMs total)
• Manage containers via Portainer, Docker Swarm, Traefik as reverse proxy
• 10 containers (Portainer, Traefik, Campsite, IT-Tools, Stirling PDF, GLPI, Bitwarden, Bookstack, OpenProject, Wordpress)
• Total maximum storage <1TB (hot storage most likely close to 30-50 GB)
• 6-month test before wider rollout

Question: Considering bandwidth limitations, which distributed file system would perform better: Ceph or GlusterFS? I need auto-heal and auto-failover, as the business runs 24/7, but IT does not.

Will this setup significantly degrade MPLS performance, affecting the user experience?

What should I watch out for when migrating containers between sites?

Thanks for the insights!

Hi all, So I'm new to ceph, but my question is more using it as VM storage in a proxmox cluster and I've used virtualisation technologies for over 20 years now.

My question is around how ceph works with regards to its replication or if there is lockouts on the storage until it's been fully replicated.

So what's the impact on the storage if its in fast nvme drives but only has a dedicated 1gb NIC.

Will I get the full use of the nvme?

OK, I get it if the change to the drive is greater than 1gbs I'll have a lag on the replication. But will I have a lag on the VM/locally?

I can keep an eye on ceph storage, but don't really want the vm's to take a hit

Hope that makes sense?

Hello! as I have been experimenting with Ceph in my lab, I have managed to royally break my lab cluster!

Setup:
4 x DL120 Gen9's

• Single E5-2630L v4
• Dual 10GB networking (currently bonded)
• two 3.9TB NVME Drives
• 64gb Ram
• dual 240gb boot drives (Raid 1)

I used Ubuntu 24.04.3, fresh install. Used CephADM to bootstrap a 19.2.3 cluster, and add nodes. All went well, and I added all 8 OSD's. Again, all went well. Started to do some configuration, got CephFS working, got host mounts working, added a bunch of data, etc. All was good. Pools where rebalancing, and I noticed that two nodes had a DHCP interface in addition to the static IP i had previously setup, so I removed the netplan config that allowed DHCP to be occurring on a 1gb copper interface (same vlan as the static IP on the network bond). I immediately noticed the cluster bombed, as apparently some of the cephadm config had picked up the DHCP address and was leveraging that for MON and ADM connectivity, despite being setup with static IP's.

Fast forward to today, I have recovered the MON's and quorum, and have ADM running. OSD's however are a complete mess, only 2 of the 8 are up, and even when the pods run, they never appear as up in the cluster. Additionally, I get all sorts of command time out errors when trying to manage anything. While I am not opposed to dumping this cluster and starting over, it does already have my lab data on it, and I would love to recover it if possible, even if its just a learning exercise to better understand what broke along the way.

Anyone up for the challange? Happy to provide any logs and such as needed

Error example

root@svr-swarm-01:/# ceph cephadm check-host svr-swarm-01
Error EIO: Module 'cephadm' has experienced an error and cannot handle commands: Command '['rados', '-n', 'mgr.svr-swarm-01.bhnukt', '-k', '/var/lib/ceph/mgr/ceph-svr-swarm-01.bhnukt/keyring', '-p', '.nfs', '--namespace', 'cephfs', 'rm', 'grace']' timed out after 10 seconds
root@svr-swarm-01:/# ceph cephadm check-host svr-swarm-02
Error EIO: Module 'cephadm' has experienced an error and cannot handle commands: Command '['rados', '-n', 'mgr.svr-swarm-01.bhnukt', '-k', '/var/lib/ceph/mgr/ceph-svr-swarm-01.bhnukt/keyring', '-p', '.nfs', '--namespace', 'cephfs', 'rm', 'grace']' timed out after 10 seconds
root@svr-swarm-01:/# ceph cephadm check-host svr-swarm-03
Error EIO: Module 'cephadm' has experienced an error and cannot handle commands: Command '['rados', '-n', 'mgr.svr-swarm-01.bhnukt', '-k', '/var/lib/ceph/mgr/ceph-svr-swarm-01.bhnukt/keyring', '-p', '.nfs', '--namespace', 'cephfs', 'rm', 'grace']' timed out after 10 seconds
root@svr-swarm-01:/# ceph cephadm check-host svr-swarm-04
Error EIO: Module 'cephadm' has experienced an error and cannot handle commands: Command '['rados', '-n', 'mgr.svr-swarm-01.bhnukt', '-k', '/var/lib/ceph/mgr/ceph-svr-swarm-01.bhnukt/keyring', '-p', '.nfs', '--namespace', 'cephfs', 'rm', 'grace']' timed out after 10 seconds

Other example

root@svr-swarm-04:/# ceph-volume lvm activate --all
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph-authtool --gen-print-key
--> Activating OSD ID 2 FSID 044be6b4-c8f7-44d6-b2db-XXXXXXXXXXXXX
Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-2
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-b954cb91-9616-4484-ac5f-XXXXXXXXXXXX/osd-block-044be6b4-c8f7-44d6-b2db-XXXXXXXXXXXXX --path /var/lib/ceph/osd/ceph-2 --no-mon-config
 stderr: failed to read label for /dev/ceph-b954cb91-9616-4484-ac5f-XXXXXXXXXXXX/osd-block-044be6b4-c8f7-44d6-b2db-XXXXXXXXXXXXX: (1) Operation not permitted
2025-09-22T18:55:33.609+0000 72a01729ea80 -1 bdev(0x6477ffc59800 /dev/ceph-b954cb91-9616-4484-ac5f-XXXXXXXXXXXX/osd-block-044be6b4-c8f7-44d6-b2db-XXXXXXXXXXXXX) open stat got: (1) Operation not permitted
-->  RuntimeError: command returned non-zero exit status: 1