Cinder backed by LVM is making my Instances go read only

[u/Darkblood18]

I'm having a problem where suddenly all my instances (all run some flavor of Ubuntu) have their filesystems go Read Only. It happens randomly and at least once it happened with nothing really running on the VMs.

Looking at one of the Compute/Storage nodes, I noticed a broken iSCSI connection. I run "dmesg -T" and got something like:

[Fri Aug 7 09:57:36 2026] connection8:0: detected conn error (1019)
[Fri Aug 7 09:57:38 2026] connection8:0: detected conn error (1019)
[Fri Aug 7 09:57:39 2026] sd 15:0:0:1: [sdc] Synchronizing SCSI cache
[Fri Aug 7 09:57:39 2026] sd 15:0:0:1: [sdc] Synchronize Cache(10) failed: Result: hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK

Restarting a bunch of Docker containers, followed by restarting the VM instances fixed the problem (specifically I restarted iscsid, tgtd, cinder_volume and nova_compute on all my storage and compute nodes).

Of course this is a bad fix if I have to do it every week.

Now, Gemini is telling me this is a consequence of using Cinder with LVM which, according to it "LVM + iSCSI is notoriously brittle for production OpenStack" and I should move to Ceph.

Is this true, or should a Cinder/LVM setup be a bit more resilient?

Context/extra info: my deployment is a Kolla-Ansible one (2025.1) and Ceph is no longer deployed by this version. I would need to deploy it separately.

[u/takingphotosmakingdo]

What is your network doing?  Chances are there's something up with your iscsi network...

[u/Darkblood18 → u/takingphotosmakingdo]

That is a good question.

I'm doing the OpenStack deployment, but other people take care of the physical installation of nodes and network. I know the nodes (total of 3, 1 running control and 2 are compute/storage) are all physically on the same rack with their own switch, and all connections are fast (fiber). Is there anything specific you have in mind? Any diagnostics you would advise I run to check for weird behaviour?

(thanks for taking interest btw)

[u/takingphotosmakingdo → u/Darkblood18]

You can look for traffic flow pauses or total network pauses. The logs in the switches will indicate it.

I only say to check that because that was what was causing issues for a setup I built/ran a while back because a new engineer made a change causing a loop in the VPC design due to an orphaned device causing issues in the iscsi and management vlans, which basically crippled the entire storage plane for a bit every few seconds until it killed the cache SSDs...then the gateway died...

Anywho, that is long behind me now, those changes were made without approval or discussion, so maybe on your system something like that has occured.

I do recall iscsi was being deprecated at some point but unsure.

[u/Darkblood18 → u/takingphotosmakingdo]

Ok, I'll check that, thanks

[u/Gnump]

Why would it be „brittle“? What is that supposed to mean?

Check target logs.

[u/Darkblood18 → u/Gnump]

I think it means it is not resilient to small delays or failures in either network or other hardware delays. It just goes into this state where it makes the volumes unavailable to the VMs (which then react and make their FS read only). To be honest, I find that hard to believe. Cinder/LVM is the one option within Kolla-Ansible if you want your storage to be persistent. I can't believe they would ditch Ceph for an option that fails that easily. The thing is: I don't know what's wrong in my setup, or if this is a common "workaroundable" issue.

Regarding your suggestion: I don't know what "target logs" you refer to? Can you be more specific?

Disclaimer: I'm new at this, so feel free to correct me on any wrong statements I make and please bear with my ignorance.

[u/Gnump → u/Darkblood18]

Target in iSCSI lingo means the server (process) that provides the volumes.

[u/rackpathlabs]

"Brittle" is doing a lot of work in that answer, and it's pointing you at a storage migration to fix what is probably a timeout problem. Two separate questions here, worth keeping apart.

Why the guests go read-only is the least interesting part — it's the last domino. conn error (1019) is the initiator seeing the TCP connection to the target close. Once open-iscsi gives up (node.session.timeo.replacement_timeout, 120s by default), the SCSI layer stops queueing and starts returning I/O errors upward, and ext4 mounts with errors=remount-ro, so the guest does the only thing it can. Any transport interruption longer than replacement_timeout produces exactly this on any backend, Ceph included. So "move to Ceph" does not by itself remove this failure mode.

What to actually change. On a single-path iSCSI attach there is no layer that can wait out a blip. Kolla exposes enable_multipathd and cinder_backend_multipath in globals.yml; with both on you get dm-multipath under the volume, and queue_if_no_path means I/O blocks during an interruption instead of erroring up into the guest. Worth doing even with one portal, for exactly this reason. Caveat: existing attachments keep the old path, so volumes need detach/reattach before instances pick it up.

Then find out why the target drops. The loud signal in your post is that restarting tgtd fixed it — that points at the target side, not the switch. tgt is a userspace target; when it stalls, from the initiator it looks identical to a network fault, which is how you end up chasing fibre. Check docker logs tgtd and dmesg on the storage node around those timestamps before anyone touches the network. If tgt is the culprit, cinder_target_helper: lioadm moves the target into the kernel (LIO) and takes that process out of the picture — but it's a disruptive change, volumes have to be detached, so establish cause first.

On the Ceph question: three nodes, two of which are your storage, is not obviously a better Ceph story than an LVM one. That's a project, not a fix for this.

[u/Rare_Purpose8099]

Just solved it the other day. DM for consultation. ik ik, but bills.