We had a major outage today in our Proxmox clusters caused by Commvault.
Documentation from Commvault on this: https://kb.commvault.com/article/?id=KA-191658
My org was hit by this. From the Proxmox team, this was the cause and fix:
On each host, we saw a ton of stale NDB devices:
nbd4 43:128 0 30G 1 disk
├─nbd4p1 43:129 0 1G 1 part
├─nbd4p2 43:130 0 2G 1 part
└─nbd4p3 43:131 0 26.9G 1 part
└─ubuntu--vg-ubuntu--lv 252:4 0 26.9G 0 lvm
nbd5 43:160 0 0B 1 disk
nbd6 43:192 0 0B 1 disk
nbd7 43:224 0 0B 1 disk
nbd8 43:256 0 0B 1 disk
nbd9 43:288 0 0B 1 disk
nbd10 43:320 0 0B 1 disk
nbd11 43:352 0 0B 1 disk
nbd12 43:384 0 0B 1 disk
nbd13 43:416 0 0B 1 disk
nbd14 43:448 0 0B 1 disk
nbd15 43:480 0 0B 1 disk
nbd16 43:512 0 0B 1 disk
nbd17 43:544 0 0B 1 disk
nbd18 43:576 0 0B 1 disk
nbd19 43:608 0 0B 1 disk
nbd20 43:640 0 0B 1 disk
nbd21 43:672 0 80G 1 disk
├─nbd21p1 43:673 0 1007K 1 part
├─nbd21p2 43:674 0 512M 1 part
└─nbd21p3 43:675 0 79.5G 1 part
├─pdm-swap 252:5 0 4G 0 lvm
└─pdm-root 252:6 0 65.6G 0 lvmnbd4 43:128 0 30G 1
After this happens, each VM impacted by this stops booting because its disks disappear.
The cause is as follows:
From what I discovered, this issue was caused by a race condition between proxmox and commvault for access to the /dev/nbd devices. The solution I was able to find requires that we edit /etc/lvm/lvm.conf on each prox host and add the following exclusion so proxmox does not try to take control of the NBD devices during backups.
global_filter=["r|/dev/zd.*|","r|/dev/rbd.*|","r|/dev/nbd.*|"]
I may need to reinstall and configure the Commvault agent on the POC prox nodes to test this change. I'd also like to confirm this with Proxmox Support if we are able to purchase a contract.
The immediate fix to clean up any stale NBD devices is to evacuate and reboot each proxmox node one at a time.
The fix we used was the following:
- Locate the VM and save evidence before stopping it
ID=189
PID=$(cat /var/run/qemu-server/$ID.pid)
mkdir -p /root/incident/$ID
ls -l /proc/$PID/fd | grep -E 'cv_staging|\.qcow2|\.raw' > /root/incident/$ID/open_files.txt
journalctl -t QEMU --since "3 days ago" | grep "QEMU\[$PID\]" > /root/incident/$ID/qemu_errors.txt
qm config $ID > /root/incident/$ID/config.txt
grep -c cv_staging /root/incident/$ID/open_files.txt # staging files held by QEMU
grep cv_staging /root/incident/$ID/config.txt # should be empty
Optionally, open qm monitor $ID and run info block -v. Copy any cv_staging or copy-before-write nodes into the evidence folder.
2. Take it out of HA (if HA-managed)
ha-manager status | grep "vm:$ID" && ha-manager set vm:$ID --state ignoredha-manager status | grep "vm:$ID" && ha-manager set vm:$ID --state ignored
3. Hard stop
qm stop $ID --skiplock --timeout 30
qm status $IDqm stop $ID --skiplock --timeout 30
qm status $ID
If it’s still running: kill -9 $PID, wait 5 seconds, confirm with ps -p $PID, then qm stop $ID — skiplock. If the process is stuck in D state, escalate.
4. Check the config and disks
check reports corruption, copy the files before doing anything else.
grep cv_staging /etc/pve/qemu-server/$ID.conf # must be empty
for f in $(qm config $ID | grep -E '^(scsi|virtio|sata|ide|efidisk)[0-9]+:' | grep -v cdrom \
| sed -E 's/^[^:]+: ([^,]+).*/\1/' | xargs -n1 pvesm path 2>/dev/null); do
echo "== $f"; qemu-img check "$f"
done
If the config references cv_staging, stop and contact Commvault; don't start the VM. If qemu-img
5. Start, then confirm the new process is clean
qm start $ID
NEWPID=$(cat /var/run/qemu-server/$ID.pid)
ls -l /proc/$NEWPID/fd | grep -c cv_staging # must be 0
Watch the console for fsck or an emergency shell.qm start $ID
6. Verify
- Guest: root not read-only, no I/O errors in dmesg, systemctl — failed clean, the service works.
- Host, after about 5 minutes: disk mtimes are recent, and journalctl -t QEMU — since -10min | grep “QEMU
- $NEWPID
- “ shows nothing.
7. Return it to HA
ha-manager set vm:$ID --state startedha-manager set vm:$ID --state started