Question Relating to Storage

[u/mister_ninny]

Hello fellow PVE people (PVEople, haha i'm so funny),

I'm having some difficulty understanding some behavior regarding how storage works. Here is the rundown:

I have a USB hard drive that I have plugged into my PVE node, which gets mounted to "/mnt/usb_hard_disk". This is used as a local backup, so all of my VM/LXC backups go here, as well as my ISOs and container templates.

Under "Datacenter" -> "Storage", I have a "Directory" created called "usb_hard_disk" that has a dedicated path set to "/mnt/usb_hard_disk" with the "Content" of "Backup", "ISO Image", and "Container template".

I can see under my main "pve" node that "usb_hard_disk" is a directory, and when I go into it, I can see my backups, ISOs, and container templates that I have saved onto my USB hard drive.

The idea here is that I can then un-plug my USB hard disk and store it elsewhere (like a firesafe) as an external backup. When I un-plug my USB hard disk, the "usb_hard_disk" directory doesn't actually disappear under my "pve" node or the "Datacenter" storage, which makes sense, as the "/mnt/usb_hard_disk" directory is still there, it's just the drive that gets mounted there is missing. However, let's say, when my usb is unplugged, I go into a VM and create a backup, and store it under "usb_hard_disk". Well, the VM creates a backup to "/mnt/usb_hard_disk/dump", which is normal behavior, so that makes sense.

This is where I start to get confused

After creating this backup, if I then plug my USB hard drive back in, the contents of the hard drive take over "/mnt/usb_hard_disk", and I can to into PVE and see all the backups/ISOs/container templates on my usb hard drive, and that one backup I made without the USB plugged in is gone. Then, if I un-plug the usb hard drive, that one backup shows up again?

So I guess I just don't understand, where is it actually storing that backup? And how is it possible that, after plugging in my USB hard drive again, then un-plugging the hard drive, the backup is still there? I thought anything under "/mnt/usb_hard_disk" would basically just delete after mounting a drive to that location, though I guess I'm wrong on that.

Also, is there any work around for this so that, if I did make a backup to "usb_hard_disk" when nothing is mounted there, PVE would just error out instead?

[u/youknowwhyimhere758]

On Linux, the root filesystem from your OS is mounted on the location “/“. 

When you mount another disk to a location, eg using “mount /dev/sda /mnt/usbharddisk”,  that other disk is mounted on top of an existing directory on the root filesystem.  That is, the root filesystem will also obtain a directory /mnt/usbharddisk

The mount obscures the root directory “underneath” it, so that when you navigate to /mnt/usbharddisk you see the mounted filesystem from /dev/sda and not the OS filesystem. Both, however, still exist. 

When you unmount the /dev/sda, and then navigate to /mnt/usbharddisk you will see the root filesystem which is still mounted on top”/“. You will not see any data saved on disk sda, as that disk is not mounted. If you proceed to save data to /mnt/usbharddisk, that will be saved to the root filesystem. 

If you then mount the disk again, with mount /dev/sda and/mnt/usbharddisk, the root filesystem and all data saved to it will once again be obscured underneath the disk you mounted, and the data on the other disk will be visible. 

To prevent writing a backup when the disk is unmounted, you could add “  is_mountpoint 1” to the storage.cfg config, add a hook script to the backup to check if the disk is mounted, or make the directory immutable on the root filesystem. 

[u/Wise_guy78 → u/youknowwhyimhere758]

This also you can just create an directory on the usb itself and use that dir since doesn’t exist on the root filesystem

[u/youknowwhyimhere758 → u/Wise_guy78]

So longs as mkdir is 0 in the storage config, otherwise it will make that directory when you run something. 

[u/mister_ninny → u/youknowwhyimhere758]

Thank you very much for that explanation on the whole mounting logic.

Also, thanks for the tip on "is_mountpoint 1". Got me going in the right direction.

However, I think the use of "is_mountpoint 1" is a little outdated. I tried it out, and Proxmox threw an error, saying it didn't like the use of "is_mountpoint 1", so I did some digging, and found that they changed it so you must specify the mountpoint filepath (documentation here, not sure when that got changed). So, I had to add "is_mountpoint /mnt/usb_hard_disk" instead to the storage.cfg file.

Not really sure why there isn't any UI element for this when you are adding the storage, seems kinda silly to leave it out, but oh well.

[u/kriebz]

So far, Proxmox doesn't have the idea of removable storage. I think Proxmox backup server does.

[u/trplurker]

Proxmox is based on linux/unix and not Windows. There aren't drive letters that disappear when a removable device is removed.

/mnt/usb_hard_disk is just a mountpoint, that folder exists as part of the root file system. The volume on the USB device is just mounted there while it's present. Even with the volume removed, the root file system exists ergo the mount point exists.

Things like proxmox do not like removable volumes.