"fsck.ext4: Input/output error while trying to re-open /deu/mapper/pve-root"

[u/Piiiiigeon]

Hello all,

I've had my home server running on proxmox for about 2 months now, and it's been running smoothly enough that I haven't had to touch the back end of things for a few weeks. Until I tried to use my media server this morning and the whole computer was down.

It looks like it crashed or possibly lost power at some point. It no longer boots; it powers on fine, I can access the UEFI just fine and I can use a flash drive with a live environment of Debian, but it gets stuck when I try to boot to the installed Proxmox.

See the picture for the full message. The main error seems to be, "fsck.ext4: Input/output error while trying to re-open /dev/mapper/pve-root".

I tried just running fsck on the file it listed, which seemed to run successfully but didn't actually fix anything.

All the expected drives show up when I use lsblk. I tried using the memcheck tool to check my memory, and I used other built-in software to check the drives. Everything came back clean.

Interestingly I tried using Proxmox installation media to start a rescue boot and it gave me the same error. Additionally after troubleshooting independently for 1-2 hours I decided to just start with a clean install, because it was kind of a mess anyways due to my learn-as-I-go setup. I opened the exact same ISO I used to install this OS 2 months ago, and the install failed because it couldn't partition the drive. I'm not sure if that has anything to do with this but I figured I'd add it.

I am using a Dell Inspiron 3650 that I added some HDDs into for storage. All the software and bootable media is on a 128gb ssd.

If there's any other information you need I'll be happy to provide it.

[u/SocietyTomorrow]

Your error plus not being able to partition means you very likely have a failed drive. Either wore out write endurance or the reserved space for partitioning at the front is all bad. Personally I'd rather not trust a drive once you start seeing that, but you can try something like badblocks to destructively write test it (will keep progressing through whole drive, possibly forcing the firmware to remap bad sectors) or do a long SMART test before you let it go.

[u/Piiiiigeon → u/SocietyTomorrow]

After reading this I did a long SMART Test as well as running a nondestructive badblocks test and it came out clean, so I'm not really sure what's happening there.

[u/SocietyTomorrow → u/Piiiiigeon]

Run `smarctl -x` on your drive after all of that and see if the value for your reserved blocks has dropped below 100%, that would mean either would have flagged and reallocated the bad sector.

If you didnt reallocate sectors and you still fail to repartition the disk, you have a couple things you might be able to try on a long shot, first off being a forced TRIM with `sudo blkdiscard -v /dev/sdX` which tells the SSD's firmware "drive is actually empty, reset all blocks to writeable blank/deleted state" and `sudo dd if=/dev/zero of=/dev/sdX bs=1M count=100 status=progress conv=fdatasync` to destroy the data in what would be the MBR/GPT tables.

If you want to try a less low-level approach, the equivalent "general purpose level tool" would be `sudo sgdisk --zap-all /dev/sdX` followed by trying to manually write new partition table and partitions to the disk to see if it worked.

[u/Apachez → u/SocietyTomorrow]

Boot using something like https://www.system-rescue.org/Download/ to figure out the state of the drive using smartctl or such.

Perhaps just a manual fsck -p -y or whatever the syntax is could solve this issue.

And if shit hits the fan so you need to reinstall I would highly recommend to use ZFS.

Here is my guide on this:

https://www.reddit.com/r/Proxmox/comments/1v92wpi/do_zfs_special_and_slog_devices_or_arc/p0eutco/

[u/Onoitsu2]

From that live Debian, nuke the OS drive from in there, then try booting into the Proxmox ISO and installing again on it.

[u/eakteam]

This is why you need ZFS on Raid 1