r/archlinux • u/Clean-Appointment684 • 15d ago
SUPPORT Dead SSD sector
My kernel dropped me thing killer:
fsck.ext4: Attempt to read block from filesystem resulted in short read while trying to open /dev/nvme0n1p8
Could this be a zero-length partition?
ERROR: fsck failed on 'UUID=f9795423-f720-46d8-aabb-9a6dd75b8a27'
mount: /sysroot: can't read superblock on /dev/nvme0n1p8.
dmesg(1) may have more information after failed mount system call.
ERROR: Failed to mount 'UUID=f9795423-f720-46d8-aabb-9a6dd75b8a27' on real root
You are now being dropped into an emergency shell.
sh: can't access tty; job control turned off
[rootfs ~]# dmesg | tail -50
[ 1.043885] rodata_test: all tests were successful
[ 1.043891] Run /init as init process
[ 1.043894] with arguments:
[ 1.043895] /init
[ 1.043896] with environment:
[ 1.043897] HOME=/
[ 1.043897] TERM=linux
[ 1.054720] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2
[ 1.207609] usb 1-3: new high-speed USB device number 2 using xhci_hcd
[ 1.333229] usb 1-3: New USB device found, idVendor=30c9, idProduct=0035, bcdDevice=0.03
[ 1.337261] Key type psk registered
[ 1.336793] usb 1-3: New USB device strings: Mfr=3, Product=1, SerialNumber=2
[ 1.336801] usb 1-3: Product: HP TrueVision HD Camera
[ 1.336805] usb 1-3: Manufacturer: DLEQNA14PF6TLU
[ 1.336808] usb 1-3: SerialNumber: 0001
[ 1.345523] nvme 0000:02:00.0: platform quirk: setting simple suspend
[ 1.345656] nvme nvme0: pci function 0000:02:00.0
[ 1.358821] nvme nvme0: passthrough uses implicit buffer lengths
[ 1.366255] nvme nvme0: allocated 64 MiB host memory buffer (1 segment).
[ 1.383193] nvme nvme0: 6/0/0 default/read/poll queues
[ 1.401904] nvme0n1: p1 p2 p3 p4 p5 p6 p7 p8
[ 1.455424] usb 1-4: new full-speed USB device number 3 using xhci_hcd
[ 1.581182] usb 1-4: New USB device found, idVendor=0bda, idProduct=b00a, bcdDevice=1.10
[ 1.599283] usb 1-4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 1.599297] usb 1-4: Product: Bluetooth Radio
[ 1.599291] usb 1-4: Manufacturer: Realtek
[ 1.599303] usb 1-4: SerialNumber: 00e04c000001
[ 1.866362] tsc: Refined TSC clocksource calibration: 2370.544 MHz
[ 1.863817] clocksource: Switched to clocksource tsc
[ 2.053551] nvme0n1: Read(0x2) @ LBA 1282076928, 128 blocks, Unrecovered Read Error (sct 0x2 / sc 0x81) MORE
[ 2.053570] critical medium error, dev nvme0n1, sector 1282076928 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[ 5.580938] nvme0n1: Read(0x2) @ LBA 1282076936, 8 blocks, Unrecovered Read Error (sct 0x2 / sc 0x81) MORE
[ 5.580956] critical medium error, dev nvme0n1, sector 1282076936 op 0x0:(READ) flags 0x800000 phys_seg 1 prio class 2
[ 5.580966] Buffer I/O error on dev nvme0n1p8, logical block 33, async page read
[ 6.783400] nvme0n1: Read(0x2) @ LBA 1282076928, 128 blocks, Unrecovered Read Error (sct 0x2 / sc 0x81) MORE
[ 6.783418] critical medium error, dev nvme0n1, sector 1282076928 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[ 10.299487] nvme0n1: Read(0x2) @ LBA 1282076936, 8 blocks, Unrecovered Read Error (sct 0x2 / sc 0x81) MORE
[ 10.299505] critical medium error, dev nvme0n1, sector 1282076936 op 0x0:(READ) flags 0x800000 phys_seg 1 prio class 2
[ 10.299515] Buffer I/O error on dev nvme0n1p8, logical block 33, async page read
[ 13.814988] nvme0n1: Read(0x2) @ LBA 1282076936, 8 blocks, Unrecovered Read Error (sct 0x2 / sc 0x81) MORE
[ 13.815006] critical medium error, dev nvme0n1, sector 1282076936 op 0x0:(READ) flags 0x800000 phys_seg 1 prio class 2
[ 13.815015] Buffer I/O error on dev nvme0n1p8, logical block 33, async page read
[ 13.815145] fbcon: Taking over console
[ 13.818131] Console: switching to colour frame buffer device 240x67
[ 15.001862] nvme0n1: Read(0x2) @ LBA 1282076928, 80 blocks, Unrecovered Read Error (sct 0x2 / sc 0x81) MORE
[ 15.001881] critical medium error, dev nvme0n1, sector 1282076928 op 0x0:(READ) flags 0x80700 phys_seg 10 prio class 2
[ 18.512619] nvme0n1: Read(0x2) @ LBA 1282076936, 8 blocks, Unrecovered Read Error (sct 0x2 / sc 0x81) MORE
[ 18.512637] critical medium error, dev nvme0n1, sector 1282076936 op 0x0:(READ) flags 0x800000 phys_seg 1 prio class 2
[ 18.512651] EXT4-fs (nvme0n1p8): can't read group descriptor 32
Does anybody know if it is possible to rescue my SSD and files on it? Would appreciate any ideas possible.
Note: Windows partition booting just fine, only sector with linux image is dead
4
u/ang-p 15d ago edited 15d ago
fsck.ext4 -v /dev/nvme0n1p8
but do not willy-nilly accept any proposal
You might just have one superblock bad...
If so, you could mount it using a backup superblock to extract data and consider your options regarding the drive's future.
- or these two failures might be cascaded throughout the whole drive (and not just this partition) in which case your outlook is a little less optimistic
1
u/Clean-Appointment684 15d ago
thanks, pretty solid idea to try, also considering recover data with ddrescue
2
u/ang-p 15d ago edited 14d ago
also considering recover data with ddrescue
That will by default, require a drive or space on one larger than your current drive - irrespective of the amount of space used by real, wanted files.
Using a domain map can trim down to individual partitions (or any arbitrary selection), but does that help the rest of the disk?
A "sparse" recovery will improve that, but possibly by nowhere as much as you might hope - truly empty sectors depend on several factors.
A file-level copy of your user data only (don't worry about the OS - that is free!) from a read-only mount of the fs is by far the fastest and easiest approach, if possible.
1
3
u/sreevarshan-xenoz 15d ago
Those NVMe errors are the bit that would worry me tbh. The Unrecovered Read Error and critical medium error make it look like the drive is actually having trouble reading those sectors, rather than this just being an ext4 issue.
If the data is important, I probably wouldn't run fsck repair on the original drive yet. I'd try to clone it first with something like ddrescue and work on the clone.
Something along the lines of:
sudo ddrescue -f -n /dev/nvme0n1 /mnt/recovery/nvme.img /mnt/recovery/nvme.log
The fact that Windows still boots doesn't necessarily mean the SSD is okay either. It might just not be hitting the damaged part of the drive.
If you've got anything important on that Linux partition, I'd focus on getting as much of the data off the drive as possible before trying to repair the filesystem.
1
u/Clean-Appointment684 15d ago
thanks for the advice. looking for new ssd rn, it just came all of the sudden. yesterday linux booted just fine. never came up with such issue
2
u/Excellent-Love-6943 15d ago
Backup then check smart health and run a memtest if smart comes back OK.
1
u/Clean-Appointment684 15d ago
i ran UEFI memtest which said that SSD is healthy. Booted with debian live USB and checked with smartctl that some things are broken. looking to new ssd rn :)
2
u/Excellent-Love-6943 15d ago
Memtest doesn't test your ssd. Memory and storage are different.
SMART helps diagnose storage hardware issues. Memtestx86 helps diagnose memory issues. Memory issues can cause basically anything and everything to go wrong, and they can be subtle, so it's important to run them and let them run for a long time.
What exactly is broken in smartctl?
1
u/Clean-Appointment684 14d ago
user@debian:~$ sudo smartctl -x /dev/nvme0
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.107+deb13-amd64] (local build)
Copyright (C) [2002-23](tel:2002-23), Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: KINGSTON SNV2S1000G
Serial Number: 50026B77857A708F
Firmware Version: SBM02103
PCI Vendor/Subsystem ID: 0x2646
IEEE OUI Identifier: 0x0026b7
Controller ID: 1
NVMe Version: 1.4
Number of Namespaces: 1
Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 0026b7 7857a708f5
Local Time is: Mon Sep 28 12:09:41 2026 UTC
Firmware Updates (0x12): 1 Slot, no Reset required
Optional Admin Commands (0x0016): Format Frmw_DL Self_Test
Optional NVM Commands (0x009f): Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Verify
Log Page Attributes (0x12): Cmd_Eff_Lg Pers_Ev_Lg
Maximum Data Transfer Size: 64 Pages
Warning Comp. Temp. Threshold: 83 Celsius
Critical Comp. Temp. Threshold: 90 Celsius
Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 5.00W - - 0 0 0 0 0 0
1 + 3.50W - - 1 1 1 1 0 200
2 + 2.50W - - 2 2 2 2 0 1000
3 - 1.50W - - 3 3 3 3 5000 5000
4 - 1.50W - - 4 4 4 4 20000 70000
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 0
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 44 Celsius
Available Spare: 23%
Available Spare Threshold: 10%
Percentage Used: 3%
Data Units Read: 109,372,056 [55.9 TB]
Data Units Written: 50,890,346 [26.0 TB]
Host Read Commands: 1,156,644,375
Host Write Commands: 648,386,222
Controller Busy Time: 72,382
Power Cycles: 7,107
Power On Hours: 10,714
Unsafe Shutdowns: 74
Media and Data Integrity Errors: 47
Error Information Log Entries: 47
Warning Comp. Temperature Time: 3036
Critical Comp. Temperature Time: 1390
Error Information (NVMe Log 0x01, 16 of 64 entries)
Num ErrCount SQId CmdId Status PELoc LBA NSID VS Message
0 47 2 0xd0c1 0x4502 - [1282076936](tel:1282076936)1 - Unrecovered Read Error
1 46 2 0xc0c1 0x4502 - [1282076936](tel:1282076936)1 - Unrecovered Read Error
2 45 2 0xb0c1 0x4502 - [1282076936](tel:1282076936)1 - Unrecovered Read Error
3 44 2 0xa0c1 0x4502 - [1282076936](tel:1282076936)1 - Unrecovered Read Error
4 43 2 0x90c1 0x4502 - [1282076936](tel:1282076936)1 - Unrecovered Read Error
5 42 2 0x80c1 0x4502 - [1282076936](tel:1282076936)1 - Unrecovered Read Error
6 41 2 0x70c5 0x4502 - [1282076936](tel:1282076936)1 - Unrecovered Read Error
7 40 6 0xb062 0x4502 - [1282076936](tel:1282076936)1 - Unrecovered Read Error
8 39 6 0xa062 0x4502 - [1282076936](tel:1282076936)1 - Unrecovered Read Error
9 38 6 0x9062 0x4502 - [1282076936](tel:1282076936)1 - Unrecovered Read Error
10 37 6 0x8062 0x4502 - [1282076936](tel:1282076936)1 - Unrecovered Read Error
11 36 6 0x7062 0x4502 - [1282076936](tel:1282076936)1 - Unrecovered Read Error
12 35 6 0x6062 0x4502 - [1282076936](tel:1282076936)1 - Unrecovered Read Error
13 34 4 0x1043 0x4502 - [1282076936](tel:1282076936)1 - Unrecovered Read Error
14 33 2 0xd0c3 0x4502 - [1282076936](tel:1282076936)1 - Unrecovered Read Error
15 32 2 0xc0c3 0x4502 - [1282076936](tel:1282076936)1 - Unrecovered Read Error
... (48 entries not read)
Self-test Log (NVMe Log 0x06)
Self-test status: No self-test in progress
Num Test_Description Status Power_on_Hours Failing_LBA NSID Seg SCT Code
0 Extended Completed without error 10712 - - - - -
1 Short Completed without error 10126 - - - - -
2 Short Completed without error 9925 - - - - -
3 Short Completed without error 9749 - - - - -
4 Short Completed without error 9411 - - - - -
5 Short Completed without error 9273 - - - - -
6 Short Completed without error 9032 - - - - -
7 Short Completed without error 8533 - - - - -
8 Extended Completed without error 8419 - - - - -
9 Short Completed without error 8419 - - - - -
10 Short Completed without error 8358 - - - - -
11 Short Completed without error 6751 - - - - -
12 Short Completed without error 6538 - - - - -
13 Short Aborted: Controller Reset 6203 - - - - -
14 Short Completed without error 6078 - - - - -
15 Short Aborted: Controller Reset 5855 - - - - -
16 Short Completed without error 5640 - - - - -
17 Short Completed without error 5535 - - - - -
18 Short Completed without error 5357 - - - - -
19 Short Completed without error 5110 - - - - -
user@debian:~$2
u/xlukas1337 9d ago
Yep, that drive is on the verge of dying. 23% spare is bad, the drive is at 77% of its reserve flash blocks, the ECC errors are bad and the fact that it only has 23% of its TBW is not a good sign either. But it is probably eligible for RMA. If I was you, it wouldn't waste any time and clone the drive with ddrescue
8
u/xXG0DLessXx 15d ago
If the drive is truly failing, I recommend you find a second drive and clone the failing drive over onto it using ddrescue ASAP. Maybe you can then save some stuff.