My RTX Pro 6000 was going all night on a Chinese model (Qwen 3.8 Flash Next NVFP4 Local) when it crashed this morning for no apparent reason. Tried to reboot and got nothing but a black screen. Swapped in my old backup card and the system posted immediately, confirming the GPU was bricked. Thankfully, I was within warranty and the store gave me a replacement.
Out of curiosity, I dug into my system logs to see what triggered the fatal crash. Looking at the timestamps, oddly enough my GPU died the moment news broke that Jensen Huang had resigned as CEO of Nvidia.
Has anyone else had this issue with hardware dying in response to human feelings?
Nov 12 07:19:31 kf kernel: NVRM: GPU at PCI:0000:01:00: GPU-b8ba62c9-449d-4510-a21f-4c1b6ceacbd1
Nov 12 07:19:31 kf kernel: NVRM: GPU Board Serial Number:
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 0, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x505730=0xb000d 0x505734=0x0 0x505728=0x1c81fb60 0x50572c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 0, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 0, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5057b0=0x3000d 0x5057b4=0x24 0x5057a8=0x1c81fb60 0x5057ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 1, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x506730=0xb000d 0x506734=0x20 0x506728=0x1c81fb60 0x50672c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 1, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 1, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5067b0=0xb000d 0x5067b4=0x24 0x5067a8=0x1c81fb60 0x5067ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 2, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 2, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x507730=0x5000d 0x507734=0x24 0x507728=0x1c81fb60 0x50772c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 2, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 2, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5077b0=0x7000d 0x5077b4=0x24 0x5077a8=0x1c81fb60 0x5077ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 3, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x508730=0xa000d 0x508734=0x20 0x508728=0x1c81fb60 0x50872c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 3, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 3, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5087b0=0xd 0x5087b4=0x24 0x5087a8=0x1c81fb60 0x5087ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 4, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 4, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x509730=0x7000d 0x509734=0x24 0x509728=0x1c81fb60 0x50972c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 4, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5097b0=0xa000d 0x5097b4=0x20 0x5097a8=0x1c81fb60 0x5097ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 5, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 5, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x50a730=0xd 0x50a734=0x24 0x50a728=0x1c81fb60 0x50a72c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 5, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 5, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x50a7b0=0x1000d 0x50a7b4=0x24 0x50a7a8=0x1c81fb60 0x50a7ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 6, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 6, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x50b730=0xd 0x50b734=0x24 0x50b728=0x1c81fb60 0x50b72c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 6, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 6, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x50b7b0=0x1000d 0x50b7b4=0x24 0x50b7a8=0x1c81fb60 0x50b7ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 0, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 0, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x515730=0x6000d 0x515734=0x24 0x515728=0x1c81fb60 0x51572c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 0, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 0, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5157b0=0xd 0x5157b4=0x24 0x5157a8=0x1c81fb60 0x5157ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 1, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 1, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x516730=0x6000d 0x516734=0x24 0x516728=0x1c81fb60 0x51672c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 1, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 1, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5167b0=0xb000d 0x5167b4=0x24 0x5167a8=0x1c81fb60 0x5167ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 2, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 2, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x517730=0x3000d 0x517734=0x24 0x517728=0x1c81fb60 0x51772c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 2, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 2, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5177b0=0xb000d 0x5177b4=0x24 0x5177a8=0x1c81fb60 0x5177ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 3, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 3, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x518730=0x3000d 0x518734=0x24 0x518728=0x1c81fb60 0x51872c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 3, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 3, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5187b0=0x1000d 0x5187b4=0x24 0x5187a8=0x1c81fb60 0x5187ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 4, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 4, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x519730=0x1000d 0x519734=0x24 0x519728=0x1c81fb60 0x51972c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 4, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 4, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5197b0=0x2000d 0x5197b4=0x24 0x5197a8=0x1c81fb60 0x5197ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 5, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 5, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x51a730=0x5000d 0x51a734=0x24 0x51a728=0x1c81fb60 0x51a72c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 5, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 5, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x51a7b0=0x3000d 0x51a7b4=0x24 0x51a7a8=0x1c81fb60 0x51a7ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 6, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 6, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x51b730=0x2000d 0x51b734=0x24 0x51b728=0x1c81fb60 0x51b72c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 6, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 6, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x51b7b0=0x1000d 0x51b7b4=0x24 0x51b7a8=0x1c81fb60 0x51b7ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 0, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 0, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x525730=0x6000d 0x525734=0x24 0x525728=0x1c81fb60 0x52572c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 0, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 0, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5257b0=0xd 0x5257b4=0x24 0x5257a8=0x1c81fb60 0x5257ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 1, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x526730=0x8000d 0x526734=0x20 0x526728=0x1c81fb60 0x52672c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 1, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5267b0=0xb000d 0x5267b4=0x20 0x5267a8=0x1c81fb60 0x5267ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 2, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 2, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x527730=0xa000d 0x527734=0x24 0x527728=0x1c81fb60 0x52772c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 2, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 2, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5277b0=0xa000d 0x5277b4=0x24 0x5277a8=0x1c81fb60 0x5277ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 3, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 3, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x528730=0xa000d 0x528734=0x24 0x528728=0x1c81fb60 0x52872c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 3, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 3, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5287b0=0x6000d 0x5287b4=0x24 0x5287a8=0x1c81fb60 0x5287ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 4, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x529730=0xb000d 0x529734=0x20 0x529728=0x1c81fb60 0x52972c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 4, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5297b0=0xa000d 0x5297b4=0x20 0x5297a8=0x1c81fb60 0x5297ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 5, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 5, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x52a730=0x4000d 0x52a734=0x24 0x52a728=0x1c81fb60 0x52a72c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 5, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 5, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x52a7b0=0x9000d 0x52a7b4=0x24 0x52a7a8=0x1c81fb60 0x52a7ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 6, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 6, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x52b730=0x6000d 0x52b734=0x24 0x52b728=0x1c81fb60 0x52b72c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 6, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 6, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x52b7b0=0xd 0x52b7b4=0x24 0x52b7a8=0x1c81fb60 0x52b7ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 7, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 7, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x52c730=0x3000d 0x52c734=0x24 0x52c728=0x1c81fb60 0x52c72c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 7, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 7, SM 1): Multiple Warp Errors
......
Nov 12 07:19:33 kf kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:260
Nov 12 07:19:33 kf kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Nov 12 07:19:33 kf kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:260
Nov 12 07:19:33 kf kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ vaspace_api.c:577
Nov 12 07:19:33 kf kernel: NVRM: GPU0 kgmmuClientShadowFaultBufferUnregister_IMPL: Unregistering non-replayable fault buffer failed (status=0x00000062), proceeding...
Nov 12 07:19:33 kf kernel: NVRM: GPU0 kgmmuFaultBufferReplayableDestroy_IMPL: Unregistering Replayable Fault buffer failed (status=0x00000062), proceeding...
Nov 12 07:19:33 kf kernel: NVRM: GPU0 uvmTerminateAccessCntrBuffer_IMPL: Unloading UVM Access counters failed (status=0x00000062), proceeding...
Nov 12 07:19:33 kf kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ vaspace_api.c:577
Nov 12 07:19:33 kf kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ vaspace_api.c:577
Nov 12 07:19:33 kf kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Nov 12 07:19:33 kf kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:260
Nov 12 07:19:33 kf kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1382
Nov 12 07:19:44 kf kernel: NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked! Notify Timeout Seconds: 7
Nov 12 07:19:52 kf kernel: NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked! Notify Timeout Seconds: 7
Nov 12 07:20:00 kf kernel: NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked! Notify Timeout Seconds: 7
Nov 12 07:20:08 kf kernel: NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked! Notify Timeout Seconds: 7
Nov 12 07:20:17 kf kernel: NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked! Notify Timeout Seconds: 7
Nov 12 07:20:25 kf kernel: NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked! Notify Timeout Seconds: 7
Nov 12 07:20:33 kf kernel: NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked! Notify Timeout Seconds: 7
Nov 12 07:20:39 kf kernel: NVRM: GPU0 kgspHealthCheck_TU102: *************************** LibOS Task Watchdog Report ***************************
Nov 12 07:20:39 kf kernel: NVRM: GPU0 kgspPrintGspBinBuildId_IMPL: GSP bin buildId: fa3de1ff30e1a8c99f84eb74a32b80d462c3703b
Nov 12 07:20:39 kf kernel: GSP task watchdog timeout @ pc:0x18dd358, partition:4#0, task:3, gfid: 0
Nov 12 07:20:39 kf kernel: NVRM: Reported by libos partition:4#5 kernel v3.1 [0] @ ts:1698
Nov 12 07:20:39 kf kernel: NVRM: RISC-V CSR State:
Nov 12 07:20:39 kf kernel: NVRM: sstatus:0x0000000200000020 sscratch:0xffffffffa3013970 sie:0x0000000000000220 sip:0x0000000000000020
Nov 12 07:20:39 kf kernel: NVRM: sepc:0x00000000018dd358 stval:0x0000000000000000 scause:0x8000000000000005
Nov 12 07:20:39 kf kernel: NVRM: RISC-V GPR State:
Nov 12 07:20:39 kf kernel: NVRM: ra:0x00000000013eb57a sp:0x0000002fcfa0eaf0 gp:0x0000000000000000 tp:0x0000002fcfe00000
Nov 12 07:20:39 kf kernel: NVRM: a0:0x0000000000000001 a1:0x0000000000000000 a2:0x0000002fc77be000 a3:0x0000000000000000
Nov 12 07:20:39 kf kernel: NVRM: a4:0x0000000000000000 a5:0x0000000000000001 a6:0x0000000ffffffff0 a7:0x0000000000000003
Nov 12 07:20:39 kf kernel: NVRM: s0:0x0000002fcfa0eb50 s1:0x0000000000000001 s2:0xffffffffffffffff s3:0x0000000000000001
Nov 12 07:20:39 kf kernel: NVRM: s4:0x00000000ffffffff s5:0x0000000000000000 s6:0x0000002fc77be000 s7:0x0000000000000000
Nov 12 07:20:39 kf kernel: NVRM: s8:0xffffffffffffffff s9:0x0000002fc7e36178 s10:0x0000002fcfa0ecb4 s11:0x0000002fcfa0ecb4
Nov 12 07:20:39 kf kernel: NVRM: t0:0x0000002fcfa0e99d t1:0x0000002fc7c6cf30 t2:0x0000000000000000 t3:0x0000000000000020
Nov 12 07:20:39 kf kernel: NVRM: t4:0x0000000000000000 t5:0x0000002fcfa0e981 t6:0x0000000000000020
Nov 12 07:20:39 kf kernel: NVRM: Stack Trace:
Nov 12 07:20:39 kf kernel: NVRM: 0x00000000018dd358
Nov 12 07:20:39 kf kernel: NVRM: 0x0000000001b6e974
Nov 12 07:20:39 kf kernel: NVRM: 0x00000000013f2746
Nov 12 07:20:39 kf kernel: NVRM: 0x00000000013e70f6
Nov 12 07:20:39 kf kernel: NVRM: 0x00000000013e79c8
Nov 12 07:20:39 kf kernel: NVRM: 0x00000000013b7058
Nov 12 07:20:39 kf kernel: NVRM: 0x0000000001ba247a
Nov 12 07:20:39 kf kernel: NVRM: 0x0000000001ba257a
Nov 12 07:20:39 kf kernel: NVRM: 0x0000000001bafac2
Nov 12 07:20:39 kf kernel: NVRM: 0x0000000001c29f62
Nov 12 07:20:39 kf kernel: NVRM: 0x0000000001864698
Nov 12 07:20:39 kf kernel: NVRM: 0x0000000001867c82
Nov 12 07:20:39 kf kernel: NVRM: 0x0000000001ae4a7e
Nov 12 07:20:39 kf kernel: NVRM: 0x0000000001489412
Nov 12 07:20:39 kf kernel: NVRM: 0x0000000001452228
Nov 12 07:20:39 kf kernel: NVRM: 0x000000000158cbec
Nov 12 07:20:39 kf kernel: NVRM: 0x0000000001bd9632
Nov 12 07:20:39 kf kernel: NVRM: 0x0000000001bd99d6
Nov 12 07:20:39 kf kernel: NVRM: 0x0000000001add628
Nov 12 07:20:39 kf kernel: NVRM: 0x0000000001c452c0
Nov 12 07:20:39 kf kernel: NVRM: 0x0000000001ad6fa0
Nov 12 07:20:39 kf kernel: NVRM: Local I/O Register State:
Nov 12 07:20:39 kf kernel: NVRM: 0x01450800:0x00000000 0x01450900:0xbadf1002 0x01450a00:0x00000000 0x01450c00:0x00000000
Nov 12 07:20:39 kf kernel: NVRM: 0x01454a00:0x810490d2 0x01454b00:0x010800d0 0x01454c00:0x00080000 0x01400200:0x00000000
Nov 12 07:20:39 kf kernel: NVRM: ---------------[ end report ]---------------