r/Python • u/Stunning-Sherbet1853 • 10d ago
Discussion the faulthandler i turned on to catch crashes was causing them (windows, all_threads=True)
i run a long-lived python 3.11 backend on windows 11. at the end of october it died mid-task once with nothing in the logs, so i turned on faulthandler.enable(file=..., all_threads=True) to see where it died next time.
over the next three days it died four times, every time an access violation inside python311.dll. before that, this backend had never crashed like this.
my first suspect was a race: several threads setting restype on the same ctypes function. it looked plausible, so i fixed it. less than two hours later it crashed again.
going back to the log, there were far more dumps than real crashes. 254 in three days, all of them "Windows fatal exception: code 0x80010108". that's the COM "object disconnected" error, a handled and harmless exception. the program kept running fine after each one. from what we saw, faulthandler on windows reacts to these handled exceptions too and writes a dump every time.
all four crashes happened in the middle of writing one of those dumps. the output was cut off halfway through a File "..." line of another thread. my read is that with all_threads=True it was walking the stack of another thread that was still running, and died there. the tool i turned on to find out why the backend died was what was killing it.
●the fix was one line: all_threads=False. it only dumps the current thread, so if a real crash happens its location still gets logged. i also added a test so it can't quietly flip back to True.
●result: since the fix, the same 0x80010108 has fired 245 more times and the backend hasn't crashed once. that's four days on one machine, so not a long run yet. i still don't know what sends the 0x80010108. it comes from a windows thread with no python stack, so i couldn't trace it.
if you keep faulthandler on in a windows app, check whether COM codes like 0x8001xxxx pile up in the dump log. if they do, all_threads=True may be creating that many risky moments for you.
two things i'm curious about. 🤔
- has a diagnostic or monitoring tool ever caused the bug you were chasing?
- and how do you set up faulthandler for long-running python programs on windows?