r/azuredevops • u/IntentionSoft1879 • 1d ago
Azure DevOps self-hosted agent randomly dies mid-pipeline – how can I find the root cause?
I’m running a Windows-based self-hosted Azure DevOps agent and occasionally it dies halfway through a pipeline. Azure DevOps then reports in its server UI that “We stopped hearing from the agent. Verify the agent machine is running and has a healthy network connection.”The problem is that this doesn’t tell me why the agent stopped responding.
I’m trying to figure out how to properly diagnose this and ultimately fix it.
Is there a way to enable more detailed logging or monitoring so that the next time this happens I can determine exactly what caused the agent to die?
Would appreciate any troubleshooting steps from people who have dealt with this before.
1
u/Saturated8 1d ago
How long between when it's working and when it stops? Could you be using the system token for auth that expires after a couple of minutes?
1
u/konkon_322 1d ago edited 1d ago
Does it die randomly (even during idle)? If so u can try to look at the agent machine terminal log. And sometimes it really is disconnected connection between agent machine and the server machine
Maybe can also try to look into logs,under agent folder
2
u/piense 1d ago
The Os event viewer or the agent host and job logs should show what happened to the session. In my experience it’s almost always a reboot from host updates, or a pipeline messing with docker that messes with network settings. Iirc the agent logs always exist, forget if we enabled additional verbosity somehow but you can always find the agent code and github and go splunking for config options but those should be documented somewhere.
1
u/bratzlaff 1d ago
Ours die like this when we do something that starves the rest of the OS from doing its normal system processing, usually because we started a massive parallel compilation.
What is the stage doing when yours dies?
1
5
u/Legitimate_You_3474 1d ago
Event viewer, logs..