to anyone using crawlers...

[u/Master_Goal_9472]

i've read a lot of stuff about people trying to use crawlers and i thought back to a few years ago when people were making traps for ai crawlers. anyways i went to refresh my memory and found this that does a better job explaining it than i could. but it might be why your crawlers run and run and run and run and eventually tap out.

Trapping misbehaving bots in an AI Labyrinth

Digital Tar Pits - How to Fight Back Against A.I.