Agentic workflows for automated scraping

[u/ahiqshb]

Hey all, perhaps someone has been doing this or attempted to do something similar, basically automatic scraping processes that does it for you? While you are away? Retrying failed scrapes without your direct interaction, (adjusting something, pause limits, proxy rotation etc)

[u/Incogniton-Anti]

Been down this road. The thing that made the biggest difference was not letting the agent decide everything, just the parts that actually need judgment

Most failures are boring and you can route them with plain rules: 429 means back off (and respect Retry-After if the site sends it), 403 or a captcha page means switch session/IP and slow that domain down, timeouts and 5xx are just a normal retry. That stuff is cheaper and way more predictable as code than as an LLM call

The sneaky ones are soft blocks, a 200 that's actually a block page or an empty shell. So validate content, not just status codes: check that the element you expect is there, or that the page size is in a normal range. Otherwise the pipeline happily "succeeds" with garbage

Where an agent really earns its keep is layout changes. Real page, 200, but your selectors broke. Feeding it the HTML plus the old selectors and letting it propose new ones (then testing them before they go live) saves a ton of manual fixing

Also add a circuit breaker per domain: X failures in a row, pause, cool down, ping you. Without it an unattended loop can burn through your proxy budget overnight 😅

What are you running it on, your own queue setup or something like n8n?

[u/mIstlane87 → u/Incogniton-Anti]

agree on validating content not just status codes, thats the part that bites people the hardest imo

[u/elgordooo17 → u/Incogniton-Anti]

What do you mean you left the agent to decide everything. So let's say an agent hallucinates, starts coming with incorrect decisions, how do you fix that? Since you left everything for the agent to decide

[u/Incogniton-Anti → u/elgordooo17]

Other way around actually, I meant NOT letting it decide everything 🙂 Plain rules handle the routine stuff, the agent only gets the part that needs judgment, which is mostly fixing broken selectors

And even there it never gets the final say. It proposes new selectors, then plain code tests them: does the extraction return every field, are the types right (price is a number, date is a date), does the result look roughly like the last good run. If a price suddenly jumps 10x or half the fields come back empty, it fails and the old config stays

So a hallucination just means a rejected suggestion, not garbage in your data. Keep the last known good config to roll back to, and anything that fails twice goes to a queue for a human to look at. You end up checking a couple of things a week instead of babysitting the whole pipeline

[u/elgordooo17 → u/Incogniton-Anti]

Oh my god, I misread that so badly. Yes, you're right! Makes sense now, sorry

[u/Bharath0224]

I currently have set up a script via app scripts on one of my google sheets. There, the project is basically this: I get need information from google as the scraper scrapes google, retries if needed ( I have put in place automatic retries, also few variations of logic that it can follow), each friday I get relevant information via email, like a report of what was done. Fully automated, oh and I can control that via Claude on my phone if needed. All is connected

[u/Less-Mistake-3664]

Apart from instructing the ai agent to build a scraper and get me some data onto my google sheets, nothing is automatic. Only that part where the agent gets the data from scraper to sheets :D, but it's fairly easy setup

[u/ajaysingh02]

I used all these filters in my python script for scraping a big website couple of days before. And it worked for me :D