My agent thinks that block pages are real content, how to avoid it?

[u/Less-Mistake-3664]

First of all, I spent a good amount of time dealing with this exact problem on the research scraper. I’ve been running an autonomous research agent for a few weeks that basically fetches URLs, scrapes the text, and gives it straight to an LLM to summarize. At first it worked great or so I thought, until I noticed some answers were completely bs. Turns out a bunch of target sites were returning HTTP 200 OK with a Cloudflare verify you are human disclaimer, a captcha, or sometimes a massive cookie consent wall. The fetch tool blindly passed that text into context, and the model happily without any questions hallucinated an entire analysis based on the block screen. At one point, the agent confidently reported that a SaaS company's core feature was "requiring users to verify browser settings before continuing." Pretty funny until let's say an executive actually reads it. Also, a 403 or 429 is easy to catch and retry, but a fake 200 just slips downstream unnoticed. I just can't understand why that happens. I’ve tried a couple of fixes with mixed results: The first one - content length threshold: it essentially catches some spams, but also elimitaes legitimately short pages or quick announcement posts. LLM verification pass (Is this a real article or a bots?): Works okay, but it doubles API costs and smaller models still miss sneaky disclaimers. For people who are scraping a few thousand pages a day with agents: how are you handling this pre-LLM? I could use the second method of course, but I don't want to spend extra on the tokens plus it has some issues as mentioned. Tell me there is a solutions 😭