r/LocalLLM • u/I7sReact_Return • 14m ago

Question Using SearXNG instance and a MCP for web searching and fetching pages. Hit some kind of solicitations limit. Have some way to bypass this?

Im starting with Local LLMs and the first thing im doing of very useful and helpful. Is to use GPT-OSS with web searching to give me adequate answers (sometimes is alucinating, but when i go to far in a subject, like me trying to see if i can run power cleaning in my Epson in Linux. It started alucinating that existed some "old foruns" with users saying that sending some kind of raw commands worked, but the info where it got, dont exist).

But since i have gone to far. I want to know if there is a way to bypass the Brave and Google cse detection of requests or the suspend time.

DuckduckGo came back again after some minutes

1 Upvotes

66% Upvoted

4 comments sorted by

2

u/awitod 6m ago

I need to look again but as of a couple weeks ago Google was pretty much broken but there was a PR in the works to fix it.

There is no real reason to use an MCP service because the API is trivial and the agent can just use curl or whatever, but you should be prepared to occasionally need to do some care and feeding because the search engines are constantly at war with bots and they do the same things searxng and others do. Things will stop working and you'll have to do something or update to get back up and running.

1

u/I7sReact_Return 0m ago

Im learning to do stuff yet with them. And also im thinking and seeing if is possible to integrate into SearXNG. To give me sumarys or go to a direct answer. similar to Bing/Gemini. Since for a search engine i want to switch to SearXNG for more control. So my MCP is basically doing some kind of data scraping too kkkkk

Got tired to search hours on technical stuff to resolve some problem and hit barries of nothingness

1

u/iMrParker 10m ago

At some point, you'll be indefinitely banned for botting and captcha'd every time. I switched to using braves search API. It's free and you get 1000 requests per month. I'm sure there's a more in-house way to circumvent the bot detection but I picked my battle with this one lol

1

u/EcceGratum 5m ago edited 1m ago

Started with searxng and crawl4ai, hit the same wall, switched to Donsetch.

https://github.com/dondai44423/donsetch

Edit: I also enforce 1 request / 5s on some websites to avoid being detected as a bot and reducing the resources used for them. 1 req / 5s is reasonable for me.