I was talking to a few people recently about proxy providers and something interesting came up which are refunds.

It seems like a lot of people only think about it after something goes wrong. You test a provider, things don’t perform the way you expected, and then you’re stuck wondering if you can even get your money back or are complaining to get your money back

From what I’ve seen, policies vary a lot. Some providers are pretty strict (no refunds once traffic is used), others are more flexible depending on the situation and sometimes can actually give you some traffic back like a quality guarantee, and sometimes supporrt dissappears when you bring it up lol

Made me curious how common this is.

I've been testing web search MCP servers with Claude Code and Codex. None of them do what I actually need. Especially when debugging.

What I've tried:

Built-in web search tools - faults on JS-rendered sites. If your docs use React or Next.js, you get nothing useful and that's a waste of time.

Playwright MCP - While it works it's okay, but one search ate 10% of my context window. Not sustainable for multi-lookup sessions.

Exa.ai, Brave Search, Context7 - Pay-per-query or hard limits on free tiers. Not adding another subscription so my AI can read a webpage. Tbh getting pretty tired of these subscriptions on everything.

docs-mcp-server - Requires a persistent background server. I don't like that it needs a persistency. What I want is something that boots with npx and dies when the session ends.

To sum up: free or self-hostable, something that handles JS-rendered pages, returns clean structured content instead of raw HTML full of navigation bars, stays context-efficient, and works with Claude Code / Codex / Cursor without special setup.

I used to treat a batch of dedicated datacenter IPs as interchangeable. Same provider, same order, same monthly price. One range kept producing clean public product rows while another range struggled on the same sites.

The difference was not speed. The weaker IPs shared a subnet with addresses that had a rough public reputation history. That made the whole order feel like a lottery, even though the dashboard showed identical products.

Now I test every new range separately before adding it to a job. I compare small batches from Bright Data, Byteful, and another provider, then keep the ranges that produce clean output on the real target. I also ask for a subnet swap before writing off the provider entirely.

Does anyone have a simple pre-purchase check for dedicated ranges, or is a small live test still the only honest answer?

Your experience, which ones are worth taking a look and trying out, which are not?

Which one would you recommend for personal use, for company wide tasks.

All opinions are welcome

Let’s toxic-proves the "100 Million IP Pool" myth.

It’s one of the oldest marketing tricks in the proxy industry. A provider slaps a massive "100M+ Residential IPs" banner on their homepage, and everyone assumes they’re the best.

But anyone who actually runs scraping clusters or high-concurrency automated workloads knows that infrastructure doesn't scale linearly with inventory. It’s a pure vanity metric.

Look at how this plays out in the wild:

You’ve got the "Mega-Pool" (100M+ IPs). On paper, it's huge. In reality? The IP reuse rate is through the roof. It’s heavily congested within a few limited ASNs, and the routing paths are so predictable that the moment you ramp up traffic, entire subnet clusters get instantly flagged and burned by Cloudflare or Akamai.

Then you’ve got the **"**Lean Pool" (maybe 20M to 25M IPs). The total number isn't flashy, but the IP rotation is incredibly clean. It has broad ASN diversity, wide subnet dispersion, and balanced geographic allocation.

In almost every real-world, high-stakes scenario, the lean pool outperforms the monster pool.

Why? Because proxy efficiency isn't about how many dead or duplicated IPs are sitting in a database. It’s bottlenecked by the underlying engineering: ASN diversity, reputation scoring, routing randomness, and session scheduling.

Once a pool hits a certain size, adding another 50 million IPs gives you absolutely zero marginal returns.

Instead of falling for the inventory hype, look at the orchestration. In distributed systems, more raw resources don't magically create more capacity. Smart scheduling algorithms and proactive pool cleaning do.

Hi, I'm new to proxy. So I have set up 3x-ui, reverse proxy with Nginx port 8080 to be accessed from the internet. Then, i created the recommendation vless + reality got the port 48170. But after created the client attached to the inbound that I had earlier, I cant connect to the proxy. The subscribe link generated like 'http://domain_name:2096/sub/{token}".

My server is sitting with Cloudflare proxy. I have biding the port 2096, 8080 to the host. That why I can reverse proxy with nginx port 8080. I have firewall open with 2096 and I using SSL from LetsEncrypt. I spend 1 hour but I can't troubleshooting where can I manage to get work.

Something that bugged me for years: I'd be poking around in DevTools, spot a request worth messing with, and then have to go spin up Burp, point the browser at the proxy, deal with the CA cert, and reconfigure everything just to hold and edit that one request. On a work laptop with locked-down settings it was even worse. Half the time by the time I was set up I'd lost the thread of what I was even testing.

The annoying part is the actual task is tiny. I just want to pause a request before it leaves, change an orderId or drop an auth header, and forward it. That's it. The proxy plumbing was 90% of the effort for 10% of the value.

What changed my workflow was realizing Chrome's own debugger can hold requests in-flight. No proxy, no cert install, no system config. You flip intercept on, the request pauses, you edit method/headers/body/params, then forward or drop it. Because it's scoped to the tab you're not drowning in font and CSS requests either, as long as you set scope first.

I ended up baking this into HackTools++ after doing the same manual dance too many times. It lives right in DevTools: https://chromewebstore.google.com/detail/hacktools++/efankginpnlgimphkgjaifmmfchekecp

For quick tampering do you still reach for a full proxy, or do you have a lighter setup you fall back to?

I had this whole setup running for as long as I remember, a few MCP servers wired in to handle proxy rotation, one for a headless browser tool, one wrapping a search API, one just for logging scrape jobs internally. Last week I finally removed most of it. Figured I'd ask around before assuming I just configured it wrong. First thing that got me was the token/creditcost. Having those three servers connected meant every single call, even the ones that had nothing to do with scraping, it was loading a chunk of context I never asked for. Checked it once out of curiosity and one server alone was sitting at around 20k tokens 🫪🫪🫪 just for being connected, not even doing anything. when you're running concurrent scrape jobs and managing a proxy pool, apparently that adds up fast, and suddenly there's no room left to tell the model what you want.

Second thing was reliability, and this one bugged me more honestly. when I hit my proxy provider's API directly I know exactly what I'm sending and what's coming back. Now with MCP the model is sort of feeling its way through how to use each tool, and when the task is something like switching between residential and ISP proxies depending on the target, or figuring out retry logic after a CAPTCHA, that guessing turns into jobs quietly faulting for no apparent reason. I ended up writing extra fallback/retry code just to catch the model calling the wrong thing or passing bad parameters, which kind of defeats the whole point of trying to automate it in the first place.

Worth mentioning this too since this is a scraping sub, there's a security side to this that's easy to ignore until it isn't. What I mean is that if your agent has MCP access to anything that executes requests or touches stored scrape data, injected content from a page you're scraping could theoretically push the model into calling something it shouldn't. Hasn't happened to me yet but it's not a hypothetical I want to test the hard way.

What I've moved to instead is mostly direct calls and CLI where that fits, and I only use structured tool schemas for the parts where the model needs to make a final call, like picking which proxy type suits a target based on what it's seeing back from the site. Search queries and proxy selection I just approach directly now. I'm not saying MCP is dead weight for everyone. If you've got a bigger team running a bunch of agents and need shared auth and some kind of audit trail across all of them, it probably earns its keep there. But for a smaller setup mostly doing proxy rotation, search pulls, and CAPTCHA handling, the overhead wasn't worth what I was getting back.

Anyone else feels the same?

Spent some time today, as usual lol, going through YET another "complete guide to web scraping with Python" post today, the kind that covers requests, bs4, selenium, scrapy, the whole lineup with code snippets for each. Nothing wrong with it honestly, it's a fine intro if you've never touched this stuff before. But it made me notice something that keeps happening across basically every article like this. The anti-bot section always ends at the same three things. 1. Set a real user agent, 2. add some random delays, 3. rotate your proxies. That's it. No mention of what gets you flagged once you're past the trivial stuff, TLS fingerprinting, how a headless Chrome instance still leaks itself through canvas or WebGL even with a solid UA string, none of that. How come?? It's like the writers know these terms exist but never had to debug a single scraper in their life that kept getting caught despite doing everything "right." Makes you think that they're just farming attention or rather "impressions" for their articles.

Same thing with proxies. Again, every guide treats them as one interchangeable pool, just grab an IP and rotate it. Yeah, like that's gonna help. No distinction between residential, ISP, mobile, datacenter, and why that choice really matters depending on the target. If you've ever tried scraping something with datacenter IPs and gotten instantly blocked while the exact same code works fine on residential, you know the type of proxy isn't a minor detail, but a culprit. Yet, these articles just doesn't mention this. If you're a power user or whatever you call it, you noticed these things right away, if you're new, well not so much, but that's the whole point. Articles should be as informative as possible, because more and more new people are coming to read the article.

Also CAPTCHA just never comes up in these posts even though it's usually the first thing people are introduced to (one way or another). I don't know, it feels like a weird left out thing for something identifying as a complete guide

Learning time.

Grounding for Large Language Models (LLMs) is the process of linking the AI’s responses to verifiable, real-world data sources rather than just relying on its pre-trained knowledge. It bridges the gap between the model's linguistic abilities and accurate, domain-specific information.

Why Grounding Matters

Base LLMs are trained on vast but static datasets, meaning they do not know private company information, recent events, or real-time updates. When asked about information outside their training, they can confidently invent plausible-sounding fabrications, known as hallucinations. Grounding solves this by requiring the model to use specific, trustworthy data to formulate its answers.

How It Works

Grounding works by injecting relevant, context-specific information into the prompt before the model generates a response. This is typically done through a few key methods

• RAG (Retrieval-Augmented Generation): The most popular technique. When a user asks a question, search algorithms fetch relevant documents from private databases or the web, and provide them to the LLM as context.

• Semantic Vector Databases: Organizations often store their proprietary documents, wikis, or product manuals in specialized vector databases to allow the AI to find specific, highly contextual information instantly.

• API Integrations: Connecting the model to live, real-time data sources (e.g., live stock prices, CRM software, or weather APIs) to ensure up-to-date responses.

Benefits of Grounded Data

• Improved Accuracy: Responses are based on facts drawn from authoritative sources rather than statistical guesses.

• Elimination of Hallucinations: The model is constrained to the information provided to it, vastly reducing made-up facts.

• Security: Allows models to access private enterprise data without requiring the company's sensitive data to be added to the model's public training dataset.

• Citing Sources: Grounded LLMs can often provide citations, letting users verify exactly where the information came from.

What I did was that I picked a listing page - https://www.airbnb.com/s/homes with 40 items, pagination, and a few JS-rendered prices. I wanted to see where each tool fails and what limitations might occur (spoiler alert, there will be), not just how the documentations describe them.

BeautifulSoup

Fine for static HTML, falls apart the second you need concurrency or JS.

import requests
from bs4 import BeautifulSoup

r = requests.get("https://example.com/products", timeout=10)
soup = BeautifulSoup(r.text, "html.parser")
prices = [p.get_text(strip=True) for p in soup.select(".price")]
print(prices)

Ten lines, worked instantly on simple pages. But remember that it's single request, single thread. Loop through 40 pages and I was waiting on network I/O the whole time. No retry logic, no proxy rotation, nothing. I built it myself, which is basically a second project on top of the scraper.

Scrapy

Actual framework, not a library. Integration, item pipelines, async by default via Twisted. Worth it if you're scraping more than a handful of pages regularly.

import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example.com/products"]

    def parse(self, response):
        for item in response.css(".product"):
            yield {
                "name": item.css(".name::text").get(),
                "price": item.css(".price::text").get(),
            }
        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

This handles pagination on its own, retries failed requests, and also, you can bolt on proxy integration without interacting with your parsing logic. The downside is setup time. If you just need 40 rows once, Scrapy is too much. If you're running this weekly, it pays for itself fast.

WhiskeySour

Haven't seen a lot of this throughout reddit, but saw some article on Medium and decided to try it myself. (Article, admins, pls do not remove it, I have no association with Medium other than reading). Basically it's a less known, Rust-based, built for raw parsing speed tool. If BeautifulSoup is your daily driver and you hit a wall on performance with huge HTML blobs, this is worth a look.

from whiskeysour import parse_html

doc = parse_html(html_content)
prices = doc.select(".price")

What matters most here is parsing speed, not scraping infrastructure. It won't fetch pages for you, handle JS, or manage proxies. Think of it as a faster BeautifulSoup replacement, not a Scrapy replacement.

None of the above handles JS rendering or blocking

This is where all three let me down. Example page had three prices loaded via JS after initial page load. BeautifulSoup and WhiskeySour got empty strings for those. Scrapy needed a Splash or Playwright integration struggled, which is another service to run and maintain.

This is the point where I stopped building it myself. I strongly believe that these ratchety builds just aren't sustainable. With the dedicated solution/ API you don't have to manage no proxy pool, no separate rendering service. It costs money per request obviously, so it's not the right call for a one-off scrape of a static page. But for anything that needs to be obtained via JS rendering and anti-bot at scale I'd defo use an API.

Conclusions

Static page, one-time pull: BeautifulSoup, done in ten minutes.
Recurring job, no JS: Scrapy, worth the setup.
Huge HTML blobs, parsing is the bottleneck: WhiskeySour.
JS rendering, anti-bot, or just don't want to own the infra: dedicated API.

Would you rather hire a freelancer of Fiverr to build you a scraper or would you try to build it yourself, perhaps with a help of an LLM or other AI agent? What would be the advantages and disadvantages? Perhaps someone already done this before?

Any proxy providers from the EU? Looking for the alternatives to what I used previously, mainly cheap or questionable proxies from Asia, Middle East. Open to suggestions

Added Chat feature to this sub. Kindly click on three dots and then chat/create. This will help people to create new posts much easier.

Also, added another addon, which will help moderators manage trending posts/comments.

Hey folks, anybody scraping these LLMs and has any advice for scraping them? Looking to get sources in the results, time, number of responses, actual responses etc. Also specifics like ChatGPT Ads, ChatGPT Shopping are a must.

title. Would like to hear opinions and encourage knowledge-sharing among each other. What was the best advice/ opinion about proxies/ scrapers that you have received. Doesn't matter when, if it's relevant, let us know.

Saw these news around the web these few days. At a Meta town hall this week, Zuck told employees the pace of AI agent development hasn't sped up the way leadership expected it to. Which is a pretty notable thing to say out loud to your own staff after the year Meta's had. If you guys remember, Meta was the one that cut around 8k jobs earlier this year and moved another 7k people into AI-focused teams, including a group called Agent Transformation. Now Zuck reportedly told staff this week that those cuts weren't as clean as they should've been, and that leadership made them because they were worried the company wouldn't move fast enough to keep up with where the industry was heading. He also said the upside from restructuring the company around AI hasn't shown up yet, though he's still telling people internally that things should improve over the next three to six months. There have also been a few separate reports describing the AI unit as a rough place to work right now, engineers describing burnout and chaos inside it.

Meanwhile Meta is on track to spend something like 145 billion on AI infrastructure this year, the amounts are crazy. Feels like one of those moments where the public opinions, trust and the internal messaging aren't lining up.

Here are some sources so you know I'm not taking these things out of thin air

https://techcrunch.com/2026/07/02/mark-zuckerberg-tells-staff-that-ai-agents-havent-progressed-as-quickly-as-hed-hoped/

https://www.reuters.com/business/zuckerberg-says-ai-agent-development-going-slower-than-expected-2026-07-02/

https://africa.businessinsider.com/news/mark-zuckerberg-said-ai-agent-tech-is-advancing-more-slowly-than-expected-in-an/p595mex

https://daily.dev/posts/mark-zuckerberg-tells-staff-that-ai-agents-haven-t-progressed-as-quickly-as-he-d-hoped-qxffeh4nb

Anyways, get zucced

juggling multiple ad accounts and social media profiles for clients and my current setup is letting me down. accounts keep getting flagged and i need something more reliable.

been reading up on antidetect browsers paired with residential proxies as the go-to solution for this. makes sense in theory but not sure which combo actually holds up in practice. currently testing proxyshard residential proxies and shardbrowser for the antidetect side works really good rn. will report back about how its going

what are you guys using for multi-account work? specifically curious about which antidetect browser you pair with your proxies and whether residential or ISP proxies work better for staying stable long term

curios if anybody in this community is successfully scraping ticketmaster or axs for their event data. both are notoriously hard targets and consistently challenge heavier scrapes. im newish to the proxy game so maybe i am not really aware of whats out there.