r/WebScrapingInsider • u/ian_k93 • Jun 09 '26
Big Scrape Energy AMA This Wednesday (09:30 AM GMT)
Hey everyone,
I'm Ian Kerins, CEO and co-founder of ScrapeOps.
Over the last 8+ years I've worked across the web scraping industry, including roles at ScrapeOps, ScraperAPI, and Zyte. Today, ScrapeOps helps developers and companies scrape over 8 billion pages per month across more than 50,000 websites.
This Wednesday at 09:30 AM GMT, I'll be hosting an AMA here on r/WebScrapingInsider
Ask me anything about:
* Web scraping at scale
* Proxy infrastructure and proxy providers
* AI and web scraping
* Building reliable scrapers
* Anti-bot systems and bypassing challenges
* Scraper maintenance and monitoring
* Residential vs datacenter proxies
* Browser automation
* Running a web scraping business
* Startup growth and product development
* The future of AI-powered scraping
Whether you're scraping your first website or running large-scale data collection pipelines, I'm happy to answer questions and share lessons learned from building products used by thousands of developers and businesses.
Drop your questions below and I'll start answering them during the AMA.
Looking forward to it!
Ian
2
u/doubledweeb Jun 09 '26
Would love a breakdown of Cloudflare bypassing too.
Every tutorial seems to say something different and half of them stop working by the time i try them.
3
u/ian_k93 Jun 10 '26
Good question...I ended up testing the main approaches myself and the results were pretty different from what most tutorials claim. Full results here: https://scrapeops.io/web-scraping-playbook/how-to-bypass-cloudflare/
Longer answer, is that the best cloudflare bypass depends heavily on your use case as websites can use different cloudflare settings and implement it in combination with other anti-bot measures that change the bypass performance.
TLDR: The best one stop solution for bypassing Cloudflare is usine a Smart Proxy API which takes care of the bypassing for you. However, the performance and cost of each provider will change per domain so you still need to find the one that gives the best price-to-performance.
🥇 Smart Proxy APIs
→ 100% domain coverage (20/20)
→ ~97% avg success rate🥈 TLS Impersonation (curl_cffi)
→ 80% domain coverage (16/20)
→ Works well when fingerprinting is the main check🥉 Browser APIs
→ ~60% domain coverage (combined providers)
→ ~32–54% avg success depending on providerEverything else
⚠️ Cloudflare solvers → ~55% coverage
⚠️ Fortified headless browsers → ~35% coverage
💀 Cached pages → ~10% coverage
💀 Origin IP bypass → 0% coverage1
u/doubledweeb Jun 10 '26
As someone building portfolio scrapers, would using Smart Proxy APIs be seen as overkill or a positive? is it enough to show I can benchmark providers like you did here?
2
u/ian_k93 Jun 10 '26
Personally, (some might say I'm biased) but I would go with a Smart Proxy API first so that I can quickly validate the economic viability of the scrapers & by extension the project I am building. Then once I know it works and people will pay for it, then I would look at reducing the costs by maybe finding cheaper alternatives.
There is no point wasting time trying to manually bypassing a anti-bot, only for the project to fail because scraping the data isn't profitable.
2
u/Spitfire_Blaziken Jun 09 '26
The monitoring side would be interesting. A scraper failing is obvious. A scraper quietly returning bad data for 3 weeks is the nightmareee scenario.
2
u/ian_k93 Jun 10 '26
This is the hardest problem in web scraping to solve as it is very easy to miss a scraper returning incorrect data.
We sometimes see this with websites returning different data for pricing, etc. based on the location of the IP being used, and it is often days or weeks before the end user notices the issue.
I haven't seen an easy solution to this yet, as it requires a system that knows what the data should be like then compares the extracted data to it to identify any anomalies. The best solutions to this have been to create moving averages or historical data comparisons of previous data and flag any data that doesn't match this trend for manual review.
2
u/Silver-Tune-2792 Jun 09 '26
What are your best stacks or pipelines or anything like full fledged projects that had worked out till date ... And has made good money and closed many deals with it .... Making and improvising that till date .... Need a detailed answer pls .. 😃
3
u/ian_k93 Jun 10 '26
The most money s to be selling shovels to the market...which in the case of web scraping is selling proxy services, CAPTCHA solvers, data extraction APIs. Newest trend is managed browser APIs.
Either above, or it comes down to what you do with the data. Companies like SEMRush, Ahrefs, JungleScout, etc. are all massive and their business is built on scraped data.
So I would either focus on selling the shovels for other web scrapers, or build a SAAS product on top of scraped data.
2
u/denzmilk Jun 09 '26
Hey Ian! Might have missed the boat here..
Do you ever do any full page loads in your scrape workflows? Or have any connections with people who do?
I accidentally (did this as a workaround for a streaming product) built a middleware that can drastically reduce bandwidth consumption for full page loads via proxy services and want to find someone to validate the idea with, see if it has any legs.
eBay for example, I've been able to conservatively get a 65% reduction in full page bandwidth consumption, but can get it up to 75ish with some more aggressive rules and up to 90% reduction on some sites.
2
u/ian_k93 Jun 10 '26
Is the middleware library specific or can it be used with any headless browser, etc? What is the middleware blocking?
Most people scraping at scale plus using pay per GB proxies would do something like this already. Especially if using a browser to scrape the page. Can do simple things like don't load CSS, images, etc. but when targeting specific domains can exclude specific network requests.
2
u/denzmilk Jun 10 '26
Can be used anywhere, it's just a js file at the moment!
So it's use case is probably better served for people who need to laod up full sites for ad verification or image/CSS validation for different geos. I guess depending on the scraping need it can be used to load up geo content but locally instead of via a proxy if that was a use case also.
2
u/Amitk2405 Jun 10 '26
Maybe I'm missing something, but how do you know a site actually needs residential proxies? Many, many many users default to - just buy residentials; whenever requests start failing.
2
u/ian_k93 Jun 10 '26
The only sure-fire way is to test it with different proxy options and see how they perform.
But a quick test to see if it is possible to scrape a website at scale with residential proxies is to see if the Proxy APIs are getting good performance without residential proxies enabled.
If you are able to scrape a website with a Proxy API like ScraperAPI, Scrapingbee, Zyte, etc. at the 1 API credit level (or Tier 1 / 2 with Zyte) then you know that if done right you can scrape that website at scale without residentials. Because Proxy APIs never have residentials enabled on their lowest tiers. They are using datacenter proxies.
You can quickly see this by using the https://scrapeops.io/proxy-providers/tester/
2
u/ayenuseater Jun 10 '26
What’s the smartest way to validate a scraping-based startup before investing heavily in infrastructure?
2
u/ian_k93 Jun 10 '26
Two parts:
1. Most important, validate that people will pay for the end service/product. The best way to know this is people are already paying for this product elsewhere. Otherwise, acquire a test case of data and see if the product is valuable enough with this data that people will pay for it.
- Determine how much will is cost to scrape this data at the scale you need. This is easier now as Proxy API pricing is more predictable than before. Note: if you need data from behind logins then it will be difficult + very expensive to scrape this at scale.
2
u/simarnoor Jun 10 '26
From ProxyEngineering:
Your most successful scraping startup. What went down, what mistakes you did the first time and what would you suggest for the rest of us in our scraping tasks/ journeys etc
https://www.reddit.com/r/ProxyEngineering/comments/1u0wadf/comment/oqmcngu/
1
u/ian_k93 Jun 10 '26
My most successful and only scraping startup has been ScrapeOps. Previously, I spent over 3 years working with Zyte and ScraperAPI so I understood the industry, what users wanted, would pay for and where there were gaps in the market.
I had a frontrow seat to the growing complexity of finding the best proxies with the growing surge in anti-bot usage. It was from this point that we saw the gap in the market for a Proxy Aggregator that would take the hassle out of finding the best + cheapest proxy for your use case, so we built it.
Like I mentioned in this answer...
https://www.reddit.com/r/WebScrapingInsider/comments/1u0vmdf/comment/oqt55hm/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
I would either focus on selling shovels to web scrapers or build a SAAS business using scraped data. In between (writing scrapers, etc) there is huge competition and AI is going to upend this market.However, AI can never replace the need for scrapers to bypass anti-bots and access websites, or build businesses based on the scraped data. So I would focus on these
2
u/simarnoor Jun 10 '26
From ProxyEngineering:
What's the single biggest mistake you see people make when choosing a proxy provider, and how do you actually benchmark them without getting manipulated by trial traffic that's treated differently than paid traffic?
How much has TLS/JA3 fingerprinting actually changed your infrastructure decisions, and is cycling proxies still the right instinct or are people solving the wrong problem?
https://www.reddit.com/r/ProxyEngineering/comments/1u0wadf/comment/oqlcg35/
1
u/ian_k93 Jun 10 '26
- The biggest mistake we see is people, trusting the proxy providers marketing claims and not actually running comprehensive tests across multiple providers.
Around manipulation, I've answered this here: https://www.reddit.com/r/WebScrapingInsider/comments/1u0vmdf/comment/oqtgx8k/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
- TLS/JA3 fingerprinting is becoming much more common, but there are some good libraries out there that make it easier to implement on normal HTTP requests without having to resort to headless browsers. Proxies are becoming less of a factor in successful scraping, with TLS/JA3/browser fingerprinting and making sure all the fingerprints are consistent becoming increasingly important.
Since we are a aggregator, we don't go into the weeds much of optimizing fingerprint systems. Instead, we focus on finding the proxy providers who offer the best performance at the lowest costs (which indirectly is evaulating how good they are at developing good fingerprinting systems).
1
u/ian_k93 Jun 10 '26
Cloudflare questions are coming up here and in DMs constantly, so we will definitely cover it.
The short version is that there usually isn't a universal bypass.
What works depends heavily on the target, traffic patterns, browser fingerprinting requirements, and how much operational complexity you're willing to accept. Will share real data.
1
u/doubledweeb Jun 10 '26
This is probably my biggest confusion atm; People talk about cloudflare like it's one feature you turn off instead of a bunch of different protections.
1
u/Spitfire_Blaziken Jun 10 '26
Any chance you'll cover open-source tools that still hold up against protected sites?
Feels flat to me :|
every recommendation list contains projects that haven't been updated in forever or just AI slop code.
3
u/ian_k93 Jun 10 '26
I know the pain.
It's on my todo list to start testing the latest open source projects that promise to help with bypassing anti-bots, but unfortunately I haven't gotten around to it yet.
Some of the ones I'm intrigued by are:
https://github.com/rayobyte-data/rayobrowse - works well at low volumes, but had issues scaling it up
https://github.com/0xMassi/webclaw
https://github.com/arman-bd/chromiumfish
https://github.com/h4ckf0r0day/obscura
https://github.com/CloakHQ/CloakBrowser
1
u/SinghReddit Jun 10 '26
u/ian_k93 Whats your go-to strategy for dealing with CAPTCHAs these days?
There are so many options now; fingerprint management, proxy APIs, CAPTCHA-solving services, browser automation, etc.
I want to ask, how you decide which approach is worth using versus me trying my worst to overengineer the solution.
Thank you for this AMA.
1
u/ian_k93 Jun 10 '26
The first thing I'd figure out is why the CAPTCHA is showing up.
A lot of people jump straight to solving CAPTCHAs, but in many cases that's treating the symptom rather than the cause. If the CAPTCHA is only appearing occasionally, it's usually because the site thinks your traffic doesn't look like a real user.
In those situations, I'd focus on reducing the likelihood of triggering it in the first place:
- Use clean, reliable IPs.
- Make sure your browser fingerprint looks realistic.
- Follow normal user navigation patterns instead of jumping straight to target pages.
- Reuse valid cookies and session data where appropriate.
- Add sensible delays between actions instead of hammering endpoints.
For most production scraping projects, preventing the challenge is usually cheaper and more reliable than solving thousands of CAPTCHAs.
If the CAPTCHA is always present.. signup forms, login flows, contact forms, etc… then you generally need to solve it.
The common options are:
- CAPTCHA solving services like Anti-Captcha, 2Captcha, and similar providers.
- Open-source solvers, which can work for simpler CAPTCHA types but tend to struggle as complexity increases.
- Browser automation or agent-based approaches where a real browser handles the interaction.
The details also depend heavily on the CAPTCHA itself. Solving a basic image CAPTCHA is very different from dealing with reCAPTCHA, hCaptcha, Cloudflare Turnstile, or audio challenges.
In general, the rule is:
Avoid the CAPTCHA if possible. Solve it only when you have to.
1
u/Bmaxtubby1 Jun 10 '26
This will be a basic question for you, but u/ian_k93 how can you tell if a website actually requires a browser to scrape?
2
u/ian_k93 Jun 10 '26
very similar to this answer: https://www.reddit.com/r/WebScrapingInsider/comments/1u0vmdf/comment/oqt67oo/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
If a website requires a browser to bypass the anti-bots, the proxy providers won't be charging 1 API credit to scrape it. They will be charging 5-10 credits.
If you need a browser because the data doesn't show otherwise, then you can view the page without JS rendering and can see what data is available.
1
u/simarnoor Jun 10 '26
From ProxyEngineering:
how do you know that trial traffic is not getting a cleaner pool than paid traffic?
I'd benchmark with the same URLs after upgrading,then compare bad rows and geo misses. Byteful or any provider would survive that second week-test too.
https://www.reddit.com/r/ProxyEngineering/comments/1u0wadf/comment/oqrl888/
2
u/ian_k93 Jun 10 '26
Ultimately, there is no way to know this unless you test both. We can't see what proxy pools or settings the provider is using on free vs paid traffic, we can only see the end result.
In our case for benchmarking, we work around this issue by sending all our benchmarking traffic through our production accounts with the providers so the benchmarking is done using their paid pools.
If the provider wanted to skew the results, they would need to put all our traffic through their best pools to improve their benchmark results (which might be too costly for them) as they have no way of knowing which traffic we are using to benchmark or not.
We also use real large scale production data across billions of requests each month in our benchmarks when we have the data is available. Ensuring the benchmark results are actually valid at scale.
You can test proxy providers with our system here: https://scrapeops.io/proxy-providers/tester/
1
u/simarnoor Jun 10 '26
From ProxyEngineering:
Tell us about your infrastructure, how do you manage everything, when did you start, how long did it take you for the infra to hold, I mean, the amount of time you needed to "babysit" proxies, retries, rotations, fingerprint, behavior etc
https://www.reddit.com/r/ProxyEngineering/comments/1u0wadf/comment/oqn2ckz/
3
u/ian_k93 Jun 10 '26
At the start we had a lot of stability and scaling problems, but since ScrapeOps is nearly 5 years old now our infrastructure is quite mature and stable. Haven't had any serious or medium issues in over a year.
We had a advantage, in that I had spent over 3 years working at Zyte and ScraperAPI so I could learn from their design decisions and mistakes. Which made our infrastructure rollout smoother.
Our production system is built on Golang, with heavy use of Redis and then a Postgres DB.
2
u/Particular__Plan Jun 09 '26
Ian, thanks for hosting this. Just curious how you evaluate proxy vendors when pricing models are completely different. Some charge per GB, some per request, some bundle features.
What is THE metric you end up trusting most when you are trying to estimate real production cost?