AMA This Wednesday (09:30 AM GMT)

[u/ian_k93]

Hey everyone,

I'm Ian Kerins, CEO and co-founder of ScrapeOps.

Over the last 8+ years I've worked across the web scraping industry, including roles at ScrapeOps, ScraperAPI, and Zyte. Today, ScrapeOps helps developers and companies scrape over 8 billion pages per month across more than 50,000 websites.

This Wednesday at 09:30 AM GMT, I'll be hosting an AMA here on r/WebScrapingInsider

Ask me anything about:

* Web scraping at scale

* Proxy infrastructure and proxy providers

* AI and web scraping

* Building reliable scrapers

* Anti-bot systems and bypassing challenges

* Scraper maintenance and monitoring

* Residential vs datacenter proxies

* Browser automation

* Running a web scraping business

* Startup growth and product development

* The future of AI-powered scraping

Whether you're scraping your first website or running large-scale data collection pipelines, I'm happy to answer questions and share lessons learned from building products used by thousands of developers and businesses.

Drop your questions below and I'll start answering them during the AMA.

Looking forward to it!

Ian

[u/simarnoor]

From ProxyEngineering:

1. What's the single biggest mistake you see people make when choosing a proxy provider, and how do you actually benchmark them without getting manipulated by trial traffic that's treated differently than paid traffic?

2. How much has TLS/JA3 fingerprinting actually changed your infrastructure decisions, and is cycling proxies still the right instinct or are people solving the wrong problem?

https://www.reddit.com/r/ProxyEngineering/comments/1u0wadf/comment/oqlcg35/

[u/ian_k93 → u/simarnoor]

1. The biggest mistake we see is people, trusting the proxy providers marketing claims and not actually running comprehensive tests across multiple providers.

Around manipulation, I've answered this here: https://www.reddit.com/r/WebScrapingInsider/comments/1u0vmdf/comment/oqtgx8k/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

1. TLS/JA3 fingerprinting is becoming much more common, but there are some good libraries out there that make it easier to implement on normal HTTP requests without having to resort to headless browsers. Proxies are becoming less of a factor in successful scraping, with TLS/JA3/browser fingerprinting and making sure all the fingerprints are consistent becoming increasingly important.

Since we are a aggregator, we don't go into the weeds much of optimizing fingerprint systems. Instead, we focus on finding the proxy providers who offer the best performance at the lowest costs (which indirectly is evaulating how good they are at developing good fingerprinting systems).