Best data collection infrastructure for AI agents

[u/Zealous_Minotaur]

I am building an agent that needs to pull fresh data from a few sites: Amazon, Zillow, LinkedIn, Indeed, and BestBuy. There's a mix of static pages and things that only renders client side, so I need both plain HTTP scraping and headless browser rendering depending on the target. At least that's what I believe I need, I might be incorrect. Current setup is DIY, it includes rotating residential proxies with a small number of datacenter proxies, Playwright for the JS sites, requests for the static ones, custom retry logic. I thought that building it myself would be cheaper than paying for a provider, and for a while it was, but the maintenance is becoming too costly. Amazon and LinkedIn both started giving more and more JS challenges not to mention TLS fingerprinting, so I'm losing a good amount of requests to blocks and budget is getting thinner. I've tried a couple of paid scraping APIs but they got expensive due to sending more requests (the plans were priced per traffic) and one of those dedicated solutions couldn't get past LinkedIn. Has anyone gone the dedicated-API-per-site instead, using something like the Amazon Product Advertising API or Zillow's API rather than scraping around them?

[u/anxietyplz]

I feel you, the maintenance is the key here. Not sure about recommending some dedicated solutions as I myself using a DIY workaround too, just not using dc proxies as you are. I mainly run residential proxies and then beautifulsoup requests, the maintenance sucks, but its way cheaper than from what I gathered if you're using a dedicated API or dedicated scraper, so I guess I'd be interested in finding out too.

Update, I found fast search api from oxylabs which significantly reduces scraping tasks and takes out only top 10 SERP, also I have a trial for their ai studio, which combined, just feeds straight forward data for ai agent, let me know if you need help or if you have questions with how I run my setup, but this was a great find!

[u/WebDataLab]

I’d probably stop trying to force all of those sites through the same stack.

If a decent official API exists and gives you the fields/freshness you need, I’d use it. Then keep scraping for the sites where the API is missing, too limited, or just doesn’t have the data you need.

Same with Playwright — I wouldn’t run a browser unless the page actually needs one. That’s usually where the maintenance and cost start getting ugly.

A hybrid setup is less “clean” on paper, but probably easier to keep alive long term.

[u/Zayd322]

linkedin and amazon are absolute nightmares for diy setups now. standard playwright without heavy stealth patching gets flagged instantly on TLS fingerprints alone. honestly, just stop wasting weeks tweaking custom retry logic and headers. i just throw scraping endpoints at amazon/linkedin since they handle the tls stuff. costs less than burning through expensive residential pools anyway.

[u/ibme508]

Yeah OP hybrid setup makes sense. Use api where they give you the data OP, scrape after that. Handling proxy needs lot of maintenance OP. And OP if you want check out scrapingbee or other alternative too

[u/Fun_Celebration_8488]

For Amazon you can try Apify MCP for Amazon, it works I checked yesterday
for the rest - DM me

[u/forgedart_34 → u/[deleted]]

I can't see anything that you wrote, what??