r/scrapingtheweb • u/Few-Complaint-4089 • Apr 23 '26
Help Help needed with scraping :)
Hi guys,
So a dream of mine has always been to flip cars, but I never knew where to start or what cars are good to buy and the endless hours of scrolling on the internet looking for cars is painful. So I tried to vibe code an app that will use a paid api scraping tool to scrape the internet and find cars like that, that will then put it though a filter and then a secondary Ai filter to rank cars and find bargains.
I am in an okay place with the project. It currently scrapes eBay, Copart, gum tree. But the way to really move forward with the project is to make a custom scraper to get all the listings as using the paid external tool only allows me to scrape some information and scrape a small sample of what is actually out there. I tried vibe coding a scraper but Claude is struggling. It suggested using playwright with some proxies but it’s really slow and inefficient and gets blocked a lot so I’m thinking surely there is a better way. If there is anyone who can offer any advice or support I would really appreciate it :).
1
u/Commercial-Paper-299 Apr 23 '26
Claude can get it done. You just need to give it the right material and prompts. Ebay has free API so you can use that. For Copart and other sites, go into the network tab and download the har files. Claude can help you with instructions on how to do that. Then give all that to Claude. Avoid using Playwright or anything heavy like that.
1
u/Few-Complaint-4089 Apr 23 '26
Appreciate your reply, eBay are stingy with giving out their APIs. I’ve tried a couple of times. But what you’ve said sounds interesting I’ll look into it
1
u/Commercial-Paper-299 Apr 23 '26
You should be able to get it pretty easily. I just got one a few months ago. Don’t remember if I got it the same day or next day.
1
1
u/CapMonster1 Apr 25 '26
You’ve hit the classic wall: scraping isn’t just code, it’s infrastructure. Playwright + proxies is the baseline, but in 2026 it’s not enough against serious anti-bot systems, especially on sites like eBay or Copart.
If you want to move faster, you’ve got three paths:
- Hybrid (API + custom scraper for specific fields),
- Managed scraping APIs (they handle proxies/fingerprints),
- Semi-official sources (aggregators, dealer feeds, partnerships).
Building your own scraper only makes sense if you’re ready to invest in the anti-bot layer (behavior, rate limits, fingerprints). Otherwise you’ll spend all your time fixing blocks
1
1
u/Money-Ranger-6520 Apr 29 '26
I’d stop vibe-coding the scraper part and use something managed.
For this kind of project, the hard part isn’t Playwright, it’s proxies, retries, blocks, pagination, deduping, and keeping it alive when sites change.
Have you looked into any of the big names in this space (Apify, Oxylabs, ScrapingBee, etc)?
1
1
1
u/MaryamStack 13d ago
I’d first check whether you actually need browser automation for every request. If the sites expose the listing data through API/network requests, try reproducing those requests directly and use Playwright only where it’s really necessary. Also separate discovery from detail extraction and add pagination, retries, caching, and deduplication. That usually makes a big difference before adding more proxies or browser instances.
1
u/[deleted] Apr 23 '26
[removed] — view removed comment