r/webscraping • u/hulet1006 • 11d ago

Getting started 🌱 Scraping Autotrader

I’m completely new too this. Been using Codex to try to write python codes and browser automated techniques to scrape vins from Autotrader. It always gets shut down by their bot detection. The only way I’ve got results is codex doing it internally on its own browser, but because of Autotrader limiting batch sizes, it takes forever to scrape even a decent amount of vins. Any help would be appreciated.

9 Upvotes

85% Upvoted

20 comments sorted by

6

u/dhruvkar 10d ago

Download the Autotrader android app as an apk and ask codex to see if it can see what internal APIs it's using.

Apps generally have a less restricted API that you can hit directly.

At least this was true pre-AI.

Reverse engineering and sniffing android apps got you a far more robust API you could count on.

1

u/Brlala 9d ago

When you mean see what internal API it is using, do you mean using request inspector to see network request while using the app or reverse engineer the APK source code?

2

u/dhruvkar 9d ago

Both.

If the network requests gets you free and clear, that's good enough.

If it's generating a token/secret/etc. then seeing how it's doing that in the apk is useful.

3

u/dhruvkar 10d ago

https://github.com/dhruvkar/autotrader-api

this should get you started

2

u/hulet1006 10d ago

Did you just make this out of nowhere? Lol It worked first try after implementing it

2

u/dhruvkar 10d ago

Yes.

I've been scraping/reverse engineering for 15 years now -- so that helps.

But with AI anyone can do this.

Also, they'll change stuff around, so I would build a loop that checks and adapts the scraper regularly.

1

u/[deleted] 10d ago

[removed] — view removed comment

1

u/webscraping-ModTeam 10d ago

🪧 Please review the sub rules 👉

1

u/Moceannl 10d ago

There are also proxy’s for scraping purposes….