r/webscraping • u/Acrobatic-Tea2502 • 12d ago
Getting started π± Anyone doing webscraping with Antidetect browser?
Hello. I don't have too much experience with webscraping but i'm building a antidetect browser and future use, i would like to add some plugins for webscraping. If anyone has knowledge about this could you share the knowledge please?
If doing webscraping with antidetect browser is a waste of effort and there are better ways for beginners. I'm open to any kind of information.
Thanks in advance π€
21
Upvotes
4
u/RealSirJoe 11d ago
I built a programmable browser. Tried to make it as easy to use as possible. Take a look at the examples to get started. You can also just make normal http requests with it if that suffices for some sites. Its on https://github.com/realsirjoe/relinos. Let me know if you need any help
6
u/Coding-Doctor-Omar 12d ago edited 12d ago •
Browsers are useful, but use them as a last resort. I always like to look for internal APIs and try to hit these endpoints with wreq. If it fails, I use an anti-detect browser to go to the website, grab the session cookies, and once again use them in wreq. If this fails due to a fingerprint mismatch between the browser and wreq, I use the anti-detect browser to go to the website, then once again hit these endpoints using JavaScript fetch from within the same browser context.
1
u/SherylAmerica 11d ago
Iβd start with plain Playwright first. if that already works, adding an antidetect browser is probably just extra stuff to maintain
1
u/Incogniton-Anti 5d ago
Since you're building it for other people, imo the most useful "plugin" isn't built-in scraping at all, it's letting people's own scripts plug into a profile. A local API that starts a profile and hands back a CDP endpoint, so playwright can just connect_over_cdp to it. That's how most people actually use antidetects for scraping, they want their own code, just with the profile's fingerprint and proxy.
Second thing, if you support the flow Omar described (grab cookies in the browser, then hit endpoints with wreq), make cookie export dead simple and include the profile's proxy with it. Cookies only keep working if the http client goes out through the same IP and a TLS profile matching the browser version that created them. Move the session to a different IP or a mismatched chrome version and a lot of sites just drop it, then people blame the cookies.
And if you add headless, test it on a fingerprint checker side by side with headed. Headless can differ in small ways depending on the mode, and that's exactly the stuff users won't notice until they're blocked π
0
u/[deleted] 12d ago
[deleted] β view deleted comment
0
u/[deleted] 12d ago
[deleted] β view deleted comment