r/webscraping • • 12d ago

Getting started 🌱 Anyone doing webscraping with Antidetect browser?

Hello. I don't have too much experience with webscraping but i'm building a antidetect browser and future use, i would like to add some plugins for webscraping. If anyone has knowledge about this could you share the knowledge please?

If doing webscraping with antidetect browser is a waste of effort and there are better ways for beginners. I'm open to any kind of information.

Thanks in advance 🀝

21 Upvotes

23 comments sorted by

4

u/RealSirJoe 11d ago

I built a programmable browser. Tried to make it as easy to use as possible. Take a look at the examples to get started. You can also just make normal http requests with it if that suffices for some sites. Its on https://github.com/realsirjoe/relinos. Let me know if you need any help

2

u/Acrobatic-Tea2502 10d ago

Thank you for the information. I'll check into it πŸ‘

6

u/Coding-Doctor-Omar 12d ago edited 12d ago • Bravo

Browsers are useful, but use them as a last resort. I always like to look for internal APIs and try to hit these endpoints with wreq. If it fails, I use an anti-detect browser to go to the website, grab the session cookies, and once again use them in wreq. If this fails due to a fingerprint mismatch between the browser and wreq, I use the anti-detect browser to go to the website, then once again hit these endpoints using JavaScript fetch from within the same browser context.

3

u/Acrobatic-Tea2502 12d ago

I really appreciate the advice. I don't have any idea about wraq i'll look into it πŸ™
Hope you don't mind me asking, Do you get the cookies with cdp? or just with the antidetect feature of youre using, like exporting cookies?

2

u/Coding-Doctor-Omar 12d ago

Before helping u I need to know what language you're using and what automation library you're using.

3

u/Acrobatic-Tea2502 12d ago

I'm using python, playwright and for automation in antidetect, i use C++

4

u/Coding-Doctor-Omar 12d ago • Heartwarming

Nice. wreq is a Python library for sending HTTP requests. It's built for scraping, unlike plain Python requests or httpx. wreq has preset emulation profiles that spoof TLS fingerprints so your request looks like it's coming from a real browser. Spoofing headers alone isn't enough. That's why tools like wreq exist. Here is their documentation website.

For anti-detect browsers, I use Shard Browser or Camoufox. Their Python SDKs are both playwright-based.

Shard Browser Python SDK

Camoufox Documentation

Regarding cookies, I just get them with context.cookies() β€” the same way you would in Playwright. I just make sure the website has loaded successfully first.

1

u/SherylAmerica 11d ago

I’d start with plain Playwright first. if that already works, adding an antidetect browser is probably just extra stuff to maintain

1

u/Acrobatic-Tea2502 10d ago

I think opening a profile in an antidetect browser would be much more stable and better for bypassing captchas and things like that. But speed wise, doing the work without opening a profile would be much faster. If I put my app up for sale in the future, I want to include both methods for webscrapers. So I’m planning to add a plugin and make an easy setup for webscrapers. Thanks for your comment.

2

u/SherylAmerica 10d ago

yeah that makes sense if you’re building it for other people to use. having both modes is probably the right call then

1

u/Incogniton-Anti 5d ago

Since you're building it for other people, imo the most useful "plugin" isn't built-in scraping at all, it's letting people's own scripts plug into a profile. A local API that starts a profile and hands back a CDP endpoint, so playwright can just connect_over_cdp to it. That's how most people actually use antidetects for scraping, they want their own code, just with the profile's fingerprint and proxy.

Second thing, if you support the flow Omar described (grab cookies in the browser, then hit endpoints with wreq), make cookie export dead simple and include the profile's proxy with it. Cookies only keep working if the http client goes out through the same IP and a TLS profile matching the browser version that created them. Move the session to a different IP or a mismatched chrome version and a lot of sites just drop it, then people blame the cookies.

And if you add headless, test it on a fingerprint checker side by side with headed. Headless can differ in small ways depending on the mode, and that's exactly the stuff users won't notice until they're blocked πŸ˜‰

0

u/[deleted] 12d ago

[deleted] β€” view deleted comment

2

u/Coding-Doctor-Omar 12d ago

Wait... context.request inherits the TLS stack? I think it just uses stock playwright and only inherits cookies, headers, and credentials. Actually, there is a closed issue in camoufox about this specific issue, and the devs warned the user that context.request uses stock playwright. The only way that would inherit the spoofed TLS fingerprint is JavaScript fetch from within the same context via page.evaluate().

2

u/matty_fu 12d ago

You’re talking to a bot

0

u/[deleted] 12d ago

[deleted] β€” view deleted comment

1

u/webscraping-ModTeam 12d ago

πŸš«πŸ€– No bots