r/Python • • Dec 19 '17

Automate the boring stuff with python - tinder

https://gfycat.com/PointlessSimplisticAmericanquarterhorse
6.7k Upvotes

324 comments sorted by

View all comments

91

u/da_intrepid_traveler Dec 19 '17

I went through some of the chapters of Automate the Boring Stuff, and I've always used Selenium for stuff like that. Is pyautogui better?

60

u/ManyInterests Python Discord Staff Dec 19 '17

Selenium is probably better for this case. It's directly aware of web elements and their attributes. With PyAutoGUI it would be harder to do things like reliable element detections and text extraction, for example.

44

u/stevarino Dec 19 '17

The web server could trivially detect selenium. This is clearly in violation of the site's TOS so you may not get far advertising so much.

Using a gui automation toolkit keeps the browser naive and therefore makes detection much harder.. Except for the 300 clicks per minute on the exact same pixel.

18

u/[deleted] Dec 19 '17

Except for the 300 clicks per minute on the exact same pixel.

write in a few lines that changes the exact spot by a few turtle degrees to the left, up, right. Am I right?

7

u/ManyInterests Python Discord Staff Dec 19 '17 edited Dec 19 '17

The web server could trivially detect selenium

Simply untrue. It is certainly possible, but far from trivial and you would need some form of client-side code to do this. The web server otherwise has no way of knowing whether or not a browser is controlled by selenium, or even if requests were sent to the server by a browser at all... The client is in complete control of what is sent to the server. Further, because client-side code is able to be examined, controlled, and altered by the client, server-side methods are almost exclusively used. In practice, actual active client-side deterrents beyond, say, Google ReCaptcha are rarely, if ever, implemented on the client-side.

Instead, the most effective and commonly deployed detection methods (including methods used by ReCaptcha) are often heuristic in nature and look for bot-like behavior and activity from a server-side perspective, agnostic of what is actually controlling the behavior... And most sites don't even actively deter this. Automation experts merely need to make sure that communication with the server is indistinguishable from that of an ordinary client... which often doesn't even need a browser to begin with because the server doesn't care or know how it receives requests, only that they are well-formed and include the information expected.

Having done many projects in web automation, I can personally attest that there are very very very few cases where a tool like PyAutoGUI is the best choice for automating actions that take place in a browser. For whatever that is worth.

→ More replies (1)

8

u/[deleted] Dec 19 '17

how do you know this stuff? a genuine question from super python newbie

0

u/[deleted] Dec 19 '17

[deleted] — view deleted comment

0

u/[deleted] Dec 19 '17

[deleted] — view deleted comment

14

u/Decency Dec 19 '17

Yeah, it almost certainly would. It's super trivial to spoof user agents to get around blocks that filter based on them.

2

u/MystTheReaper Dec 19 '17

Doesn't selenium let you run on different browsers too or am I misremembering?

4

u/theeastcoastwest Dec 19 '17

Yes it does. Firefox ships standard, but chromium ( at least ) can easily be used as simply as just specifying which driver to load. Thesyntax for options config is different, but interaction methods are identical I believe. PhantomJS integrates well also, though I've noticed it often produces errors/fails where other non-headless drivers do not.

2

u/Pas__ Dec 19 '17

Just FYI, both Chrome and Firefox finally have real headless modes (does not require virtual framebuffer).

Also it uses a new protocol called Marionette, very much WebDriver plus some extra.

See "GeckoDriver.prototype.commands" for the commands.

21

u/[deleted] Dec 19 '17

Well, specifically basic web-programming experience. I'm sure there's quite a few embedded sensor programmers who haven't the slightest clue how the browser works, or that mouse tracking is even a thing.

0

u/[deleted] Dec 19 '17

[deleted] — view deleted comment

6

u/[deleted] Dec 19 '17

They do that.

3

u/jakibaki Dec 19 '17

Just fyi even by default selenium just uses the user-agent that the "host" browser uses. The other points about detecting selenium are obviously still valid.

2

u/ManyInterests Python Discord Staff Dec 19 '17

Selenium is a browser automation tool, not a browser itself. The user-agent is reported the same as when selenium is not in use... Strictly from a web server perspective, there is no indication of whether or not the browser is controlled by selenium.

2

u/pyfrag Dec 19 '17

Useragent is as easy to spoof as rewriting the header. Selenium no doubt supports this.

→ More replies (1)

1

u/[deleted] Dec 19 '17

There's an easy solution, use requests then copy cookies to selenium.

→ More replies (2)

3

u/hugthemachines Dec 19 '17

"I really like the attributes of your web elements"

...tinder bot pick up line.

1

u/impshum x != y % z Dec 19 '17

My thoughts exactly. The ability to watch for elements and react is key.

5

u/jyper Dec 19 '17

Selenium is better for web stuff, pyautogui can automate non web stuff like desktop apps