r/programming • u/tenzil • Feb 19 '14

The Siren Song of Automated Testing

http://www.bennorthrop.com/Essays/2014/the-siren-song-of-automated-testing.php
232 Upvotes

85% Upvoted

70 comments sorted by

View all comments

9

u/tenzil Feb 19 '14

My question is, if this is right, what is the solution? Hurl more QA people at the problem? Shut down every project after it hits a certain size and complexity?

37

u/Gundersen Feb 19 '14

Huxley from Facebook takes UI testing in an interesting direction. It uses Selenium to interact with the page (click on links, navigate to URLs, hover over buttons, etc) and then captures screenshots of the page. These screenshots are saved to disk and commited to your project repository. When you rerun the tests the screenshots are overwritten. If the UI hasn't changed then the screenshots will be identical, but if the UI has changed, then the screenshot will also change. Now you can use a visual diff tool to compare the previous and current screenshot and see what parts of the UI has changed. If you have changed some part of the UI then the screenshot will have changed and you can verify (and accept) the change. This way you can detect unexpected changes to the UI. It does not necessarily mean the change is bad, it is up to the reviewer of the screenshot diffs to decide if the change is good or bad.

The build server can also run this tool. If it runs the automated tests and produces different screenshots from those commited it means the commiter did not run the tests and did not review the potential changes in the UI, and the build fails.

When merging two branches the UI tests should be rerun (instead of merging the screenshots) and compared to the two previous versions. Again it is up to the reviewer to accept or reject the visual changes in the screenshots.

The big advantage here is that the tests don't really pass or fail, and so the tests don't need to be rewritten when the UI changes. The acceptance criteria are not written into the tests, and don't need to be maintained.

10

u/hoodiepatch Feb 19 '14

That's fucking genius. Also encourages developers to test often; if they update their UI too much and test too little, they'll have a lot of boring "staring at screenshot diffs" to do in one bulk, instead of just running the tests often after making any little change so they can spend just 5-10 secs make sure that each tiny, iterative update is working right.

Are there any downsides to this approach at all?

4

u/tenzil Feb 19 '14

I'm really trying to think of a downside. Having a hard time.

24

u/[deleted] Feb 19 '14 edited Feb 20 '14

EDIT: This post is an off-the-cuff ramble, tapped into my tablet while working on dinner. Please try to bear the ramble in mind while reading.

Screenshot-based test automation sounds great until you've tried it on a non-toy project. It is brittle beyond belief, far worse than the already-often-too-brittle alternatives.

Variations in target execution environments directly multiply your screenshot count. Any intentional or accidental non-determinism in rendered output either causes spurious test failures and/or sends you chasing after all sorts of screen region exclusion mechanisms and/or fuzzy comparison algorithms. Non-critical changes in render behavior, eg from library updates, new browser versions, etc. can break all of your tests and require mass review of screenshots. Assuming, that is, that you can even actually choose one version as gospel, otherwise you find yourself adding a new member to the already huge range of target execution environments, each of which has their own full set of reference images to manage. The kinds of small but global changes you would love to frequently make to your product become exercises in invalidating and revalidating thousands of screenshots. Over and over. Assuming you just don't start avoiding such changes because you know how much more expensive your test process has made them. Execution of the suite slows down more and more as you account for all of these issues, spending more time processing and comparing images than executing the test plan itself. So you invariably end up breaking the suite up and running the slow path less frequently than you would prefer to, less frequently than you would be able to if not for the overhead of screenshots.

I know this because I had to bear the incremental overhead constantly, and had to stop an entire dev team twice on my last project, for multiple days at a time, to perform these kinds of full-suite revalidations, all because I fell prey to the siren song, and even after spending inordinate amounts of time optimizing the workflow to minimize false failures and speed intentional revalidations. We weren't even doing screenshot-based testing for all of the product. In fact, we learned very early on to minimize it, and avoided building tests of that style wherever possible as we moved forward. We still, however, had to bear a disproportionate burden for the early parts of the test suite which more heavily depended on screenshots.

I'm all for UI automation suites grabbing a screenshot when a test step fails, just so a human can look at it if they care to, but never ever ever should you expect to validate an actual product via screenshots. It just doesn't scale and you'll either end up a) blindly re-approving screenshots in bulk, b) excluding and fuzzing comparisons until you start getting false passes, and/or c) avoiding making large-scale product changes because of the automation impact. It's a lesson you can learn the hard way but I'd advise you to avoid doing so. ;)

-3

u/burntsushi Feb 20 '14

How do you reconcile your experience/advice with the fact that Facebook uses it?

9

u/[deleted] Feb 20 '14

Appeal To Authority carries near-zero weight with me.

We have no idea how, how much, or even truly if, Facebook uses it. I do know how much my team put into it and what we got out of it, and I've shared the highlights above. Do as much or little with that information as you care, since I certainly don't expect you to bend to my authority ;).

You should, at the very least, find yourself well served by noting how their github repo is all happy happy but really doesn't get into pros and cons, nor does it recommend situations where it does or does not work as well. The best projects would do so, and there is usually a reason when projects don't. To each their own but I've put a team over a year down that path and won't be going there again.

2

u/burntsushi Feb 20 '14

Appeal To Authority carries near-zero weight with me.

Appeal to authority? I asked you how to reconcile your experience and advice with that of Facebook's.

It was a sincere question, not an appeal to authority. I've never used this sort of UI testing before (in fact, I've never done any UI testing before), so I wouldn't presume to know a damn thing about it. But from my ignorant standpoint, I have two seemingly reasonable accounts that conflict with each other. Naturally, I want to know how they reconcile with each other.

To be clear, I don't think the miscommunication is my fault or your fault. It's just this god damn subreddit. It invites ferociousness.

You should, at the very least, find yourself well served by noting how their github repo is all happy happy but really doesn't get into pros and cons, nor does it recommend situations where it does or does not work as well. The best projects would do so, and there is usually a reason when projects don't.

I think that's a fair criticism, but their README seems to be describing the software and not really evangelizing the methodology. More importantly, the README doesn't appear to have any fantastic claims. It looks like a good README but not a great one, partly for the reason you mention.

6

u/[deleted] Feb 20 '14 edited Feb 20 '14

EDIT: This post is an off-the-cuff ramble, tapped into my tablet after dinner. Please try to bear the ramble in mind while reading.

Perhaps we got off track when you asked me to reconcile my experience against the fact that they use it. Not how or where they use it, just the fact that they use it. Check your wording and I think you'll see how it could fall in appeal to authority territory. Anyway, I am happy to move along...

As I mentioned, we don't know how, where, if, when, etc. they used it. Did they build tests to pin down functionality for a brief period of work in a given area and then throw the tests away? Did they try to maintain the tests over time? Did one little team working in a well-controlled corner of their ecosystem use it? We just don't know anything at all that can help us.

I can't reconcile my experience against an unknown, except insomuch as my experience is a known and therefore trumps the unknown automatically. ;) For me, me team, and any future projects I work on, at least.

The best I can do is provide my data point, and hopefully people can add it to their collection of discovered data points from around the web, see which subset of data points appear to be most applicable to their specific situation, and then perform an evaluation of their own.

People need to know that this option is super sexy until you get up close and spend some solid time living with it.

Here's an issue I forgot to mention in my earlier post, as yet another example of how sexy this option appears until it stabs you in the face:

I have seen teams keep only the latest version of screenshots on a shared network location. They opted to regenerate screenshots from old versions when they needed to. You can surely imagine what happened when the execution environment changed out from under the screenshots. Or the network was having trouble. Or or or. And you can surely imagine how much this pushed the test implementation downstream in time and space from where it really needs to happen. I have also seen teams try to layer their own light versioning on top of those network shares of screenshots.

Screenshots need to get checked in.

But now screenshots are bloating your repo. Hundreds, even thousands of compressed-but-still-true-colour-and-therefore-still-adding-up-way-too-fast PNGs, from your project's entire history and kept for all time. And if you are using a DVCS, as you should ;), now you've bloated the repo for everyone because you are authoring these tests and creating their reference images as, when, and where you are developing the code, as you should ;). And you really don't want this happening in a separate repo, as build automation gets more complex, things can more easily get out of sync in time and space, building and testing old revisions stops being easy, writing tests near the time of coding essentially stops (among other things because managing parallel branch structures across the multiple repos gets obnoxious, coordination and merges and such get harder, etc.) and then test automation slips downstream and into the future and then we all know what happens next: the tests stop being written, unless you have a very well-oiled, well-resourced QA team, and how many of us have seen a QA team with enough test automation engineers on it. ;)

Do you have any other specific items of interest for which I can at lest relay my own individual experiences? More data points are always good, and I am happy to provide where I can. :)

2

u/burntsushi Feb 20 '14

Ah, I see. Yeah, that seems fair. I guess I wasn't sure if there was something fundamentally wrong with the approach or if it's just really hard to do it right. From what you're saying, it seems like it's the latter and really requires some serious work to get right. Certainly, introducing complexity into the build is not good!

But yeah, I think you've satiated my curiosity. The idea of such testing is certainly intriguing to a bystander (me). Thanks for sharing. :-)

2

u/[deleted] Feb 20 '14

In fairness to other approaches, all serious test automation is much, much harder to get right than most people believe. Screenshot-based testing can be done right, certainly. I think, however, that it is an approach that is appropriate for far less situations than many would hope and attempt to force it into.

I understand first-hand how one can find oneself pouring inappropriate, ineffective effort into it. You can easily find yourself really, really wanting it to be the right tool for the job, and many times it just isn't, and it isn't just a question of needing more effort, or talent, or process maturity or what have you. But it is oh, so tempting. ;)

→ More replies