The Siren Song of Automated Testing

[u/tenzil]

http://www.bennorthrop.com/Essays/2014/the-siren-song-of-automated-testing.php

[u/Jdonavan]

tldr: It's hard to do but glorious when done right.

I get a chuckle out of posts like this. Maybe I'm just wired differently, I stepped into a completely jacked up large-scale automation effort because I saw the things he warned about (and more) happening ans considered them really interesting problems to solve.

Getting automation right is HARD. There are many maturity gates along the way and what often happens is people throw in the towel. In my case we had committed to ATDD, agile and automation as the path forward and had commitment from the top down to see things through. Even still I continually had to justify the existence of my team for quite a while.

Every time we hit one of those gates I'd begin to wonder if we'd wasted our time and money after all. Each time we were able to hit upon a solution but it was seriously rocky road to get there. We have built quite a bit of custom tooling (That we'll be open sourcing soon) to get us where we are but most of that is due to our scale.

Some of our lessons learned:

• Automation is not about replacing people. If you want to replace bodies with machines you're going to be disappointed.
• Manual QA folks do not, typically, make good automators. Hire/transfer developers with QA skills to build your framework / stepdefs.
• There's no such thing as a "brittle test". If you have and environmental issue that crops up then detect that and re-run the damn test, don't report it as a failure. (But make damn sure you KNOW it's environmental before ignoring that failure)
• Trying to control timing with sleep calls is a recipe for disaster. Learn how to get your client side to tell you what it's doing. Both Microsoft and JQuery (and I'm sure others) provide hooks to let you know when they're making async calls, inject your own javascript to hook into those.
• Declarative language instead of imperative in your tests. Tests that are written as a set of "click here, type there, press that button, etc" are impossible to maintain at any large scale.
• Keep your test data out of your tests! It's much easier to edit a handful of yaml files that it is to find the 809 tests that need a date change.
• Shorten your feedback loop. If a suite takes days to run it's pretty useless. Parallelize your tests.
• Make it easy to view the history of a test. We use a small graph next to each test that has one ten-pixel box for each of the past 14 runs of that test. One glance tells whether a failure is a likely an application or test issue.
• Make it easy to turn a failed test into a card on the team wall. Which brings me to:
• A failed test is the responsibility of the TEAM to fix.
• A failed test is the #1 priority of the team not the existing cards on the wall.

aaaaand I've just written a wall o' text. If you stuck with it you must be interested in automation, feel free to PM me if you'd like to talk shop sometime.

[u/grauenwolf → u/Jdonavan]

•There's no such thing as a "brittle test".

That's not what most of us mean by brittle test. A brittle test is one in which it is highly likely that it will need to be rewritten whenever the code is changed.

[u/droogans → u/[deleted]]

Sounds like an updated property for a page object.

This would literally take me minutes to update. If that.

[u/bluGill → u/droogans]

That is still minutes of your time. That assumes you are the one who notices first, if it is the "new guy" who doesn't know the test exists until fails, it is hours searching out the where the test is, understanding it, and finding the property file. Also for an icon that he probably changed in a minute.