Is automated acceptance testing harmful?

Iwein Fuld

A lot of automated acceptance testing pioneers have come around and denounced their fate in heavy automated test suites. A recent article on InfoQ sums up the trend quite nicely. I am not going to jump on that bandwagon, but I will try to find the safe middle ground between the overzealously created maintenance burden and anarchy. The main point is that automating acceptance tests is the way to go, you just shouldn't automate and maintain useless tests. The tricky part is to find out what tests are useful and what tests are not.

Before I start let me emphasize the difference between automated acceptance testing and automated integration testing or unit testing. Unit testing is absolutely essential. Anyone that tells you otherwise is either ignorant because he hasn't tried yet, or a moron. We can have a long discussion on how to do unit testing properly, but that's not the topic of this post. Automated integration testing (within reason) is extremely useful, and essential in some areas. Where you touch external systems or things that are otherwise very expensive to mock, some of reservations might be defensible. Automated acceptance testing is testing where the aim is to simulate user interaction with the system. When and how this is useful is at stake here.

Let's list the arguments pro automated acceptance testing first:

  1. If you don't test it will break, the shorter the iteration the more you will have to test, hence in the shorter the iteration the faster the ROI.
  2. If you use an automated test framework that is understandable to the customer you can delegate test writing back to them, making properly defined functionality their problem.
  3. Human beings are very easily persuaded to overlook certain details, automated tests are impervious to that sort of corruption

Then there are the cons:

  1. Tests are expensive to maintain. They will never ROI because the more you change the more you need to change your tests ultimately increasing the cost of the project.
  2. Customers don't understand the test tools, so the developers end up writing and maintaining these tests anyway
  3. Tests fail all the time when you change little things where a tester would use his head and leave you alone if the change was sensible

Ad 1. It is true on the one hand that manual testing is repetitive work. But if there are many changes the level of repetition might be lower than was assumed before. It only makes sense to automate repetitive work, but it only makes sense to automate it once you have verified that it is repetitive. It follows that writing automated tests up front (when you haven't verified that they will be repeated) is a bad idea. Not writing them at all is a bad idea too, since you're excluding automation of repetitive work up front. We need to define exactly when it makes sense to automate a test. I'll revisit this later in Ad 3.

Ad 2. Current automated test frameworks are not understandable to the customer. At least not all by themselves. I've seen plenty of counter examples though of Product Owners recording selenium tests to get them through boring flows and customer testers writing Fitnesse fixtures to see if they could break a headless application. This only works when a team puts in some serious effort to help the customer to use the tools. This I blame primarily to the quality of those tools, we're not there yet. But if that initial investment is done there is a special kind of interaction with the stakeholders that would otherwise be impossible.

Ad 3. There is no way that you can have both well defined specs and Agile evolving software. You need to choose what is variable and what is fixed. A tester should not be the one responsible for this choice it should be the stakeholder. I think that when the stakeholder is happy with a certain feature and he wants to keep it, then is the time to automate the test. You might wait with this until the first bug arises breaking said feature, but you cannot allow regression after regression or manual test run after manual test run to stack up the costs. Once it's done it's done and you can cast it in stone.

There are two types of cost to consider when writing automated acceptance tests. What is it going to cost me initially what is going to cost me to maintain those tests. If this outweighs the benefits in savings you shouldn't invest. In most projects that go for more than a handful of iterations though the cost of regressions are quite steep, so I'd say setting things up properly is efficient more often than not.

Once you have a setup that can be used for automated acceptance testing, there is still the question of what to test. I like the idea of focusing on automating away repetitive work. Because of that a human tester might be your most likely expert on what needs to be automated, simply because he gets to do the repetitive work.

If automated testing becomes mainly a cost saving measure, it becomes rather moot who does the implementation of the tests. If developers can do it more efficiently than the customer because the tools are still too clunky for mere mortal usage, just let the developers do it until they pick or make better tools. Developers are lazy enough to figure out when that gets efficient.

Comments (5)

  1. Geert Bossuyt - Reply

    April 14, 2010 at 8:06 pm

    Very helpfull. Tx.

    In the situation where you have automated the acceptance tests, would you still combine these automated tests with exploratory testing from time to time ? How do you make sure your tests are working correctly ?

  2. Iwein Fuld - Reply

    April 15, 2010 at 6:21 am

    There is no replacement for exploratory testing. I think the more you can get the customer involved with the application (clicking around, using it etc) the better it is.

  3. Frank Silbermann - Reply

    April 16, 2010 at 2:43 pm

    I have used products that record user-GUI interactions to replay them as automated tests. Because many use cases change the data store, programmers have to modify the tests to ensure that the application data is set up before the test and cleaned up afterward.

    Nevertheless, the automatic testing products did indeed prove all but useless, because every cosmetic change to the GUI broke the test. This is because we have no standards for generating from GUIs the persistent APIs which automated tests should work from. The identification of a widget to its location is simply too fragile a link.

    The problem can be solved in theory. Suppose that a GUI toolkit allowed the programmer to associate with each widget an identifier to be used by automated testing tools, and suppose that each screen, form or frame could provide a map of widget identifiers to screen real estate. If the test script were recorded in terms of these external widget identifiers and read the identifier-to-location map at run time, developers could reposition their widgets without breaking the tests. We would also want a way to "turn off" the availability of the widgetID->location map in production if we don't want users to treat our GUI as an API for integration.

    (Test scripts would have to be fixed if widgets are removed or replaced, but that is unavoidable. Those would be changes to the interface's "business logic".)

    I suppose someone could build a GUI tool kit and automated testing tool as a pair (a good research project for a grad school student), but we would probably want to standardize the format of the map so that we could use any compatible GUI toolkit with any compatible automatic testing product.

  4. Giorgio Sironi - Reply

    April 17, 2010 at 12:35 pm

    Refactoring is feasible also for test suites, to maintain their capability to change and adapt. Agile embraces changing requirements, so when you change the requirement you change the acceptance tests. I like TDD at the acceptance level because it gives a definition of done, and applies the test-first approach with all its benefits to design and actual testability.

  5. Iwein Fuld - Reply

    April 21, 2010 at 10:39 am

    @Frank:
    I agree with you, the harder (or more brittle) the link between a visual component and the test code, the more reluctant you should be to write detailed tests. For selenium with an xpath expression on the element class I'd be much more comfortable than with an exact location on the screen or something like Mercury or Eggplant. That's why I propose to emphasize the cost savings in terms of preventing repetive work instead of the true TDD test-first approach.
    @Giorgio:
    You're right, it is feasible to make the tests refactorable, but this usually means that they become less strict with regards to exact button placement etc. I think this is a good thing. In theory I can like the test first approach also for Acceptance tests, but you'd have to show me the first Product Owner who knows what he wants up front. Having a lenient definition of DONE in the face of unclear requirements seems practical. Otherwise you'd be surfing a lot while the Product Owner prepared the backlog.

Add a Comment