Every so often, discussions in the online testing community turns to the “manual testing is dead” theme. Another iteration of this was kicked off earlier this week in the Ministry of Testing discussion group (https://club.ministryoftesting.com/t/automation-is-completely-taking-away-manual-qa/29150/5). I tried to ignore that inflammatory thread title for as long as I could without replying. I lasted about two days.
Ultimately, using automated testing will deliver applications that work perfectly, but only as long as the users know how to use them. A user coming to the app cold, either without training (if the app is a business-related piece of software that demands it), or as a first-time user from the general population who will expect to be guided through the app intuitively, or with detailed instructions, may find the app unhelpful at best, and unusable at worst.
A couple of years ago, I encountered a supermarket customer service terminal that I gave up on in disgust. First it needed to read my loyalty card – but the reader was broken. So I had to enter my address manually. But instead of entering the (UK) postcode first and first line of the address second (thus drilling down from an area to an individual address*), it asked for first line of address first and then postcode second. (So this was validating the address with the postcode rather than using the postcode to validate the address.) But my address appears on different databases formatted differently, and one of them is “wrong” (not the address I was given when I moved into the property). So I entered first line of address and postcode, only to be constantly told “No such address at that postcode”.
(*UK postcodes have sufficient granularity to cover at the most some fifty different property addresses. In rural areas, some postcodes may only cover a handful of properties, or possibly even only one.)
After giving up on this terminal the first time, I referred to some correspondence from the supermarket to see which version of my address they were using. Armed with this information, I went back and tried again.
Although this was run on a touchscreen terminal, with an onscreen keyboard, you were given two fields at the top of the page, and a “Next” button, in the form of an arrow to the right at the bottom right of the page, like so:
Field 1 (top left of screen)+++++++++++++++++++++++(top right of screen) Field 2
The insert point was already in the first field when the screen was displayed.
So with a touchscreen, you’d expect to touch field 1 to enter text, then touch field 2 to enter text, and then go to the bottom of the screen to touch ‘Next’ and go to the next page.
Except you didn’t. You actually had to use the ‘Next’ button to navigate from field 1 to field 2, even though it didn’t look as if that was the correct workflow. After three different attempts to use this terminal on three different visits, and encountering other problems on subsequent pages that meant I was unable to complete the workflow, I gave up and wrote to the supermarket’s CEO to complain. It was clear to me that this app had been through automatic testing, because everything worked the way it was designed to. But it was also clear that the app had never been subject to any manual testing, as that would have soon shown that the app was totally useless to even an above-average user (he said, modestly) unless you’d read the specification document!
(Those of us who work in IT, as I assume most readers of this blog do, are in only about the top 20% of the population who use computers of any sort on a regular basis. If a system that we come across in the real world flummoxes us, then ordinary members of the public have little chance.)
When I wrote to the CEO, I pointed out what my Day Job was, and hinted what my usual invoice for that sort of report would be. When they wrote back, the CEO admitted that the app was due to be replaced within the next three months but didn’t offer me any money, not even a supermarket voucher. I’ve not tried its replacement yet.
Automation is great for checking, as a previous poster said. But any app that is only subject to automated testing is going to fail sooner or later when faced with real world users in real world situations. And to prevent that, you need “manual” testers. Or you need to be able to explain to your CEO why they are getting snotty letters – or even invoices! – from members of the public.