The trees are evil and must be punished

Someone asked this morning on the MoT site about the relative importance of documentation in testing. “Do we have to spend so much time documenting our tests? It’s so time-consuming: can we do without it?”

If the application you are testing is only ever going to be used once by one person for a completely trivial purpose and never used again, then no documentation is required. In every other instance, some sort of documentation will be needed:

  • When users report bugs in production but no-one who worked on the app is still with the company;
  • When version 2 needs to be developed but the devs who built version 1 have left the company;
  • When users need help or support in the ordinary operation of the app (traditional test scripts are actually very good for helping to create Help files or a wiki);
  • When a third party (a client, a regulator, a lawyer, the police, the Government) requires evidence of tests performed or some other tangible guarantee of “quality”;
  • When a user blames the app for something bad happening and escalates their complaint to:
    • Your CEO, or
    • a court of law; or
  • When something happens in real life (on a scale of 1 to 10 where 1 is “everybody is laughing at your company” and 10 is “somebody died”) and your company is blamed for it in the media.

These are the times when you will be thankful for your test documentation. I know, I’ve seen all of these (though fortunately, on the last one, my personal involvement was around a 5). Risk analysis will help companies and managers take a reasoned decision on just how much documentation is essential to create and retain. If your risk on the last one is a 10, then be prepared for a lot of printing which you hope no-one will ever have to look at again.


The Way of the Tester: 9. Do nothing which is of no use.

In Musashi’s time, this principle, the final one in his Way of Strategy, was exhorting his pupils to fill each shining hour with valuable, objective-oriented activity. Practice, train, do – that was the mantra. Our modern age might encourage us to add: learn, create, teach, share.

But much of our corporate life might be taken up with meaningless activity. No matter how streamlined and agile our teams may be, we may find ourselves working with partner companies whose ideas don’t match our own. In a recent testers’ meetup, I heard an account of working with a digitial media partner whose internal compliance criteria required the submission for scrutiny of all the content provider’s test cases and scripts. Their reaction was, shall we say, interesting.

But most of our thinking has gone beyond this. “Do nothing that is of no use” could equally well apply to our working practices. We should look for ways of working that eliminate pointless activity and add as much value to our products or processes as possible. If there is one thing I took away from thirty years’ working in the UK Civil Service, it was that there are actually good reasons for a lot of bureaucracy; but over time, these get forgotten; then new managements bring in “reform” and “sweeping away the cobwebs”, and this works reasonably well until suddenly one morning, people wake up and things have all gone horribly wrong. And then we find ourselves hastily re-inventing the bureaucratic wheel. I’m going to be a bit heretical here; it’s my opinion that the public sector, having been under continual political pressure to “cut out pointless bureaucracy”, has actually got rather good at streamlining processes to get the maximum benefit for the least effort (at least in areas where political interference is kept to a minimum). The worst bureaucracies are to be found, in my opinion, in the supposedly efficient private sector, especially in finance and insurance.

These latter two sectors are good examples of complex organisations. And it is those organisations that severely test our ability to “do nothing that is of no use”. But the more complex a system is, the more likely it is to demonstrate situations where there are unexpected system behaviours and outcomes.

James Christie has written a series of blog posts on the subject of “The dragons of the unknown”, about the broader testing issues exposed in a career that started out working for one of the major corporate accountancy firms and then moving into business audit work before specialising in IT audit, which is rather like software testing but coming to systems that are already in place and at a time when they aren’t actually broken (well, at least not in the eyes of their admins…) 

In the second blog, he looks at the characteristics of complex systems:

And he writes about the nature of IT audit work in the final post of the sequence:

All this goes to show that there is value even in the most menial of tasks if you can extract lessons from it that you might be able to apply elsewhere. I wrote in earlier posts about the value of experience; well, this is the “information gathering” phase, where to get the experience to inform your later decisions, you first have to put the time in at the workface. Experience cannot be bought, only taught.

There is a story that if you asked someone sweeping up leaves outside NASA’s Manned Spaceflight Center in Houston what they did, they would say “I help launch the Space Shuttle”. That is an example of knowing about the whole organisation’s objectives and how an individual fits into the wider system. It also suggests an appreciation of the interconnectedness of things, which again is something that the modern world relies on. An appreciation of the relationships within complex systems will help show areas where tests are in or out of scope, because of interactions with a different app, or an upstream or downstream system, or an external provider, or a third-party application. There will be points of hand-off between all these, and criteria for successful hand-off. Often, in complex sytstems, these are the points where failures can occur. James Christie writes about not relying on the system specifications as Holy Writ to be adhered to, but such documentation can reveal where someone overlooked some data packet parameters when designing a downstream application. At some point, a tester may well have to report that what looks like a bug may be a configuration issue, or a problem in another application; and the fix may not be an IT matter but a business decision.

Very few of our systems work in complete isolation. And the more everything is connected to everything else, the more likely it is that the solution to a problem might lie outside the immediate environment. And that is where we might find that nothing we ever learn about the real world is ever “of no use”. Even in a 16th century manual of strategy for samurai.

The Way of the Tester: 8. Pay attention even to trifles.

Nowadays, we say “The Devil is in the detail”. In Musashi’s time, things could be a bit more drastic than that. A small clue – a bird suddenly bursting into flight, a change in the fall of shadows, a movement of a branch or someone hanging back from stepping through a certain doorway – might indicate the presence of a hidden attacker. Trifles like these could save your life; or more importantly, your lord’s.

For the most part, our testing jobs are  not quite so life-and-death these days. Though I’ve worked on medical applications which were subject to very stringent regulation to safeguard patients and ensure that defects were minimised; in another job, if I’d gotten my testing wrong, there could ultimately have been a knock-on effect that would have affected utility bills paid by every household in the country. The financial sector also places a strong emphasis on accuracy because of both financial probity and the need to try to ensure legal compliance with money laundering regulations.

But for the most part, the ‘trifles’ we encounter are likely to be quite minor manifestations of error, though if these are things that occur in outputs, a minor error may be an outward sign of more serious issues in the code. And if the app handles a large volume of transactions, a rounding error four decimal places down may become subject to a multiplier effect and have a material impact on decisons taken in the real world using those numbers.

Even if you are not dealing with transactional apps or working in business areas subject to regulatory oversight, paying attention to small details is highly worthwhile. People notice small things. A small feature that gives them pleasure makes them think favourably of your app; a minor error that sticks out for one person may affect decisions taken about your app, your work or your company. If two apps are under consideration for purchase, and one has minor typographical or presentational issues in it and the other doesn’t, the final decision may hinge on those errors. As the saying goes, “you only have one chance to make a first impression”. Any compromise on presentation, spelling or grammar is going to be important to somebody – and they may be the decision-taker in your dealings with them.

The Way of the Tester: 7. Perceive those things which cannot be seen.

In this principle, Musashi was thinking about using all the senses – hearing, smell and the other, more peripheral senses – to supplement sight when in combat. It might even apply to not allowing what you are seeing to distract you from other non-visual clues about the what is happening around you. For most applications, that won’t apply to software testing, though of course if your app involves different media channels, that may well apply in its most direct sense.

But for most testers, “perceiving what cannot be seen” will be a rather harder task. We’re back to that old standby, experience.

So many software issues occur when two things interact. Finding that interaction is one of the hardest jobs in testing – “Well, it works on my machine.” There are always hidden outcomes, display issues, and accessibility problems. All of these may fall into the category of “things that cannot be seen”.

Cross-browser testing is one area which may expose “that which cannot be seen”. An app may display differently on one browser to another; one I have seen recently literally has a feature which cannot be seen in one popular browser but is OK in others.

Developing insight is another of these things that comes with experience; and just as I said in an earlier post, it’s never too early to start gaining that experience. Everyone has to start somewhere.

There is also the question of unintended consequences. You will never be able to think through all the possible permutations, but you should try to anticipate some of the more obvious possible ones. The more you think of them, the more new possibilities may occur to you. This is an area where mind mapping or risk storming may be useful techniques.

The Way of the Tester: 6. Develop an intuitive judgement and understanding for everything.

“But what,” you might ask, “should I do in the afternoon?”

That’s a little unfair of me. But this is one of Musashi’s principles that can only come with experience. It stands against the snap judgement, “we have to do something; this is something; therefore we should do it” position that you get in so many situations.

It may not be about deciding to do a thing; it may be about deciding not to do a thing. I came across a bug yesterday that I thought was happening when the system did a certain thing. But examining it further showed that this was a coincidence and the bug was happening without the certain thing happening as well. My intuitive judgement was to look a bit further and try out a number of scenarios before committing the bug report back to the devs.

I have had a working career of nearly fifty years. I’m not for one moment suggesting that people with less experience than me are not suitable to be testers – far from it. But I’m often finding that I draw on experience dating back before I was a tester, or sometimes even before we had computers in the workplace, to inform me about user behaviour and business needs. It is a question of developing the art of seeing, a capacious memory for odd things that people do at work, and being able to develop the intuition to link the two.

That skill doesn’t come overnight. But neither does the repository of experience. Keen testers need to start developing the habits of observing and finding out about other people’s lives, jobs and transactions. You never know when a bit of obscure knowledge will come in handy.

It’s no good thinking that “my company makes software for soft toy manufacturers, so I need to learn everything there is to know about soft toys.” That may be so, but restricting your hunt for knowledge only to soft toys will not make you a better tester. One day, a problem will arise that can only be solved with reference to the outside world.

A tester can never have too much knowledge.


The Way of the Tester: 5. Distinguish between gain and loss in worldly matters.

This principle has a number of different possible interpretations. The one that first came to mind was the artistic tenet of minimalism: “Less is more”. But I think that a better example for business the Law of Diminishing Returns, such as the extra test that satisifies a managerial target for “test coverage” but actually adds nothing to the value of the tests done.

In any case, managerial metrics such as “test coverage” are merely statistical constructs, and have little or no bearing on how effective the testing has actually been.

On reflection, though, I think this can be read as applying far more to risk analysis. What are the most important functions of the app? Which ones would cause the most trouble if they didn’t work? (Do not overlook the impact of minor but highly visible bugs. Reputational damage is still damage.) How much time is available for testing, and which functionalities of the app have to be tested before that deadline? Is that deadline an internal one – a milestone that must be passed, but where some presentational aspects can be deferred because the app isn’t going to be seen in public – or is it a stakeholder or public demo, or a release, where functionalities have to be in place because they have been in some way promised, or where there are stakeholder or customer expectations?

Business managers should be taking these things into account when allocating test resources to a project. Their appreciation of the nature of risk to the business should, we assume, be all the greater because of their outward-facing focus. They should understand the benefit to be gained from deploying testing resources and balance the loss to the company (through overheads) of providing that resource against the gain of not having to do rework, rectification or just plain fire-fighting.

Weighing these things properly in the balance, I think, meets Musashi’s idea of “gain and loss in worldly matters”.

The Way of the Tester: 4. Know the Ways of all professions.

At first, I thought this was sufficiently similar to no.3 (“become acquainted with every art”) as not to need a separate post. But not.

Certainly, just as the good samurai should have a proper appreciation of all the arts, so as to make good judgements on behalf of his lord, his daimyo, then the same applied to the professions (from builders and other craftsmen to medicine and law). As the manager of his domain, this was required. But for the professions, this went further.

Knowing the Ways of the professions meant having a deep understanding of what it was those professions had to do to deliver their goods and services. There was no value in trying to cut corners, because the Ways of professions were part of the warp and weft of life.

So it is now. If we are testing an application which addresses a particular function in law, or education, or medicine, then it is no good imagining that we can test independently of the special requirements of that profession. Business knowledge is invaluable in understanding how users will react with an app, what they want from it and how they will use it. If we are lucky, these things are in the spec or the requirements. But sometimes they are not. Or we may not have access to the original specification; or, indeed, to end users themselves.

For that reason, there is a good case for companies to ensure that a testing team has at least one person who has business knowledge; preferably of the business specialism itself, if the app is so specialist, but otherwise any sort of wider business knowledge will be useful.

If the company only has one tester, then business knowledge is essential. Indeed, it may be preferable to technical knowledge. Technical knowledge can be taught; experience cannot.

But if this is not possible, then at the very least it will help if the tester can visualise the end user and the way they will probably want to interact with the app. Any sort of life experience may be turned to this end.