(This review is cross-posted from my book review blog, Deep Waters Reading.)
I bought a copy of this book on the recommendation of a colleague in the software testing community, and I struggled with it until bailing out at around the 65% mark. The basic premise, that we have a bicameral mind with two different ways of thinking, and that we rely on the first way of thinking most of the time, which is fine when it’s right but not so good when it’s wrong, is important and needs saying. So much of our thinking about things that are complicated seems severely influenced by what Kahneman describes as System 1 thinking, which jumps to conclusions and takes the easy way out. We certainly seem to be living in a System 1 world right now.
Kahneman then goes on to describe all the different sorts of biases that can fool us. This is an important area for software testers in particular, because these biases influence the way we look at software applications under test, and the assumptions we make about how a particular application works, ought to work, or how it will be used by people out in the Real World who just want to open the software and use it without any further thought or preparation, like any other simple tool, from the stone axe onwards. However, computer software is just that bit more complex than the stone axe, and that’s where the problems start.
So far for the book, so good. But I started running into problems with it from the outset. I rapidly came to the conclusion that someone, most likely the publisher, dumbed it down. (It took me a few days to discover the notes at the back because someone decided that it would be better to take all the referencing out of the text – but without the referencing, the book often reads to me like pseudo-science because of the way Kahneman keeps saying “Studies have shown…” or “Scientists in San Francisco found…”; without knowing that there actually is a solid, valid reference behind these statements, they look like the sorts of things pseudo-scientists say to “prove” that you can extract sunbeams from cucumbers).
I did find the text rather old-fashioned; it read like a 1970s psychology textbook,. and indeed that’s when Kahneman and his collaborator Amos Tversky did a lot of their initial work. In any case, I got as far as Chapter 16 and then seriously considered abandoning the book. But I rested it for a few days and then went back to it, which seemed to coincide with what looked like a change in direction in the text, to a more anecdotal style. But that was something of a false dawn, because Kahneman then dived into analyses of risk and gaming, and we ended up with a series of examples of questions like “Would you rather have a 50% chance of winning $50 and a 10% chance of having $10 taken away from you, or a 60% chance of winning $10 and a 35% chance of winning $85?” and after the fourth or fifth example of that – which seemed to occupy much of the rest of the book – I gave up. This is not something I regularly do.
Partly, I suspect I may not be the book’s intended audience; I found myself challenging too many of his examples and I saw through the perspective exercise in chapter 9 (Figure 9 – page 100 in my UK paperback edition) and was then amused to see that the author recognises that “experienced photographers have the skills of seeing the drawing as an object…” and I am such a photographer!
Or perhaps I was applying my tester’s mindset to the problems, which may be over-thinking them, trying to find real-world solutions instead of just letting my own Systems 1 and 2 battle it out between them.
I did find the text excessively US-centric, to the point where I complained loudly over one question that was put as an example quite early on: “How many murders are committed in the state of Michigan?”, to which I replied “No idea – I’d usually Google that one.” Well, of course, the question ought to have been “What is your estimate of the annual number of murders in the state of Michigan?”; but that aside, then Kahneman saying “Well, of course you only thought of Michigan and forgot that Detroit is in Michigan and so has its tremendous number of murders counted in the state-wide total” just struck me as the sort of geographical bias we try to eliminate when doing testing work, and indeed seemed to expose the very biases he went on to discuss later on in the book. There are other examples but this was the worst.
But Chapters 14 and 15 brought me to a juddering halt with two examples of what testers call “testing personas”, invented characters who are used to represent typical real-world users. The first, “Tom W”, is based around a set of assumptions about IT systems developers from the 1970s! I’ve worked in IT for twenty-five years, and the sort of stereotyping that Kahneman bases his expectations on died out long ago, certainly in the organisations and companies I’ve worked in. And then we had the “Linda problem”. Kahneman set up a fictional character, Linda, with a fairly detailed backstory and life circumstances, but then when people say “Yes, Linda could be a feminist bank teller”, he says that is the wrong answer! I’m sorry, I got angry with him at that point. I’ve met plenty of Lindas (of both genders) with strongly-held political beliefs that drive their existence, and they are quite capable of holding down comparatively menial jobs. If anything, their political beliefs support them in their jobs and give them a focus outside of those jobs that helps them cope. It was at this point that I realised that Kahneman was applying economic criteria to his cases. Thinking of the probabilities of the quantum of feminist bank tellers as a proportion of all bank tellers, the likelihood of Linda being a feminist bank teller was arrived at statistically. Yet Kahneman created her with a backstory where those feminist values would be sufficiently important to her for her to hold to her feminism – in the real world. Kahneman’s explanation at the end of chapter 15, that the sort of objections I raise actually aren’t relevant to the argument he’s trying to make, just irritated me more. Are we supposed to be reading this book to find out interesting facts about human nature when working with human beings and their artifacts, or just to admire how clever Daniel Kahneman is?
(His ‘less is more’ example in that chapter – can you charge more for a tea service with fewer pieces, all of which are perfect, or for one with more pieces, a significant number of which are imperfect – made me smile because if the author had had much experience of actually selling things in sets, he would have realised that a smaller but complete set is worth more than a larger, but incomplete and/or flawed set, especially to more discerning customers.)
Perhaps I was reading it too fast. One of the blurbs on the back of the UK edition says “Buy it fast. Read it slowly.”; and indeed the person who recommended it to me in the first place stretched his reading of it out over a series of weeks. As I said, perhaps i’m not the book’s intended audience. Perhaps I’ll just take it to the office and see if any of my other testing colleagues want to have a try at it.
Postscript
In Chapter 34, Kahnemann refers to ‘nudge theory’, as described in the 2008 book by Richard Thaler and Cass Sunstein, “Nudge: Improving Decisions about Health, Wealth, and Happiness“. (Kahnemann and Thaler worked together at Stanford University in 1977-78.) Thaler is generally credited with creating nudge theory. Yet I have recently uncovered evidednce that the idea originated a long way away and much earlier.
I’ve recently been reading a collection of the amateur writings by an Irish science fiction fan, Walt Willis. His day job was as a senior civil servant in the devolved Northern Irish Stormont parliament. In 1969, Willis described an idea he’d had:
“...the principle is one I follow in my day-to-day work as a senior civil servant concerned with the problem of making people behave better. Privately I think of it as Nudgism, the theory that people can be induced voluntarily to do things you couldn’t force them to do.”
Willis only ever described his idea once in print, in a limited-circulation small press fanzine (Warhoon 26, February 1969). I’ve seen no evidence that either Thaler or Sunstein ever saw that, and at this stage it would probably be impossible to access what circulation records there might be (if, indeed, any ever existed) to see how the idea got passed on. I suspect it submerged into the collective unconscious and only surfaced some 35-40 years later in a conversation with one of the authors that went “I read somewhere about an idea…”
When they wrote back, the CEO admitted that the app was due to be replaced within the next three months but didn’t offer me any money, not even a supermarket voucher. I’ve not tried its replacement yet.
Probe probare - to properly test 