The 10 Things Wrong with Quantitative Research

The Problem With Metrics Today

It infuriates me the way organisations love their metrics. It’s not that I am anti-measurement, it’s more of a cultural thing; the way companies behave is similar to the farmer who constantly weighs a pig in the hope that the mere act of measurement will change its weight! Now I certainly get the appeal of metrics but it seems to me that the ‘real deal’ with many metric processes is not about executing change but more about creating an infrastructure to keep someone’s job. As one wag in patient experience said, the point about Customer Satisfaction is to give that guy in marketing his bonus.

On the face of it, measurement does sound reasonable. Surely if we are scoring low on CSAT/NPS® attitudinal scores the correct action to take is “Do something to improve them!” But have you seriously asked yourself why bother? If you improve the attitudinal scores and there is no accompanying change in the business-related criteria, such as an increase in spend or long-term tenure, what’s the point, it’s a waste of money.

Likewise, I would critique the view that value can be derived by obsessing about your scores across the ‘whole customer-journey and every part of it’. Oh really! ‘I’ can buy an electric toaster and score low on NPS® or CSAT for many parts of the experience – the drive to the store, the 5 minute queue, the search for the white goods section etc. It’s a toaster for God’s sake, a 6 out of 10 type of product! But I can still like the price and the convenience; Job done! It’s good enough for me; you’re wasting your money if you do anymore.

• So the first point about measurement is: are you measuring for measurements sake, or are you considering what are or ‘could’ be the drivers and destroyers of value?  

Yes these can be found across many parts of the journey but not over the whole thing.

For me there seems to be a problem in Customer Experience Management if you take the “let’s measure everything to make everything perfect” viewpoint. Sure this stands up if you believe that CEM is “the sum of every experience”, but in reality not every experience is going to be equally valuable (or could be equally valuable) to the consumer.

The measurement paradigm also omits a key purpose of Customer Experience Management. As I highlighted in my paper: ‘What is Customer Experience Management: did Pine and Gilmore get it wrong?’ CEM is about Innovation and “extracting value from your whole offering wherever that may lie,” So, if you obsess about the ‘as is’ state you will never get to “create a differential value proposition” in the mind of the consumer, something they are willing to pay money for, because you fail to think about what ‘could’ be. Culturally you will end up obsessing about fixing breakages rather than seeing the opportunies.

Of course this is not the fault of the Maths; it’s the fault of how the Maths is interpreted. It’s also a problem of business philosophy: too little attention has been paid to the limitations of numbers because the philosophy of “what gets measured gets done” has not been sufficiently critiqued. In principle I have no problem with the mantra, after all a measurement can be a qualitative observation in which case all it is saying is “what gets managed, gets managed”.  But I believe the term has been bastardised, a better corporate philosophy for things nowadays would be “what gets quantitatively measured is of little or no use at best and at worst seriously prevents action.” The reason: because quantification itself has serious flaws when you deal with customer psychology, let me explain further:

So What’s Wrong With My Metrics!

Issue 1: The philosophical problems of ‘Traditional Quantitative Insights’  

The tradition of quantitative research is based on the Natural Sciences (such as understanding how a rock weathers). Yet it is the most fundamental of flaws to assume that the way people psychologically react to your Customer Experience is the same as in the Natural Sciences.  Rocks don’t have brains after all.

Let’s examine this idea in a little more detail:

Natural Science Phenomena

You put a rock outside in the sun and the rain. You can measure its patterns of weathering and relate this back with accuracy to the amount of sunlight and rainfall received.

You can do this because the ‘rules’ are grounded. Rainfall and sunlight are specifically defined variables; the weathering response is an objective, mechanical response that remains pretty constant and predictable through time.

In much the same way certain short-term market actions predict behaviour in a Natural Science way.

‘I’ often go overseas for work, my mobile operator has recognised this and put a limit on the amount of data spend I have to bear when I am abroad.  The mobile operator has used my past use to create a moment of delight – they have thought about me.

Here then, some short-term behaviour is predictable. Indeed, Big Data operates within this paradigm and so do some aspects of hard phenomena such as price or speed that can be traded off (at least in the short-term).

However, there is another phenomenon called Customer Psychology. The key difference between this and the Natural Sciences is that customers do not react in just a mechanical way; they have brains which think and feel and critically operate in a ‘future predictive’ not ‘past predictive’ way. Hence, unlike the Natural Sciences where rules are immutable – the rain and sunshine are always the same as before and mechanical rules apply– ‘new rules’ can be created as customers react to the subjective question ‘what does this product or service mean to me.’

Here is an example of what this means:

Customer Psychology Phenomena

Customers buying a TV in the 1990s base their buying decision on price and perceived brand quality. Picture quality is a given, the construction of the set with a large cathode ray tube at the back is not important. In the 2000s, however, prices rise, High Definition is a differentiator; Flat Screens are de rigour and essential to TV value.

As we can see from the above example, in Customer Psychology, rules are not grounded and new rules are possible. This means that any dependency on past prediction alone runs the risk of damaging innovation! By way of another example, think of how bendable phones are potentially coming into the market in the next decade. Is phone bendability important now? Not at the moment, but will it become important in the future? Probably!

For managers, therefore, Customer Psychology must be a consideration particularly when most information on a firm’s goods and services is received and subjectively filtered by customers in this way – think about concepts such as ‘ease of doing business’ or ‘quality of call centre.’ These are not ‘mechanical concepts’, they require a subjective interpretation.

Hence, businesses are creating experiences that are constantly open to change; a situation further complicated by the fact that even Natural Science- like information is indirectly impacted by customer’s subjectivity; for example a price rise is perceived better for Apple than for Microsoft.

So, when dealing with customer psychology, the use of quantitative statistics requires careful consideration. It is not just about deduction from the past, but induction to the future. This involves creating and testing new rules without a precedent to see what works, even if customers themselves have not yet expressed any demand.

Issue 2: Correlations in many instances fail

If a customer makes a decision, but does not know why they made it then your correlations must fail.

Much of what customers perceive and act on exists at a subconscious or in-the-moment emotional level. After the event, any traditional research risks picking up, at least in part, the post-hoc rationalisations of consumers, not the real reasons. This is because customer’s lack perfect knowledge of their actions.

Hence, you are at risk if you depend on correlational analysis alone. Sure you will pick up important reasons that are ‘true and fair’ like it was the price, but they will not be the entire reason for action by a long way. They will lack nuance.

Issue 3: Complexity destroys most linear models

Most statistical approaches will not be able to model the complex world of the consumer.   Many just take a few variables and make huge assumptions about how consumers behave; for instance the classic assumption that relationships between variables are linear and exist in isolation i.e., if quality of call centre rises, customer satisfaction always rises by the same proportion.  Likewise, there is a tendency for a great deal of information on an experience to be omitted e.g.  the emotional and subconscious reactions to an experience and the psychological touch points.

If a company’s model fails the complexity test, are you sure you can trust the results?

Issue 4: Customers do not behave logically all the time; hence your logical assumptions fail

It is now widely accepted that consumers do not behave logically all of the time. For the most part consumer decision making is full of heuristic strategies and biases, as outlined by Kahnemen (see the book ‘Thinking, Fast and Slow’, Daniel Kahneman). The point is, has your data taken account of these effects.  For instance, if in reality a few people (key influencers) lead the direction of the market, then basing decisions on an average effect is false. Look at Apple, a business that comes out with innovations no other firm has thought of before and no consumer focus group or dataset has identified. The reality is where Apple leads, others follow. That’s a creative not a VOC analytical approach.

One further example of psychology in action is the sensitivity of customers to negative influences – if something goes wrong it is remembered, if something goes right it is usually forgotten! Unfortunately most companies operate bipolar customer satisfaction scores (very unsatisfied to very satisfied) that fail to tease out levels of dissatisfaction. For instance, if I phone a call centre and speak to the rep and they give me the answer I want but in a surly manner, I am both satisfied and dissatisfied over the same item. I am likely to answer with a bland 5 out of 10 on a bi-polar scale when in reality I would answer highly if there was a separate dissatisfaction scale.  This is something we raised in our webinar ‘Researching Satisfaction Measuring with Unintended Bias”

Issue 5: Future expectations are more important than past predictions

If customers were rocks then we could use past rainfall to nicely predict weathering patterns. But as we have brains, we do not just come to your store based on what has happened before but also based on what we expect to happen in the future. There is of course an overlap with a past experience approach to decision-making but the future expectations approach is important and to my mind closer to reality.

Hence with future expectations a customer would say, ‘I’ think ‘If’ I go there….. ‘Then’ I will get a nice coat (see the If…Then rules of emotion in Baumeister, R.F., Vohs, K.D., et al. (2007), “How emotion shapes behaviour: feedback, anticipation, and reflection, rather than direct causation”, Personality and Social Psychology Review, May 2007 Vol. 11.no.2 pgs167-203). This conforms to how humans are always predicting the future in their behaviour. This also means at some point there is likely to be a disjoint between what has happened in the past and what they would like to happen.

Beyond Philosophy has actually done a comparison between what a driver analysis based on past predictive analytics would say and a conjoint analysis of what customers would like to do. The difference was on average around 30%. This means 30% of your touch points are potentially being deemed unimportant, when in fact they are critical.

Think of it this way:

‘Imagine a Ford car showroom in the 1920s. All cars would have been available in one colour only – black – and in one product type – the Model T. Any decision to buy would have been made on price. If you ask customers directly what they would like, they would say alternative colours and product types.  There is a difference between what is offered and what could drive value.’

Issue 6: There are no set laws, hence any change will change the multicollinearities and ‘doom’ your nice statistical assumptions

One assumption in the Natural Science approach is that the sun and the rain remain the same.  Now let’s look at that thinking and how it applies to the world of Customer Experience. In a Natural Science model you could say, Quality of Call Centre drives Satisfaction, so if we improve Quality of Call Centre then we get an increase in Customer Satisfaction.   But an improvement in Quality of Call Centre means what exactly? Remember if you make a change to quality of call centre, say by putting in more empathetic scripts, this means you can no longer depend on your modelling. Why? Because the very thing you have been modelling against has been the ‘old’ Quality of Call Centre. Now you have changed it, all your relationships will change.

Issue 7: You can’t split an experience into components

You might now be thinking, if past predictions hold restrictive value, perhaps we can model how customers react to possible future changes in a product or service? Perhaps we can model future expectations. Perhaps, if we looked at say buying a restaurant meal, we could establish the best combination of characteristics to gain market share over the competition i.e., comfortable seats, quality meal, mid range service.

In theory this sounds attractive. But is it so easy; if customers are attracted to buy because of the ‘quality of meal’ can all the key components of quality, really be disaggregated and their relative values discerned if they were changed?   Let’s call this the linear view.  Or does each component act in context to the other components such that its value can only be discerned through its relationship to these other components. Let’s call this the complex view.

• In the linear view, you can look at say a car and treat its components such as design as being separate from the Mercedes-Benz brand marquee.
• In the complex view, you cannot treat car design as separated from the Mercedes-Benz brand marquee.

My view is that the complex view holds. This is something I mentioned in the book ‘Future Trends and Insights’ with the example of the Mexican Restaurant meal:

“Maybe you go to a Mexican restaurant with low expectations, but actually get an excellent meal. Now you start to become attuned to the experience. The quality of the food is something that no longer acts in isolation to raise your level of satisfaction, but also starts to have an effect on other aspects of the experience. You start to notice how good the décor is; perhaps you even notice negative things, like the waiter’s poor customer service, and how you wish they played authentic Mexican folk music.”

The complex view, views items not as isolated but existing ‘in relation’ to other aspects of the environment, this is why I call this, entangled value; Quantum marketing anyone?

Hence in the restaurant example, the elements that comprise quality are entangled with each other. They work in combination to form in the customer’s head a higher order value (here named quality). Furthermore as they cannot be separated, they can create an experience halo amongst other items of the experience. The simplest demonstration of this is how Virgin Media is consistently rated highly as a telecommunications operator and First Direct for their ATM service, even though both do not have these activities!

Of course, there will be things in the customers mind that are not part of this entangled value such as if I was to buy a pair of shoes, the internal rubber beneath the heel leather.   But those elements that are in a state of entanglement hold a relationship, a meaning or potential meaning, in the mind of the consumer to quality, the key rateable (and discussable) higher order driver of value.

This is a different view to the way goods and services are perceived mathematically by firms as linear combinations.  This is also different in that entangled value critiques the conception of some things as being hygienic i.e., only affecting value when missed. To give an example, in a shoe purchase, colour of shoelace would be perceived as a hygiene factor. However, if we view it instead as a ‘potentiality’, you could start to think how you could change the meaning of ‘shoe quality’ i.e., if we put in different styles and quality of shoelace it might attract attention.

To illustrate the same point here is another example. One Mobile Phone provider we dealt with put the 28 days terms and conditions for return of phone up front and easily readable on their letter to new customers. This acted as an important clue, informing the client of their honesty and transparency. This was a very different use of the experience than a competitor that just considered return times as hygienic and therefore buried in line 238 of their T’s and C’s.

The point of all this is that you cannot look for a disaggregated ROI for every investment in Customer Experience.   You start from the gestalt ROI (i.e., quality) and then define how that gestalt could be formed.

‘When you go shopping in a luxury store there are many ‘experiences’ that go to make up the concept of ‘luxury store quality.’ There is the smile of the assistant, the look of the carpet, the way the goods are displayed as well as the price and product features.  The point is that these elements, as well as others, come together to form the concept ‘store quality’. You cannot disaggregate ‘smile’ from ‘store quality’ it is entangled with it, indirectly helping define it. If you looked for an ROI of a smile it would be a ridiculous question.  What you need to do is look at the broader concept of ‘quality’ and define what that gestalt is made up of in the mind of the consumer both directly and indirectly. In this way business is more of an art than a science.’

Understanding an Experience can be rather like a chemistry experiment where you start off with two gases (hydrogen and oxygen) and end up with something entirely different when they are put together i.e., water. The point is if you were to just disaggregate them you would assume you would end up with a gas.

Issue 8: Different contexts hold different meanings even if the number is the same

All customer metrics exist in context. A score of 5 out of 10 on say trust towards an electricity and gas utility does not necessarily mean the same thing as 5 out of 10 for a grocery store. In the former the category of good is ‘standard’ and ‘unappealing’ hence a lower score is to be expected and customers will not change their buying behaviour. In the latter example however, because customer expectations are different, the lower score means customers will now go and buy their groceries from somewhere else. This means that companies must take the context or meaning of the score into account.

Here are two examples of why this is important:

1. Comparison problems: one of the problems with cross-comparison of scores between companies is that it can be fairly meaningless if the goods or services in question do not operate in or near the same customer goal state.
2. So what problems: as we saw in the example of the ‘electric toaster’ you can score low on some part of the customer journey but that does not necessarily mean the effect of that low score on customer value is the same as a low score in another part of the journey. So what if it was a 5 out of 10!

However, an advantage of talking about what the number means rather than just what the number is, is that even if we score currently highly on a quality rating – let’s say on trust – we can still create a new experience and different meaning that scores the same level of trust (or even lower!) but drives more value!

To demonstrate this point think of the high trust levels that used to apply to local shops pre-supermarkets. My local butcher would have scored 9 out of 10, great produce and he knows the family. Now a supermarket arrives and ‘I’ go there. I have a lower level of trust say 8 out of 10 for the same products, but it has other benefits, it’s a different way of shopping.

Wither the Impact – Performance chart, a vestige of linear thinking.  Consider more about the meaning of your experience.

Issue 9: Non-variance can hide critical influences

If you go to a cafe at lunchtime to buy a sandwich, one of the main drivers to purchase will be ‘sandwich quality’ which you score at say 8 out of 10.  Now let’s say you go every day for a month and every day you score 8 out of 10 (you like the cheese and pickle sandwich). Let’s imagine that you are also asked about satisfaction and that tends to vary a little with superfluous short-term effects, such as one day they had some boxes in the way of the entrance. This doesn’t change the fact that when you spend money it’s related to the ‘quality of sandwich.’

Unfortunately, because the quality scores are stable over the long-term and there is no or very limited variance, the importance of ‘sandwich quality’ is completely missed, or worse it is considered hygienic when in fact it is critical. You can only see its impact if it changes, and because it hasn’t changed you cannot see its impact. Therefore it does not become a focus for intervention and ideation.  Worse because short-term effects on Customer Satisfaction are deemed more important, money is spent on keeping the aisles clear rather than trying to create new sandwich ranges. This even though the relationship between Customer Satisfaction and spend is not established or poorly established.

Managers must be aware of the fact that non-variance is not the same as not important.

Of course there is also the fact that you are only asking customers of your store and not the vastly higher percentage of consumers who go to your competitors for more compelling reasons!  Again, to get a fuller picture you must consider the drivers of value of your competitors.

Issue 10: The problem of Homeostatic Regulation

You don’t feel as good on the 10th time you visit your favourite store as your first, and neither should you! In other words, emotions and attitudes don’t keep on increasing at the same rate as they are subject to homeostatic regulation. This doesn’t mean you are less likely to visit your store; what it does mean is that returns on emotion and attitude are curvilinear: after a tipping point returns decline. Baumeister, et al. (“How emotion shapes behaviour: feedback, anticipation, and reflection, rather than direct causation”, 2007) evidences homeostatic regulation in emotional response as follows:

“There is evidence that people spontaneously regulate their emotions (Forgas & Ciarrochi, 2002). Immediately after an emotional event, people in both happy and sad moods experience more mood-congruent than mood–incongruent thoughts. With time, however, the content of people’s thoughts moves toward the opposite valence. That is, after a few minutes, participants induced to feel sad were having happy thoughts, whereas those put into a happy mood had relatively more sad thoughts. This homeostatic emotion regulation fits nicely with the current analysis: mood-congruent thoughts help people learn the lessons of their previous behaviour, but adaptive future behaviour requires that emotion regulation take place.”

The management implications of this are that:

1. Firms have to model non-linear effects. This may sound like a technical argument, but many firms allocate resources in the belief that returns will be constant – the more a touch point improves the more positive the effect on the overall Customer Experience as measured through KPI’s such as Customer Satisfaction and NPS®.  In fact, if a tipping point has been reached and you cannot get anymore rise in spend, CSAT or NPS®, predicted on an investment in call centre quality then perhaps you need to stop wasting money!
2. Firms have to critique their insistence on achieving ever higher scores on say CSAT, NPS® or other attitudinal indicators.  Many companies believe that their current high levels of satisfaction are simply not good enough and need to be kept on the path to improvement. Homeostatic regulation would imply that in fact ‘good’ can be ‘good enough’.  Depending of course on your starting point!  A consequence of homeostatic regulation is that all measures are ‘sticky upwards’ since once a positive effect has spiked the figures once (say moving scores from 7 to 9 out of 10), on the 10^th visit scores settle down to say 8 out of 10 but with a different meaning.