October is almost over, so you might ask, “why am I still looking back on September?”. At the beginning of the year I promised myself to share more. Every month so far I have managed to write down what inspired me in the previous month. You might judge me for procrastinating on it this time. I can confess that’s it is really due to a really busy period with little time to recollect my thoughts.

So what have I learnt last month? Work wise I have been encouraged to focus more on ‘the why’ of customer behaviour for some months now. I have dedicated more than a fair share of thought on delving deeper into the why: Why do customers choose this brand or the other? Why do they leave and why do they stay? In this context here are three lessons on the topic. 

3 steps to get to the ‘why’:

One book I started reading in September is The Book of Why by Judea Pearl. One section in the book which I keep returning to is the Ladder to Causation. 

Before it gets too technical, watch this scene from Small Foot. In the scene, Migo’s dad misses the gong. All his life he thought that he was the one causing the sun to come up by sounding the gong. On that morning when he catapulted himself and missed the gong, the sun still come up. He is devastated to learn that it is not his gong causing the sky snake to come up. He shared his feelings with Migo, who is amazed at the discovery. 

For Pearl the first cognitive ability needed to get to the cause is observation. Most insights research is based on observing associations, such as ‘the gong is always heard just before the sun comes up’. Similar observations highlight association but very little on the causal relationship. 

The second ladder rung is doing something about what you observe to understand better the association. One would ask “If I sound the gong, will the sun come up?”. In a different context, “If I take aspirin, will my headache go away?”. That’s a step up the ladder but we’re not fully there yet. Even if my headache does go away, I cannot fully declare that aspirin was the cause. 

That gets us to the third rung, countering the fact through imagination and understanding. “What if I did not have the aspirin, would my headache still have gone away?”. Back to the movie I would ask “if I do not sound the gong will the sun still come up?”.

The promise of machine learning and AI

When we realise how much we do not yet understand the cause and effect relationship of events, particularly human behaviour, we have to also acknowledge how hard it would be for us to get that answer through data science. Why is that so: 

Primarily any machine learning model is trained by humans. The saying “garbage in, garbage out” is most appropriate here. If the input is incomplete, so is the output. I attended AI Experience Conference in Sydney last month. One keynote speaker reported that the vast majority of data models are actually never productionised well enough to give any form of return on investment. Most of them never see the light of day and a chunk of those that do, do not do so in a timely way. By the time some useful insight comes out, the business would have moved on to the next problem.  

Even more of a reason not to hope that a machine can solve what a collection of human brains can’t is the 3 step ladder to causation itself. A machine may be trained to observe associations at a scale and speed beyond what humans can. Machines can also be made to run tests more efficiently than humans can. However the journey to causation requires imagination, retrospection and understanding. As in Small Foot, it requires enough brain power to challenge the facts with counter-facts. As yet, this seems to be far from what machines are able to do!

The emotional side of the ‘why’. 

There was one more statistic which made me realise how hard it is to understand human behaviour, as much as it is fundamental. Humans are emotional beings – 95% of the time they decide based on emotions, the remaining 5% they make a very logical and rational choice. This means that to really understand what causes humans to behave in one way or the other requires us to understand theemotional state accompanying their behaviour. 

This inherently means that even when we manage to get the machine to do the observation, we need to capture as much of the human emotional state as possible. Not surprising that we debate the lack of ‘experience performance indicators’ to properly understand behavioural triggers.

Why is it so important to get to the cause? In the same scene from Small Foot, whilst Migo’s father is devastated to learn that he does not cause ‘the sky snail’ (the sun) to come up, Migo is amazed that “another stone is wrong”, another long standing assumption is being tested. Whilst his dad is scared to ask the question, Migo hopes to discover something even better than what everyone thought they knew. In his own words, “maybe there’s something even better than banging your head against the gong”. 

And there is truly something even better than relying on what we think we know. We just need to stop being the busy gong ringer and start focusing on getting to the why.