Reducing bias in your data analysis

Hi everybody, I’m Neil from the Data and Insight team here at RocketMill and today I’m going to be talking about how to reduce bias in your data analysis.

Now this is a subject that I’m really passionate about, and this is because bias influences every piece of data that we analyse. Not only that, bias effects each and every one of us, so you could be the most talented analyst out there, but your work is going to be significantly impacted by bias. Now why is this? Fundamentally it’s due to the way our brains process information. Now, recent research suggests that our brains make up to 35,000 decisions a day, that’s right, 35,000 decisions a day. That is on average a decision being made every two and a half seconds. So, if we consider this, that we need to have lots of information to make decisions, and we’ve got to make decisions at speed, it’s no wonder that our brains need to take shortcuts in order to reach decisions at speed.

Cognitive Bias

These decisions are effectively cognitive biases. So a cognitive bias is our brain’s way of simplifying what is processed, enabling us to reach decisions at speed. Now you may be mistaken for thinking that cognitive bias is a single thing, a single element, however it isn’t. There are in fact 189 different cognitive biases that our brains are susceptible to. Now broadly speaking they can cover four key areas, so our brains need to take shortcuts in order to reach decisions at speed in the following scenarios, where we simply have too much information to process, where we don’t have enough meaning, where we need to act quick and fast, or based off of our own memories or pre-conceptions or beliefs.

So I think it’s safe to say that cognitive bias is here to stay, we can’t overcome it, but I do believe we can train our minds to spot the signs of cognitive bias. This will enable us to become more mindful of cognitive bias and the effects that it has, which can really help in terms of us making robust decision making. Now I believe this is really, really important when it comes to data analysis in particular, and throughout my career as a digital analyst, I’ve come across the following three cognitive biases very, very regularly. So I’d like to talk a little bit about each of these today.

So the three cognitive biases that I’m going to be talking about are the anchoring effect, the bandwagon effect and confirmation bias.

Anchoring Effect

So I’m going to start by talking about the anchoring effect. Now the anchoring effect is where we make a decision anchored on one single piece of information. Now this piece of information could be the first piece of information you’re exposed to, but it doesn’t necessarily have to be the case. So let’s take an example, we’re looking at our web analytics data and we’ve seen in our data that sessions are down 20 percent year on year. The anchoring effect kicks in which means we solely focus on the sessions metric to understand why sessions are down 20 percent year on year. However, in order to reduce the impact of the anchoring effect, what we need to do is keep an open mind and make decisions using all available data sources. So taking our example, sessions are down 20 percent year on year, how about we look at engagement metrics? Are engagement metrics improved despite the traffic going down? How about our conversion rate? We could actually be doing a lot better job at converting our audience, even though there are fewer of them. And it could just be that our sessions are down year on year because we’ve invested less in paid media.

Bandwagon Effect

So considering all of those different data sources is going to really, really help in terms of making that robust decision and helping to reduce the impact of the bias. So in terms of our second bias, it is known as the bandwagon effect. Now the bandwagon effect is where we adopt a decision by following the majority. So let’s take another example to explain this. So let’s say we’re looking at our organic search traffic performance and we’ve seen that organic search traffic has shown a decline, and we know at the same time that there’s been a Google algorithm update. So it could be really simple to draw the conclusion that our traffic has actually in fact been hit by a Google algorithm update. And if lots and lots of people are believing this, it’s very, very easy for the bandwagon effect to take control, and for you to just believe it because everybody else is.

So what can we do to reduce the impact of the bandwagon effect? Well, I think we need to be sceptical, we need to challenge, first of all, if that decision has been made based on opinion or data, if it has been made on data, we need to ask certain questions, such as what data was analysed? Was there anything that was missing that we need to consider? We should also look at how we’ve analysed that data as well, are there alternative ways of analysing the data that will give us the same conclusion? In fact, it could be are there any other factors that have affected this trend which have nothing to do with the algorithm update themselves? So all of these elements need to be looked at and investigated thoroughly, and I believe by doing that it helps in terms of made a robust decision. If we’ve done all of that and our hypothesis still holds true, then great. If we’ve done that and it doesn’t, then we’ve avoided a potentially costly decision being made.

Confirmation Bias

The final area I want to talk about today is around confirmation bias. Now confirmation bias is where we make a decision by favouring information that confirms our own preconceptions and beliefs. So again, let’s take another digital marketing example, let’s say we’re doing a lot of AB testing on our website around website colour button and every test that we’ve done so far is showing that orange is the best colour for a website button. Now, as we’ve seen this happening, and happening, and happening we’re building up this preconception and belief that orange is the best colour. So it might go down to the extreme that we just stop AB testing altogether and just put orange buttons all over our website, or we continue AB testing, but instead of looking broadly at different colours we focus on micro shades of orange to see if we can get that little bit more conversion rate coming up.

So what do we do to reduce the impact of confirmation bias? Now I believe we need to constantly challenge and question our data, but also look to seek evidence that contradicts our preconceptions. So for example, let’s look at the test that we’ve run so far, have we got a broad sample of pages? Actually, when we look at those tests, were there some that were only a marginal difference, or some that have a really big difference? For those ones that had marginal differences, could there be something else out there which actually helps improve the conversion rate even further? Also, it’s looking outside of our own website as well, so for example, if our competitors aren’t using orange, is there potentially another colour we could test? So all these different elements should be challenged, should be questioned and should be tested out, again just to ensure that we’re making a really robust decision based off of what we’ve seen.

Overview

So bring this altogether and helping to answer the question I posed at the start, how do we reduce the impact of bias in our data analysis, I believe these four core principles should be adopted at all times to help with this. So first and foremost, always keep an open mind, I believe that underpins each of the three cognitive biases that I’ve talked about today. In order to reduce the impact of the anchoring effect it’s about using all available data to reach a decision, think broadly, thinking about the metrics we’re using, how they interlink with each other and use that to your benefit to make a really strong and robust decision.

In terms of reducing the impact of the bandwagon effect, don’t be afraid to be sceptical. If you see a data analyst being sceptical, they’re not being awkward, in fact I think that’s a sign of a great analyst who’s aware of bias and the impact that that can have on decision making. So don’t be afraid to be sceptical, challenge, pose the right questions.

Finally, in terms of reducing the impact of confirmation bias, it’s all about challenging your preconceptions and beliefs. It’s about thinking broadly and keeping the open mind. And I believe if we combine these four points together, not only will it help to reduce the impact of the three biases that I’ve talked about today, but it’ll go some way to help improving the other 186 cognitive biases which I haven’t talked about today.

Thank you very much