Most people have heard of data-driven decision making – the admirable desire to make better decisions through data. The conventional justification is that, given the complexity of the world, organizations need data to make decisions.

Such ideas exist in all organizations, from governments to corporations to hospitals. In fact, they are wrong in principle and misleading in practice.

Data-driven decision making puts emphasis in the wrong place and means organizations focus on the wrong thing. Let’s take a look at why.

Drowning in data

In 2010 we created 2ZB (zettabytes) of data globally. More than 100 ZB will be created this year. That’s about 50 times more – and roughly the equivalent of every human being generating an entire copy of the Library of Congress every year.

If data-driven decision making were correct, this growth should lead to significantly improved organizational performance.

Ideally, you could be hoping for a 50x performance increase over 2010. Did that happen? Clearly not. In fact, has there been any improvement since 2010? If there is, it is not obvious.

In hundreds of conversations with companies, no one has ever said to me: “Data has made my life easier. Suddenly, I have all the answers. “In reality, the conversations are more like,” The data is overwhelming. It’s all in silos. People pick the pieces to help their cause. “

So what’s wrong?

The history of decision making

Let’s go back in time to understand why. For thousands of years it has been thought that understanding, and therefore decisions, came from divine mandate. About 400 years ago, philosophers realized that collecting data to create understanding was a good thing.

However, they also thought that the data alone was enough to establish what the world was like and predict what would happen next, a process called induction. They thought that a broader understanding of what was happening didn’t matter.

Note that this is the same statement made for data-driven decision making, but we know that a broader understanding is important.

Will the stars appear in the sky because they did it yesterday? Well yes, for a while. But at some point they will run out. What was an obvious extrapolation is suddenly no longer true.

This view changed with the philosopher Karl Popper, who claimed that we don’t inductively extrapolate from data, because that’s impossible. In fact, he guesses what’s going on, so we find the data to falsify that theory.

This is a crucial change. Suddenly, the focus is on the theory, not the data. This means that the theory can be very different from an extrapolation from the data.

Thus, the first stage of decision making was divine truth is everything, the second stage the data is everything and the third stage the theories proposed and refuted through the data.

Companies have moved from phase one to phase two in the past few decades. Moving from CEO mandates to data analytics is fairly straightforward, but we haven’t yet made the leap from data to theories.

Why this is important

For the first time ever, machine learning offers us the ability to help people build more complex explanatory theories, test them to see what’s likely to happen, and see what we can do about it. It sounds abstract without an example, so let’s take the pandemic.

Understanding when Covid cases will rise and fall has guided decisions affecting trillions of pounds and billions of lives.

In March 2020, the NHS had to make life-or-death decisions based on understanding whether the curves were trending up or down. For the first time ever, they connected data streams from the entire system.

This is a potentially overwhelming amount of data, but most importantly, it hasn’t been paralyzed by this sudden torrent. They have not succumbed to data-driven decision making. In a pandemic, it would have been catastrophic. Why was that?

Well, working together with the NHS, we have applied approaches from a nascent field called decision intelligence. We used this data to build the understanding the NHS needed to make confident decisions about which hospital wards to open or close and which resources to send where.

Instead of just showing data on current levels of patient demand, we could also show demand for tomorrow or next month, why demand was changing, how this affected resources like beds or ventilators, and what actions they could take. .

Rather than making data-only decisions, the NHS made them based on an understanding of how and why events were unfolding. The collective achievements have been widely credited for saving thousands of lives.

Decision intelligence is the future

Decision intelligence is one of the most important contemporary applications of artificial intelligence (AI). This next generation of technology will have far-reaching consequences on how organizations make decisions and comes at a much needed time.

We need to go beyond the hell of the data-driven decision spreadsheet. We need a revolution in the way our leaders approach decision making.

And, just as Popper taught us, we need to move to understanding-based decisions, and that means true decision-making intelligence.

Marc Warner is the CEO of Faculty, which he co-founded in the belief that the benefits of AI should extend to everyone. He has since overseen the growth of the Faculty to become one of the leading artificial intelligence companies in Europe. He has led many data science projects, with clients ranging from multinational companies such as easyJet and Siemens to the UK government and the NHS. Outside of faculty, Warner advises the government through the AI ​​Council.

Leave a Reply

Your email address will not be published. Required fields are marked *