ELI5: explainable AI



If an AI confuses a kitten with a giraffe, it’s relatively harmless. Comical, even. But what if that AI mistakes you for a wanted criminal?

AI is not perfect. It gets things wrong, learns biases and is easy to trick. Punctuating this is the fact that we largely don’t know how AI comes to the answers it does.

This is a worrying concept when you consider that AI could soon decide what medicine you need. Or whether you get that loan you applied for. Or even how to navigate a road accident.

Enter explainable AI.


What is explainable AI?

Explainable AI (XAI) is artificial intelligence with an understandable decision-making process. That is, a human can understand and explain how an AI program reached the output it provides.

There’s a key distinction here. An AI program is not explainable AI because a human can justify its answers. Rather, it’s XAI when the program comes with clarity as part of its design.

This means that the human can look at the criteria the AI has used, the chosen decision-making process and the potential for error. They aren’t justifying the answer, they are simply understanding how the AI reached that answer.


The AI black box

Explainable AI is an answer to the AI black box problem. When it comes to an AI-powered decision or answer, we tend to only see the start (input) and the result (output). The process in the middle isn’t visible to us. This is the AI black box.

The black box problem is most rampant in machine learning technology and neural networks. This is where machines spot patterns and learn from data over time. So, when you give the machine input, it filters through ‘hidden layers’, applying the patterns and rules the machine has learned.

But we can’t see these layers. So, we don’t know what patterns the AI applies, or how it applies them. The idea of explainable AI, then, is to shine a light on these hidden layers and processes. But why is this important?


Explainable AI vs learned bias

With artificial intelligence growing in use, the need for explainable AI becomes more pressing. We’re reaching a point where a flawed AI-based decision can have a heavy impact on someone’s life. For instance, consider medical diagnosis AI, where a wrong answer could result in needless suffering.

If we can’t understand why the AI gives a certain diagnosis, how can we trust it?

AI, much like humans, isn’t infallible — it can make mistakes. And when we can’t explain the reason for the mistake, we can’t fix it.

For example, AI programs can accidentally learn biases — and so generate discriminatory results. We’ve seen recruitment AI that discards applications from women. Or consider the facial recognition AI that cannot correctly identify people of colour, for instance. 

With explainable AI, we can understand the cause of biased results, and take steps to fix it.


The need for transparency

The rise in AI and the issue of the AI black box has legal implications, too.

When it comes to the kinds of big decision AI is starting to tackle, there’s the right to explanation to consider. That is, an individual has the right to get an explanation for the output of an algorithm. But if we don’t know how AI makes its decisions, it’s impossible to explain.

Then there’s the issue of accountability. If an AI makes an incorrect or discriminatory decision, who is accountable for the fallout? Explainable AI would give organisations easier control over their AI tools. This means it’s easier to hold them accountable if their AI offers discriminatory, detrimental or dangerous answers.


Explainable AI, explained

Explainable AI is, in its simplest terms, AI with none of its hidden workings hidden away. It keeps clear the reasoning behind its answers.  So, with it, we can make sure that the decisions made by machines are the best ones.


Useful links

What is machine learning? A beginner’s guide

ELI5: what is an artificial neural network?

AI fails: why AI still isn’t ready to take your job