Is AI transparency helpful or harmful?

People are still learning to trust artificial intelligence. As such, the goal of many tools and discussions revolve around encouraging widespread AI acceptance.

The thing is, for many AI tools, we can’t see how they reach the answers they do. Known as the AI black box, it’s one of the biggest problems in AI today. We can see the input that an AI tool receives. We can see the output it generates. But what happens in the middle is all too often a mystery.

So, in the pursuit of acceptance and ethical AI, we’re striving for AI transparency. But could transparent AI cause harm?

What is AI transparency?

AI transparency is an answer to the AI black box problem. This means that it’s all about why and how AI decides something. After all, if we can’t see how a computer has reached an outcome, then how can we trust its suggestions? How do we ensure that the answers are correct and free from bias?

AI transparency would help users recognise when an AI decision isn’t correct. It would make sure that someone was authorising AI.

So, it’s needed if we’re ever to achieve ethical AI uses. It’s needed before we can confidently trust AI tools. In short, AI transparency is essential for a healthy AI future.

Terms in AI transparency

Glass box AI, as the name suggests, is where the inner workings of the AI algorithm are visible. This means that you can see the route an AI has taken to reach its output. Currently, most glass box AI tools are those that use linear reasoning, flow charts, and so on. They don’t involve hundreds of layers of processing.

Explainable AI (XAI) is another term you may come across when talking about AI transparency. XAI is about ensuring that a human can understand and explain an AI decision. This means that it provides interpretable information into the criteria, weights and processes that it’s used.

The need for AI transparency

It’s often believed that AI is objective. But the reality is that AI is only as objective as its training data. It’s prone to algorithmic bias; biased data means biased AI. Similarly, AI is not infallible. It can and does make mistakes from time to time.

Without AI transparency, we can’t see when a result stems from an error. For unimportant tasks, this is less of a problem. Yes, it’s better if AI is always correct and fair. But an AI’s mistaken conclusion that you’d like to watch a programme about fishing will never negatively impact the rest of your life.

Now, though, it’s entering important and critical applications. It’s influencing convictions, medical treatment, financial decisions. The kind of things that have a heavy impact on a person’s life. And suddenly, it’s crucial we understand — and can verify — the reasoning behind an AI’s output. If AI gets it wrong, it could hurt lives.

AI transparency: the problem

As with any tool, the problem lies in how we might use AI transparency.

Many AI work using ANNs – artificial neural networks. These act a bit like a filter. They consist of hidden layers of nodes which each process the information and communicate their output to each other. Different nodes have different weights, making their output either more, or less, important to the end answer.

In short, there’s a LOT of processing happening behind the scenes. So, what could happen if this processing becomes transparent? Well, in a 2018 study, it was found that transparent AI models achieved the opposite of what they set out to do. That is, they made it harder to spot and fix mistakes and errors. But why?

Information overload and automation bias

With how complex AI neural networks can get, too much transparency can quickly translate to information overload. This is where there’s so much information about something that it’s harder to decipher and understand.

Applied to AI transparency, this explains how a glass-box approach could become harmful. A person will struggle to process every node and layer of processing involved in an AI decision.

A follow-up study published in 2020 saw that automation bias also comes into play. Automation bias is a phenomenon in which humans favour suggestions from computers. So much so, in fact, that they ignore information to the contrary, even if it’s correct. The study saw that people would trust AI, even when they didn’t understand the explanations. Further, participants regularly didn’t question clearly incorrect output.

In other words, there’s a tendency to over-trust automated decisions.

Mitigating the risksthe importance of interpretability

The issue is, AI doesn’t work or think the same way we do. Understanding why it does what it does, then, isn’t intuitive. For an AI transparency to work, it must also be interpretable.

It needs to be easy for the average user to understand the explanations an explainable AI provides. One method of achieving this is by enabling the AI to provide natural language explanations. This means that it explains itself in the same way a human might, rather than with complicated calculations.

Beyond that, it’s also important that we put practices in place to help AI users avoid complacency. Merely providing an explanation doesn’t make the AI correct all the time. As such, there should always be someone to analyse the explanation and make sure it’s not displaying signs of bias.

There’s still a need for AI transparency

Make no mistake, AI transparency is important. It stands to help us recognise bias and incorrect outcomes. But that doesn’t mean we should blindly trust an AI that explains itself. We mustn’t fall foul of over-trust.

By being aware of the potential risks that come with AI transparency, it’s possible to put practices in place that mitigate them.

Useful links

The AI black box problem

Are AI ethics impossible?

ELI5: explainable AI