Why is handwriting recognition so difficult for AI?

Alongside digital data, you may also have plenty of handwritten information to manage. Cue the slog of digitising handwritten information from letters and forms.

Elsewhere, meanwhile, someone is desperately trying to decipher what grandma has written in her letter this week.

There are countless potential uses for handwriting recognition. But so far, it’s proving an elusive ability for artificial intelligence (AI) to handle.

Here, we explore what makes handwriting recognition so difficult for AI systems.

What is handwriting recognition?

First things first, what is handwriting recognition?

Handwriting recognition is a computer’s ability to recognise and interpret handwritten input. It’s sometimes known as ‘handwritten text recognition’.

This could be a scanned handwritten document or a photo of a handwritten note, for instance. The growth and proliferation of touch screens add another way to input handwriting.

The goal of handwriting recognition has been around since the 80s — and has suffered from accuracy issues from the beginning.

There are two types of handwriting recognition. First is the older of the two, known as offline handwriting recognition. This is where the handwritten input is scanned or photographed and given to the computer.

The second is online, which is where the writing is input through a stylus/touchscreen. This offers the computer more clues about what’s being written. (For instance, stroke direction and pen weight.)

How it works

There are a few different ways that handwriting recognition works. In general, it’s about allowing the computer to turn handwriting into a format that the computer understands.

One way for this to happen is handwriting OCR, or optical character recognition. This is where the computer zooms in on each character and identifies it by comparing it to a database of known characters and words.

This is why you often need to print your answers in ‘BLOCK CAPITALS’ on forms. That’s the easiest kind of writing to program a computer to recognise. It reduces the range of differences in the writing and keeps each character distinct and separate from the last.

The method used for hand-written address interpretation (HWAI) in the processing of letter mail by the US Postal Service

The problems with handwriting recognition

Handwriting recognition tends to have problems when it comes to accuracy. People can struggle to read others’ handwriting. How, then, is a computer going to do it?

The issue is that there’s a wide range of handwriting – good and bad. This makes it tricky for programmers to provide enough examples of how every character might look. Plus, sometimes, characters look very similar, making it hard for a computer to recognise accurately.

Joined-up handwriting is another challenge for computers. When your letters all connect, it makes it hard for computers to recognise individual characters. Consider, for instance, an ‘r’ and an ‘n’. Joined up, these letters could be mistaken for an ‘m’.

In the case of handwriting recognition from photos, there are also awkward angles to consider. The angle the photo is taken could obscure the character, making it harder for the computer to identify.

The upcoming solutions

It’s clear, then, that for computers to recognise and digitise handwritten documents and messages, there’s a lot to learn. There are the different letters, characters and digits. But there’s also the importance of being able to identify them despite differences due to different handwriting styles.

This is where deep learning and neural networks are coming into handwriting recognition. Deep learning allows machines to learn over time, and adapt their output using weights.

In other words, the machine can learn how to identify letters despite different handwriting. More weight can sit on the factors that stay largely the same across handwriting. This means that deep learning is more adaptable to handwriting changes.

Handwriting recognition

In a world running on data, accurate handwriting recognition could become a powerful tool.  With it, hastily scribbled notes and formal, handwritten letters become readable by a computer.

We aren’t there yet.

But, with the help of deep learning, AI could one day greatly improve the accuracy and ability of handwritten text recognition.

Useful links