A horizontal row displaying six handwritten numerical digits: 5, 0, 4, 1, 9, and 2. The digits vary in stroke thickness and slight slant, reflecting natural human handwriting styles.
Most people effortlessly recognize those digits as 504192. That ease is deceptive. In each hemisphere of our brain, humans have a primary visual cortex, also known as V1, containing 140 million neurons, with tens of billions of connections between them. And yet human vision involves not just V1, but an entire series of visual cortices—V2, V3, V4, and V5—doing progressively more complex image processing. We carry in our heads a supercomputer, tuned by evolution over hundreds of millions of years, and superbly adapted to understand the visual world. Recognizing handwritten digits isn’t easy. Rather, we humans are astoundingly good at making sense of what our eyes show us. But nearly all that work is done unconsciously. And so we don’t usually appreciate how tough a problem our visual systems solve.
The difficulty of visual pattern recognition becomes apparent if you attempt to write a computer program to recognize digits like those above. What seems easy when we do it ourselves suddenly becomes extremely difficult. Simple intuitions about how we recognize shapes – “a 9 has a loop at the top, and a vertical stroke in the bottom right” – turn out to be not so simple to express algorithmically. When you try to make such rules precise, you quickly get lost in a morass of exceptions and caveats and special cases. It seems hopeless.
Artificial neural networks (ANNs) approach the problem in a different way. The idea is to take a large number of examples, known as training examples (as seen above), and then develop a system which can learn from those training examples. In other words, the neural network uses the examples to automatically infer rules. To understand how this works, we must look at the basic neuron models.
Why do we need neural networks for something as simple as recognizing numbers? Think about trying to write strict code for every possible way someone could write a loop or a curve. A person’s handwriting might be slanted, cramped, or messy, completely breaking rigid programmatic rules. Neural networks bypass this by learning the flexible features of each digit straight from data, much like how humans learn to read handwriting through exposure rather than memorizing a rulebook.
To get started, we’ll explain a type of artificial neuron called a perceptron. Perceptrons were developed in the 1950s and 1960s by the scientist Frank Rosenblatt, inspired by earlier work by Warren McCulloch and Walter Pitts.
A way you can think about the perceptron is that it’s a device that makes decisions by weighing up evidence. A perceptron takes several binary inputs and produces a single binary output (either a 0 or a 1). To determine this output, Rosenblatt introduced two key parameters:
These terms are further discussed in section 6.2. By varying the weights and the bias, we can get different models of decision-making. A many-layer network of perceptrons can engage in sophisticated decision-making, where a perceptron in the second layer can make a decision at a more complex and more abstract level than perceptrons in the first layer.
While perceptrons are a useful model, a network of perceptrons cannot easily learn. The problem is that a small change in the weights or bias of any single perceptron in the network can sometimes cause the output of that perceptron to completely flip, say from 0 to 1.
That flip may then cause the behavior of the rest of the network to completely change in some very complicated way. That makes it difficult to see how to gradually modify the weights and biases so that the network gets closer to the desired behavior. Luckily there’s a clever way of getting around this problem.
We can overcome this problem by introducing a new type of artificial neuron called a sigmoid neuron. Sigmoid neurons are similar to perceptrons, but modified so that small changes in their weights and bias cause only a small change in their output. That’s the crucial fact which will allow a network of sigmoid neurons to learn.
Just like a perceptron, the sigmoid neuron has inputs, weights, and a bias. But instead of being just 0 or 1, the inputs and output can take on any real numbers between 0 and 1 (for example, 0.638...).
A Cartesian coordinate plot illustrating the sigmoid function curve. The vertical axis ranges from 0 to 1, while the horizontal axis represents inputs, demonstrating a smooth, continuous S-shape transition from 0 up to 1.
To calculate this output, the sigmoid neuron uses a specific function called the sigmoid function (or logistic function). The mathematical hook of the sigmoid function is its shape when plotted on a graph: it is a smooth, S-shaped curve.
Because of this geometric smoothness, a small change in the weights or bias of a sigmoid neuron will produce a small change in the output. In fact, the output changes linearly and predictably with small modifications. This smooth responsiveness is what makes it possible for a neural network to gradually tune its parameters, learn from data, and improve its performance over time.
