Crandi

Backpropagation Explained Step By Step In Google Colab 2025 How

Leo Migdal
-
backpropagation explained step by step in google colab 2025 how

Neural networks look magical from the outside — you feed in data, and they somehow learn to recognize cats, translate languages, or even drive cars. But behind the scenes, the learning process is powered by one key algorithm: Backpropagation. In this blog, we’ll break down backpropagation step by step, understand why it works, and see how it teaches neural networks to get better over time. Backpropagation (short for backward propagation of errors) is the algorithm that enables neural networks to learn from mistakes. Here’s the idea: Repeat this enough times, and the network learns.

Let’s go through the full training cycle. An interactive, Colab-ready notebook that builds a tiny autograd engine (like micrograd) from scratch, spells out backpropagation step by step, and makes gradients visible. 🚀 Designed as a teaching + portfolio project to demystify backprop and showcase engineering clarity. A neural network consists of a set of parameters - the weights and biases - which define the outcome of the network, that is the predictions. When training a neural network we aim to adjust these weights and biases such that the predictions improve. To achieve that Backpropagation is used.

In this post, we discuss how backpropagation works, and explain it in detail for three simple examples. The first two examples will contain all the calculations, for the last one we will only illustrate the equations that need to be calculated. We will not go into the general formulation of the backpropagation algorithm but will give some further readings at the end. This post is quite long because of the detailed examples. If you want to skip some parts, these are the links to the examples. Before starting with the first example, let’s quickly go through the main ideas of the training process of a neural net.

The first thing we need, when we want to train a neural net is the training data. The training data consists of pairs of inputs and labels. The inputs are also called features and are usually written as $X = (x_1, \dots, x_n)$, with $n$ the number of data samples. The labels are the expected outcomes - or true values - and they are usually denoted as $y = (y_1, \dots, y_n)$. Training a neural net is an iterative process over a certain number of epochs. In each epoch, the training data is processed through the network in a so-called forward pass, which results in the model output.

Then the error - loss - of model output compared to the true values is calculated to evaluate the model. Finally, in the backward pass - the backpropagation - Gradient Descent is used to update the model parameters and reduce the loss. Note, that in practice, generally no pure gradient descent is used, but a variant of it. We are not going into detail here, but important to understand is that some optimization algorithm is used to update the weights and biases. For a general and more detailed introduction to Deep Learning terms and concepts, please refer to Introduction to Deep Learning. If not mentioned differently, we use the following data, activation function, and loss throughout the examples of this post.

We consider the most simple situation with one-dimensional input data and just one sample $x = 0.5$ and labels $y = 1$. Last week, we really went to town on activation functions, the functions which provide us the means to model non-linear patterns with neural networks. Nice! This time round, we’ll discuss the process by which a neural network’s weights and biases are updated (aka how a neural network learns): backpropagation. I would strongly recommend you catch up on my previous two articles if you haven’t already: 🔐 An introduction to what neural networks are can be found in Neural Networks 101

🔐 Be sure to check out Introducing non-linearity in neural networks with activation functions (it does what it says on the tin) Neural networks are like brain-inspired math machines they learn by trial and error. But how do they know what to fix when they get something wrong? The answer is backpropagation — the math magic behind almost every AI breakthrough today. This guide breaks it down simply, with no PhD required. Whether you’re just starting out or need a refresher, you’ll walk away knowing how AI learns, and why backprop is so powerful.

In the 1980s, neural networks struggled to handle more than a few layers — they couldn’t “learn deeply.” Backpropagation solved this by letting errors move backward through the layers, allowing networks to learn complex patterns. That breakthrough laid the foundation for deep learning as we know it. Backpropagation, short for Backward Propagation of Errors, is a key algorithm used to train neural networks by minimizing the difference between predicted and actual outputs. It works by propagating errors backward through the network, using the chain rule of calculus to compute gradients and then iteratively updating the weights and biases. Combined with optimization techniques like gradient descent, backpropagation enables the model to reduce loss across epochs and effectively learn complex patterns from data.

Back Propagation plays a critical role in how neural networks improve over time. Here's why: The Back Propagation algorithm involves two main steps: the Forward Pass and the Backward Pass. In forward pass the input data is fed into the input layer. These inputs combined with their respective weights are passed to hidden layers. For example in a network with two hidden layers (h1 and h2) the output from h1 serves as the input to h2.

Before applying an activation function, a bias is added to the weighted inputs. Each hidden layer computes the weighted sum (`a`) of the inputs then applies an activation function like ReLU (Rectified Linear Unit) to obtain the output (`o`). The output is passed to the next layer where an activation function such as softmax converts the weighted outputs into probabilities for classification.

People Also Search

Neural Networks Look Magical From The Outside — You Feed

Neural networks look magical from the outside — you feed in data, and they somehow learn to recognize cats, translate languages, or even drive cars. But behind the scenes, the learning process is powered by one key algorithm: Backpropagation. In this blog, we’ll break down backpropagation step by step, understand why it works, and see how it teaches neural networks to get better over time. Backpro...

Let’s Go Through The Full Training Cycle. An Interactive, Colab-ready

Let’s go through the full training cycle. An interactive, Colab-ready notebook that builds a tiny autograd engine (like micrograd) from scratch, spells out backpropagation step by step, and makes gradients visible. 🚀 Designed as a teaching + portfolio project to demystify backprop and showcase engineering clarity. A neural network consists of a set of parameters - the weights and biases - which d...

In This Post, We Discuss How Backpropagation Works, And Explain

In this post, we discuss how backpropagation works, and explain it in detail for three simple examples. The first two examples will contain all the calculations, for the last one we will only illustrate the equations that need to be calculated. We will not go into the general formulation of the backpropagation algorithm but will give some further readings at the end. This post is quite long becaus...

The First Thing We Need, When We Want To Train

The first thing we need, when we want to train a neural net is the training data. The training data consists of pairs of inputs and labels. The inputs are also called features and are usually written as $X = (x_1, \dots, x_n)$, with $n$ the number of data samples. The labels are the expected outcomes - or true values - and they are usually denoted as $y = (y_1, \dots, y_n)$. Training a neural net ...

Then The Error - Loss - Of Model Output Compared

Then the error - loss - of model output compared to the true values is calculated to evaluate the model. Finally, in the backward pass - the backpropagation - Gradient Descent is used to update the model parameters and reduce the loss. Note, that in practice, generally no pure gradient descent is used, but a variant of it. We are not going into detail here, but important to understand is that some...