Unraveling Backpropagation Calculus

Introduction:

I have published my first book, "What Everone Should Know about the Rise of AI" is live now on google play books at Google Play Books and Audio, check back with us at https://theapibook.com for the print versions, go to Barnes and Noble at Barnes and Noble Print Books!

Ever wondered how calculus plays a vital role in deep learning's backpropagation process? I love math and because of my love for math, AI/ML math and data sciences at its core is made up of math and statistics.  In this article, we will discuss sensitivity analysis in neural networks.  Don't let "calculus" scare you if you don't like math, because I am about to share with you practical and easy to follow demonstrations of the role calculus plays in all of this.

The Cost Function in a Simple Network

In a nutshell, the cost of a simple network for a single training example boils down to the squared difference between the network's output (a^(L)) and the desired output (y). This video peels back the layers to reveal how the last activation, determined by a weight (w^(L)) and bias (b^(L)), influences this cost function.

Imagine you're trying to teach a computer program to recognize handwritten numbers. You feed it images of handwritten digits along with the correct labels (e.g., the image of a handwritten "3" is labeled as "3"). The computer program, also known as a neural network, tries to learn from these examples so that it can correctly identify the digits in new images it hasn't seen before.

Now, let's focus on just one example: you show the neural network an image of a handwritten "3," and you want it to correctly recognize it as a "3." After processing the image, the neural network produces an output, let's call it  ( )a 

(L)

 , which represents its guess for what the digit is.

However, the neural network might not always get it right. The difference between what the network guessed ( ( )a 

(L)

 ) and what it should have guessed (the actual label  y) tells us how far off the network's prediction was. This difference is the basis for evaluating how well the network is performing for this particular example.

To quantify this error, we use something called a cost function. In the case of a simple network, the cost function measures the squared difference between the network's output and the desired output. So, for our handwritten "3" example, the cost function would calculate how much  ( )a 

(L)

  differs from "3" and square that difference.

But how does the network's output ( ( )a 

(L)

 ) relate to the cost function? Well, the output of a neural network is influenced by weights ( ( )w 

(L)

 ) and biases ( ( )b 

(L)

 ), among other things. These weights and biases determine how the input data is transformed as it passes through the network, ultimately affecting the final output. So, by adjusting the weights and biases, we can minimize the difference between the network's output and the desired output, thus reducing the cost function and improving the network's performance.

In simpler terms, think of it like this: if the network initially guesses that the handwritten "3" is actually a "5," the cost function would give a higher score because the network's guess is further from the correct answer. By tweaking the weights and biases in the network, we aim to make it more accurate so that it consistently produces the correct outputs for a wide range of inputs.

Unpacking Sensitivity to Weight Changes

Delving deeper, understanding how the cost function responds to changes in weight w^(L) is critical. Visualize this sensitivity as the ratio of a tiny change in the inner z^(L) to the corresponding tiny change in weight w^(L). Here, the derivative of z^(L) with respect to w^(L) takes the spotlight.

The Chain Rule and Relevant Derivatives

The chain rule's role in determining the sensitivity of the cost function to small changes in weight w^(L) cannot be overstated. The derivative of C with respect to a^(L) is intricately related to the network's output vs. the desired output, emphasizing the ripple effect a weight carries in the network.

Impacts of Neurons and Weight Interplay

Delve into the interconnectedness of neurons and weights: 'neurons that fire together wire together.' The influence of a weight in the final layer is intertwined with the strength of the preceding neuron, showcasing the network's adaptability and how it refines its predictions through intricate connections.

Grasping the Cost Function's Sensitivity

While we can't directly alter the previous activation, the chain rule's expansion enlightens us on the sensitivity of the cost function to prior weights and biases. Calculating the squared differences between last layer activations and the desired output unveils the meticulous balancing act within the network.

Conclusion:

Unraveling backpropagation calculus showcases the intricate dance of derivatives, weights, and activations in neural networks. Understanding this process not only enhances our grasp of machine learning but also unveils the beauty of calculus driving deep learning forward.

Check out this great video on this topic for visual overview:

by the https://www.youtube.com/@3blue1brown Youtube Channel!