Deep Feed Forward Neural Networks and the Advantage of ReLU Activation Function
How to build a Deep Feed Forward (DFF) Neural Network in Python using Tensorflow Keras API and how to choose between different activation...
Neural Networks
How to build a Deep Feed Forward (DFF) Neural Network in Python using Tensorflow Keras API and how to choose between different activation functions

Intro
We, as Data Scientists, always get excited about any performance gains we can achieve in our models. Chasing these improvements drives the community to keep experimenting, often leading to breakthroughs.
Many of the earlier experiments focused on the depth of the networks and how we could make them more efficient and get them to produce more accurate results.
In this article, I will take you through the structure of Deep Feed Forward (DFF) Neural Networks. Also, I will take a closer look at ReLU (Rectified Linear Unit) activation function and show you how to build a DFF Neural Network in Python using Tensorflow and Keras libraries.
This article is a continuation of my previous one on Feed Forward Neural Networks. If you are not familiar with the Neural Network basics, you may want to read this first:
Feed Forward Neural Networks - How To Successfully Build Them in Python
Contents
Deep Feed Forward Neural Network's place within the universe of Machine Learning
The difference between Feed Forward (FF) and Deep Feed Forward (DFF) Neural Networks
What is the purpose of activation functions, and why has ReLU become a default for Deep Neural Networks?
Python example of how to build and train your own DFF Neural Network
DFF Neural Networks within the universe of Machine Learning
While Neural Networks are most frequently used in a supervised manner with labeled training data, I felt that their unique approach to Machine Learning deserves a separate category.
Within the Neural Network branch, DFF comes under the subcategory of Feed Forward Neural Networks, which are also often called Multilayer Perceptrons (MLPs).
The below graph is interactive, so please click on different categories to enlarge and reveal more👇 .
If you enjoy Data Science and Machine Learning, please subscribe to get an email whenever I publish a new story.
The difference between Feed Forward (FF) and Deep Feed Forward (DFF) Neural Networks
Structure
The structure of a DFF is very similar to that of an FF. The major difference between them is the number of hidden layers.
Currently, people refer to a Neural Network with one hidden layer as a "shallow" network or simply a Feed-Forward network. Meanwhile, a Neural Network with multiple hidden layers (2+) is called a Deep network.
If we were to draw a diagram of structural comparison between FF and DFF, it would look like this:

What is the point of depth?
Typically you will find that deep NNs perform better than shallow ones. However, it is not always necessary to use a deep network. The choice will largely depend on the task you have at hand.
If you are working with many inputs, such as image data, then using a Deep Feed Forward (DFF) or a Convolutional Neural Network (CNN) would likely yield better results than a simple Feed Forward network.
However, suppose your task is to do some basic classification with a limited number of inputs. In that case, you may be better off using a simple FF network or even a tree-based algorithm such as XGBoost, Random Forest, or a single Decision tree.
So, going back to the point of depth, the simple answer is that deeper networks tend to deliver better performance on more complex tasks. There are multiple hypotheses on why deep networks perform better, ranging from efficiency to improved ability to learn more abstract representations.
Here are some hypotheses described in this detailed answer on Stackexchange:
A shallow network may require more neurons than the deep one.
A shallow network may be more difficult to train with our current algorithms (e.g. it may have a more nasty local minima, or the convergence rate might be slower)
Perhaps a shallow architecture does not fit to the kind of problems we are usually trying to solve (e.g. object recognition is a quintessential "deep", hierarchical process)?
Maybe there is a different reason not mentioned above.
Do not hesitate to share if you come across definitive proof of why deeper is better.😃
What is the purpose of activation functions, and why did ReLU become a go-to for Deep Neural Networks?
Challenges presented by deep networks
A Neural Network architecture contains activation functions inside the hidden nodes and output nodes. In short, the activation function takes the input value entering the node, performs a transformation, and then passes the result onwards to the next set of neurons.
Here is a simple illustration of a Feed Forward Neural Network that shows what activation functions look like (in this case, softplus and sigmoid) and how they transform data within the Neural Network.

Traditionally, the two most widely used nonlinear activation functions were the sigmoid and the hyperbolic tangent (tanh). However, using these activation functions with Deep Neural Networks presented a problem of vanishing gradient.
The error is backpropagated through the network during the training process and is used to update the weights. Unfortunately, sigmoid and tanh activation functions tend to saturate.
It means that large negative and large positive values are transformed to 0 and 1 by sigmoid and -1 and 1 by tanh. The saturation often happens regardless of whether the inputs provided to the node contain useful information or not.
As the functions saturate, the derivate becomes close to zero. Hence, there is essentially no gradient left to propagate back through the network, making it challenging for the learning algorithm to continue adapting the weights.
Here is an illustration of the three activation functions:

Rectified Linear Unit (ReLU) activation function
ReLU has been introduced as a solution to a vanishing gradient problem and quickly became a default option for most Deep Feed Forward (DFF) and Convolutional Neural Networks (CNN).
It has a very simple function that sets all negative values to 0 while returning the same value for all positive inputs.
ReLU is, of course, a nonlinear function. However, it is very close to being linear, enabling it to preserve many of the properties that make linear models easy to optimize with gradient-based methods. At the same time, it tends to generalize well too.
Tensorflow implementation of ReLU allows you to set a few parameters to tune ReLU to your liking. E.g.:
You can set the saturation threshold by specifying max_value or;
You can turn ReLU into a Leaky ReLU by setting an alpha parameter, where alpha governs the slope for values lower than 0 (or another chosen threshold).
Here is an illustration of ReLU with max_value set to 3 and Leaky ReLU with alpha at 0.01.

You may choose to use ReLU with a max cap if you run into a problem of exploding gradient while training your model. As you may have guessed, the exploding gradient is the opposite problem to the vanishing gradient, and it results from having large weights.
Although, typically, you can avoid the issue of exploding gradient if you use He weight initialization with a standard ReLU. He initialization (see: HeNormal, HeUniform) ensures that the initial weights are small enough to minimize the risk of exploding gradient.
Meanwhile, a Leaky ReLU can be beneficial when you want to avoid having "dead" neurons, which result from negative inputs being set to 0 by a standard ReLU.

Python example of how to build and train your own DFF Neural Network
It's now time to have some fun and develop our own Deep Neural Network capable of recognizing MNIST digits.
Setup
We'll need the following data and libraries:
MNIST handwritten digit data (copyright held by Yann LeCun and Corinna Cortes under the Creative Commons Attribution-Share Alike 3.0 license; the original source of the data: The MNIST Database)
Matplotlib for displaying handwritten digits
Tensorflow/Keras for Neural Networks
Scikit-learn library for some basic model evaluation
Let's import all the libraries:
The above code prints package versions used in this example:
Next, we ingest MNIST handwritten digits data and display the first ten digits with their true labels above the images.
This is what we get running the above code:

As you can see, we have 60,000 images in the training set and 10,000 in the test set. Note that their dimensions are 28 x 28 pixels. However, we need to reshape the data before using it to train the DFF Neural Network.
The new shapes:
Training and Evaluating a Deep Feed Forward Neural Network
I have provided extensive commentary within the code, so I will not repeat the same in the body of the article. However, there are a few things that I would like to highlight:
Softmax activation in the output layer - this function takes values from all ten nodes and converts them to values between 0 and 1, where the sum of those ten values equals 1. Hence, you can imagine the output as a sort of "probability." Since each output node represents a different digit between 0–9, the one with the highest value ("probability") is what the Neural Net believes to be the correct answer.
Use of a "SparseCategoricalCrossentropy" for loss function - when your target data is binary, you should use "BinaryCrossentropy." When you have two or more label classes, and your data is OneHot encoded, you should use "CategoricalCrossentropy." In our scenario, we had ten classes to predict, but our target data was not OneHot encoded. Hence, we needed to use "SparseCategoricalCrossentropy."
Predicting labels - since the prediction gives us ten values (one for each output node), we need to pass the result through the argmax function that takes it as an input and returns the digit with the highest value ("probability").
Here is a summary and evaluation metrics for our Deep Feed Forward Neural Network, which we trained over five epochs.

While 97% accuracy on test data is pretty decent, you could improve the results by training over more epochs or experimenting with the network structure. So give it a go and let me know if you managed to beat my score!
Conclusions
Congratulations! You can now successfully create Deep Feed Forward Neural Networks, experiment with the network structure, choose an activation function and use your network to make predictions.
Feel free to use the code provided in this article for your own projects. You can also download the entire Jupyter notebook from my GitHub repository.
As I try to make my articles more useful for readers, I would appreciate it if you could let me know what has driven you to read this piece and whether it has given you the answers you were looking for. If not, what was missing?
Cheers! 👏 Saul Dobilas
Feed Forward Neural Networks - How To Successfully Build Them in Python
How to Benefit from the Semi-Supervised Learning with Label Spreading Algorithm
SVM Classifier and RBF Kernel - How to Make Better Models in Python








