Data Science

Random Forest Models: Why Are They Better Than Single Decision Trees?

A detailed explanation of how random forest machine learning algorithm works, what makes it superior to decision trees, and how you can...

Saul Dobilas
February 7, 20219 min read

Machine Learning

A detailed explanation of how random forest machine learning algorithm works, what makes it superior to decision trees, and how you can build them in Python

Random Forest. Image by author.
Random Forest. Image by author.

Intro

If you want to be a successful Data Scientist, you need to understand the nuances of different Machine Learning algorithms.

This story is part of a series where I provide an in-depth look at how such algorithms work. This includes simple examples, 3D visualizations, and complete Python code for you to use in your Data Science projects.

The story covers the following topics:

  • The category of algorithms Random Forest classification belongs to

  • An explanation of how Random Forest classification works and why it is better than a single decision tree

  • Python code examples and visualizations

What category of algorithms does it belong to?

Similar to some other algorithms, Random Forest can handle both classification and regression. While in this story, I focus on classification, the same logic largely applies to regression too.

A quick recap on the difference between classification and regression :

  • With classification, we attempt to predict a class label. In other words, it is used for problems where the output (target variable) takes a finite set of values, e.g., whether it will rain tomorrow or not. Corresponding algorithm: [sklearn.ensemble.RandomForestClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn-tree-decisiontreeclassifier)

  • Meanwhile, regression is used to predict a numerical label. This means your output can take an infinite set of values, e.g., a house price. Corresponding algorithm:[sklearn.ensemble.RandomForestRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html)

Both cases fall under the supervised branch of machine learning algorithms. The below graph is interactive, so make sure to click on different categories to enlarge and reveal more👇 .

If you enjoy Data Science and Machine Learning, please subscribe to get an email whenever I publish a new story.

How does the Random Forest classification algorithm work?

Under the hood, the random forest is essentially a CART algorithm (Classification and Regression Trees), except it creates an ensemble of many trees instead of just one. This provides several advantages such as:

  • Improved performance (the wisdom of crowds)

  • Improved robustness (less likely to overfit since it relies on many random trees)

If you are not familiar with CART, you can find everything about it in my previous story:

CART: Classification and Regression Trees for Clean but Powerful Models

Wisdom of crowds - what is it and why is it good?

Random Forest algorithm uses majority agreement prediction for the class label, which means that each tree predicts whether the observation belongs to 'Class 0' or 'Class 1'. If 55 trees out of a hundred predicted 'Class 1' and 45 predicted 'Class 0', then the final model prediction would be 'Class 1'.

Multiple Decision Trees. Image by author.
Multiple Decision Trees. Image by author.

The majority prediction from multiple trees is better than an individual tree prediction because the trees protect each other from their individual errors. This is, however, dependant on the trees being relatively uncorrelated with each other.

The process of building multiple trees

The question is - how do we build many trees from the same data pool while keeping them relatively uncorrelated? There are two parts to it:

  1. Bootstrap aggregation (random sampling with replacement)

  2. Feature randomness

Bootstrap aggregation

Decision trees are so-called high-variance estimators, which means that small changes to the sample data can greatly impact the tree structure and its prediction.

Ideally, you want to turn it into a low-variance estimator by creating many trees and using them in aggregation to make the prediction.

Bootstrapping is a technique where instead of taking the whole training data to build one tree, you use sampling with replacement to create many samples of the same size. Then you can use these samples to build separate trees. For example:

text
Whole data (10 observations): [1,2,2,2,3,3,4,5,6,7]
text
Bootstrap sample 1 (10 obs): [1,1,2,2,3,4,5,6,7,7]Bootstrap sample 2 (10 obs): [1,2,2,2,2,3,3,3,3,6]...Bootstrap sample n (10 obs): [2,2,2,3,4,5,6,6,7,7]
text
Note, how each of your samples consist of 10 observations.

In this case, you randomly pick one number from the whole sample and record what you have picked. Then you repeat the action until you have recorded 10 observations.

An equivalent would be randomly picking a ball from a bucket of red and blue balls, recording its color, and then returning it to the bucket before repeating the same action.

This method produces many samples with the same observations but different distributions. In the end, the aggregation of these trees leads to a reduction in the variance of your model prediction, turning it into a low-variance estimator.

Feature randomness

The second part in reducing the correlation between the trees comes from using a random selection of features when looking for the best split for each node.

For example:

text
Full list of features: [feat1, feat2, ..., feat10]
text
Random selection of features (1): [feat3, feat5, feat8]Random selection of features (2): [feat1, feat5, feat10]...
text
The split in the first node would use the most predcitive feature from a set of [feat3, feat5, feat8]The splint in the second node would use the most predictive feature from a set of [feat1, feat5, feat10]...
text
Note, when building a Random Forest model, you can control the number/proportion of features that are used to find the best split for each node.

This process allows the creation of less correlated trees and, combined with bootstrap aggregation, produces a Random Forest.

Python examples of Random Forest classification models

Leaving theory behind, let us build a Random Forest model in Python.

Setup

We will use the following data and libraries:

Let's import all the libraries:

python
import pandas as pd # for data manipulationimport numpy as np # for data manipulationfrom sklearn.model_selection import train_test_split # for splitting the data into train and test samplesfrom sklearn.metrics import classification_report # for model evaluation metricsfrom sklearn import tree # for decision tree modelsfrom sklearn.ensemble import RandomForestClassifierimport plotly.express as px  # for data visualizationimport plotly.graph_objects as go # for data visualizationimport graphviz # for plotting decision tree graphs

Then we get the Australian weather data from Kaggle, which you can download following this link: https://www.kaggle.com/jsphyg/weather-dataset-rattle-package.

We ingest the data and derive a few new variables for usage in the models.

python
# Set Pandas options to display more columnspd.options.display.max_columns=50# Read in the weather data csvdf=pd.read_csv('weatherAUS.csv', encoding='utf-8')# Drop records where target RainTomorrow=NaNdf=df[pd.isnull(df['RainTomorrow'])==False]# For other columns with missing values, fill them in with column meandf=df.fillna(df.mean())# Create a flag for RainToday and RainTomorrow, note RainTomorrowFlag will be our target variabledf['RainTodayFlag']=df['RainToday'].apply(lambda x: 1 if x=='Yes' else 0)df['RainTomorrowFlag']=df['RainTomorrow'].apply(lambda x: 1 if x=='Yes' else 0)# Show a snaphsot of datadf
A snippet of Kaggle's Australian weather data with some modifications. Image by author.
A snippet of Kaggle's Australian weather data with some modifications. Image by author.

Next, let's build a model by following these steps:

  • Step 1 - select model features (independent variables) and model target (dependent variable)

  • Step 2 - split data into train and test samples

  • Step 3 - set model parameters and train (fit) the model

  • Step 4 - predict class labels on train and test data using our model

  • Step 5 - generate model summary statistics

python
##### Step 1 - Select data for modelingX=df[['MinTemp', 'MaxTemp', 'Rainfall', 'Evaporation', 'Sunshine', 'WindGustSpeed',       'WindSpeed9am', 'WindSpeed3pm', 'Humidity9am', 'Humidity3pm', 'Pressure9am',        'Pressure3pm', 'Cloud9am', 'Cloud3pm', 'Temp9am', 'Temp3pm', 'RainTodayFlag']]y=df['RainTomorrowFlag'].values##### Step 2 - Create training and testing samplesX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)##### Step 3# Set model and its parametersmodel = RandomForestClassifier(criterion='gini',                                bootstrap=True, # enabling bootstrapping                               random_state=0, # random state for reproducibility                               max_features='sqrt', # number of random features to use sqrt(n_features)                               min_samples_leaf=1000, # minimum no of observarions allowed in a leaf                               max_depth=4, # maximum depth of the tree                               n_estimators=1000 # how many trees to build                              )# Fit the modelclf = model.fit(X_train, y_train)##### Step 4# Predict class labels on training datapred_labels_tr = model.predict(X_train)# Predict class labels on a test datapred_labels_te = model.predict(X_test)##### Step 5 - Model summary# Basic info about the modelprint('*************** Model Summary ***************')print('Classes: ', clf.classes_)print('No. of outputs: ', clf.n_outputs_)print('No. of features: ', clf.n_features_in_)print('No. of Estimators: ', len(clf.estimators_))print('--------------------------------------------------------')print("")print('*************** Evaluation on Test Data ***************')score_te = model.score(X_test, y_test)print('Accuracy Score: ', score_te)# Look at classification report to evaluate the modelprint(classification_report(y_test, pred_labels_te))print('--------------------------------------------------------')print("")print('*************** Evaluation on Training Data ***************')score_tr = model.score(X_train, y_train)print('Accuracy Score: ', score_tr)# Look at classification report to evaluate the modelprint(classification_report(y_train, pred_labels_tr))print('--------------------------------------------------------')

The above code generates the following output that summarizes the model performance.

Random Forest model performance. Image by author.
Random Forest model performance. Image by author.

As we can see, the model generalized quite well with similar performance on test data when compared to training data. The number of estimators was 1000, which means that our Random Forest consists of 1000 individual trees. Here is what a tree 552 looks like. If you wish, you can generate tree diagrams for each one of them by changing the index.

python
# Use graphviz to plot the tree graphdot_data = tree.export_graphviz(clf.estimators_[552], out_file=None,                             feature_names=X.columns,                             class_names=[str(list(clf.classes_)[0]), str(list(clf.classes_)[1])],                            filled=True,                             rounded=True,                             rotate=False,                           ) graph = graphviz.Source(dot_data)graph
Decision Tree Graph. Image by author.
Decision Tree Graph. Image by author.

Bonus code

While we cannot easily visualize all model predictions using 17 features, we can do it when we build a random forest with just 2 features. For this, we first need to build another model using the code above, remembering to select only 2 features:

text
X=df[['WindGustSpeed', 'Humidity3pm']]

After building a model, we use the following code to create a 3D visualization with Plotly:

python
##### Define a function to generate a 3D graphdef Plot_3D(X, X_test, y_test, clf, x1, x2, mesh_size, margin):    # Specify a size of the mesh to be used    mesh_size=mesh_size    margin=margin    # Create a mesh grid on which we will run our model    x_min, x_max = X.iloc[:, 0].fillna(X.mean()).min() - margin, X.iloc[:, 0].fillna(X.mean()).max() + margin    y_min, y_max = X.iloc[:, 1].fillna(X.mean()).min() - margin, X.iloc[:, 1].fillna(X.mean()).max() + margin    xrange = np.arange(x_min, x_max, mesh_size)    yrange = np.arange(y_min, y_max, mesh_size)    xx, yy = np.meshgrid(xrange, yrange)    # Calculate predictions on grid    Z = clf.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]    Z = Z.reshape(xx.shape)    # Create a 3D scatter plot with predictions    fig = px.scatter_3d(x=X_test[x1], y=X_test[x2], z=y_test, opacity=0.8, color_discrete_sequence=['black'])    # Set figure title and colors    fig.update_layout(#title_text="Scatter 3D Plot with Random Forest Prediction Surface",                      paper_bgcolor = 'white',                      scene = dict(xaxis=dict(title=x1,                                              backgroundcolor='white',                                              color='black',                                              gridcolor='#f0f0f0'),                                   yaxis=dict(title=x2,                                              backgroundcolor='white',                                              color='black',                                              gridcolor='#f0f0f0'                                              ),                                   zaxis=dict(title='Probability of Rain Tomorrow',                                              backgroundcolor='lightgrey',                                              color='black',                                               gridcolor='#f0f0f0',                                               )))    # Update marker size    fig.update_traces(marker=dict(size=1))    # Add prediction plane    fig.add_traces(go.Surface(x=xrange, y=yrange, z=Z, name='Random Forest Prediction',                              ccolorscale='Bluered',                              reversescale=True,                              showscale=False,                               contours = {"z": {"show": True, "start": 0.5, "end": 0.9, "size": 0.5}}))    fig.show()    return fig##### Call the above functionfig = Plot_3D(X, X_test, y_test, clf, x1='WindGustSpeed', x2='Humidity3pm', mesh_size=1, margin=1)

The above generates the following graph.

Random Forest model prediction surface. Image by author.
Random Forest model prediction surface. Image by author.

From the above visualization, we can clearly see that chances of rain tomorrow increase as the humidity at 3 pm and wind gust speed increases. Also, while not completely smooth, the decision surface has fewer big step changes compared to using just one decision tree (see graph below).

Single Decision Tree model prediction surface. Image by author.
Single Decision Tree model prediction surface. Image by author.

Conclusion

Random Forest is amongst the best performing Machine Learning algorithms, which has seen wide adoption. While it is a bit harder to interpret than a single Decision Tree model, it brings many advantages, such as improved performance and better generalization.

Thanks for reading, and feel free to use the above code and materials in your own Data Science projects.

Cheers! 👏 Saul Dobilas


Also, you may be interested to see how Random Forest compares to AdaBoost:

AdaBoost Machine Learning Algorithm: How to Improve Performance of Hard to Predict Cases

Other Boosting Algorithms:

Gradient Boosted Trees for Classification - One of the Best Machine Learning Algorithms

XGBoost: Extreme Gradient Boosting - How to Improve on Regular Gradient Boosting?

Related Articles