Data Science

LOWESS Regression in Python: How to Discover Clear Patterns in Your Data?

A detailed guide to using Locally Weighted Scatterplot Smoothing (LOWESS) algorithm in Python

Saul Dobilas
December 6, 202011 min read

Machine Learning

LOWESS algorithm finding the trend. Image by author.
LOWESS algorithm finding the trend. Image by author.

Intro

Machine Learning is making huge leaps forward, with an increasing number of algorithms enabling us to solve complex real-world problems.

This story is part of a deep dive series explaining the mechanics of Machine Learning algorithms. In addition to giving you an understanding of how ML algorithms work, it also provides you with Python examples to build your own ML models.

This story covers the following topics:

  • What category of algorithms does LOWESS belong to?

  • How does Locally Weighted Scatterplot Smoothing work?

  • How can I use LOWESS to identify patterns and predict new data in Python?

What category of algorithms does LOWESS belong to?

Locally Weighted Scatterplot Smoothing sits within the family of regression algorithms under the umbrella of Supervised Learning. This means that you need a set of labeled data with a numerical target variable to train your model.

The below graph is interactive, so make sure to click on different categories to enlarge and reveal more👇 .

If you enjoy Data Science and Machine Learning, please subscribe to get an email whenever I publish a new story.

Since LOWESS is a non-parametric fitting technique, you do not need to assume that data follows any specific distribution. However, at the same time, non-parametric fitting means that at the end, you will not have a global equation for you to use to predict the values of new data points. Not to worry, though, as I provide a workaround to this issue in the Python section later in this story.

Given the above advantages and disadvantages, LOWESS is often used to perform the following analysis:

  • A regression analysis where simple linear regression cannot produce a line of good fit due to data attributes not following a linear relationship. Note, a special case of multiple linear regression known as polynomial regression could also be used in this case.

  • Data exploration and analysis in the social sciences, such as elections and voting behavior.

  • Fitting a line to a scatter plot or time plot where noisy data values, sparse data points, or weak interrelationships interfere with your ability to see a line of best fit.

How does the Locally Weighted Scatterplot Smoothing algorithm work?

While writing this story, I have assumed that you are already familiar with the ordinary least squares (OLS) regression. Hence, in this section, I only intend to provide an intuitive explanation of how LOWESS splits up the data to perform linear regression on local sections of the data.

If you would like a refresher on OLS regression, feel free to take a look at my linear regression story:

Linear regression made easy. How does it work and how to use it in Python?

Said that one important addition to LOWESS over OLS is that it applies weightings as you might have guessed from the algorithm's name - Locally Weighted Scatterplot Smoothing.

Typically, the algorithm uses a tri-cube weight function (see below), although other functions can also be used.

text
Weight function
text
w(x) = (1 - |d|³)³, 
text
where d is is the distance of a given data point from the point on the curve being fitted, scaled to lie in the range between 0 and 1

The weight function gives the most weight to the data points nearest to the point of estimation and the least weight to the data points that are furthest away. It is based on the idea that points near each other in the explanatory variable space are more likely to be related to each other in a simple way than points that are further apart.

Intuitive explanation

We will use an example to illustrate how LOWESS works. Let us start by creating a scatterplot where the data points follow a sine wave pattern, but they have some random noise added, making the pattern less obvious.

After that, we perform a LOWESS regression analysis a couple of times using different hyperparameters and add LOWESS curves to the plot:

LOWESS approximation of the sine wave. Image by author.
LOWESS approximation of the sine wave. Image by author.

As you can see, there is a lot of noise in the data, with the relationship between x and y being non-linear (periodic, in fact). Clearly, simple linear regression would not give us a meaningful result here since it can only produce a straight line (unlike LOWESS or polynomial regression).

The first thing that LOWESS does is selects a subset of the data. We can control the size of the subsets by specifying the 'fraction' hyperparameter. The two lines that we plotted above have hyperparameter values of 2/3 and 1/5.

Let's take the orange line as an example, with a default hyperparameter value of 2/3. This means that the LOWESS algorithm takes a subset of two-thirds of the entire data and performs a weighted linear least squares regression over the span of that data while moving by one point at a time and repeating the same process until it has gone through every single point.

Starting from the very left and going right one point at a time, we can see how the orange line remains straight until x=-2.1. This is because the entire data ranges from -2π to +2π (-6.28 to +6.28) with an initial two-thirds window ranging from -6.28 to +2.1. Hence, only when the algorithm goes beyond -2.1 (midpoint between -6.28 and +2.1), it starts encountering new data points within its window, changing the curve's slope.

The two-thirds window shifting along as LOWESS moves across the data performing regression. Image by author.
The two-thirds window shifting along as LOWESS moves across the data performing regression. Image by author.

If we now look at the red line (frac=1/5), we can see how narrowing the window to one-fifth of the data makes the algorithm more sensitive towards local trends. The featured gif image at the beginning of this story illustrates how this red line (frac=1/5) moves across the data performing weighted linear regression. This is the same process as described for the orange line (frac=2/3) except, the window the algorithm sees is narrower.

While in this example, making the window smaller helped us get a better approximation of the sine wave, it is not always desirable to make a 'fraction' hyperparameter smaller. This largely depends on the data you are analyzing, as making a window smaller runs a risk of overfitting. You will see this in my next example, where 2/3 works much better than 1/5.

How can I use LOWESS to identify patterns and predict new data in Python?

Now that you have a general understating of how the algorithm works, it is time to have some fun and build a model in Python

We will use the following:

  • House price data from Kaggle

  • Scikit-learn library to build a simple linear regression model (so we can compare the result to LOWESS)

  • statsmodels library for LOWESS algorithm

  • Plotly library for visualizations

  • scipy library for data interpolation

  • Pandas and Numpy

Setup

Let us start by importing the required libraries.

python
import pandas as pd # for data manipulationimport numpy as np # for data manipulationfrom sklearn.linear_model import LinearRegression # to build a LR model for comparisonimport plotly.graph_objects as go # for data visualizationimport plotly.express as px # for data visualization import statsmodels.api as sm # to build a LOWESS modelfrom scipy.interpolate import interp1d # for interpolation of new data points

Next, we download and ingest the data that we will use to build our LOWESS and linear regression models. (source: https://www.kaggle.com/quantbruce/real-estate-price-prediction?select=Real+estate.csv)

python
# Read in datadf = pd.read_csv('Real estate.csv', encoding='utf-8')# Print Dataframedf
House price data from Kaggle. Image by author.
House price data from Kaggle. Image by author.

LOWESS vs. Linear Regression

We will take 'X3 distance to the nearest MRT station' as our input (independent) variable and 'Y house price of unit area' as our output (dependent) variable and create a scatterplot to visualize the data.

python
# Create a scatter plotfig = px.scatter(df, x=df['X3 distance to the nearest MRT station'], y=df['Y house price of unit area'],                  opacity=0.8, color_discrete_sequence=['black'])# Change chart background colorfig.update_layout(dict(plot_bgcolor = 'white'))# Update axes linesfig.update_xaxes(showgrid=True, gridwidth=1, gridcolor='lightgrey',                  zeroline=True, zerolinewidth=1, zerolinecolor='lightgrey',                  showline=True, linewidth=1, linecolor='black')fig.update_yaxes(showgrid=True, gridwidth=1, gridcolor='lightgrey',                  zeroline=True, zerolinewidth=1, zerolinecolor='lightgrey',                  showline=True, linewidth=1, linecolor='black')# Set figure titlefig.update_layout(title=dict(text="House Price Based on Distance from the Nearest MRT",                              font=dict(color='black')))# Update marker sizefig.update_traces(marker=dict(size=3))fig.show()
Scatterplot of X and Y. Image by author.
Scatterplot of X and Y. Image by author.

We can clearly see the relationship between x and y with the house price (per unit area) decreasing as the distance from the nearest MRT station increases.

Since the relationship is somewhat linear, the question is whether LOWESS can give us a better approximation than simple linear regression. Let us find out by fitting the two models.

python
# ------- Select variables -------# x values for Linear RegressionX=df['X3 distance to the nearest MRT station'].values.reshape(-1,1) # Note, we need X to be a 2D array, hence reshape# x values for LOWESSx=df['X3 distance to the nearest MRT station'].values # y values for bothy=df['Y house price of unit area'].values# ------- Linear Regression -------# Define and fit the modelmodel1 = LinearRegression()LR = model1.fit(X, y)# Predict a few points with Linear Regression model for the grpah# Create 20 evenly spaced points from smallest X to largest Xx_range = np.linspace(X.min(), X.max(), 20) # Predict y values for our set of X valuesy_range = model1.predict(x_range.reshape(-1, 1))# ------- LOWESS -------# Define a modellowess = sm.nonparametric.lowess# Generate y_hat values using lowess, try a couple values for hyperparametersy_hat1 = lowess(y, x) # note, default frac=2/3y_hat2 = lowess(y, x, frac=1/5)

With fitting done, let us draw simple linear regression and LOWESS lines on the graph.

python
# Create a scatter plotfig = px.scatter(df, x=df['X3 distance to the nearest MRT station'], y=df['Y house price of unit area'],                  opacity=0.8, color_discrete_sequence=['black'])# Add the prediction linefig.add_traces(go.Scatter(x=x_range, y=y_range, name='Linear Regression', line=dict(color='limegreen')))fig.add_traces(go.Scatter(x=y_hat1[:,0], y=y_hat1[:,1], name='LOWESS, frac=2/3', line=dict(color='red')))fig.add_traces(go.Scatter(x=y_hat2[:,0], y=y_hat2[:,1], name='LOWESS, frac=1/5', line=dict(color='orange')))# Change chart background colorfig.update_layout(dict(plot_bgcolor = 'white'))# Update axes linesfig.update_xaxes(showgrid=True, gridwidth=1, gridcolor='lightgrey',                  zeroline=True, zerolinewidth=1, zerolinecolor='lightgrey',                  showline=True, linewidth=1, linecolor='black')fig.update_yaxes(showgrid=True, gridwidth=1, gridcolor='lightgrey',                  zeroline=True, zerolinewidth=1, zerolinecolor='lightgrey',                  showline=True, linewidth=1, linecolor='black')# Set figure titlefig.update_layout(title=dict(text="House Price Based on Distance from the Nearest MRT with Model Predictions",                              font=dict(color='black')))# Update marker sizefig.update_traces(marker=dict(size=3))fig.show()
Linear regression and LOWESS model comparison. Image by author.
Linear regression and LOWESS model comparison. Image by author.

Visually we can see that LOWESS can fit the data better. This is because the price drop is steeper initially, which then starts to flatten out as the distance from the nearest MRT goes beyond 1000.

Simple linear regression has only one slope parameter meaning that it has the same steepness of the curve throughout. Meanwhile, LOWESS can adjust the curve's steepness at various points, producing a better fit than that of simple linear regression.

Let us now zoom in on the graph to see the difference between the two LOWESS models.

LOWESS model comparison (zoomed in). Image by author.
LOWESS model comparison (zoomed in). Image by author.

We can clearly see that setting the fraction hyperparameter to 1/5 makes LOWESS a bit too sensitive to the local data fluctuations, giving us a model that is overfitting. This is the opposite of the sine wave example, emphasizing that we need to be careful with hyperparameters and ensure that we choose the best values for the given data.

Predicting new values

Unfortunately, the lowess algorithm from statsmodels does not provide us with a predict() method. Hence, we cannot use the model that we have built to give us y values for the new set of x values (i.e., data points that were not in the original data).

Fortunately, we have a solution for that. We can use the interpolation method from the scipy library.

First, we create a function using the interp1d method by passing x and y values from the LOWESS output. Then, we use that newly created function to take new x values and generate y values for them.

Note, in the below code, we used a couple of different options for interpolation. Differences are explained in the code comments. One of them is drawing a line between the two nearest points and calculating a new y value for a given x value, and the other one simply taking the y value from the nearest available point.

At the same time, we have set fill_value to 'extrapolate,' which means that if you pass an x value outside of the original range (like 6400 in the example below), you will still get a result based on the extrapolated line. Without it, the algorithm would return 'NaN.' However, be careful with extrapolation as you may not always get good results for values far outside the original data range.

python
# ------- Define interploation functions -------# Linear - draws a line between the two nearest points and calculates y value based on the slope of that linef_linear = interp1d(y_hat1[:,0], y=y_hat1[:,1], bounds_error=False, kind='linear', fill_value='extrapolate') # Nearest - finds the nearest available point and takes its y valuef_nearest = interp1d(y_hat1[:,0], y=y_hat1[:,1], bounds_error=False, kind='nearest', fill_value='extrapolate') # Create a new set of points with x values xnew = [300, 600, 900, 1200, 1500, 1800, 2100, 6400]# Find y values based on the two different interpolation methodsynew_linear = f_linear(xnew)ynew_nearest = f_nearest(xnew)# Print resultsprint(ynew_linear)print(ynew_nearest)
text
Printed output of ynew_linear and ynew_nearest:
text
[45.17484583 38.88067785 33.63954152 30.70005122 28.90428712 27.31620311 26.02059902 11.5419846]
text
[45.02258129 38.86385487 33.43419447 31.09566559 28.91596696 27.30837281 26.0121316  11.55394747]

Conclusion

LOWESS is not something that you may want to use in all of your regression models as it follows a non-parametric approach and is quite computationally intensive. However, it is a good way to model a relationship between two variables that do not fit a predefined distribution and have a non-linear relationship.

I hope you enjoyed this story and can use the code and explanations for your own data science projects. If you have any feedback or questions, feel free to reach out.

Cheers! 👏 Saul Dobilas


Related stories you may like:

MARS: Multivariate Adaptive Regression Splines - How to Improve on Linear Regression?

Support Vector Regression (SVR) - One of the Most Flexible Yet Robust Prediction Algorithms

Related Articles