14: Root Mean Squared Error

Residual Plots

While we can plot the residual for each data point with a red dashed line, it's easy to imagine that all the red lines from all the data points could overlap and create a cluttered plot, making it difficult to make good observations of them.

Instead, we can draw residual plots which display all the residuals in a more organized way. A residual plot is a scatterplot of the predictor on the x-axis, and the residuals on the y-axis. The plot also contains a horizontal line at residual = 0 to represent a perfectly accurate prediction. Points above this line have positive residuals and points below this line have negative residuals.

First we find the predicted MPG, then we find the residuals.

# find predicted MPG
y_pred = reg_line_slope * cars['weight'] + reg_line_intercept
# find residuals
residuals = cars['mpg'] - y_pred

Output?

Plot weight vs MPG and weight vs residuals side-by-side.

plt.figure(figsize=(8, 4))
# plot weight vs mpg
plt.subplot(1, 2, 1)
plt.scatter(cars['weight'], cars['mpg'], alpha=0.6)
plt.plot(cars['weight'], y_pred, color='red', linewidth=2)
plt.xlabel('Weight')
plt.ylabel('MPG')
plt.title('Weight vs MPG')
plt.grid()

# plot weight vs residuals
plt.subplot(1, 2, 2)
plt.scatter(cars['weight'], residuals, color='green', alpha=0.6)
plt.axhline(0, color='red', linewidth=2)
plt.xlabel('Weight')
plt.ylabel('Residual')
plt.title('Weight vs Residuals')
plt.grid()

plt.tight_layout()
plt.show()

Output?

Every point in the original scatterplot has a corresponding point in the residual plot. The x-coordinate stays the same, while the y-coordinate changes from the actual MPG to the residual.

Looking at the Weight vs MPG plot, we observe that:

• At the two ends of the weight range, around 2000 lb and 5000 lb, many of the actual MPG values are above the regression line. This is reflected in the residual plot, with many residuals lying above the zero line.
• In the middle of the weight range, between about 2500 lb to 4000 lbs, many of the actual MPG values are below the regression line. This is reflected in the residual plot, with many of the residuals lying below the zero line.

When the residuals form a curved pattern instead of being randomly scattered around the zero line, it suggests that the relationship between the variables is curved rather than linear. A residual plot can help us detect when a linear regression model may not be a good fit for the data.

We also observed from the Weight vs MPG plot that:

• From about 4000 lb to 5000 lb, all the data points tend to be closer to the regression line. This is reflected in the residual plot, where the data are clustered more closely for the same weight range.
• Below 4000 lb quite a few data points are farther away from the linear regression line. This is reflected in the residual plot, where the data points are spread out more.

Residual plots can be used to detect uneven spread of prediction errors, such as when the residuals are larger for a specific range of x-values.

MSE

As each residual is the prediction error for one data point, a summary of all the residuals, or their average, gives us a measure of the overall prediction error.

We simply can't calculate the average of the residuals by summing all the residuals and dividing by the number of residuals. This is because the regression line passes through the center of the data points, so the positive and negative residuals effectively cancel each other out. As a result, the average residual is 0 even when many predictions are far from the actual data.

For example, if two residuals are +5 and -5, then

\[\frac{-5 + 5}{2} = 0 \nonumber\]

The result shows no prediction error, even though both predictions are actually off by 5 units.

Instead we'll use the same method as the calculation of the standard deviation:

• square the error values to remove the negative sign
• find the average of all the squares

This average is the Mean Squared Error or the MSE. A smaller MSE means the predictions are generally closer to the actual value.

MSE is useful because it summarizes the prediction errors in a single number, and it is the value used by many regression models to find the best regression line.

However, because the errors are squared, the units are also squared. For example, if the MPG prediction errors are measured in miles per gallon, then MSE is measured in \(\text{miles per gallon}^2\) Since people normally think of prediction errors in miles per gallon rather than square miles per gallon, MSE is not easy to interpret.

RMSE

Just as with the standard deviation calculation, we can take the square root of the MSE. This converts the squared units back to the original units of the data, making the result easier to interpret. The result is called the Root Mean Squared Error or RMSE. Its long name describes how it is calculated: square the residuals, then find their mean, and finally take the square root.

rmse = np.sqrt(np.mean(residuals**2))
print("RMSE:", round(rmse, 2))
actual_mpg_mean = np.mean(cars["mpg"])
print("Average actual MPG:", round(actual_mpg_mean, 2))
print("RMSE as a percentage of average MPG:", round(rmse / actual_mpg_mean * 100), "%")

Output?

RMSE: 4.32
Average actual MPG: 23.45
RMSE as a percentage of average MPG: 18 %

The RMSE is 4.32. This means that the regression model's predictions differ from the actual MPG values by about 4.32 miles per gallon, on average. A smaller RMSE indicates that the predictions are closer to the actual values and that the regression model is more accurate.

The average MPG of all the cars in the dataset is 23.45 MPG. The RMSE is about 18%, or about one-fifth of the average MPG. This gives us a sense of the size of the prediction error relative to a typical MPG value in the dataset.

The RMSE is most useful for comparing different regression models on the same dataset. In general the model with the smaller RMSE makes more accurate predictions.