Residual Analysis for Diagnostics

A linear regression model shows a strong R², but your business intuition says something is off. How do you uncover the hidden flaws? Residual analysis is the key to understanding if a model's underlying assumptions hold, even with a high R².

The Model's Leftovers: What Are Residuals?

When a linear regression model makes a prediction, it aims to capture the underlying relationship between variables. However, no model is perfect; there will always be some difference between the actual observed values and the values the model predicts.

These differences are called residuals. Think of them as the 'error' or the 'unexplained' portion of the response variable that the model couldn't account for. Analyzing these leftovers provides crucial insights into how well the model truly fits the data and if its fundamental assumptions are met.

Formal Definition: Residual

A residual, denoted as

e_i

, is the difference between an observed value

y_i

and its corresponding predicted value

\hat{y}_i

from a regression model.

Example: If a model predicts a house price of $300,000, but the actual sale price was $310,000, the residual is

e_i = 310,000 - 300,000 = 10,000

Why Residuals Matter: Validating Model Assumptions

Linear regression models rely on several key assumptions about the error terms (which residuals approximate). If these assumptions are violated, the model's coefficients, standard errors, and overall inferences can be unreliable, even if the R² value appears high.

Analyzing residual patterns helps us visually check for violations of assumptions such as linearity (the relationship is truly linear), independence (errors are not correlated), homoscedasticity (errors have constant variance), and normality (errors are normally distributed).

The Go-To Plot: Residuals vs. Fitted Values

The most fundamental diagnostic tool for residuals is a scatter plot of residuals against fitted (predicted) values. This plot helps assess the linearity and homoscedasticity assumptions of the model.

A healthy residual plot shows points randomly scattered around the horizontal line at zero, with no discernible pattern. This indicates that the model has captured the systematic relationship in the data, and the remaining errors are just random noise, fulfilling the model's assumptions.

Well-Behaved Residuals vs. Fitted Values

This scatter plot shows residuals randomly distributed around the zero line, indicating a good fit and that the linear regression assumptions of linearity and homoscedasticity are likely met.

Loading chart...

Key Insight: Random scatter around zero indicates a good model fit, suggesting linearity and constant variance.

Pattern Alert: Diagnosing Non-Linearity

If the relationship between the predictor and response variables is not truly linear, but a linear model is fitted, the residuals will often reveal this mismatch. A common indicator of non-linearity is a curved pattern in the residuals vs. fitted plot.

For example, a U-shaped or inverted U-shaped pattern suggests that the linear model is systematically under-predicting or over-predicting at certain ranges of the fitted values. This means the model is missing a non-linear component, and a transformation of variables or a non-linear model might be more appropriate.

Residuals Showing Non-Linearity (U-Shape)

This plot shows a distinct U-shaped pattern in the residuals, indicating that the linear model is failing to capture a non-linear relationship present in the data.

Loading chart...

Key Insight: A U-shaped pattern indicates that the linear model is missing a non-linear relationship in the data.

Check Your Understanding

What does a U-shaped pattern in a residual plot primarily suggest?

Expanding Errors: Identifying Heteroscedasticity

Homoscedasticity is the assumption that the variance of the error terms is constant across all levels of the independent variables. When this assumption is violated, it's called heteroscedasticity.

In a residuals vs. fitted plot, heteroscedasticity often appears as a 'fanning out' or 'cone' shape. This means the spread of residuals increases (or decreases) as the fitted values change. Heteroscedasticity can lead to inefficient parameter estimates and incorrect standard errors, making hypothesis tests unreliable.

Residuals Showing Heteroscedasticity (Cone Shape)

This scatter plot shows residuals fanning out as fitted values increase, forming a cone shape. This indicates that the variance of the errors is not constant, violating the homoscedasticity assumption.

Loading chart...

Key Insight: A 'fanning out' pattern indicates heteroscedasticity, meaning error variance changes with fitted values.

Are Errors Normal? The Q-Q Plot

Another crucial assumption for linear regression, particularly for inference (confidence intervals and p-values), is that the error terms are normally distributed. A Quantile-Quantile (Q-Q) plot is used to check this assumption.

A Q-Q plot compares the quantiles of your residuals against the quantiles of a theoretical normal distribution. If the residuals are normally distributed, the points on the plot will generally fall along a 45-degree reference line. Deviations from this line, especially at the tails, suggest non-normal errors, which can affect the validity of statistical tests.

Q-Q Plot: Normally Distributed Residuals

This Q-Q plot shows residuals closely following the 45-degree reference line, indicating that the residuals are approximately normally distributed, satisfying a key linear regression assumption.

Loading chart...

Key Insight: Points closely following the 45-degree line indicate normally distributed residuals.

Q-Q Plot: Non-Normally Distributed Residuals (Heavy Tails)

This Q-Q plot shows points deviating from the 45-degree line, particularly at the tails, suggesting that the residuals have heavier tails than a normal distribution, indicating non-normality.

Loading chart...

Key Insight: Deviations from the 45-degree line, especially at the tails, indicate non-normal residuals.

Implementing Residual Diagnostics in Python

pythonGenerating Diagnostic Plots

Try It Yourself

Modify the provided Python code to analyze a new dataset. Generate X and y where y has a clear linear relationship but with increasing variance (heteroscedasticity). Observe how the residual plot changes.

python

import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
import seaborn as sns

# Original data (non-linear)
# np.random.seed(42)
# X = np.random.rand(100) * 10
# y = 2 * X + 0.5 * X**2 + np.random.normal(0, 5, 100)

# TODO: Generate new data with linear relationship and heteroscedasticity
np.random.seed(43)
X_new = np.random.rand(100) * 10
# Hint: Make noise variance dependent on X_new
y_new = 3 * X_new + np.random.normal(0, 1 + X_new * 0.5, 100)

# Fit a simple linear regression model
X_sm_new = sm.add_constant(X_new)
model_new = sm.OLS(y_new, X_sm_new)
results_new = model_new.fit()

# Get residuals and fitted values
residuals_new = results_new.resid
fitted_values_new = results_new.fittedvalues

# Plot Residuals vs. Fitted Values
plt.figure(figsize=(12, 5))

plt.subplot(1, 2, 1)
sns.scatterplot(x=fitted_values_new, y=residuals_new)
plt.axhline(y=0, color='r', linestyle='--')
plt.xlabel('Fitted Values')
plt.ylabel('Residuals')
plt.title('Residuals vs. Fitted Values (Heteroscedastic)')

# Plot Q-Q Plot of Residuals
plt.subplot(1, 2, 2)
sm.qqplot(residuals_new, line='45', fit=True)
plt.title('Normal Q-Q Plot of Residuals (Heteroscedastic)')

plt.tight_layout()
plt.show()

This exercise uses a charting library (e.g. seaborn, plotly) that the in-browser runner can't display — it shows text output only. Google Colab renders plots inline, so you can run this and actually see the figures there.

Expected

You should see a residual plot where the spread of residuals widens as the fitted values increase, forming a cone shape. The Q-Q plot should still look relatively normal if the errors are otherwise Gaussian.

Key Takeaways: Your Model's Health Check

Residuals are the differences between observed and predicted values, representing the unexplained portion of the data.
Residual analysis is a critical diagnostic tool for visually inspecting linear regression assumptions.
A residuals vs. fitted values plot is the primary tool to check for linearity and homoscedasticity.
Random scatter around zero in a residuals plot indicates a well-fitting model.
A curved pattern (e.g., U-shape) in residuals signals a violation of the linearity assumption.
A fanning out or cone shape in residuals indicates heteroscedasticity (non-constant error variance).
A Q-Q plot compares residual quantiles to theoretical normal quantiles, assessing the normality assumption.

Residual Analysis for Diagnostics

The Model's Leftovers: What Are Residuals?

Why Residuals Matter: Validating Model Assumptions

The Go-To Plot: Residuals vs. Fitted Values

Pattern Alert: Diagnosing Non-Linearity

Expanding Errors: Identifying Heteroscedasticity

Are Errors Normal? The Q-Q Plot

Implementing Residual Diagnostics in Python

Ready to keep this from fading?