Assumptions of Linear Regression
A data scientist presents a linear regression model with excellent R-squared, only to find its predictions wildly inaccurate for certain customer segments. This common pitfall occurs when the model's fundamental assumptions about the data are violated.
Linear regression models, despite their apparent simplicity, rely on several key assumptions about the data and the error term. These assumptions are not mere technicalities; they are foundational to the model's validity. Ignoring them can lead to misleading interpretations of coefficients, incorrect p-values, and ultimately, unreliable predictions, even if initial performance metrics like R-squared appear favorable.
The Five Pillars of Linear Regression
Linearity: The Relationship's Shape
The first assumption, linearity, states that the relationship between the independent variables and the mean of the response variable is linear. This means the response variable can be expressed as a linear function of the predictors, plus an error term. If the true relationship is non-linear, a linear model will systematically misrepresent the data, leading to biased coefficient estimates and poor predictive performance, especially at the extremes of the predictor range. A common diagnostic involves plotting the residuals (the differences between observed and predicted values) against the fitted values (the predicted values).
Independence: No Hidden Order
The independence of errors assumption states that the error terms are uncorrelated with each other. This is particularly important in time-series data, where observations collected sequentially often exhibit autocorrelation (errors from one period are correlated with errors from a previous period). Violations of this assumption lead to underestimated standard errors for the regression coefficients, making them appear more statistically significant than they truly are. This can result in incorrect conclusions about the impact of predictors. A common diagnostic involves plotting residuals against the observation order or time.
Homoscedasticity: Consistent Error Spread
The assumption of homoscedasticity requires that the variance of the error terms is constant across all levels of the independent variables. When this assumption is violated, it's called heteroscedasticity, meaning the spread of residuals changes as the fitted values change. Heteroscedasticity does not bias the coefficient estimates themselves, but it leads to inefficient and biased standard errors. This impacts the reliability of confidence intervals and hypothesis tests, making it difficult to assess the true precision of the model's estimates. The residuals vs. fitted values plot is also a key diagnostic tool here.
Normality of Residuals: Errors' Distribution
The assumption of normality of residuals states that the error terms are normally distributed. This assumption is primarily important for the validity of hypothesis tests (p-values) and confidence intervals, especially with smaller sample sizes. For large sample sizes, the Central Limit Theorem often ensures that the sampling distributions of the coefficients are approximately normal, even if the errors themselves are not. However, severe non-normality can still indicate underlying issues such as the presence of significant outliers or a misspecified model. A histogram of residuals or a Q-Q plot can help diagnose this assumption.
No Multicollinearity: Independent Predictors
The assumption of no multicollinearity states that the independent variables should not be highly correlated with each other. When two or more predictor variables are highly correlated, it becomes difficult for the model to determine the individual effect of each predictor on the response variable. This leads to unstable and uninterpretable coefficient estimates, where small changes in the data can result in large changes in the coefficients. While multicollinearity does not affect the model's overall predictive power, it severely hinders the ability to understand the unique contribution of each predictor. A correlation matrix or Variance Inflation Factor (VIF) are common diagnostic tools.
When Assumptions Break: Remedies
| Assumption Violated | Diagnostic Clue | Potential Remedy |
|---|---|---|
| Linearity | Curved pattern in residuals vs. fitted values plot | Transform variables (e.g., log, square root), add polynomial terms (, ), or use a non-linear model. |
| Independence of Errors | Wave-like pattern in residuals vs. observation order plot (autocorrelation) | Include lagged variables, use time series models (e.g., ARIMA), or generalized least squares (GLS). |
| Homoscedasticity | Fan or cone shape in residuals vs. fitted values plot | Transform the response variable (e.g., log), use weighted least squares (WLS), or robust standard errors. |
| Normality of Residuals | Skewed or bimodal histogram of residuals, non-linear Q-Q plot | Check for outliers, transform the response variable, or use non-parametric methods. Less critical for large sample sizes. |
| No Multicollinearity | High correlation coefficients (>0.8) in correlation matrix, high VIF values | Remove highly correlated predictors, combine predictors, or use Principal Component Analysis (PCA). |
Linear regression relies on five core assumptions: linearity, independence of errors, homoscedasticity, normality of residuals, and no multicollinearity.
Violating these assumptions can lead to misleading coefficient interpretations and unreliable predictions, even if metrics like R-squared look good.
Linearity is diagnosed by checking for curved patterns in residuals vs. fitted values plots.
Independence of errors is diagnosed by looking for patterns (e.g., wave-like) in residuals vs. observation order plots.
Homoscedasticity is diagnosed by checking for a 'fan' or 'cone' shape in residuals vs. fitted values plots.
Normality of residuals is diagnosed by examining the shape of a residuals histogram or Q-Q plot for skewness or bimodality.
Multicollinearity is diagnosed by high correlations between independent variables in a correlation matrix.