Your Correlation Matrix Is Not Feature Selection

Almost every regression project I've seen in the last decade starts the same way. Load the data, df.corr(), plot the heatmap, scan for red squares, drop one variable from every pair above 0.8. Then we move on, feeling like we've handled multicollinearity.

We haven't. We've handled the version of it that happens to show up in pairs.

The three-line example that should end the ritual

Take two independent features and construct a third as their sum:

import numpy as np
import pandas as pd

rng = np.random.default_rng(42)
x1 = rng.normal(size=10_000)
x2 = rng.normal(size=10_000)
x3 = x1 + x2

pd.DataFrame({"x1": x1, "x2": x2, "x3": x3}).corr().round(3)

The correlation matrix shows roughly 0.7 between x1 and x3, roughly 0.7 between x2 and x3, and approximately 0 between x1 and x2.

Every pair passes a 0.8 threshold. Every pair passes a 0.75 threshold. And yet x3 is perfectly determined by the other two. The design matrix is singular. There is no unique solution for the coefficients — your software will either return garbage or silently regularise its way out of the problem.

The heatmap saw nothing, because there was nothing to see pairwise. Correlation is a two-variable statistic. Multicollinearity is a property of the entire design matrix. Those are different questions, and the heatmap only answers the first one.

This isn't a contrived edge case. It's every ratio you've engineered, every "total" column that's the sum of its parts, every set of channel spends that add up to a budget, every one-hot encoding you forgot to drop a level from. In practice, near-dependence across three or four features is far more common than a single scary pair — and it is exactly what the heatmap is blind to.

What VIF actually measures

The Variance Inflation Factor asks the right question. For each feature, regress it on all the other features and take:

That $R^2_j$ is multivariate by construction. In the example above, regressing x3 on x1 and x2 gives $R^2 = 1$, so the VIF is infinite. The heatmap said fine; the VIF says the model is unidentifiable.

The name is literal, which is the part people miss. A VIF of 10 means the variance of that coefficient estimate is ten times what it would be if the feature were orthogonal to the others — so the standard error is $\sqrt{10} \approx 3.2$ times larger. That's the whole mechanism. Your coefficient isn't wrong on average; it's just so unstable that it's useless. Resample the data and it swings, sometimes across zero.

from statsmodels.stats.outliers_influence import variance_inflation_factor
import statsmodels.api as sm

X = sm.add_constant(pd.DataFrame({"x1": x1, "x2": x2, "x3": x3}))
[variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

A note on the thresholds, in the spirit of this whole post: VIF > 5 and VIF > 10 are conventions, not findings. They're no more principled than the 0.8 on the heatmap. Use them to rank features by severity, not as a pass/fail gate.

The part nobody says out loud: this is an inference problem, not a prediction problem

If you only care about predictions, multicollinearity mostly doesn't matter. Correlated features don't bias the fitted values, don't inflate test error, and don't need to be removed. A gradient boosting model on collinear features will predict perfectly well.

Multicollinearity only hurts when you need to read the coefficients — when the model is going to answer "how much should we spend on this channel" or "what happens if we raise price by 5%." That's where unstable coefficients become bad decisions.

So the first question isn't "which features are correlated." It's "am I predicting or explaining?" Half the arguments about multicollinearity are people answering different questions and not realising it.

Why I sometimes drop a variable that correlates strongly with the target

Building Marketing Mix Models, I started the way everyone does: rank variables by correlation with sales, keep the strong ones. It worked, in the sense that the model fit beautifully. It also quietly pushed every media driver out of the model.

The pattern was consistent. Distribution, seasonality, holiday flags — these sat at 0.8 and above. The media variables I actually needed to report on sat between 0.4 and 0.6. Ranked on correlation, the media never stood a chance.

The problem is that in any real market, promotions, media and seasonal demand all move together. December sales are up. So is TV spend, so is display, so is the holiday flag. Every one of those variables is competing to explain the same peak, and the one that wins the competition is the one most tightly coupled to it — which is almost always the variable you can't control.

So I dropped variables with high correlation to the target, deliberately, and accepted a worse fit. Because a marketing mix model whose coefficients say "sales are seasonal" is not a model. Nobody can act on it. The entire point of the exercise was the media coefficients, and the high-correlation variables were crowding out the only part anyone was going to read.

This is the inference-versus-prediction distinction from earlier, arriving with a budget attached. If I were forecasting sales, keeping seasonality and dropping media would be the right call — better fit, better forecast, done. But the model existed to answer "what should we spend on this channel next quarter," and that question is answered by coefficients, not by fit.

What's left is a judgement call that no diagnostic makes for you: balancing model fit against business expectation and explainability. The correlation matrix can't help here. It ranks variables by their relationship to the target and has nothing at all to say about which relationships you need to be reportable.

The general principle I've settled on: feature selection is a question about what the model is for, not just what's in the data. A variable that improves fit but destabilises the coefficient you're going to act on is a bad trade, regardless of what the heatmap says.

The mirror-image mistake: dropping features with low target correlation

While we're dismantling this ritual, the other half of it deserves the same treatment. People keep the features that correlate strongly with the target and cut the ones that don't.

But a feature can have near-zero correlation with the target and still be one of the most important variables in the model. These are suppressor variables: they don't predict the target directly, they explain away noise in another predictor, sharpening its signal. Cut it on a univariate screen and the model gets worse.

The pattern is the same error in both directions — using a pairwise statistic to make a multivariate decision.

Where eigenvalues come in

If you want the honest, single-number version of "is my design matrix in trouble," it's the condition number: the ratio of the largest to the smallest singular value of your scaled feature matrix. Values above ~30 are the usual flag.

That's not a coincidence — it's the same underlying idea. Near-dependence among your features means the matrix $X^\top X$ has eigenvalues close to zero, and inverting something with a near-zero eigenvalue is what blows the coefficient variances up in the first place. VIF and the condition number are two views of one geometric fact: your features don't span the space you think they do.

What I'd actually do instead

1. Decide first whether you're predicting or explaining. If predicting, most of this is moot.
2. Skip the heatmap as a decision tool. Keep it for exploration — it's genuinely useful for spotting data errors and leakage.
3. Compute VIF on the full feature set. Rank, don't threshold.
4. Check the condition number as a whole-matrix sanity check.
5. For anything with a high VIF, ask why it's dependent. Engineered ratio? Sum of components? Dummy trap? The fix is usually structural, not deletion.
6. Only then decide what to drop — and decide it against what the model is for.

None of this is more work than making the heatmap. It's just less satisfying, because there's no red square to point at.

Want the underlying concepts, properly? Explore Correlation & Covariance, Linear Regression, and Eigenvalues in the Bitelrn Open Library — no sign-up.

--