What PCA is actually doing to your data

Most explanations of PCA stop at "it reduces dimensions." That is true, and it is roughly as useful as saying a regression "fits a line." It tells you what happens without telling you what the method is doing or when it will fail you.

Here is the whole idea in one sentence: PCA finds the directions your data varies in most, by taking the eigenvectors of its covariance matrix.

Everything else is detail. But that sentence only helps if you know what a covariance matrix is and what an eigenvector does, so let's build it up — and if the prior question is why you'd reduce dimensions at all, start there.

Start with the covariance matrix

If you centre your data — subtract the mean from every column — the covariance matrix is:

For $p$ features you get a $p \times p$ matrix. The diagonal holds each feature's variance. Everything off the diagonal holds the covariance between a pair of features: how much they move together.

That matrix is the entire input to PCA. Not the raw data — the covariance structure. Which means PCA only ever sees how your features vary and co-vary, and is blind to anything else about them.

What the eigenvectors are doing

For a square matrix $\Sigma$, an eigenvector $v$ is a direction that the matrix does not rotate:

Multiplying by $\Sigma$ only stretches $v$, by a factor of $\lambda$. Every other direction gets turned.

When $\Sigma$ is a covariance matrix, those special directions have a concrete meaning: they are the axes along which the data varies independently, and $\lambda$ is how much variance lies along each one.

So sorting the eigenvectors by their eigenvalues sorts the directions of your data from "most spread out" to "least." The first eigenvector is the first principal component. It is the single direction that captures more of your data's variance than any other.

That is the whole method. Compute the covariance matrix, take its eigenvectors, order them by eigenvalue, keep the top few. The full derivation is worth walking through once.

Why the components are uncorrelated

A covariance matrix is symmetric, and symmetric matrices have orthogonal eigenvectors. Every principal component is at right angles to every other.

Orthogonal directions have zero covariance. So your new features are, by construction, completely uncorrelated with each other — a property none of your original features are likely to have.

Hold onto that. It is the source of PCA's most underrated use, which we'll come to.

Choosing how many components to keep

Each eigenvalue is the variance along its component, so the fraction of total variance a component explains is just its share of the sum:

Plot the cumulative version and keep however many components clear the bar you care about — 90%, 95%, whatever the application justifies. There is no correct threshold, which is worth saying plainly rather than pretending an elbow plot settles it. More on choosing the number of components.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X_scaled = StandardScaler().fit_transform(X)   # not optional — see below
pca = PCA().fit(X_scaled)

print(pca.explained_variance_ratio_.cumsum())

PCA and feature selection

This is where terminology causes real confusion, so let's be exact.

PCA is not feature selection. It is feature extraction.

• Selection keeps a subset of your original columns. You start with 50 features, you end with 12 of the original 50, and you can still say "this model uses income and tenure."
• Extraction builds new features out of the old ones. PC1 is not one of your columns; it is a weighted blend of all of them — something like $0.4 \cdot \text{income} + 0.3 \cdot \text{tenure} - 0.2 \cdot \text{age} + \ldots$

So PCA never selects anything. It replaces your feature set with a smaller, rotated one. If someone tells you they "used PCA for feature selection," they mean one of the three things below.

1. Dimensionality reduction before modelling

The standard use. Fifty correlated features become ten components carrying 95% of the variance, and the model trains on those. It serves the same purpose as selection — fewer inputs, less overfitting, faster training — while doing something mathematically different.

2. Reading the loadings to inform real selection

Each component has loadings: the weight of every original feature within it. If one feature dominates the top components, that is evidence it carries a lot of the variance, and you might keep it in a genuine selection step.

This is legitimate but weaker than it looks. High loading means high variance contribution, not high predictive value — and those are not the same thing. See interpreting component loadings.

3. Removing multicollinearity

The one people underuse. Because the components are orthogonal, they have zero correlation with each other by construction. Regressing on principal components instead of raw features makes multicollinearity structurally impossible.

If you have ever watched a VIF climb past 10 and had to decide which of two near-identical predictors to drop, PCA sidesteps that decision entirely — it is one of the few honest answers to that violated regression assumption, rather than a coin toss between two predictors.

Advantages

• No multicollinearity, guaranteed by orthogonality — not something you check for afterwards.
• Fewer inputs means less overfitting and faster training, particularly when $p$ approaches $n$.
• Noise reduction. Low-variance components are often mostly measurement noise; dropping them can improve signal.
• It keeps information that selection throws away. Dropping a column discards everything in it. PCA compresses all fifty features into ten components, so a feature's contribution survives even when it isn't individually important.
• It's unsupervised, so it can run before you have labels, and it cannot leak your target into the transformation.

The costs, which are not small

You lose interpretability. PC1 is a blend of everything. You can no longer say "a year of tenure is worth £400." For anything that needs explaining to a regulator, a clinician, or a stakeholder, that alone can rule PCA out.

PCA maximises variance, not predictive power. It never sees your target. The direction your data varies in most is not necessarily the direction that predicts $y$ — a low-variance component can easily be the one that matters, and dropping it because it explains 2% of variance can quietly cost you the model.

It only finds linear structure. Data on a curved manifold won't be captured well. That's what kernel PCA, t-SNE and UMAP are for.

The mistake almost everyone makes once

Standardise your features first.

PCA maximises variance, and variance has units. A feature measured in rupees will have a variance millions of times larger than one measured in years — so the first principal component will point almost exactly along the rupee axis, and you will have discovered nothing except which column has the biggest numbers.

Scaling puts every feature on equal footing so the components reflect structure rather than measurement units. The only time you skip it is when your features are already in the same units and their relative scales are meaningful — the rest of the preprocessing PCA expects is short but non-negotiable.

Where to go from here

PCA is the point where linear algebra stops being abstract. Eigenvectors go from a homework exercise to the thing deciding which ten of your fifty columns survive.

If any of the pieces felt shaky, fix them before you use this in anger — the covariance structure and the eigen-decomposition do all the work here, and everything else is bookkeeping:

• Principal Component Analysis — the full topic, free to read
• Understanding covariance — the matrix PCA actually consumes
• Understanding correlation — why orthogonal means uncorrelated
• Linear regression assumptions — the multicollinearity problem PCA dissolves