Determining Optimal Components

Which number of principal components balances data reduction with information retention? The process of principal component selection addresses this by identifying the optimal subset of components that capture most of the original data's variance while significantly reducing dimensionality. This decision is critical because too few components can lead to significant information loss, while too many components might retain noise and defeat the purpose of dimensionality reduction.

Visualizing Variance: The Scree Plot

A scree plot is a line plot that displays the explained variance for each principal component, ordered from largest to smallest. It helps visualize how much information each successive component contributes. To interpret a scree plot, look for an 'elbow point' where the slope of the line changes dramatically, flattening out. Components before this elbow are typically considered the most significant, as they explain a substantial amount of variance, while subsequent components contribute much less.

Scree Plot and Cumulative Explained Variance

This plot shows the individual explained variance ratio for each principal component, revealing an 'elbow' where the gain in explanation diminishes. It also displays the cumulative explained variance, indicating how much total information is retained as more components are added.

Loading chart...

Key Insight: The 'elbow' in the individual explained variance helps identify the optimal number of components, often aligning with a high cumulative explained variance threshold.

Check Your Understanding

Based on a scree plot, what indicates the optimal number of components?

The Kaiser Criterion: A Rule of Thumb

The Kaiser criterion offers a simple quantitative rule for selecting principal components. It suggests retaining only those components whose eigenvalues are greater than 1. The rationale behind this rule is that any component with an eigenvalue less than 1 explains less variance than a single original variable, assuming the data has been standardized. Therefore, such components are often considered to contribute little meaningful information.

📐 Kaiser Criterion

Retain principal components where the eigenvalue $\lambda > 1$ .

Setting a Cumulative Explained Variance Threshold

Another common quantitative approach involves setting a cumulative explained variance threshold. This method focuses on retaining enough components to explain a predefined percentage of the total variance in the dataset, typically between 85% and 95%. For example, if you aim to retain 90% of the information, you would select the minimum number of principal components whose combined explained variance ratio sums up to at least 0.90. This method is straightforward and ensures a specific level of information retention.

Implementing Component Selection in Python

pythonPCA and Component Selection

Try It Yourself

Modify the code to find the number of components required to explain 95% of the total variance. How many components are selected now?

python

import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_classification

# 1. Generate synthetic data
X, y = make_classification(n_samples=100, n_features=15, n_informative=8, n_redundant=2, random_state=42)

# 2. Standardize the data
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# 3. Perform PCA (without specifying n_components to get all)
pca = PCA()
pca.fit(X_scaled)

# 4. Get explained variance ratios and eigenvalues
explained_variance_ratio = pca.explained_variance_ratio_
eigenvalues = pca.explained_variance_
cumulative_variance = np.cumsum(explained_variance_ratio)

# 5. Plot Scree Plot and Cumulative Explained Variance (optional, for visualization)
# plt.figure(figsize=(10, 6))
# plt.plot(range(1, len(explained_variance_ratio) + 1), explained_variance_ratio, marker='o', linestyle='-', label='Individual Explained Variance')
# plt.plot(range(1, len(cumulative_variance) + 1), cumulative_variance, marker='x', linestyle='--', label='Cumulative Explained Variance')
# plt.xlabel('Principal Component Number')
# plt.ylabel('Explained Variance Ratio')
# plt.title('Scree Plot and Cumulative Explained Variance')
# plt.axhline(y=1/X.shape[1], color='r', linestyle=':', label='Kaiser Criterion (Eigenvalue > 1/p)')
# plt.grid(True)
# plt.legend()
# plt.show()

# 6. Apply Kaiser Criterion (eigenvalue > 1)
kaiser_components = np.where(eigenvalues > 1)[0] + 1
print(f"Components selected by Kaiser Criterion (eigenvalue > 1): {len(kaiser_components)} components ({kaiser_components})")

# 7. Apply Cumulative Explained Variance Threshold (e.g., 90%)
threshold = 0.90 # CHANGE THIS VALUE
n_components_threshold = np.where(cumulative_variance >= threshold)[0][0] + 1
print(f"Components selected for {threshold*100}% cumulative variance: {n_components_threshold} components")

Beyond the Rules: Context and Trade-offs

No single method for principal component selection is universally perfect; the best approach often depends on the specific dataset and the problem's context. The scree plot provides a visual, intuitive guide, but the 'elbow' can sometimes be ambiguous. Quantitative rules like the Kaiser criterion offer clear cutoffs but might be too aggressive or too conservative depending on the data's structure. Similarly, a cumulative variance threshold is straightforward but requires an arbitrary choice of percentage. Ultimately, domain knowledge plays a crucial role in making an informed decision, as it helps interpret the significance of components and the acceptable level of information loss for a given application.

Comparing Principal Component Selection Methods

Method	Pros	Cons
Scree Plot	Provides a visual, intuitive understanding of variance contribution.	The 'elbow' point can be subjective and hard to pinpoint clearly.
Kaiser Criterion	Offers a simple, objective rule (eigenvalue > 1).	Can be overly simplistic; may retain too many or too few components depending on data structure.
Cumulative Explained Variance Threshold	Ensures a specific percentage of total variance is retained.	Requires an arbitrary choice for the variance threshold (e.g., 85%, 90%).

A comparison of common methods for determining the optimal number of principal components.

Key Takeaways

Scree plots visually display explained variance per component, with the 'elbow' indicating an optimal cutoff.
The Kaiser criterion suggests retaining components with eigenvalues $\lambda > 1$ , meaning they explain more variance than a single original variable.
A cumulative explained variance threshold selects the minimum number of components needed to reach a desired percentage of total variance (e.g., 90%).
Python's sklearn.decomposition.PCA allows easy calculation and plotting of explained variance for component selection.
No single method is universally superior; the best choice balances data reduction with information retention, often guided by domain knowledge and the specific problem context.

← All lessons in Principal Component Analysis