Preprocessing for K-Means

A data scientist clustered customer data, expecting to find distinct behavioral segments. The resulting clusters, however, were primarily driven by income, completely overshadowing other key features due to scale differences. This common problem highlights why feature scaling is a critical preprocessing step for K-Means clustering.

K-Means is an algorithm that groups data points based on their proximity to cluster centroids. This proximity is typically measured using Euclidean distance, which calculates the straight-line distance between points in a multi-dimensional space. When features in your dataset have vastly different numerical ranges, the feature with the largest range will contribute disproportionately more to this distance calculation. This dominance effectively allows one feature to dictate the clustering outcome, masking the influence of other potentially important attributes.

Unscaled Features Distort Cluster Perception
This scatter plot shows customer data with 'Income' (0-100,000) and 'Product_Rating' (1-5). Although there are underlying patterns related to product rating, the vast difference in scale between the two features causes 'Income' to visually dominate the data's spread. K-Means, being distance-based, will primarily form clusters along the income axis, obscuring the influence of product rating.
Loading chart...
Key Insight: Features with larger numerical ranges disproportionately influence K-Means' distance calculations, leading to clusters driven by scale rather than true underlying patterns.
📐 Euclidean Distance and Scale

The Euclidean distance between two points P1=(x1,y1)P_1 = (x_1, y_1) and P2=(x2,y2)P_2 = (x_2, y_2) is given by:

d(P1,P2)=(x1−x2)2+(y1−y2)2d(P_1, P_2) = \sqrt{(x_1 - x_2)^2 + (y_1 - y_2)^2}

If, for example, the difference in income (x1−x2)(x_1 - x_2) is 50,000 and the difference in product rating (y1−y2)(y_1 - y_2) is 2, the (x1−x2)2(x_1 - x_2)^2 term (2.5×1092.5 \times 10^9) completely overshadows (y1−y2)2(y_1 - y_2)^2 (4), making the yy dimension's contribution negligible to the total distance.

To prevent features with larger scales from dominating the clustering process, we apply feature scaling. This preprocessing technique transforms numerical features to a similar scale, ensuring that each feature contributes equally to the distance calculations. The goal is to equalize the 'voice' of each feature, allowing K-Means to identify clusters based on the true underlying relationships in the data, rather than arbitrary differences in measurement units or ranges.

StandardScaler (Z-score Normalization)
StandardScaler transforms data such that its distribution has a mean of 0 and a standard deviation of 1. It achieves this by subtracting the mean and then dividing by the standard deviation of each feature.
Example: If a feature has values [10, 20, 30] with a mean of 20 and standard deviation of 8.16, StandardScaler would transform them to approximately [-1.22, 0, 1.22].
MinMaxScaler
MinMaxScaler scales data to a fixed range, typically between 0 and 1. It achieves this by subtracting the minimum value of the feature and then dividing by the range (maximum value - minimum value).
Example: If a feature has values [10, 20, 30], MinMaxScaler would transform them to [0, 0.5, 1], assuming 10 is the min and 30 is the max.
StandardScaler vs. MinMaxScaler
CharacteristicStandardScalerMinMaxScaler
Effect on OutliersSensitive; outliers can significantly shift the mean and standard deviation.Less sensitive; outliers are compressed within the 0-1 range but can distort the distribution of other points.
Output RangeNo fixed range; typically centered around 0 with unit variance.Fixed range, usually [0, 1] or [-1, 1].
When to UseWhen data follows a Gaussian (normal) distribution or algorithm assumes zero-mean, unit-variance data.When a specific bounded range is required, or when the algorithm is robust to the shape of the distribution.
Suitability for K-MeansGenerally preferred for K-Means as it handles varying scales well and centers data.Can be used, but sensitive to outliers defining the min/max of the scaling range.
Choosing the right scaling method depends on your data's distribution and the presence of outliers.
Check Your Understanding
You are preprocessing data for K-Means, and one of your key features has several extreme outliers. Which scaling method is more likely to be heavily influenced by these outliers, potentially compressing the range of your 'normal' data points?
Scaled Features Reveal True Cluster Patterns
This scatter plot shows the same customer data as before, but now with both 'Income' and 'Product_Rating' features scaled using StandardScaler. The data points are now distributed more evenly across both dimensions, allowing K-Means to identify clusters based on the combined influence of both features, revealing the distinct customer segments more accurately.
Loading chart...
Key Insight: Feature scaling normalizes the contribution of each feature to distance calculations, enabling K-Means to discover meaningful clusters that reflect underlying data structures.

Beyond feature scaling, handling outliers is another critical preprocessing step for K-Means. Outliers are data points that significantly deviate from other observations. These extreme values can disproportionately affect the calculation of means and standard deviations used in scaling methods like StandardScaler, leading to suboptimal transformations. More importantly, in K-Means, outliers can either form their own small, spurious clusters or unfairly pull cluster centroids towards them, distorting the overall clustering structure and misrepresenting the true groupings within the data.

Identifying Outliers in Annual Purchases
This box plot illustrates the distribution of 'Annual Purchases'. The central box represents the interquartile range (IQR), with the line inside indicating the median. The 'whiskers' extend to 1.5×IQR1.5 \times IQR from the quartiles. Data points falling outside these whiskers are considered outliers, suggesting unusually high or low purchase values that could skew analysis.
Loading chart...
Key Insight: Box plots provide a clear visual method to detect outliers, which are data points significantly deviating from the majority of observations.

Several strategies exist for handling outliers, depending on their nature and impact. One common approach is removal, if the outliers are few and clearly erroneous data entry mistakes. Alternatively, winsorization involves capping outlier values at a specified percentile (e.g., replacing values above the 99th percentile with the 99th percentile value). Transformation, such as applying a logarithmic transform to highly skewed data, can also reduce the impact of outliers by compressing their range. For K-Means, addressing outliers before clustering helps ensure that centroids accurately represent the dense regions of the data.

pythonApplying StandardScaler to a DataFrame
Try It Yourself
Modify the provided code to use MinMaxScaler instead of StandardScaler on the same dataset. Observe and describe the differences in the scaled output values compared to StandardScaler.
python
Key Takeaways
  • K-Means is highly sensitive to feature scales because its distance calculations are dominated by features with larger numerical ranges.

  • Feature scaling, using methods like StandardScaler or MinMaxScaler, ensures all features contribute equally to the clustering process.

  • StandardScaler (Z-score normalization) transforms data to a mean of 0 and standard deviation of 1, suitable for normally distributed data.

  • MinMaxScaler scales data to a fixed range (e.g., 0 to 1), useful when a specific bounded range is desired.

  • Outliers can significantly skew scaling parameters and distort cluster centroids, making their identification and handling crucial before scaling and clustering.

  • Proper preprocessing prevents K-Means from forming distorted clusters driven by arbitrary feature ranges, allowing it to reveal true underlying patterns, as seen in the opening scenario.

← All lessons in K-Means Clustering

Ready to keep this from fading?

Bitelrn turns lessons like this into a full course — quizzes, a knowledge map, and spaced review.

Get started free