Vector Spaces (Data Representation Context)

Every piece of data we analyze in machine learning, from a customer's purchase history to an image's pixel values, can be represented as a vector. But for these vectors to be useful—to be added, scaled, transformed, or compared—they need to exist within a structured environment. This environment is called a vector space, a mathematical construct that provides the rules and properties necessary for consistent vector operations.

Defining a Vector Space

A vector space is formally defined as a set $V$ of objects, called vectors, along with two operations: vector addition and scalar multiplication. These operations must satisfy ten specific axioms, ensuring that the set behaves predictably under these manipulations. Think of it as a playground with very strict rules, where every move you make with a vector is guaranteed to keep you within the playground itself.

💡 The Ten Axioms

The axioms ensure closure (results stay in the space), associativity, commutativity, existence of a zero vector and additive inverse, and distributive properties for scalar multiplication. These are the bedrock for all linear algebra operations.

Vectors as Data Points

In data science, a vector typically represents a single observation or a set of features for an entity. Each component of the vector corresponds to a specific feature. For instance, a customer's profile might be a vector where components represent age, income, and number of purchases. This allows us to treat each customer as a distinct point in a multi-dimensional space.

pythonRepresenting Customer Data as a Vector

Scalar Multiplication: Scaling Features

Scalar multiplication involves multiplying a vector by a single real number (a scalar). This operation scales the magnitude of the vector, making it longer or shorter, but it does not change its direction (unless the scalar is negative, which reverses it). In data preprocessing, this is crucial for operations like normalization or standardization, where we adjust feature scales to prevent certain features from dominating calculations.

pythonScaling a Vector (Feature Normalization)

Check Your Understanding

What happens to a vector's direction when multiplied by a negative scalar?

Vector Addition: Combining Data Influences

Vector addition combines two vectors by adding their corresponding components. This operation is fundamental when we need to aggregate different influences or features. For example, if we have a vector representing a user's preferences and another representing a friend's preferences, adding them could create a combined preference vector, useful in recommendation systems. The result is a new vector that also resides within the same vector space.

pythonCombining Feature Influences with Vector Addition

The Zero Vector and Additive Inverse

Every vector space must contain a zero vector, which is a vector where all components are zero. When added to any other vector, it leaves that vector unchanged, acting as an identity element for addition. Additionally, for every vector $v$ in the space, there must exist an additive inverse $-v$ , such that $v + (-v)$ equals the zero vector. These properties are critical for ensuring that vector subtraction is well-defined and that the space is 'complete' under addition.

📌 Identity and Inverse

The zero vector is the additive identity, and the additive inverse allows us to 'undo' vector addition. These are fundamental for maintaining the structure of the vector space.

The Real Coordinate Space $\mathbb{R}^n$

The most commonly encountered vector space in data science and machine learning is the real coordinate space, denoted as $\mathbb{R}^n$ . This is the set of all $n$ -dimensional vectors where each component is a real number. Here, $n$ directly corresponds to the number of features or attributes in our dataset. If you have a dataset with 10 features, each data point (row) is a vector in $\mathbb{R}^{10}$ .

Check Your Understanding

If a dataset describes houses using 7 numerical features, what is the dimensionality of its feature space?

Basis Vectors and Span

A basis for a vector space is a minimal set of linearly independent vectors that can span the entire space. To 'span' means that any vector in that space can be uniquely expressed as a linear combination (scalar multiplication and addition) of these basis vectors. For $\mathbb{R}^n$ , the standard basis consists of $n$ vectors, each with a 1 in one position and 0s elsewhere, effectively aligning with the coordinate axes.

💡 Standard Basis in $\mathbb{R}^n$

In $\mathbb{R}^2$ , the standard basis vectors are $e_1 = (1,0)$ and $e_2 = (0,1)$ . In $\mathbb{R}^3$ , they are $e_1 = (1,0,0)$ , $e_2 = (0,1,0)$ , and $e_3 = (0,0,1)$ . Any vector can be built from these fundamental directions.

Basis Vectors and Linear Combination in

\mathbb{R}^2

This plot shows the standard basis vectors

e_1

and

e_2

in a 2D space. A third vector,

v

, is shown as a linear combination of these basis vectors, with its components projected onto the axes.

Loading chart...

Key Insight: Any vector in a space can be uniquely formed by scaling and adding the basis vectors, defining its position.

Linear Independence and Dimensionality

A set of vectors is linearly independent if no vector in the set can be written as a linear combination of the others. If one vector can be formed from the others, it's redundant and doesn't add a new 'direction' to the space. The number of vectors in any basis for a given vector space is always the same, and this number defines the dimensionality of that vector space. This concept is crucial for understanding feature redundancy and techniques like Principal Component Analysis (PCA).

✨ Multicollinearity

In machine learning, highly correlated features are often linearly dependent. This can lead to issues like multicollinearity in regression models, making coefficient interpretations unstable. Feature selection or dimensionality reduction aims to find a more linearly independent set of features.

Subspaces: Focused Feature Sets

A subspace is a subset of a vector space that is itself a vector space under the same operations. It must contain the zero vector, and be closed under vector addition and scalar multiplication. In data science, a subspace can represent a reduced feature set, perhaps after applying dimensionality reduction techniques like PCA, where the data is projected onto a lower-dimensional space that still captures most of the variance. This allows for more efficient computation and often better model generalization.

Conditions for a Subset to be a Subspace

Loading diagram...

A subset must satisfy three conditions to be considered a vector subspace, ensuring it maintains the core properties of a vector space.

Why Vector Spaces are Essential for Data

Understanding vector spaces provides the mathematical foundation for almost every machine learning algorithm. Operations like calculating distances between data points (e.g., Euclidean distance in K-Nearest Neighbors), transforming data (e.g., feature scaling, PCA, SVD), and projecting data onto different planes all rely on the consistent properties guaranteed by a vector space. Without this framework, the mathematical rigor and predictability of these algorithms would collapse, making data analysis unreliable.

Key Takeaways

A vector space is a set of vectors with defined addition and scalar multiplication operations that satisfy ten axioms, providing a consistent environment for data manipulation.
In data science, individual data points or feature sets are represented as vectors, typically residing in the real coordinate space $\mathbb{R}^n$ , where $n$ is the number of features.
Scalar multiplication scales vector magnitudes (e.g., feature normalization), while vector addition combines influences (e.g., aggregating preferences), both crucial for data preprocessing.
The zero vector (additive identity) and additive inverse ensure that vector spaces are complete and consistent under addition and subtraction.
A basis is a minimal set of linearly independent vectors that can span the entire space, with the number of basis vectors defining the space's dimensionality.
Subspaces are subsets of a vector space that are themselves vector spaces, often representing reduced feature sets after dimensionality reduction.
Vector spaces provide the mathematical bedrock for machine learning, enabling reliable distance calculations, transformations, and projections of data.

← All lessons in Linear Algebra: Vectors