Linear Combinations and Span

In linear algebra, vectors are more than just arrows; they are fundamental building blocks. Just as we combine numbers with arithmetic operations, we can combine vectors using scalar multiplication and vector addition. This process, known as forming a linear combination, allows us to generate new vectors from a given set. Understanding how to combine vectors and what region of space they can collectively "reach" is crucial for grasping concepts like basis vectors, dimensionality, and even how features interact in machine learning models.

A linear combination of a set of vectors is a new vector formed by multiplying each vector by a scalar (a real number) and then adding the results. Think of it as mixing ingredients: each vector is an ingredient, and its scalar multiplier is the quantity. By adjusting these scalar quantities, we can create an infinite variety of new vectors. This operation is foundational because it describes how vectors can be expressed in terms of others, revealing underlying relationships within a vector space.

In a 2D plane, scalar multiplication stretches or shrinks a vector, potentially reversing its direction if the scalar is negative. Vector addition then places the tail of the second scaled vector at the head of the first, with the resultant vector extending from the origin to the head of the final vector. This geometric interpretation helps us see how different choices of scalars $c_1$ and $c_2$ allow us to "reach" various points in the plane. The resultant vector is simply the coordinates of that final point.

The concept of linear combinations extends naturally to any number of vectors and any dimension. For example, in 3D space, we can combine three 3D vectors. If these vectors are linearly independent (meaning none can be formed as a linear combination of the others), they can form any point in that 3D space. This ability to construct any vector from a set of base vectors is fundamental to understanding coordinate systems and basis vectors.

The span of a set of vectors is the collection of all possible vectors that can be formed by taking linear combinations of those vectors. It represents the entire geometric space or subspace that the given vectors can "reach" or "generate." If you imagine a set of vectors as a set of tools, their span is everything you can build with those tools. Understanding span is critical for defining vector spaces and determining the dimensionality of data, which directly impacts how efficiently we can represent and process information.

If you have only one non-zero vector, say $\mathbf{v}$, its span is simply all possible scalar multiples of $\mathbf{v}$. Geometrically, this forms a line passing through the origin and extending infinitely in both directions along the path of $\mathbf{v}$. Any point on this line can be reached by choosing an appropriate scalar. For example, if $\mathbf{v} = [1, 1]$, its span includes $[2, 2]$, $[-3, -3]$, and $[0.5, 0.5]$—all points on the line $y=x$.

When you have two vectors that are not collinear (meaning one is not a scalar multiple of the other), their span expands beyond a line. In 2D space, two non-collinear vectors can form any point in the entire 2D plane. In 3D space, two non-collinear vectors will span a plane that passes through the origin. This is because their independent directions allow them to "stretch" and "reach" across a two-dimensional surface.

If you have a set of vectors where all vectors are collinear (i.e., they all lie on the same line through the origin), adding more vectors to the set will not increase their span. Even if you have ten collinear vectors, their combined linear combinations will still only form a line. This is because each additional vector is essentially redundant; it doesn't introduce a new direction that allows you to reach points off the existing line. This concept is closely related to linear dependence.

The concept of span is intrinsically linked to the dimension of a vector space. The dimension of a vector space is the minimum number of linearly independent vectors required to span that space. These vectors form a basis for the space. For instance, to span a 2D plane, you need at least two non-collinear vectors. To span 3D space, you need at least three non-coplanar vectors. In data science, understanding the span of your feature vectors helps identify the true dimensionality of your data, which is crucial for techniques like Principal Component Analysis (PCA) that aim to reduce dimensionality while preserving the data's essential "reach."