Linear Algebra: Vectors

In fields ranging from physics simulations to machine learning algorithms, quantities often require more than just a single number to describe them. Consider the force applied to an object, or the velocity of an airplane. These aren't just about "how much" (magnitude); they also involve "in what direction." This is where vectors become indispensable. They provide a precise mathematical tool to represent quantities that possess both magnitude and direction.

Geometrically, a vector is visualized as a directed line segment, or an arrow. The length of this arrow represents the vector's magnitude (or size), while the way the arrow points indicates its direction. A crucial aspect of this geometric definition is that a vector's identity is independent of its starting position. As long as its magnitude and direction remain the same, it's considered the same vector, regardless of where it originates in space.

While the geometric view is intuitive, for calculations and computational work, we represent vectors algebraically. An algebraic vector is an ordered list of numbers, called components, that describe the vector's displacement along each axis of a coordinate system. For instance, in a 2D Cartesian plane, a vector might be represented as $(x, y)$, where $x$ is the displacement along the x-axis and $y$ is the displacement along the y-axis. This representation allows us to perform precise mathematical operations.

The number of components in a vector determines its dimension. A vector with two components, like $[x, y]$, exists in a 2D space. A vector with three components, $[x, y, z]$, exists in 3D space. In machine learning, it's common to encounter vectors with hundreds or even thousands of components, representing high-dimensional spaces where each component might correspond to a feature or attribute. The principles remain the same, regardless of the number of dimensions.

The magnitude of an algebraic vector, often denoted as $||\mathbf{v}||$ or $|\mathbf{v}|$, is its length. For a 2D vector $\mathbf{v} = [x, y]$, we calculate its magnitude using the Pythagorean theorem: $\sqrt{x^2 + y^2}$. This concept extends to higher dimensions; for an n-dimensional vector $\mathbf{v} = [v_1, v_2, ..., v_n]$, the magnitude is $\sqrt{v_1^2 + v_2^2 + ... + v_n^2}$. This is also known as the Euclidean norm or L2 norm.

While magnitude is a single number, direction is more complex. For a 2D vector, direction is often described by the angle it makes with the positive x-axis. In higher dimensions, this becomes less intuitive. A more general way to represent a vector's direction is through a unit vector. A unit vector is a vector with a magnitude of 1 that points in the same direction as the original vector. You can obtain a unit vector by dividing each component of the original vector by its magnitude.

A position vector is a special type of vector that always originates from the origin of a coordinate system (e.g., $(0,0)$ in 2D, $(0,0,0)$ in 3D). Its components directly correspond to the coordinates of a specific point in space. For example, the position vector $[3, 2]$ points directly to the point $(3, 2)$. These are particularly useful for locating points and describing their displacement from a fixed reference point, such as in GPS systems or object tracking.

In contrast to position vectors, free vectors represent pure displacement or a physical quantity like force or velocity, independent of a specific starting location. As long as a free vector maintains its magnitude and direction, it is considered the same vector, regardless of where its tail is placed. The geometric visualization we saw earlier, with two identical vectors starting at different points, perfectly illustrates the concept of a free vector. They are useful for describing relative motion or forces acting on objects without needing to fix them to an origin.