Matrix Multiplication

As a data scientist building a recommendation engine, you might first apply a feature scaling transformation to normalize user ratings. Following this, you could apply a dimensionality reduction transformation to extract key patterns from the scaled data. Both of these operations are represented mathematically by matrices.

But does the order of these transformations matter? What if you apply dimensionality reduction before scaling the features? Understanding matrix multiplication reveals how these sequential operations combine and why their order can drastically change the final outcome.

Beyond Element-Wise: The Core Idea

Unlike scalar multiplication, which simply scales each element of a matrix, matrix multiplication is not an element-wise operation. Instead, it represents a more complex interaction, combining rows from the first matrix with columns from the second. Think of it as applying a series of weighted sums.

Each element in the resulting matrix is a product of this interaction, reflecting how the components of one transformation influence the components of another. This fundamental difference is what allows matrix multiplication to model complex sequential operations, like the data transformations in our recommendation engine.

When Can You Multiply? Compatibility Rules

For two matrices, say matrix A and matrix B, to be multiplied, they must satisfy a specific dimension compatibility rule. The number of columns in the first matrix (A) must exactly match the number of rows in the second matrix (B). If this condition is not met, multiplication is undefined.

If matrix A has dimensions $m \times n$ (m rows, n columns) and matrix B has dimensions $n \times p$ (n rows, p columns), their product C will have dimensions $m \times p$ . The inner dimensions ( $n$ ) must match, and the outer dimensions ( $m$ and $p$ ) determine the size of the result.

Check Your Understanding

If matrix P is

4 \times 3

and matrix Q is

3 \times 5

, what are the dimensions of P Q?

The Row-Column Dot Product

The core mechanism of matrix multiplication relies on the dot product of vectors. To find an element at position $(i, j)$ in the resulting product matrix C, you take the $i$ -th row of the first matrix (A) and the $j$ -th column of the second matrix (B).

You then multiply corresponding elements from this row and column and sum these products. This sum becomes the single value at $C_{ij}$ . This process is repeated for every position in the product matrix, systematically combining each row of A with each column of B.

Calculating a Matrix Product (2x2 Example)

Define Matrices

Let's multiply two

2 \times 2

matrices, A and B, to get matrix C. We'll find each element of C one by one.

A = [[1, 2],
     [3, 4]]

B = [[5, 6],
     [7, 8]]

C = [[c11, c12],
     [c21, c22]]

Calculate C₁₁

To find the element in the first row, first column (

C_{11}

), we take the dot product of the first row of A and the first column of B. Multiply corresponding elements and sum them.

C₁₁ = (A[0][0] * B[0][0]) + (A[0][1] * B[1][0])
C₁₁ = (1 * 5) + (2 * 7)
C₁₁ = 5 + 14
C₁₁ = 19

Calculate C₁₂

For the element in the first row, second column (

C_{12}

), we use the first row of A and the second column of B.

C₁₂ = (A[0][0] * B[0][1]) + (A[0][1] * B[1][1])
C₁₂ = (1 * 6) + (2 * 8)
C₁₂ = 6 + 16
C₁₂ = 22

Calculate C₂₁

For the element in the second row, first column (

C_{21}

), we use the second row of A and the first column of B.

C₂₁ = (A[1][0] * B[0][0]) + (A[1][1] * B[1][0])
C₂₁ = (3 * 5) + (4 * 7)
C₂₁ = 15 + 28
C₂₁ = 43

Calculate C₂₂

Finally, for the element in the second row, second column (

C_{22}

), we use the second row of A and the second column of B.

C₂₂ = (A[1][0] * B[0][1]) + (A[1][1] * B[1][1])
C₂₂ = (3 * 6) + (4 * 8)
C₂₂ = 18 + 32
C₂₂ = 50

Resulting Matrix C

Combining all calculated elements gives us the product matrix C.

C = [[19, 22],
     [43, 50]]

The Formal Definition

📐 Matrix Multiplication Formula

Given an $m \times n$ matrix A and an $n \times p$ matrix B, their product C is an $m \times p$ matrix where each element $C_{ij}$ is calculated as:

C_{ij} = \sum_{k=1}^{n} A_{ik} B_{kj}

This means the element in the $i$ -th row and $j$ -th column of C is the sum of the products of corresponding elements from the $i$ -th row of A and the $j$ -th column of B.

Matrix Product

The result of multiplying two matrices, A and B, where the element at row

i

and column

j

of the product matrix is the dot product of the

i

-th row of A and the

j

-th column of B.

Example: If matrix A is a

2 \times 3

matrix and matrix B is a

3 \times 4

matrix, their matrix product C will be a

2 \times 4

matrix.

Order's Impact: Non-Commutativity

A critical property of matrix multiplication is its non-commutativity. This means that for two matrices A and B, the product A B is generally not equal to B A. This differs significantly from scalar multiplication, where $2 \times 3$ is always the same as $3 \times 2$ .

This non-commutative property directly relates to our opening scenario. Applying a scaling transformation followed by dimensionality reduction (A B) is not the same as applying dimensionality reduction followed by scaling (B A). The sequence of operations fundamentally changes the final transformed data, impacting the recommendation engine's output.

pythonDemonstrating Non-Commutativity with NumPy

Check Your Understanding

If matrix A represents a rotation and matrix B represents a scaling, what does A B = B A imply?

Multiplying Matrices in Python

pythonUsing NumPy for Matrix Multiplication

Try It Yourself

Using the matrices from the previous example, calculate matrix_Y @ matrix_X. What are the dimensions of the result, and is it the same as matrix_X @ matrix_Y?

python

Real-World Impact: Transformations and Networks

Matrix multiplication is a cornerstone in many computational fields. In computer graphics, it's used extensively for linear transformations like scaling, rotation, and translation of 3D objects. Each transformation can be represented by a matrix, and applying multiple transformations sequentially involves multiplying these matrices.

In machine learning, particularly in neural networks, matrix multiplication is the primary operation within each layer. When an input vector passes through a layer, it's multiplied by the layer's weight matrix, and a bias vector is added. This process transforms the input into a new representation, allowing the network to learn complex patterns and make predictions.

Key Takeaways

Matrix multiplication is a fundamental operation that combines rows of the first matrix with columns of the second, not an element-wise product.
For matrices A ( $m \times n$ ) and B ( $n \times p$ ), multiplication is only possible if the number of columns in A equals the number of rows in B.
The resulting product matrix C will have dimensions $m \times p$ .
Each element $C_{ij}$ is computed as the dot product of the $i$ -th row of A and the $j$ -th column of B.
Matrix multiplication is generally non-commutative, meaning A B is typically not equal to B A.
This operation is vital for modeling sequential linear transformations in fields like computer graphics and forms the computational core of neural networks, where the order of operations significantly impacts the outcome.

← All lessons in Linear Algebra: Matrices