CBOW Model

How does a neural network learn a rich vector representation for a word by predicting it from its surrounding context?
The Continuous Bag-of-Words (CBOW) model provides an elegant solution by training a shallow neural network to predict a target word based on its surrounding context words.

The Intuition: Context is King

The words that appear around a target word often provide strong clues about its meaning and grammatical role. For example, if you see the words "hot" and "coffee" frequently together, you can infer a relationship. The CBOW model leverages this fundamental idea: a word's meaning can be inferred from its neighbors.

Instead of trying to predict context from a target word, CBOW takes the surrounding words as input and attempts to predict the central word. This approach helps the model learn dense, numerical representations for words, capturing their semantic and syntactic properties based on how they are used in language.

The CBOW Architecture

The CBOW model operates with a simple, shallow neural network. It begins by taking a window of context words around a target word. Each of these context words is first converted into a one-hot encoded vector, a sparse representation where a single element is 1 and all others are 0, corresponding to its position in the vocabulary.

These one-hot vectors are then multiplied by an input weight matrix, effectively looking up their corresponding word vectors. In the projection layer, these individual context word vectors are averaged together to form a single, dense context vector. This averaging step is crucial as it creates a unified representation of the surrounding words, which then serves as the input for the next layer.

Word Embeddings

Dense, low-dimensional vector representations of words that capture semantic and syntactic relationships. Words with similar meanings or contexts are mapped to similar vectors in the embedding space.

Example: After training, the word "king" might have an embedding vector like

[0.2, -0.5, 0.8, 0.1]

and "queen" might be

[0.3, -0.4, 0.7, 0.2]

, showing their proximity in the vector space.

The combined context vector from the projection layer is then fed into the output layer. This layer uses a softmax function to produce a probability distribution over all words in the vocabulary. The goal is for the model to assign the highest probability to the actual target word that was originally surrounded by the input context words.

During training, the model adjusts its internal weights (which are the word embeddings themselves) to maximize the probability of predicting the correct target word. This process allows the model to learn meaningful vector representations that encode the contextual relationships between words.

Learning Word Relationships

📐 CBOW Loss Function

The CBOW model minimizes the negative log-likelihood of predicting the target word $w_O$ given its context words $w_I$ . This is typically achieved using the cross-entropy loss function:

L = -\log P(w_O | w_I) = -\log \left( \frac{\exp(v_{w_O}^T h)}{\sum_{w' \in V} \exp(v_{w'}^T h)} \right)

Here, $v_{w_O}$ is the output vector of the target word, $h$ is the averaged context vector from the projection layer, and $V$ is the entire vocabulary. The term $\sum_{w' \in V} \exp(v_{w'}^T h)$ normalizes the probabilities across all words.

CBOW Training Process

Input Context Words

Select a target word and its surrounding context words from the training corpus. Convert each context word into a one-hot encoded vector.

Compute Context Vector

Look up the vector representation for each context word from the input weight matrix. Average these vectors to create a single, dense context vector

h

in the projection layer.

Predict Target Word

Feed the context vector

h

through the output layer, which uses a softmax function to calculate the probability distribution of all words in the vocabulary being the target word.

Calculate Loss

Compare the predicted probability distribution with the actual one-hot encoded target word. Calculate the loss using the negative log-likelihood (cross-entropy) function.

Backpropagate Error

Propagate the calculated error backward through the network. This determines how much each weight (including the word vectors) contributed to the error.

Update Word Vectors

Adjust the input and output weight matrices (which contain the word embeddings) using an optimization algorithm like stochastic gradient descent to minimize the loss. This iterative update refines the word vectors.

Check Your Understanding

What is the primary role of the projection layer in the CBOW model?

Visualizing Learned Embeddings

Sample Word Embedding Matrix (5 Dimensions)

This heatmap illustrates a small portion of a hypothetical word embedding matrix. Each row represents a word, and each column represents a dimension in the embedding space. The color intensity indicates the numerical value of that dimension for the specific word, with darker shades often representing higher values and lighter shades representing lower or negative values.

Loading chart...

Key Insight: Word embeddings are dense numerical vectors where each dimension contributes to encoding semantic and syntactic properties, allowing words to be represented as points in a continuous space.

The heatmap above provides a glimpse into the structure of word embeddings. Each row is a word's embedding vector, and each column is a dimension. While individual dimensions might not have a human-interpretable meaning, the overall pattern of values within a vector is what matters.

Crucially, words with similar meanings or that appear in similar contexts tend to have similar embedding vectors. This means that in the high-dimensional space, "king" and "queen" would be close to each other, and "apple" and "banana" would also be close, but "king" and "apple" would be far apart. This proximity reflects the semantic relationships learned by the CBOW model.

CBOW vs. Skip-gram: A Quick Look

CBOW vs. Skip-gram Model Comparison

Feature	CBOW (Continuous Bag-of-Words)	Skip-gram
Input/Output	Predicts target word from context words	Predicts context words from target word
Training Speed	Generally faster for large corpora	Generally slower, especially with large window sizes
Performance on Rare Words	Less effective at representing rare words	Better at capturing representations for rare words
Context Handling	Averages context word vectors	Treats each context word independently

A comparison highlighting the key differences between the CBOW and Skip-gram models, two popular approaches for learning word embeddings.

Check Your Understanding

When might the CBOW model be a more suitable choice than Skip-gram?

Key Takeaways

The CBOW model learns word embeddings by predicting a target word from its surrounding context words.
Its architecture involves an input layer for context words, a projection layer that averages context vectors, and an output layer that uses softmax to predict the target word.
Training involves minimizing the negative log-likelihood (cross-entropy loss) of the target word given the context.
The model iteratively updates word vectors (embeddings) through backpropagation, refining their ability to capture semantic relationships.
Learned word embeddings are dense numerical vectors where words with similar meanings or contexts have similar vector representations.
CBOW is generally faster to train than Skip-gram, making it suitable for large corpora, though it may be less effective with rare words.

← All lessons in Word Embeddings