Skip-gram Model

When a computer processes text, how does it discern that 'king' relates to 'queen' in the same way 'man' relates to 'woman'? The Skip-gram model addresses this by learning word representations where words with similar contexts are positioned closely in a vector space.

Defining 'Context': The Sliding Window

To understand how words relate, the Skip-gram model first needs to define what constitutes a word's context. It achieves this using a sliding window mechanism. This window moves across a sentence, identifying a central target word and all the words immediately surrounding it within a predefined range.

Generating Word Pairs with a Context Window

Loading diagram...

A sliding window moves across the text, identifying a target word and its surrounding context words to create training pairs.

Predicting Neighbors: The Skip-gram Idea

At its core, the Skip-gram model operates on a simple yet powerful idea: given a specific word, it tries to predict the words that are likely to appear in its surrounding context. By repeatedly performing this prediction task across a large corpus of text, the model implicitly learns meaningful word embeddings. These embeddings are dense vector representations where words with similar meanings or contexts are mapped to nearby points in a high-dimensional space.

Maximizing Likelihood: The Training Goal

During training, the Skip-gram model's objective is to maximize the probability of observing the actual context words given a target word. This means the model adjusts its internal weights, which form the word embeddings, so that words frequently appearing together in the training data have higher predicted probabilities. The model essentially learns by making its predictions align as closely as possible with the real-world co-occurrence patterns of words.

📐 Skip-gram Objective Function

The Skip-gram model aims to maximize the log-likelihood of observing context words $w_{t+j}$ given the target word $w_t$ over the entire training corpus $T$ :

\sum_{t=1}^T \sum_{-c \le j \le c, j \ne 0} \log P(w_{t+j} | w_t)

Here, $c$ is the size of the context window, and $P(w_{t+j} | w_t)$ is typically computed using a softmax function over the vocabulary.

Scaling Up: The Role of Negative Sampling

A significant computational challenge in training Skip-gram arises from the softmax function in the output layer. Calculating the probability for every word in a large vocabulary for each training step is extremely inefficient. Negative sampling provides an elegant solution by transforming the multi-class classification problem into a set of binary classification tasks. Instead of predicting all context words, it trains the model to distinguish between a few actual context words (positive samples) and a small number of randomly chosen non-context words (negative samples).

Check Your Understanding

What is the primary goal of the Skip-gram model during training?

Preparing Training Data for Skip-gram

pythonPython Example: Generating Pairs

Try It Yourself

Modify the generate_skipgram_pairs function to use a window_size of 1 instead of 2. Observe how the number and specific pairs change.

python

Key Takeaways

The Skip-gram model learns word embeddings by training a neural network to predict surrounding context words given a target word.
A sliding window mechanism is used to generate (target, context) pairs from raw text, forming the training data.
The model's architecture is a simple neural network with an input layer (one-hot target), a hidden layer (the embedding), and an output layer (context word probabilities).
Training involves maximizing the log-likelihood of observing actual context words, effectively adjusting embeddings to reflect co-occurrence patterns.
Negative sampling addresses the computational cost of softmax over large vocabularies by converting the task into binary classification.
By capturing these predictive relationships, Skip-gram enables computers to understand and represent complex semantic relationships between words.

← All lessons in Word Embeddings