GloVe Embeddings

How can we learn word embeddings that effectively combine local context with global co-occurrence statistics? GloVe embeddings address this by integrating both local window-based information and overall corpus statistics. Traditional methods like Word2Vec excel at capturing local context through skip-gram or CBOW models, while techniques such as Latent Semantic Analysis (LSA) focus on global patterns. GloVe synthesizes these two perspectives, aiming for richer semantic representations that leverage the strengths of both.

The Power of Co-occurrence Counts

Co-occurrence Probability

The co-occurrence probability

P(j|i)

is the likelihood that word

j

appears in the context of word

i

within a specified window size across a corpus. These probabilities are derived from a co-occurrence matrix, which tabulates how often words appear together.

Example: In the sentence "The quick brown fox jumps over the lazy dog," with a context window of 2, the word "quick" co-occurs with "The" and "brown." If "quick" appears 100 times in the corpus and 80 of those times "brown" is within its 2-word window, then

P(\text{brown}|\text{quick})

would be

0.8

(simplified).

The core intuition behind GloVe is that ratios of co-occurrence probabilities carry more semantic meaning than the probabilities themselves. Consider two words, 'ice' and 'steam,' and a set of probe words like 'solid,' 'gas,' and 'water.' The raw probability $P(\text{solid}|\text{ice})$ might be high, and $P(\text{gas}|\text{steam})$ also high. However, comparing $P(\text{solid}|\text{ice})$ to $P(\text{solid}|\text{steam})$ reveals a stark difference.

This ratio, $P(\text{solid}|\text{ice}) / P(\text{solid}|\text{steam})$ , would be very large, indicating 'solid' is strongly associated with 'ice' but not 'steam.' Conversely, $P(\text{gas}|\text{ice}) / P(\text{gas}|\text{steam})$ would be very small. For a word like 'water,' which relates to both, the ratio $P(\text{water}|\text{ice}) / P(\text{water}|\text{steam})$ would be close to 1. These ratios effectively encode the semantic relationships between words.

Co-occurrence Probability Ratios for 'Ice' vs. 'Steam'

Probe Word (k)	$P(k\|\text{ice})$	$P(k\|\text{steam})$	Ratio $P(k\|\text{ice}) / P(k\|\text{steam})$
solid	0.85	0.03	28.33
gas	0.02	0.78	0.03
water	0.60	0.55	1.09
fashion	0.01	0.01	1.00

This table illustrates how co-occurrence probability ratios reveal semantic distinctions. A high ratio (e.g., for 'solid') indicates a strong association with the numerator word ('ice'). A low ratio (e.g., for 'gas') indicates a strong association with the denominator word ('steam'). A ratio near 1 (e.g., for 'water' or 'fashion') suggests a balanced or unrelated association.

Check Your Understanding

If the ratio

P(\text{k}|\text{Word X}) / P(\text{k}|\text{Word Y})

is very close to zero, what does it imply about Word X and Word Y relative to probe word k?

The GloVe Objective Function

📐 GloVe Objective Function

The GloVe model minimizes the following cost function:

\sum_{i,j=1}^{V} f(X_{ij}) (w_i^T \tilde{w}_j + b_i + \tilde{b}_j - \log X_{ij})^2

Where:
-

V

is the size of the vocabulary.
-

X_{ij}

is the number of times word

i

and word

j

co-occur.
-

w_i

and

\tilde{w}_j

are the word vectors for word

i

and context word

j

, respectively.
-

b_i

and

\tilde{b}_j

are the bias terms for word

i

and context word

j

.
-

f(X_{ij})

is a weighting function that gives less weight to very rare or very frequent co-occurrences.

The GloVe objective function is designed to learn word vectors such that their dot product effectively predicts the logarithm of their co-occurrence probability. The term $w_i^T \tilde{w}_j$ represents the dot product between the word vector $w_i$ and the context word vector $\tilde{w}_j$ . This dot product is intended to approximate $\log P(j|i)$ , which is related to $\log X_{ij}$ .

Bias terms, $b_i$ and $\tilde{b}_j$ , are included to account for inherent biases of words, independent of their specific context. For instance, some words might be generally more frequent or appear in more diverse contexts. The weighting function $f(X_{ij})$ is crucial; it assigns higher weights to meaningful co-occurrences while down-weighting very rare pairs (which might be noise) and very frequent pairs (which might not carry much specific information, like stop words). This function typically increases monotonically but saturates for large $X_{ij}$ values.

GloVe vs. Word2Vec: A Head-to-Head

Comparing GloVe and Word2Vec

Feature	Word2Vec	GloVe
Input/Data Source	Local context windows (sentences)	Global co-occurrence matrix
Model Type	Predictive (neural network based)	Count-based (matrix factorization)
Information Captured	Local syntactic and semantic relationships	Global semantic relationships via co-occurrence ratios
Training Mechanism	Trains on individual word pairs/windows using SGD	Trains on non-zero entries of co-occurrence matrix using AdaGrad
Strengths	Captures nuanced local context, computationally efficient for large corpora	Leverages global statistics, often better for analogy tasks, faster training on smaller corpora
Weaknesses	Does not explicitly use global statistics, can be slower for very large vocabularies	Requires building a co-occurrence matrix first, which can be memory-intensive for huge vocabularies

GloVe and Word2Vec represent different philosophies in learning word embeddings, each with distinct advantages depending on the corpus size and specific task.

Check Your Understanding

Which embedding method would likely be more efficient for a very large corpus where global co-occurrence statistics are paramount for capturing nuanced semantic relationships?

Putting GloVe to Work

pythonLoading and Using Pre-trained GloVe Embeddings

Try It Yourself

Using the loaded GloVe model, find the top 3 most similar words to 'computer' and then perform the analogy 'Paris - France + Italy = ?' (expecting a city).

python

This code uses a deep-learning or other compiled library (e.g. PyTorch, TensorFlow, Keras, spaCy, Transformers) that has no in-browser build, so it can't run here. Google Colab has these preinstalled — with a free GPU — so you can run and experiment there for free.

Expected

You should see the top 3 similar words to 'computer' (e.g., 'software', 'technology', 'internet') and the analogy result 'Rome' (or a similar city).

Key Takeaways

GloVe learns word embeddings by leveraging global word-word co-occurrence statistics.
The core idea is that ratios of co-occurrence probabilities encode semantic relationships more effectively than raw probabilities.
The GloVe objective function minimizes the squared difference between the dot product of word vectors and the logarithm of their co-occurrence count, incorporating bias terms and a weighting function.
GloVe combines the benefits of local context (like Word2Vec) with global statistical information (like LSA) into a single model.
Compared to Word2Vec, GloVe is a count-based model that can be more efficient for certain tasks and often performs well on analogy tasks.
Pre-trained GloVe embeddings are readily available and can be used to find word similarities and perform semantic analogies, directly addressing how to combine local context with global co-occurrence statistics for rich word representations.

← All lessons in Word Embeddings

GloVe Embeddings

The Power of Co-occurrence Counts

The GloVe Objective Function

GloVe vs. Word2Vec: A Head-to-Head

Putting GloVe to Work

Ready to keep this from fading?