Distributional Hypothesis
How can machines understand that 'king' and 'monarch' are similar, or that 'bank' has multiple meanings, without human intervention? This challenge is addressed by the Distributional Hypothesis, which posits that words appearing in similar contexts tend to have similar meanings. As J.R. Firth famously stated, "You shall know a word by the company it keeps."
Consider the word "bank." If you read "The bank of the river was overgrown," you immediately understand it refers to land beside water. However, if the sentence is "I deposited money at the bank," the meaning shifts to a financial institution. The words surrounding "bank"—"river" versus "deposited money"—provide the crucial clues that differentiate its meaning. This simple observation forms the bedrock of how machines learn word semantics.
Defining Context for Words
For the Distributional Hypothesis to be useful, we need a concrete way to define a word's "context." The most common approach involves a fixed-size window around a target word. This window includes a certain number of words immediately preceding and following the target word. For instance, a window size of two would capture two words before and two words after the target. More advanced methods can also consider syntactic dependencies, analyzing grammatical relationships between words rather than just their proximity.
| Sentence | Target Word | Context (Window Size 2) |
|---|---|---|
| Example 1 | The cat sat on the mat. | cat |
| Example 2 | She bought a new car yesterday. | bought |
| Example 3 | A quick brown fox jumps over the lazy dog. | quick |
From Context to Word Embeddings
The Distributional Hypothesis is the foundational principle behind modern word embeddings, which are dense vector representations of words. Algorithms like Word2Vec and GloVe analyze vast amounts of text to learn the contexts in which words appear. They then map each word to a point in a high-dimensional vector space, where the distance and direction between vectors capture semantic relationships. Words that share similar contexts, and thus similar meanings, will be positioned close to each other in this vector space.
The Distributional Hypothesis states that words appearing in similar contexts tend to have similar meanings.
The meaning of a word is not inherent but is derived from the company it keeps within text.
Context is typically defined by a fixed-size window of surrounding words, though more complex methods exist.
This hypothesis is the fundamental principle enabling the creation of word embeddings, where semantically similar words are represented by vectors that are close in a high-dimensional space.
Machines can understand word similarity and polysemy (multiple meanings) by analyzing and comparing the contexts in which words appear, without explicit human labeling.