An embedding is just a vector

Ask what a token embedding is and you'll usually get something like "it captures the meaning of a word as a vector." That sounds profound and tells you nothing you can act on.

Here is the mechanical version: an embedding is a row of a matrix, looked up by an integer. That's it. Everything interesting comes from what training does to those rows, not from the lookup.

If you know what a vector is, you already have the tools to follow the whole thing.

From text to an integer

Before anything mathematical happens, the tokeniser turns your text into integers:

"data science"  →  ["data", " science"]  →  [1223, 8622]

Those numbers aren't meaningful. Token 8622 isn't twice as anything as token 4311. They're just addresses.

The embedding matrix

The model holds a matrix $E$ with one row per token in its vocabulary:

With a 50,000-token vocabulary and $d = 768$, that's a 50,000 × 768 matrix — about 38 million numbers, all learned during training.

"Embedding token 8622" means taking row 8622. A vector of 768 numbers — the whole idea behind representing a word as a vector.

You will often see this written as a matrix multiplication instead:

where $\mathbf{o}_t$ is a one-hot vector — all zeros except a 1 at position $t$. Multiply that by $E$ and you get row $t$ back. It's the same operation, written so it composes with the matrix multiplications that follow. Nobody implements it that way; it would be 50,000 multiplications to fetch one row — and a one-hot vector on its own has problems of its own that embeddings exist to solve.

What is actually in the vector

Nothing interpretable. Dimension 400 is not "formality" and dimension 12 is not "plural."

The vectors start as random noise. Training nudges them so that tokens appearing in similar contexts drift towards similar directions, because that's what reduces the loss. The structure is a side effect of prediction, not something anyone designed — a bet formalised as the distributional hypothesis.

Which means an individual embedding, on its own, means nothing at all. All the information is in how it sits relative to the others — and that is a question about vector spaces, not about individual numbers.

Why similarity is about direction

Two embeddings are compared with cosine similarity:

Dividing by both lengths throws magnitude away and leaves only the angle between them. That's deliberate: magnitude in an embedding tracks things like how often a token appeared in training, which has nothing to do with whether two words mean similar things.

Normalise both vectors to length 1 first and the denominator becomes 1, so the cosine is the dot product. This is exactly why vector search libraries store unit vectors — it turns every similarity query into a single dot product.

The famous arithmetic, and why it's oversold

You've seen it:

That's vector addition and subtraction, nothing more. Subtracting man from king gives a direction — loosely "royalty minus maleness" — and adding it to woman lands somewhere near queen.

It genuinely works, and it's genuinely oversold. Two things are usually left out:

The three input words are excluded from the search. Without that exclusion the nearest neighbour to the result is very often just king again, because you haven't moved far. Much of the magic is in the exclusion rule.

It works for some relations and badly for others. Gender and capital-city analogies work well. Most others don't. The demo survived because the examples that work are the ones people repeat.

It's still worth knowing, because it shows the space has directional structure — some directions correspond to consistent semantic changes. That's real. It just isn't a reliable reasoning engine.

Where the linear algebra earns its keep

Two things follow that are easy to miss.

Scaling a vector doesn't change what it means. Multiplying by a scalar stretches a vector without rotating it, so a normalised embedding points the same way as the original. That is the entire justification for normalising before you store embeddings in a vector database — you're discarding magnitude precisely because it carries no meaning.

Everything downstream is a linear combination of these vectors. When attention produces its output, it is computing a weighted sum of value vectors — the weights come from the softmax, but the operation is a linear combination. When the final layer predicts a token, it's a matrix multiplication against the vocabulary. The embedding layer sets up a space, and the rest of the network moves around inside it.

One thing that confuses almost everyone

The embedding layer is context-free.

Row 8622 is the same 768 numbers whether the sentence is "river bank" or "investment bank." The lookup has no idea which one you meant.

Context arrives later, from attention, which mixes each token's vector with the vectors around it. By the middle layers the representation of bank in those two sentences has diverged sharply — but that divergence is attention's doing, not the embedding table's.

This is the difference between static embeddings and contextual ones (what a transformer produces at its later layers). The embedding matrix is static; the model is not. The static family is worth knowing on its own terms — Word2Vec learns by prediction, in either the skip-gram or CBOW direction, while GloVe factorises a global co-occurrence matrix instead.

The takeaway

There's no special mathematics in an embedding. A vocabulary-sized matrix, a row lookup, and a lot of training.

What makes it work is that a vector space is a genuinely good place to put meaning: you can measure similarity with an angle, move in directions, and combine things linearly — and all of those operations already have well-understood behaviour.

If the vector operations underneath felt shaky, that's the thing worth shoring up, because the rest of the stack is built entirely from them:

• Word Embeddings — the full topic, from the distributional hypothesis through Word2Vec and GloVe
• Vector definition — the arrow and the list of numbers
• Vector length and unit vectors — why normalising is the default
• Linear combination and span — the operation attention is built from
• Matrices — where the embedding table actually lives