NLP
Intermediate 16 min read NLP

Word Embeddings

Dense representations that encode meaning in geometry

Word2Vec, GloVe and fastText learn dense vectors where similar words sit close together and vector arithmetic captures analogy. Embeddings give models semantic generalization that counts cannot.

word2vecGloVefastTextCBOWskip-gramword vectorsanalogy
Prerequisites: Bag of Words and TF-IDF
Before we start

What you will learn

  • Explain how word2vec/Glove learn distributional embeddings
  • Use pretrained embeddings for downstream tasks
  • Handle OOV and multilingual limitations
Introduction

What is this?

Word embeddings map each word to a dense vector (typically 100–300 dims) such that distributionally similar words land near each other and relationships appear as vector offsets — famously king − man + woman ≈ queen. Word2Vec (skip-gram/CBOW) learns them by predicting neighbors; GloVe factorizes global co-occurrence counts; fastText adds subword info so rare or OOV forms still get vectors. Downstream models start from these semantic coordinates instead of raw counts.

Motivation

Why do you need it?

  • Bags of words ignore semantics; embeddings let 'car' and 'vehicle' share signal.
  • Similarity/analogy become arithmetically computable — a strong building block for search and labeling.
  • Embedding inputs let small datasets borrow knowledge from massive corpora (transfer).
  • They support approximating unseen relationships, out-of-train generalizations.
Think about it

Mental model

Imagine

A map of a city where nearby neighborhoods share profile

Think of

Embedding space

Semantically 'close' words are like adjacent blocks — 'restaurant' near 'cafe'. And just as you can navigate 'from airport to hotel then to a gym', word vectors support direction arithmetic because the geometry encodes the relationships.

Mechanics

How it works

  • Distributional hypothesis: words seen in similar contexts share meaning.
  • Word2Vec skip-gram: given the target word, predict context words within a window — the hidden layer's weights become the embeddings.
  • CBOW: the inverse — predict target from context bag.
  • GloVe: factorize a global co-occurrence matrix with weighted least squares.
  • Result: one vector per word — plug into classifiers, similarity search or as initialization for deep models.

Training contexts → vector geometry.

Corpus
Context windows
Learn
Vectors
Use
Real world

In production

  • Search indexes terms with pretrained Word2Vec to map queries to synonyms.
  • Customer-ticket clustering embeds only rare-domain vocabulary using training on that corpus (fine-tune).
  • A recommendation system initializes its embedding layers with domain word vectors before fine-tuning.
Guidelines

Best practices

  • Prefer pretrained embeddings covering your domain lexicon; train your own only when jargon dominates.
  • Use fastText/OOV-aware vectors when product or slang tokens appear live but not in training.
  • Normalize vectors (L2) before computing cosine similarities; store in a vector-index for scale.
  • Always validate embedding usefulness on the downstream task — cosine similarity alone isn't accuracy.
Watch out

Common mistakes

  • Training word2vec on 5MB of text then expecting quality — it needs corpus scale.
  • Treating embedding vectors as calibrated probabilities when feeding downstream layers.
  • Mixing vocabularies: a doc embedding from one model and token embeddings from another.
  • Assuming the same word vector works for sentiment, NER and search — each task rewires the geometry.
Root cause analysis

Common issues

Error
Nearest neighbors are all the same rare word
Root cause

Embedding undertrained or vocab too aggressive pruning

Fix

Grow the corpus/window, lower min_count, or switch to a larger pretrained model.

Error
Cosine similarity ignores sentiment ('good' ≈ 'awful')
Root cause

Distributional similarity ≠ sentiment polarity

Fix

Use task-fine-tuned embeddings or add a sentiment-adjusting layer on top.

When things break

Debugging tips

  • Sanity-test nearest neighbors: a well-trained embedding puts 'apple' near 'banana', not near 'company'.
  • Run the king-man+woman≈queen analogy check before trusting the space.
  • Large OOV share means either filter the vocab at the edge or switch to subword/fastText.
  • Anisotropy (all vectors near origin) is common — subtract the mean in batch or use background-fix tricks.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The distributional hypothesis states that:

2

Word2Vec embeddings know that 'good' is positive and 'bad' is negative.

Go deeper

Official documentation

gensim word2vec
docs
GloVe project
docs