Embeddings
Beginner 12 min read Embeddings

Embedding Basics

Meaning as coordinates

Embeddings map text to fixed-length vectors where meaning shapes geometry: similar text lands near each other. They power retrieval, clustering, dedup and classifiers without retraining a model per task.

embeddingvectorsemantic spacetext representationfixed-length features
Prerequisites: What a Neural Network Computes
Before we start

What you will learn

  • Define an embedding in one path
  • Explain why geometry encodes meaning
  • Pick when to use embeddings vs the LLM alone
Introduction

What is this?

An embedding model turns text into a fixed-size numeric vector, trained so that semantically related inputs are geometrically close (cosine or dot similarity). Embeddings are the substrate for semantic search, clustering, dedup and lightweight classifiers that never call a generate-capable model.

Motivation

Why do you need it?

  • Fixed-size vectors make text comparable in O(1) algebra instead of string matching.
  • Semantic similarity generalizes past exact keywords — 'exploding memory' matches 'OOM killing processes'.
  • Anyone can afford millions of embeddings; tokens-of-text cost mostly at index time.
  • Clustering/dedup/classifiers become standard vector math.
Think about it

Mental model

Imagine

A coat pegboard sorted by topic

Think of

Embedding space

Coats hang where their topic lives: all 'authentication' coats near each other, 'deploy' coats in another nook. Hanging a new coat where it fits is retrieval; counting how many hang near is clustering; sharing a peg is a near-duplicate signal.

Mechanics

How it works

  • Text → tokenizer → model → mean-pooled (or CLS) hidden state → vector.
  • Training pushes similar pairs together and dissimilar pairs apart (contrastive loss).
  • Similarity = cosine (L2-normalized dot) or Euclidean distance between vectors.
  • Fixed dims (e.g. 256, 768, 1536) scale dimensions vs cost and storage.
  • Different models produce different spaces — never mix vectors across embedding models.

From text to geometry.

Text
Tokenize
Model
Pool
Space
Hands-on

Examples

Cosine similarity by hand

Cosine similarity by hand
import numpy as np
 
def cosine(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
 
# pretend embeddings
e_retry   = np.array([0.9, 0.1])
e_backoff = np.array([0.8, 0.2])
e_billing = np.array([0.1, 0.9])
 
print("retry~backoff:", round(cosine(e_retry, e_backoff), 2))
print("retry~billing:", round(cosine(e_retry, e_billing), 2))

Cosine ∈ [-1, 1]; 1 = same direction. Semantic neighbors score high.

Real world

In production

  • RAG retrieval indexes document chunk embeddings at write time and queries them at read.
  • Dedup pipelines cluster embeddings to collapse near-duplicate support tickets/docs.
  • Intent routing in LLM apps classifies user input via nearest-neighbor embeddings.
Guidelines

Best practices

  • Use ONE embedding model per space, everywhere.
  • Normalize vectors; store and compare consistently.
  • Batch and cache embeddings at ingestion, not per query.
  • Verify similarity thresholds against labeled pairs before trusting them.
Watch out

Common mistakes

  • Mixing vectors from different embedding models (incomparable).
  • Skipping normalization, then thresholds mean nothing.
  • Judging a document by one coarse chunk embedding instead of many precise ones.
Root cause analysis

Common issues

Error
Search returns topic-adjacent-but-wrong item
Root cause

Over-coarse chunks or wrong similarity metric

Fix

Chunk finer, use cosine, and pick the top-k with a strict threshold.

Error
Two models' vectors concatenated
Root cause

Mixed embedding providers

Fix

Standardize on one model; recompute on change.

When things break

Debugging tips

  • Bad neighbors in search → chunk quality or threshold; inspect a labeled pair matrix.
  • O(1) geometry looks odd multiclass → embeddings capture topic, not always nuance; add classifier head if needed.
  • Dim mismatch in a pipeline → two different embedding models somewhere.
Interviews

Interview checkpoint

Practice

Hands-on lab

Embedding Geometry by Hand
25 min · beginner
In progress
Objectives
  • Run cosine similarity from first principles on tiny vectors
  • See related pairs score above unrelated pairs
  • Turn a cosine threshold into a retrieval decision

Embedding Geometry by Hand

25 min · beginner

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Embedding geometry is designed so similar text is…

2

Vectors from different embedding models are interchangeable.

Go deeper

Official documentation

Embeddings guide (OpenAI)
docs
Embeddings (Hugging Face)
docs