# Embeddings

> Turn text into vectors with an embedding model, measure similarity with cosine math, and build a small semantic search.

AI Engineer roadmap · Building with LLMs · 18 min read · https://levelcode.org/roadmaps/ai-engineer/embeddings

Keyword search fails the moment users phrase things differently from your documents: "forgot my login" won't match a help article titled "Resetting your password". **Embeddings** fix this by turning text into lists of numbers where *meaning* determines position, so texts about the same thing end up close together even when they share no words. Embeddings power semantic search, retrieval for RAG, recommendations, clustering and de-duplication. This lesson explains what embeddings are, how the models that produce them are trained, the similarity maths (with a worked example), and how to embed and search text in Python with a free, local, open-source model.

## What an embedding is

An **embedding** is a fixed-length vector of floating-point numbers that represents a piece of text. An embedding model is a function:

```text
"How do I reset my password?"  ->  [0.021, -0.087, 0.044, ..., 0.013]   (384 numbers)
```

The individual numbers mean nothing on their own; you can't point at dimension 17 and call it "about passwords". What matters is **geometry**. The model is trained so that texts with similar meaning get vectors pointing in similar directions:

- "How do I reset my password?" and "I forgot my login credentials" end up close together.
- "What's the weather in Lisbon?" ends up far from both.

Every text you embed with the same model gets a vector of the same length, whether it is three words or three paragraphs. That is what makes comparison possible: you can measure how close any two vectors are, quickly and cheaply, across millions of documents.

Embeddings are not limited to text. Images, audio and code have embedding models too, and some models place text and images in the same space so you can search photos with words. The ideas in this lesson apply to all of them.

## How embedding models are made

Embedding models are usually **encoder** transformers, a cousin of the generative LLMs from earlier lessons. Instead of predicting the next token, an encoder reads the whole input at once and produces a vector for every token. A **pooling** step, commonly averaging those token vectors, turns them into one vector for the whole text.

The model learns useful geometry through **contrastive training**. It sees huge numbers of pairs known to be related: a question and its accepted answer, a title and its article, two paraphrases. Training pulls the vectors of related pairs together and pushes unrelated texts apart. After enough examples, "similar meaning means nearby" holds for text the model has never seen.

Two consequences matter in practice:

1. **Embedding models are small and cheap** compared to chat models. The model used in this lesson, `all-MiniLM-L6-v2`, has about 22 million parameters, produces 384-dimensional vectors, and runs comfortably on a laptop CPU. Larger models produce longer vectors (768, 1,024 or more dimensions) and usually retrieve better, at the cost of speed and storage.
2. **"Similar" means whatever the training data taught.** Most general models capture *topic and intent*. They are not fact checkers. "The medication is safe for children" and "The medication is not safe for children" are about the same topic, so their vectors are very close, even though they mean opposite things.

> [!NOTE]
> Anthropic doesn't offer an embeddings endpoint. Several other providers sell hosted embedding APIs, and there are many strong open-source models you can run yourself. The concepts and the maths are identical whichever you choose.

## Measuring similarity: cosine maths

Given two vectors, how close are they? The standard measure is **cosine similarity**, the cosine of the angle between them:

```text
cos(a, b) = (a · b) / (|a| × |b|)

where  a · b = a1*b1 + a2*b2 + ... + an*bn     (dot product)
       |a|   = sqrt(a1² + a2² + ... + an²)      (length, or magnitude)
```

The result ranges from 1 (same direction) through 0 (unrelated, at right angles) to -1 (opposite directions). Because it only measures *direction*, a long text and a short text about the same thing can still score high.

To build intuition, play with a two-dimensional space first. Drag the query (or click a word to make it the query) and watch the ranking change with the *angle* between vectors, not their distance apart. Then drag the query straight towards the origin: its length shrinks, but no cosine moves.

> **Interactive: Cosine similarity in a toy 2-D space.** Words with similar meaning point in similar directions, so the top-k by cosine usually come from the cluster the query points into. Try it: https://levelcode.org/labs/embedding-space

Here is a worked example with three-dimensional vectors. Real embeddings have hundreds of dimensions, but the arithmetic is the same:

```text
a = [1, 2, 2]     b = [2, 1, 2]     c = [-1, 0, 1]

a · b = 1*2 + 2*1 + 2*2 = 2 + 2 + 4 = 8
|a|   = sqrt(1 + 4 + 4) = sqrt(9) = 3
|b|   = sqrt(4 + 1 + 4) = sqrt(9) = 3
cos(a, b) = 8 / (3 × 3) = 0.889        -> very similar

a · c = 1*(-1) + 2*0 + 2*1 = -1 + 0 + 2 = 1
|c|   = sqrt(1 + 0 + 1) = sqrt(2) ≈ 1.414
cos(a, c) = 1 / (3 × 1.414) ≈ 0.236     -> not very similar
```

The same calculation in NumPy:

```python
import numpy as np

def cosine(a: np.ndarray, b: np.ndarray) -> float:
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

a, b, c = np.array([1, 2, 2]), np.array([2, 1, 2]), np.array([-1, 0, 1])
print(round(cosine(a, b), 3), round(cosine(a, c), 3))
```

```text
0.889 0.236
```

### Normalise once, then just use the dot product

Dividing a vector by its length gives a **unit vector** with length 1. For `a`, that is `[1/3, 2/3, 2/3] ≈ [0.333, 0.667, 0.667]`. When both vectors have length 1, the denominator of the cosine formula is `1 × 1`, so **cosine similarity equals the dot product**. That's why systems normalise every embedding when it is stored. Comparing a query against a million documents then takes one matrix multiplication.

For unit vectors, Euclidean distance and cosine carry the same information: `|a − b|² = 2 − 2·cos(a, b)`. Ranking by smallest distance or largest cosine gives the same order, which is why vector databases let you choose either.

> [!WARNING]
> There is no universal threshold for "similar enough". A cosine of 0.5 might be a strong match for one model and noise for another. Pick thresholds by looking at scores on *your* data with *your* model, and re-check them whenever you change models.

## Embedding text in Python

The `sentence-transformers` library runs open-source embedding models locally. Install it (it pulls in PyTorch) and embed a few sentences:

```bash
pip install sentence-transformers
```

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")  # downloads about 90 MB on first run, then cached

sentences = [
    "How do I reset my password?",
    "I forgot my login credentials.",
    "What's the weather like in Lisbon today?",
]
embeddings = model.encode(sentences, normalize_embeddings=True)

print(embeddings.shape)
print((embeddings @ embeddings.T).round(2))  # unit vectors, so dot product = cosine
```

You should see output roughly like this. Exact values vary slightly between library versions and hardware, but the pattern is stable:

```text
(3, 384)
[[1.   0.6  0.05]
 [0.6  1.   0.04]
 [0.05 0.04 1.  ]]
```

`encode` takes a list of strings and returns a NumPy array with one row per input. Passing many texts in one call is much faster than calling it once per text, because they are processed in batches. The two account questions score high despite sharing almost no words. The weather question scores near zero against both. The diagonal is 1 because every vector is identical to itself.

After the first download, the model runs entirely on your machine: no API key, no per-call cost, and no data leaving your network. In JavaScript, **Transformers.js** (`@huggingface/transformers`) runs the same model in Node or the browser:

```ts
import { pipeline } from "@huggingface/transformers";

const embed = await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2");
const output = await embed(["How do I reset my password?"], { pooling: "mean", normalize: true });
console.log(output.dims); // [1, 384]
```

## Semantic search in a few lines

Semantic search follows two phases. **Indexing** embeds every document once and stores the vectors. **Querying** embeds the query, scores it against every document vector, and returns the best matches:

```python
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

docs = [
    "To reset your password, click 'Forgot password' on the sign-in page.",
    "Invoices are emailed on the first day of each month.",
    "You can change your plan at any time from Settings > Billing.",
    "Our mobile app is available for iOS and Android.",
    "Two-factor authentication can be enabled under Security settings.",
]
doc_vectors = model.encode(docs, normalize_embeddings=True)  # indexing: once

def search(query: str, k: int = 2) -> list[tuple[float, str]]:
    query_vector = model.encode([query], normalize_embeddings=True)[0]
    scores = doc_vectors @ query_vector          # cosine similarity with every doc
    best = np.argsort(-scores)[:k]               # indices of the top-k scores
    return [(round(float(scores[i]), 2), docs[i]) for i in best]

for score, doc in search("I can't log in to my account"):
    print(score, doc)
```

A typical run puts the password-reset article first and the two-factor article second. Neither contains the word "log", but both are about getting into an account. That is the advantage over keyword search. Its weakness is the other side: exact identifiers like error codes, SKUs or names are often better matched by keywords, which is why production systems frequently combine both (*hybrid search*).

This brute-force approach, scoring every document, is perfectly fast for thousands or even tens of thousands of documents. Beyond that you need an index built for vectors, which is the subject of the next lesson.

## Using embeddings well

- **Use one model for everything in an index.** Vectors from different models live in unrelated spaces. Comparing them gives meaningless scores, and so do vectors from two versions of the same model. Store the model name next to every vector, and re-embed the whole corpus when you switch.
- **Mind the input limit.** Every model has a maximum input length; `all-MiniLM-L6-v2` truncates input beyond 256 word pieces (roughly 200 English words), silently ignoring the rest. Long documents must be split into **chunks** that are each embedded separately, as the RAG lesson covers.
- **Check whether the model expects prefixes.** Some models are trained for *asymmetric* search, with short queries and long passages, and expect prefixes such as `query: ` and `passage: `, or a task instruction. Using them where expected, and only there, noticeably improves results. The model's documentation says which applies.
- **Match the model to the data.** For non-English or mixed-language text, choose a multilingual model. Domain-heavy text (legal, medical, code) may need a model trained on that domain.
- **Plan storage.** One million 384-dimensional float32 vectors take `1,000,000 × 384 × 4 bytes ≈ 1.5 GB`. At 1,536 dimensions, it's about 6.1 GB. Dimension count drives memory, disk and search speed.
- **Remember what similarity isn't.** High similarity means "about the same thing", not "true", "relevant to the user's intent" or "agrees". Negations, numbers and dates are weak spots. A retrieval step should feed a model or a reranker that reads the text, not make final decisions on its own.

## Practice

Build an FAQ matcher. Store at least five FAQ entries as question/answer pairs. Embed only the **questions**, since user queries resemble questions more than answers. Then write `answer(query)` that:

1. returns the answer of the most similar FAQ question if its cosine similarity is at least a threshold;
2. otherwise returns "Sorry, I don't know that one. A human will get back to you.";
3. prints the best score, so you can calibrate the threshold.

Try it on a paraphrase ("my card got charged twice"), an unrelated query ("best pizza in Naples?"), and one more of your own. Pick a threshold from the scores you observe.

**Solution**

```python
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

FAQ = [
    ("How do I reset my password?", "Use 'Forgot password' on the sign-in page; a reset link arrives by email."),
    ("Why was I charged twice?", "Duplicate charges are usually pending authorisations that drop off within 3 days."),
    ("How do I cancel my subscription?", "Go to Settings > Billing > Cancel plan. You keep access until the period ends."),
    ("Can I export my data?", "Yes. Settings > Privacy > Export creates a ZIP of your data within an hour."),
    ("Do you offer student discounts?", "Students get 50% off with a valid university email address."),
]
questions = [q for q, _ in FAQ]
question_vectors = model.encode(questions, normalize_embeddings=True)

THRESHOLD = 0.5  # calibrated by looking at printed scores, specific to this model

def answer(query: str) -> str:
    query_vector = model.encode([query], normalize_embeddings=True)[0]
    scores = question_vectors @ query_vector
    best = int(np.argmax(scores))
    print(f"  best match {scores[best]:.2f}: {questions[best]!r}")
    if scores[best] < THRESHOLD:
        return "Sorry, I don't know that one. A human will get back to you."
    return FAQ[best][1]

for query in ["my card got charged twice", "best pizza in Naples?", "how can I download everything you have on me"]:
    print(query)
    print(" ", answer(query))
```

A typical run matches the double-charge paraphrase and the data-export paraphrase with high scores, while the pizza query's best score sits well below the others, so it gets the fallback. If a valid paraphrase falls below your threshold, lower it a little. If unrelated queries pass, raise it. Write down the scores you observed so the choice is reproducible.

Embedding only the questions works because a user's query and an FAQ question are the same *kind* of text. With longer documents you would embed the passages themselves, and a model trained for asymmetric query-to-passage search would do better.

## Key takeaways

- An embedding is a fixed-length vector whose *direction* encodes meaning, so similar texts get nearby vectors even without shared words.
- Embedding models are small encoder transformers trained contrastively. They capture topic and intent, not truth, and handle negation poorly.
- Cosine similarity is `a·b / (|a||b|)`. Normalise vectors once and it becomes a plain dot product.
- `sentence-transformers` with `all-MiniLM-L6-v2` runs locally and free. Batch your `encode` calls and pass `normalize_embeddings=True`.
- Semantic search means embedding documents once, embedding each query, ranking by similarity and taking the top-k. Calibrate thresholds on your own data.
- Never mix vectors from different models, chunk long text to fit the model's input limit, and store which model produced each vector.

## Quiz

1. Vectors a = [3, 4] and b = [6, 8]. What is their cosine similarity?

   A. 0.5
   B. 2.0
   C. 1.0
   D. 50

   **Answer: C.** b is exactly 2 x a, so they point in the same direction. a·b = 18 + 32 = 50, |a| = 5, |b| = 10, and 50 / (5 x 10) = 1. Cosine measures direction, not length.

2. You re-embedded half your documents with a newer embedding model, leaving the rest on the old model. Searches now return strange results. Why?

   A. Vectors from different models live in unrelated spaces, so similarities across them are meaningless. Re-embed everything with one model
   B. The newer model has a lower similarity threshold
   C. Embeddings expire after a few months
   D. Half the documents are now normalised and half aren't, which is the only problem

   **Answer: A.** Each model learns its own geometry. Even with the same dimension count, comparing an old-model vector with a new-model vector is like comparing coordinates from two different maps. Store the model name with each vector, and re-embed the corpus when you switch.

3. "The update fixed the crash" and "The update did not fix the crash" get a cosine similarity of 0.93. What does this show?

   A. The embedding model is broken
   B. Embeddings capture topic and wording similarity more than logical meaning, so negations can score as near-duplicates
   C. Cosine similarity is the wrong formula for text
   D. The vectors weren't normalised

   **Answer: B.** Both sentences are about the same update and the same crash, so their vectors are close. Embeddings are good at finding related text, but a model that reads the text is needed to judge what it actually says.

4. Your documents are 5,000-word reports, and your embedding model has a 256-token input limit. What should you do?

   A. Embed each report as is; the model reads all of it
   B. Increase the similarity threshold
   C. Use the dot product instead of cosine
   D. Split each report into chunks within the limit and embed each chunk separately

   **Answer: D.** Models silently truncate input beyond their limit, so most of each report would be ignored. Chunking makes every part of the document searchable. The RAG lesson covers chunking strategies.