Embeddings

Keyword search fails the moment users phrase things differently from your documents: "forgot my login" won't match a help article titled "Resetting your password". Embeddings fix this by turning text into lists of numbers where meaning determines position, so texts about the same thing end up close together even when they share no words. Embeddings power semantic search, retrieval for RAG, recommendations, clustering and de-duplication. This lesson explains what embeddings are, how the models that produce them are trained, the similarity maths (with a worked example), and how to embed and search text in Python with a free, local, open-source model.

#What an embedding is

An embedding is a fixed-length vector of floating-point numbers that represents a piece of text. An embedding model is a function:

"How do I reset my password?"  ->  [0.021, -0.087, 0.044, ..., 0.013]   (384 numbers)

The individual numbers mean nothing on their own; you can't point at dimension 17 and call it "about passwords". What matters is geometry. The model is trained so that texts with similar meaning get vectors pointing in similar directions:

• "How do I reset my password?" and "I forgot my login credentials" end up close together.
• "What's the weather in Lisbon?" ends up far from both.

Every text you embed with the same model gets a vector of the same length, whether it is three words or three paragraphs. That is what makes comparison possible: you can measure how close any two vectors are, quickly and cheaply, across millions of documents.

Embeddings are not limited to text. Images, audio and code have embedding models too, and some models place text and images in the same space so you can search photos with words. The ideas in this lesson apply to all of them.

#How embedding models are made

Embedding models are usually encoder transformers, a cousin of the generative LLMs from earlier lessons. Instead of predicting the next token, an encoder reads the whole input at once and produces a vector for every token. A pooling step, commonly averaging those token vectors, turns them into one vector for the whole text.

The model learns useful geometry through contrastive training. It sees huge numbers of pairs known to be related: a question and its accepted answer, a title and its article, two paraphrases. Training pulls the vectors of related pairs together and pushes unrelated texts apart. After enough examples, "similar meaning means nearby" holds for text the model has never seen.

Two consequences matter in practice:

1. Embedding models are small and cheap compared to chat models. The model used in this lesson, all-MiniLM-L6-v2, has about 22 million parameters, produces 384-dimensional vectors, and runs comfortably on a laptop CPU. Larger models produce longer vectors (768, 1,024 or more dimensions) and usually retrieve better, at the cost of speed and storage.
2. "Similar" means whatever the training data taught. Most general models capture topic and intent. They are not fact checkers. "The medication is safe for children" and "The medication is not safe for children" are about the same topic, so their vectors are very close, even though they mean opposite things.

#Measuring similarity: cosine maths

Given two vectors, how close are they? The standard measure is cosine similarity, the cosine of the angle between them:

cos(a, b) = (a · b) / (|a| × |b|)

where  a · b = a1*b1 + a2*b2 + ... + an*bn     (dot product)
       |a|   = sqrt(a1² + a2² + ... + an²)      (length, or magnitude)

The result ranges from 1 (same direction) through 0 (unrelated, at right angles) to -1 (opposite directions). Because it only measures direction, a long text and a short text about the same thing can still score high.

To build intuition, play with a two-dimensional space first. Drag the query (or click a word to make it the query) and watch the ranking change with the angle between vectors, not their distance apart. Then drag the query straight towards the origin: its length shrinks, but no cosine moves.

Here is a worked example with three-dimensional vectors. Real embeddings have hundreds of dimensions, but the arithmetic is the same:

a = [1, 2, 2]     b = [2, 1, 2]     c = [-1, 0, 1]

a · b = 1*2 + 2*1 + 2*2 = 2 + 2 + 4 = 8
|a|   = sqrt(1 + 4 + 4) = sqrt(9) = 3
|b|   = sqrt(4 + 1 + 4) = sqrt(9) = 3
cos(a, b) = 8 / (3 × 3) = 0.889        -> very similar

a · c = 1*(-1) + 2*0 + 2*1 = -1 + 0 + 2 = 1
|c|   = sqrt(1 + 0 + 1) = sqrt(2) ≈ 1.414
cos(a, c) = 1 / (3 × 1.414) ≈ 0.236     -> not very similar

The same calculation in NumPy:

import numpy as np

def cosine(a: np.ndarray, b: np.ndarray) -> float:
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

a, b, c = np.array([1, 2, 2]), np.array([2, 1, 2]), np.array([-1, 0, 1])
print(round(cosine(a, b), 3), round(cosine(a, c), 3))

0.889 0.236

#Normalise once, then just use the dot product

Dividing a vector by its length gives a unit vector with length 1. For a, that is [1/3, 2/3, 2/3] ≈ [0.333, 0.667, 0.667]. When both vectors have length 1, the denominator of the cosine formula is 1 × 1, so cosine similarity equals the dot product. That's why systems normalise every embedding when it is stored. Comparing a query against a million documents then takes one matrix multiplication.

For unit vectors, Euclidean distance and cosine carry the same information: |a − b|² = 2 − 2·cos(a, b). Ranking by smallest distance or largest cosine gives the same order, which is why vector databases let you choose either.

#Embedding text in Python

The sentence-transformers library runs open-source embedding models locally. Install it (it pulls in PyTorch) and embed a few sentences:

pip install sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")  # downloads about 90 MB on first run, then cached

sentences = [
    "How do I reset my password?",
    "I forgot my login credentials.",
    "What's the weather like in Lisbon today?",
]
embeddings = model.encode(sentences, normalize_embeddings=True)

print(embeddings.shape)
print((embeddings @ embeddings.T).round(2))  # unit vectors, so dot product = cosine

You should see output roughly like this. Exact values vary slightly between library versions and hardware, but the pattern is stable:

(3, 384)
[[1.   0.6  0.05]
 [0.6  1.   0.04]
 [0.05 0.04 1.  ]]

encode takes a list of strings and returns a NumPy array with one row per input. Passing many texts in one call is much faster than calling it once per text, because they are processed in batches. The two account questions score high despite sharing almost no words. The weather question scores near zero against both. The diagonal is 1 because every vector is identical to itself.

After the first download, the model runs entirely on your machine: no API key, no per-call cost, and no data leaving your network. In JavaScript, Transformers.js (@huggingface/transformers) runs the same model in Node or the browser:

import { pipeline } from "@huggingface/transformers";

const embed = await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2");
const output = await embed(["How do I reset my password?"], { pooling: "mean", normalize: true });
console.log(output.dims); // [1, 384]

#Semantic search in a few lines

Semantic search follows two phases. Indexing embeds every document once and stores the vectors. Querying embeds the query, scores it against every document vector, and returns the best matches:

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

docs = [
    "To reset your password, click 'Forgot password' on the sign-in page.",
    "Invoices are emailed on the first day of each month.",
    "You can change your plan at any time from Settings > Billing.",
    "Our mobile app is available for iOS and Android.",
    "Two-factor authentication can be enabled under Security settings.",
]
doc_vectors = model.encode(docs, normalize_embeddings=True)  # indexing: once

def search(query: str, k: int = 2) -> list[tuple[float, str]]:
    query_vector = model.encode([query], normalize_embeddings=True)[0]
    scores = doc_vectors @ query_vector          # cosine similarity with every doc
    best = np.argsort(-scores)[:k]               # indices of the top-k scores
    return [(round(float(scores[i]), 2), docs[i]) for i in best]

for score, doc in search("I can't log in to my account"):
    print(score, doc)

A typical run puts the password-reset article first and the two-factor article second. Neither contains the word "log", but both are about getting into an account. That is the advantage over keyword search. Its weakness is the other side: exact identifiers like error codes, SKUs or names are often better matched by keywords, which is why production systems frequently combine both (hybrid search).

This brute-force approach, scoring every document, is perfectly fast for thousands or even tens of thousands of documents. Beyond that you need an index built for vectors, which is the subject of the next lesson.

#Using embeddings well

• Use one model for everything in an index. Vectors from different models live in unrelated spaces. Comparing them gives meaningless scores, and so do vectors from two versions of the same model. Store the model name next to every vector, and re-embed the whole corpus when you switch.
• Mind the input limit. Every model has a maximum input length; all-MiniLM-L6-v2 truncates input beyond 256 word pieces (roughly 200 English words), silently ignoring the rest. Long documents must be split into chunks that are each embedded separately, as the RAG lesson covers.
• Check whether the model expects prefixes. Some models are trained for asymmetric search, with short queries and long passages, and expect prefixes such as query: and passage: , or a task instruction. Using them where expected, and only there, noticeably improves results. The model's documentation says which applies.
• Match the model to the data. For non-English or mixed-language text, choose a multilingual model. Domain-heavy text (legal, medical, code) may need a model trained on that domain.
• Plan storage. One million 384-dimensional float32 vectors take 1,000,000 × 384 × 4 bytes ≈ 1.5 GB. At 1,536 dimensions, it's about 6.1 GB. Dimension count drives memory, disk and search speed.
• Remember what similarity isn't. High similarity means "about the same thing", not "true", "relevant to the user's intent" or "agrees". Negations, numbers and dates are weak spots. A retrieval step should feed a model or a reranker that reads the text, not make final decisions on its own.

#Practice

Build an FAQ matcher. Store at least five FAQ entries as question/answer pairs. Embed only the questions, since user queries resemble questions more than answers. Then write answer(query) that:

1. returns the answer of the most similar FAQ question if its cosine similarity is at least a threshold;
2. otherwise returns "Sorry, I don't know that one. A human will get back to you.";
3. prints the best score, so you can calibrate the threshold.

Try it on a paraphrase ("my card got charged twice"), an unrelated query ("best pizza in Naples?"), and one more of your own. Pick a threshold from the scores you observe.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

FAQ = [
    ("How do I reset my password?", "Use 'Forgot password' on the sign-in page; a reset link arrives by email."),
    ("Why was I charged twice?", "Duplicate charges are usually pending authorisations that drop off within 3 days."),
    ("How do I cancel my subscription?", "Go to Settings > Billing > Cancel plan. You keep access until the period ends."),
    ("Can I export my data?", "Yes. Settings > Privacy > Export creates a ZIP of your data within an hour."),
    ("Do you offer student discounts?", "Students get 50% off with a valid university email address."),
]
questions = [q for q, _ in FAQ]
question_vectors = model.encode(questions, normalize_embeddings=True)

THRESHOLD = 0.5  # calibrated by looking at printed scores, specific to this model

def answer(query: str) -> str:
    query_vector = model.encode([query], normalize_embeddings=True)[0]
    scores = question_vectors @ query_vector
    best = int(np.argmax(scores))
    print(f"  best match {scores[best]:.2f}: {questions[best]!r}")
    if scores[best] < THRESHOLD:
        return "Sorry, I don't know that one. A human will get back to you."
    return FAQ[best][1]

for query in ["my card got charged twice", "best pizza in Naples?", "how can I download everything you have on me"]:
    print(query)
    print(" ", answer(query))

#Key takeaways

• An embedding is a fixed-length vector whose direction encodes meaning, so similar texts get nearby vectors even without shared words.
• Embedding models are small encoder transformers trained contrastively. They capture topic and intent, not truth, and handle negation poorly.
• Cosine similarity is a·b / (|a||b|). Normalise vectors once and it becomes a plain dot product.
• sentence-transformers with all-MiniLM-L6-v2 runs locally and free. Batch your encode calls and pass normalize_embeddings=True.
• Semantic search means embedding documents once, embedding each query, ranking by similarity and taking the top-k. Calibrate thresholds on your own data.
• Never mix vectors from different models, chunk long text to fit the model's input limit, and store which model produced each vector.