An embedding is a list of numbers that represents the meaning of text as a point in geometric space. Text with similar meaning produces points that are close together. Text with different meaning produces points that are far apart. Every AI search system, every RAG pipeline, and every semantic similarity tool is built on this single idea: meaning as geometry.
Analysis Briefing
- Topic: Vector embeddings and semantic search foundations
- Analyst: Mike D (@MrComputerScience)
- Context: Sparked by a question from Claude Sonnet 4.6
- Source: Pithy Cyborg
- Key Question: What actually happens when AI searches by meaning instead of keywords?
How Text Becomes Numbers That Represent Meaning
An embedding model reads a piece of text and outputs a vector, a list of typically 768 to 4096 numbers. Each number represents the text’s position along one dimension of a learned semantic space. The model learned that space by training on massive amounts of text and adjusting the dimensions until similar texts consistently produced similar number lists.
The resulting space has interpretable structure even though no individual dimension has an obvious label. Words related to royalty cluster near each other. Words related to cooking cluster elsewhere. Sentences that express the same idea in different words produce vectors that point in nearly the same direction even if they share no words at all.
This is what allows AI search to retrieve “cardiac arrest” when you search for “heart attack.” A keyword search finds documents containing those exact words. An embedding search finds documents that are geometrically close in meaning space to your query, regardless of whether the exact words match. The geometric proximity is the meaning similarity.
Why Cosine Similarity Is the Measure That Makes This Work
The standard way to measure how close two embeddings are is cosine similarity, which measures the angle between two vectors rather than the distance between their endpoints. Two vectors pointing in the same direction have a cosine similarity of 1. Two vectors pointing in opposite directions have a cosine similarity of negative 1.
Cosine similarity rather than Euclidean distance is used because embedding vectors vary in magnitude for reasons unrelated to meaning. A long document produces a higher-magnitude vector than a short document expressing the same idea. Measuring the angle between vectors rather than the distance between their tips makes the similarity measure invariant to document length.
This geometry underlies every failure mode in the RAG pieces published on this site. Negation queries fail because “without encryption” and “with encryption” point in nearly the same direction in embedding space. Semantic search returns confidently wrong results when the nearest geometric neighbor is topically related but contextually wrong. The geometry is the mechanism behind both the capability and the failure.
What Embeddings Cannot Represent and Why That Matters
Embeddings compress text into a fixed-length vector, which means they compress away information. The specific information that gets compressed is exactly the information that makes individual documents distinct rather than topically similar.
Document-level embeddings average the meaning of all the text in a chunk into a single vector. A document that discusses both the advantages and disadvantages of a technology produces a vector that averages both, positioned somewhere between the advantage cluster and the disadvantage cluster in semantic space. A query about disadvantages retrieves this document because it is near the disadvantage cluster. The retrieved chunk contains both advantages and disadvantages. The model synthesizes a response that inherits that ambiguity.
Logical structure is also compressed away. Embedding models represent topics, not logical relationships between topics. “A causes B” and “B causes A” produce similar embeddings because they contain the same topical content. The causal direction, which may be the entire point of a document, is not reliably preserved in the embedding representation.
What This Means For You
- Understand embeddings as meaning geometry, not magic. When AI search retrieves the wrong document, the cause is geometric: the wrong document is closer to your query in embedding space than the right one. The fixes are always about changing that geometry.
- Use embeddings for semantic retrieval and keywords for exact retrieval. Hybrid search that combines embedding similarity with BM25 keyword matching outperforms either alone because they fail in complementary ways. Embeddings miss exact term matches. Keywords miss semantic paraphrases.
- Chunk documents at meaning boundaries, not at character count boundaries. A chunk that averages two unrelated topics produces a vector that retrieves poorly for both. Chunks that contain coherent single topics produce vectors that retrieve well for that topic.
- Read the RAG failure mode pieces on this site with the geometry model in mind. Every failure covered, negation queries, stale index poisoning, hallucination relocation, has a geometric explanation that becomes clear once you understand what an embedding actually represents.
Enjoyed this deep dive? Join my inner circle:
- Pithy Cyborg → AI news made simple without hype.