Embeddings in LLMs Explained: How Text Becomes Meaningful Numbers

Embeddings in LLMs: A Beginner's Guide

1. What is an Embedding?

An embedding is a way of converting text into a list of numbers called a vector.

For example:

"cat"

is converted into something like:

[0.21, -0.45, 0.78, 0.12, ...]

These numbers represent information about the text in a form that a computer can mathematically work with.

The important idea is:

Embedding converts text into numbers so that computers can compare the meaning of different texts.

2. Why Do We Need Embeddings?

Computers cannot directly compare the meaning of sentences the same way humans do.

Consider these sentences:

"I love dogs."

"I really like puppies."

The words are different, but their meanings are similar.

An embedding model converts both sentences into vectors:

"I love dogs."
[0.21, 0.45, 0.78, ...]

"I really like puppies."
[0.22, 0.43, 0.76, ...]

The two vectors will generally be closer to each other because the sentences have similar meanings.

Now consider:

"The server crashed."

Its vector would generally be farther away because the meaning is different.

3. What is a Vector?

A vector is simply a list of numbers.

For example:

[0.21, 0.45, -0.12, 0.89]

This is a vector containing four numbers.

Real embedding vectors can contain hundreds or thousands of numbers.

For example:

[0.021, -0.452, 0.781, 0.123, ...]

You usually do not need to understand what each individual number means.

The whole vector represents the text.

4. Embeddings and Similarity

Once text has been converted into vectors, we can mathematically compare the vectors.

For example:

"I love dogs."
↕
"I like puppies."

Their vectors may be quite similar.

But:

"I love dogs."
↕
"The database server crashed."

Their vectors would generally be much less similar.

This allows a computer to find text with similar meaning.

5. Cosine Similarity

One common method for comparing embeddings is called Cosine Similarity.

It gives us a similarity score between two vectors.

For example:

Sentence A: "I love dogs."
Sentence B: "I like puppies."
Similarity: High

While:

Sentence A: "I love dogs."
Sentence C: "The server crashed."
Similarity: Low

The exact numerical score depends on the embedding model and the data being compared.

6. Embeddings vs Tokens

This is an important distinction.

Tokenization

Tokenization breaks text into smaller pieces called tokens.

"I love dogs."

↓

["I", "love", "dogs"]

Embedding

Embedding converts text into a vector of numbers.

"I love dogs."

↓

[0.21, 0.45, -0.12, 0.89, ...]

So remember:

Tokenization:
Text → Tokens

Embedding:
Text → Vector

They solve different problems.

7. Embeddings in RAG

Embeddings are extremely important in RAG (Retrieval-Augmented Generation).

Suppose you have a PDF containing information about your projects.

First, we divide the PDF into smaller pieces called chunks.

PDF
↓
Chunk 1
Chunk 2
Chunk 3
Chunk 4

Then we create an embedding for each chunk.

Chunk 1 → [0.21, 0.43, ...]
Chunk 2 → [0.12, 0.87, ...]
Chunk 3 → [0.33, 0.51, ...]

These embeddings can then be stored in a vector database.

Examples of Vector Databases

• Pinecone
• Qdrant
• Weaviate
• Chroma
• pgvector

8. What Happens When a User Asks a Question?

Suppose the user asks:

"What frontend technologies does Shiv know?"

The question is also converted into an embedding.

User Question
↓
Embedding Model
↓
Question Vector

Then we compare the question vector with the document vectors.

Question Vector
↓
Vector Search
↓
Find similar vectors
↓
Relevant document chunks

The relevant chunks are then given to the LLM.

9. Complete RAG Flow

The basic RAG process looks like this:

Document
↓
Split into chunks
↓
Create embeddings
↓
Store embeddings in vector database
↓
User asks a question
↓
Create embedding of the question
↓
Search for similar document embeddings
↓
Retrieve relevant chunks
↓
Give chunks + question to LLM
↓
LLM generates answer

10. Simple Real-World Analogy

Imagine a huge library containing thousands of documents.

You do not want to read every document whenever someone asks a question.

Instead, you create a numerical representation for each document based on its meaning.

When a question arrives, you also create a numerical representation of the question.

Then you find the documents whose representations are closest to the question.

In simple terms:

Documents
↓
Embeddings
↓
Vector Database

Question
↓
Embedding
↓
Search for similar vectors
↓
Relevant Documents

11. Three Important Concepts

Token

A token represents a piece of text.

Text → Tokens

Embedding

An embedding represents text as numbers so its meaning can be compared mathematically with other text.

Text → Vector

LLM

An LLM processes tokens and generates text.

Tokens → LLM → Tokens

12. Key Takeaway

An embedding is a numerical representation of text that allows computers to compare the meaning or semantic similarity of different pieces of text.

Quick Summary

• Embeddings convert text into vectors.
• A vector is simply a list of numbers.
• The whole vector represents the text.
• Similar meanings generally produce similar vectors.
• Cosine Similarity is one common way to compare vectors.
• Embeddings are different from tokens.
• Embeddings are widely used in RAG systems.
• Vector databases can store and search embeddings.
• Embeddings help retrieve relevant information for an LLM.