Mar Java Mit Java

  • Uncategorised

How LLMs Work: A Deep Dive into Text Prediction

by · July 4, 2025

Large Language Models (LLMs) like OpenAI’s GPT, Google’s Gemini, or Meta’s LLaMA have become incredibly powerful at understanding and generating human-like text. But how do they actually work under the hood? What happens when you type “Once upon a time…” and expect the model to continue?

This blog demystifies the internals of LLMs — step by step, from raw text input to the final predicted word.

🧾 1. Step 1: Input Text

Let’s say we input this sentence:

cssCopyEditOnce upon a time

This is just plain text — a sequence of characters. But LLMs don’t understand characters or words directly. They speak in tokens.

🔡 2. Step 2: Tokenization

The first step is to convert the input text into tokens using a tokenizer (like Byte Pair Encoding or WordPiece). Tokens are subword units — not always full words.

Example tokenization:

cssCopyEdit"Once upon a time" → ["Once", " upon", " a", " time"]

These tokens are language-specific, trained during the model’s pretraining phase.

🔢 3. Step 3: Tokens → Token IDs

Each token maps to a unique integer ID using a vocabulary table (usually 50,000 to 100,000 entries).

cssCopyEdit["Once", " upon", " a", " time"] → [1234, 765, 98, 567]

These IDs are what go into the model.

🧭 4. Step 4: Token IDs → Embeddings

Each token ID is used to look up a vector in the model’s embedding matrix, usually a matrix of size V x D:

  • V = vocabulary size (e.g., 50,000)
  • D = embedding dimension (e.g., 768 or 4096)

So [1234, 765, 98, 567] → 4 vectors of shape [D]:

csharpCopyEdit[Vec1234, Vec765, Vec98, Vec567]

These vectors are also combined with positional encodings, since transformers don’t inherently know order.

🧠 5. Step 5: Passing Through Transformer Layers

LLMs are built from stacked transformer blocks, each containing:

  • Multi-head Self-Attention
  • Feedforward Neural Network (FFN)

A typical LLM might have 12, 32, or even 96 transformer layers.

A. Multi-Head Attention:

  • Computes how each word relates to every other word in the sequence.
  • Uses three matrices:
    • Q (query), K (key), V (value)
  • Attention = softmax(Q × Kᵀ / √d) × V

This produces context-aware vectors, letting the model “attend” to relevant parts of the input.

B. Feedforward Network:

  • Applies a simple two-layer MLP (with ReLU/GELU).
  • Think of it as transforming each token’s vector non-linearly.

Each layer outputs a new set of vectors, same shape [seq_length x D], but more “informed”.

🔄 6. Step 6: Repeat Across Layers

These updated vectors are fed to the next transformer layer, and the process is repeated:

mathematicaCopyEditLayer 1 → Layer 2 → Layer 3 → ... → Layer N

Each layer further refines the contextual meaning of each token.

🎯 7. Step 7: Final Layer Output

At the end of all layers, we have a final vector (embedding) for each input token. But we’re usually only interested in the last token’s vector (e.g., “time”) for prediction.

📈 8. Step 8: Project to Vocabulary Space

To predict the next word, the model multiplies the final vector by the transpose of the embedding matrix:

nginxCopyEditoutput_vector × embedding_matrixᵀ → logits
  • output_vector: shape [D]
  • embedding_matrixᵀ: shape [D x V]
  • logits: shape [V] → one score per vocabulary word

This gives raw logits (unnormalized scores) for all 50K+ vocabulary words.

🎲 9. Step 9: Softmax → Probabilities

The logits are passed through a softmax function to convert them into probabilities:

textCopyEditsoftmax(logits) → [0.01, 0.0002, ..., 0.09]

The highest value corresponds to the most likely next token.

✍️ 10. Step 10: Sample or Select Next Token

Depending on the decoding strategy, the model either:

  • Greedy Decoding: pick the highest probability token
  • Sampling: pick based on probability distribution
  • Top-k / Top-p sampling: restrict to most likely k or top p% tokens

This token is then appended to the input, and the whole process repeats to generate the next word.

🧩 Putting It All Together

Recap Flow:

textCopyEditInput Text → Tokens → Token IDs
→ Embeddings + Positions
→ Transformer Layers (attention + feedforward)
→ Final Token Vector
→ Multiply with Embedding Matrixᵀ
→ Logits → Softmax → Probabilities
→ Next Token

This loop continues until:

  • A stop token is generated
  • Max length is reached
  • A user-defined stop condition triggers

🔍 A Note on Parameters

Modern LLMs have billions of parameters, most of which are weights in:

  • Embedding matrices
  • Attention weights (Q, K, V, output)
  • Feedforward weights

Training these involves adjusting weights to minimize the cross-entropy loss between predicted tokens and actual ones across huge corpora.

⚡ Example: GPT-Style Next Word Prediction

Input:

textCopyEdit"The capital of France is"

After tokenizing, passing through transformer layers, and computing logits → the model might output:

textCopyEdit["Paris": 0.91, "London": 0.02, "Berlin": 0.01, ...]

The model picks “Paris” and appends it to the sequence.

You may also like...