How an LLM Works: From Text to the Next Word
How an LLM Works: From Input to Output
A practical, numerical journey through tokenization, embeddings, attention, Q/K/V, multi-head attention, feed-forward networks, and billions of parameters.
Introduction
Large Language Models (LLMs) such as ChatGPT can answer questions, write code, explain concepts and generate text. But what actually happens inside an LLM when we enter a sentence?
Let’s follow this input through the model:
We will use an illustrative Transformer with the following dimensions. These are example architecture choices, not specifications of a particular commercial model.
| Property | Value |
|---|---|
| Vocabulary size | 50,000 tokens |
| Hidden dimension | 4,096 |
| Feed-forward intermediate dimension | 14,336 |
| Transformer layers | 32 |
| Attention heads per layer | 32 |
| Dimension per attention head | 128 |
1. Tokenization: Text Becomes Token IDs
The first step is tokenization. The text is split into tokens, and each token is represented by an ID from the tokenizer’s vocabulary.
Token ID: illustrative
Token ID: illustrative
Token ID: illustrative
A token is not always a complete word. Depending on the tokenizer, it may be a word, subword, punctuation mark or another text fragment.
2. Embedding: Turning Token IDs into Vectors
A token ID is just an identifier. The model looks up a learned vector for that ID in its input embedding table.
Our example has 50,000 tokens, each represented by 4,096 numbers.
That is approximately 204.8 million embedding parameters.
For illustration, a tiny embedding could look like this:
In our example model, the real vector has 4,096 elements. Positional information is also incorporated so the model can represent token order. Different architectures handle position differently.
These initial embeddings are learned during training. Their individual dimensions are not necessarily simple, human-readable features such as “animal” or “movement.”
3. Transformer Layers: Processing Context
The embedding provides an initial representation, but the model must interpret a token in the context of the sentence.
Consider the word “bank”:
- I deposited money at the bank.
- We sat beside the river bank.
The initial embedding for the same token can be identical, while the representation after contextual processing differs.
Our example contains 32 Transformer layers. Each layer contains two major components:
Residual connections and normalization also operate within Transformer layers.
4. Attention: Understanding Relationships
Attention allows a token to gather relevant information from other tokens in the input. In a causal language model, a token can generally attend to itself and earlier tokens, but not future tokens.
4.1 What are Q, K and V?
Used to determine which other tokens are relevant.
Compared against queries to calculate relevance scores.
Carries the information that can be combined from tokens.
Q, K and V are calculated from current token representations using learned weight matrices.
The matrices are fixed during ordinary inference, but the Q, K and V vectors are calculated dynamically for each token and layer.
4.2 Numerical example
To keep the arithmetic simple, imagine the query for “sat” is:
Suppose the keys are:
Compare Q for “sat” against each key using a dot product.
| Comparison | Calculation | Score |
|---|---|---|
| Against The | 2×1 + 6×1 | 8 |
| Against cat | 2×2 + 6×1 | 10 |
| Against sat | 2×2 + 6×5 | 34 |
These scores are not probabilities. Real scaled dot-product attention divides the scores by the square root of the key dimension before applying softmax, and applies a causal mask where appropriate.
4.3 From scores to attention weights
Softmax converts the scores into weights that sum to one. For illustration, suppose the weights are:
| Token | Illustrative weight |
|---|---|
| The | 0.2 |
| cat | 0.5 |
| sat | 0.3 |
These weights are illustrative and are not the softmax result of the scores calculated above.
4.4 Use the weights to combine V
Suppose the value vectors are:
Vcat = [0, 4]
Vsat = [7, 3]
The attention output for “sat” is the weighted sum:
Q helped determine the weights. The V vectors supplied the information being combined. The result is an intermediate attention output, not yet the final vector used to predict the next token.
5. Multi-Head Attention
A model can run multiple attention calculations in parallel. These are called attention heads. Different heads can learn different patterns of relationships between tokens.
In our example, the hidden dimension is 4,096 and there are 32 heads.
The Q projection produces a 4,096-element vector, which can be viewed as 32 chunks of 128 elements. The same dimensional arrangement applies to K and V.
Each head uses its own learned projections and performs attention. The 32 head outputs are then concatenated:
What is WO?
The concatenated vector is multiplied by the output projection matrix WO.
The output is still a 4,096-element vector, but its numerical values have changed. WO learns how to combine information from the different attention heads.
6. The Feed-Forward Neural Network
Attention gathers information across token positions. The feed-forward network transforms the information in each token’s representation.
The same feed-forward weights are reused for every token position within a layer. Each token nevertheless has its own input vector and its own resulting activations.
6.1 Expand the representation
The first matrix has dimensions 4,096 × 14,336.
The resulting intermediate vector contains 14,336 numbers.
6.2 Apply a nonlinear operation
The model applies an activation function or a gated operation. For a simple ReLU example:
Modern LLMs commonly use other activation functions or gated feed-forward designs.
6.3 Project back to 4,096
The second matrix has dimensions 14,336 × 4,096.
The representation returns to 4,096 dimensions, but its values have been transformed by learned weights and nonlinear processing.
7. Why Are There 32 Transformer Layers?
Each of our 32 layers has its own attention weights and feed-forward weights.
- Layer 1 processes the initial representations.
- Layer 2 uses the updated representations and its own weights.
- The same process repeats through the remaining layers.
Consequently, Q, K and V are recalculated from the latest representations at every layer. The learned weights are normally fixed during inference.
8. Calculating Billions of Parameters
A parameter is a learned numerical value stored in the model. Let’s count the major matrices in our illustrative architecture.
8.1 Input embeddings
8.2 Attention matrices
Each layer has four major matrices: WQ, WK, WV, and WO.
8.3 Feed-forward matrices
Each layer has two major matrices:
8.4 Output projection
Assuming the output weights are separate from the input embedding table:
8.5 Total
| Component | Parameters |
|---|---|
| Input embedding table | 204.8 million |
| Attention matrices across 32 layers | 2.147 billion |
| Feed-forward matrices across 32 layers | 3.758 billion |
| Output projection | 204.8 million |
| Total | 6.315 billion |
This estimate omits biases, normalization parameters and some architecture-specific components. If input embeddings and output weights are tied, the shared matrix is counted only once.
Modern gated feed-forward networks and other attention designs can have different parameter counts.
9. Parameters Versus Calculations
Parameters and calculations are not the same thing.
The learned numbers stored in the model.
The operations performed using those numbers.
For the input “The cat sat on the mat,” the same feed-forward weights are applied to each token’s representation. The model does not create a separate feed-forward network for every token.
- More parameters mean more learned numbers.
- More input tokens generally require more computation.
- The same parameters can be reused across token positions and across requests.
A billion-parameter model does not have a billion parameters for each token.
10. The Output Layer: Predicting the Next Token
After all Transformer layers, the model uses the final hidden vector at the last input position to predict the next token.
For illustration, suppose the final vector is:
In our real-sized example, this vector would have 4,096 elements.
10.1 What is an output vector?
The output projection has a learned vector associated with each possible output token. These vectors are parameters.
Some architectures share input embedding weights with the output projection; others use separate weights. Therefore, an output vector is not necessarily the same as the token’s input embedding.
For a tiny example, suppose the output vectors are:
| Candidate | Output vector |
|---|---|
| on | [2, 1] |
| down | [1, 2] |
| quietly | [0, 1] |
10.2 Calculate the scores
| Candidate | Score |
|---|---|
| on | 10 |
| down | 8 |
| quietly | 2 |
These scores are called logits. The model calculates one for every candidate token in its vocabulary. The scores are not distances between words.
10.3 Convert scores into probabilities
Softmax converts the logits into probabilities. For our three illustrative candidates, the approximate probabilities are:
| Candidate | Probability |
|---|---|
| on | 88.0% |
| down | 11.9% |
| quietly | 0.1% |
These probabilities apply only to the tiny example, not to a real LLM vocabulary.
With greedy decoding, the model selects the highest-scoring token: “on.” The text becomes:
If more text is needed, the model predicts another token using the expanded sequence.
11. How Does Training Teach the Model?
During pretraining, the model sees large amounts of text and learns to predict the next token.
For example:
The cat sat
on
The model calculates its predictions, measures the error using a loss function, and uses backpropagation and an optimizer to update the learned parameters.
These updates affect embeddings, attention matrices, feed-forward matrices and output weights. Over many training examples, the model learns patterns that improve its predictions.
Later training stages can further improve instruction following and response quality. Training does not guarantee that every answer will be correct.
Conclusion: The Complete Journey
An LLM is not simply a dictionary of words or a system that selects the nearest embedding. It is a learned numerical system that processes context and calculates probabilities for possible next tokens.
