Transformers and GPT
Attention: let every word decide which other words matter.
Transformers replace LSTMs with self-attention: each token builds a query, key and value, and weighs other tokens by query–key similarity. GPT is a Transformer with a causal mask, so it can predict the next word.
The example computes attention over 4 words and prints the weight table — the upper-right triangle is 0 because the mask forbids looking at the future.
Attention(Q, K, V) = softmax( Q·Kᵀ / √d + mask ) · V
mask[i, j] = −∞ if j > i
| Block | Role |
|---|---|
| Multi-head attention | Several heads look at different relationships at once |
| Positional encoding | Tells the model token order |
| Layer norm + skip | Lets many layers stack stably |
| Encoder / Decoder | BERT = encoder, GPT = decoder, T5 = both |
Key takeaways
- Attention sees all words at once, so training is parallel and long context works
- ChatGPT is a large GPT plus instruction tuning (RLHF)
import torch
torch.manual_seed(0)
tokens = ["the", "cat", "sat", "down"]
d = 8
x = torch.randn(len(tokens), d) # token embeddings
Wq, Wk, Wv = (torch.randn(d, d) / d ** 0.5 for _ in range(3))
Q, K, V = x @ Wq, x @ Wk, x @ Wv
scores = Q @ K.T / d ** 0.5 # how much each token looks at each other
mask = torch.triu(torch.ones(len(tokens), len(tokens)), diagonal=1).bool()
scores = scores.masked_fill(mask, float("-inf")) # causal mask: no peeking at the future
weights = torch.softmax(scores, dim=-1)
out = weights @ V
print("attention weights (row = query token):")
print(" " + "".join(f"{t:>7}" for t in tokens))
for t, row in zip(tokens, weights):
print(f"{t:>6}" + "".join(f"{w:7.2f}" for w in row))
print("output shape:", tuple(out.shape))Output
attention weights (row = query token):
the cat sat down
the 1.00 0.00 0.00 0.00
cat 0.40 0.60 0.00 0.00
sat 0.38 0.59 0.04 0.00
down 0.09 0.13 0.68 0.10
output shape: (4, 8)
- #GenerativeAI
- #DeepLearning
- #PythonສຳລັບຄົນLaos
- #AI
- #ບອກຕໍ່ຄວາມຮູ້