第 05 课 · 六大核心方法

自回归模型：LSTM 与 PixelCNN

一次生成一个 token 的文本或图像。

自回归模型把 p(x) 分解为 p(x₁)·p(x₂|x₁)·p(x₃|x₁,x₂)…，然后逐个预测下一个元素。LSTM 对文本这样做，PixelCNN 对图像像素这样做。

示例训练一个字符级 LSTM，并用不同温度生成文本：温度低 = 稳妥但重复，温度高 = 有创意但凌乱。

p(x) = Π p(xₜ | x₁ … xₜ₋₁) 采样: softmax(logits / T) T < 1 ⇒ 更确定, T > 1 ⇒ 更多样

LSTM / GRU用门控决定记住或遗忘什么，因此能处理长文本

PixelCNN掩码卷积只看前面的像素，预测下一个像素

要点

自回归模型能给出精确似然，但生成慢，因为必须逐步进行
第 9 课的 Transformer 同样是自回归模型

generative-ai-ep05-autoregressive.py

import torch, torch.nn as nn

torch.manual_seed(0)
text = ("python is fun. python is easy. learning python is fun and easy. " * 20)
chars = sorted(set(text)); idx = {c: i for i, c in enumerate(chars)}
data = torch.tensor([idx[c] for c in text])

class CharLSTM(nn.Module):
    def __init__(self, v, h=64):
        super().__init__()
        self.emb, self.lstm, self.out = nn.Embedding(v, 16), nn.LSTM(16, h, batch_first=True), nn.Linear(h, v)
    def forward(self, x, state=None):
        y, state = self.lstm(self.emb(x), state)
        return self.out(y), state

model = CharLSTM(len(chars))
opt = torch.optim.Adam(model.parameters(), lr=1e-2)
for step in range(300):
    i = torch.randint(0, len(data) - 41, (32,))
    x = torch.stack([data[j:j + 40] for j in i]); y = torch.stack([data[j + 1:j + 41] for j in i])
    logits, _ = model(x)
    loss = nn.functional.cross_entropy(logits.reshape(-1, len(chars)), y.reshape(-1))
    opt.zero_grad(); loss.backward(); opt.step()
print(f"final loss {loss.item():.3f}")

def generate(start, n=50, temperature=1.0):
    x, state, out = torch.tensor([[idx[c] for c in start]]), None, start
    for _ in range(n):                      # one token at a time: p(x_t | x_<t)
        logits, state = model(x, state)
        probs = torch.softmax(logits[0, -1] / temperature, dim=0)
        c = torch.multinomial(probs, 1).item()
        out += chars[c]; x = torch.tensor([[c]])
    return out

for t in (0.2, 1.5):
    print(f"T={t}:", repr(generate("python ", temperature=t)))

输出结果

final loss 0.044
T=0.2: 'python is fun. python is easy. learning python is fun and'
T=1.5: 'python is fun and easy. python is fun. python is easy. le'

#GenerativeAI
#DeepLearning
#PythonສຳລັບຄົນLaos
#AI
#ບອກຕໍ່ຄວາມຮູ້

主菜单

切换语言

货币

自回归模型：LSTM 与 PixelCNN

要点