Energy-based models
Score real data with low "energy", then walk downhill to create new samples.
An energy-based model learns a function E(x): low for real data, high for fakes. p(x) ∝ exp(−E(x)), but the normaliser can't be computed, so two tricks are used: contrastive divergence for training and Langevin dynamics for sampling.
The example starts from random noise and walks down the gradient of E until the samples sit near the real data at (2, −1).
p(x) = exp(−E(x)) / Z (Z is intractable)
train: loss = E(x_real) − E(x_fake)
sample: x ← x − η·∇ₓE(x) + noise (Langevin)
Key takeaways
- Training: push real data down, push fakes up (contrastive divergence)
- Sampling: start from noise and walk downhill step by step — the idea that leads to diffusion models
import torch, torch.nn as nn
torch.manual_seed(0)
real = lambda n: torch.randn(n, 2) * 0.3 + torch.tensor([2.0, -1.0]) # data near (2, -1)
E = nn.Sequential(nn.Linear(2, 64), nn.SiLU(), nn.Linear(64, 64), nn.SiLU(), nn.Linear(64, 1))
opt = torch.optim.Adam(E.parameters(), lr=1e-3)
def langevin(x, steps=30, step_size=0.05):
"""Walk downhill on the energy, plus a little noise."""
x = x.clone().requires_grad_(True)
for _ in range(steps):
grad = torch.autograd.grad(E(x).sum(), x)[0]
x = (x - step_size * grad + 0.01 * torch.randn_like(x)).detach().requires_grad_(True)
return x.detach()
for step in range(501):
fake = langevin(torch.rand(128, 2) * 8 - 4) # start from uniform noise
# contrastive divergence: real -> low energy, fake -> high energy
loss = E(real(128)).mean() - E(fake).mean() + 0.1 * (E(real(128)) ** 2).mean()
opt.zero_grad(); loss.backward(); opt.step()
if step % 250 == 0:
print(f"step {step:3d} fake samples centred at {fake.mean(0).numpy().round(2)}")Output
step 0 fake samples centred at [0.02 0.3 ] step 250 fake samples centred at [ 1.98 -0.98] step 500 fake samples centred at [ 2.02 -0.98]
- #GenerativeAI
- #DeepLearning
- #PythonສຳລັບຄົນLaos
- #AI
- #ບອກຕໍ່ຄວາມຮູ້