Energy-Based Models
ໃຫ້ຄະແນນ "ພະລັງງານ" ຕ່ຳກັບຂອງຈິງ ແລ້ວເດີນລົງເນີນເພື່ອສ້າງຂອງໃໝ່.
Energy-based model ຮຽນຟັງຊັນ E(x): ຂອງຈິງໄດ້ຄ່າຕ່ຳ, ຂອງປອມໄດ້ຄ່າສູງ. p(x) ∝ exp(−E(x)) ແຕ່ຕົວຫານຄິດໄລ່ບໍ່ໄດ້ ຈຶ່ງໃຊ້ 2 ເທັກນິກ: contrastive divergence ສຳລັບຝຶກ ແລະ Langevin dynamics ສຳລັບສຸ່ມ.
ໂຄດຕົວຢ່າງເລີ່ມຈາກ noise ທົ່ວໄປ ແລ້ວເດີນລົງຕາມ gradient ຂອງ E ຈົນຈຸດສຸ່ມໄປຢູ່ໃກ້ຂໍ້ມູນຈິງ (2, −1).
p(x) = exp(−E(x)) / Z (Z ຄິດໄລ່ບໍ່ໄດ້)
train: loss = E(x_real) − E(x_fake)
sample: x ← x − η·∇ₓE(x) + noise (Langevin)
ສິ່ງທີ່ຄວນຈື່
- ຝຶກ: ດັນຂອງຈິງລົງ, ດັນຂອງປອມຂຶ້ນ (contrastive divergence)
- ສຸ່ມ: ເລີ່ມຈາກ noise ແລ້ວເດີນລົງເນີນເທື່ອລະກ້າວ — ແນວຄິດນີ້ນຳໄປສູ່ diffusion model
import torch, torch.nn as nn
torch.manual_seed(0)
real = lambda n: torch.randn(n, 2) * 0.3 + torch.tensor([2.0, -1.0]) # data near (2, -1)
E = nn.Sequential(nn.Linear(2, 64), nn.SiLU(), nn.Linear(64, 64), nn.SiLU(), nn.Linear(64, 1))
opt = torch.optim.Adam(E.parameters(), lr=1e-3)
def langevin(x, steps=30, step_size=0.05):
"""Walk downhill on the energy, plus a little noise."""
x = x.clone().requires_grad_(True)
for _ in range(steps):
grad = torch.autograd.grad(E(x).sum(), x)[0]
x = (x - step_size * grad + 0.01 * torch.randn_like(x)).detach().requires_grad_(True)
return x.detach()
for step in range(501):
fake = langevin(torch.rand(128, 2) * 8 - 4) # start from uniform noise
# contrastive divergence: real -> low energy, fake -> high energy
loss = E(real(128)).mean() - E(fake).mean() + 0.1 * (E(real(128)) ** 2).mean()
opt.zero_grad(); loss.backward(); opt.step()
if step % 250 == 0:
print(f"step {step:3d} fake samples centred at {fake.mean(0).numpy().round(2)}")ຜົນລັບ
step 0 fake samples centred at [0.02 0.3 ] step 250 fake samples centred at [ 1.98 -0.98] step 500 fake samples centred at [ 2.02 -0.98]
- #GenerativeAI
- #DeepLearning
- #PythonສຳລັບຄົນLaos
- #AI
- #ບອກຕໍ່ຄວາມຮູ້