Lesson 03 · Six core methods

Variational autoencoders

Squeeze images into a latent space, then draw new ones from random points.

An autoencoder has an encoder that squeezes an image into a small vector (the latent) and a decoder that rebuilds it. But its latent space has gaps, so random samples look poor.

A VAE fixes this: the encoder outputs a distribution (μ, σ) and a KL-divergence term pulls it towards N(0, I). After training, just sample z from N(0, I) and decode it to get a new image. The example draws a new 8×8 digit as text.

z = μ + σ · ε, ε ~ N(0, I) ← reparameterization trick loss = reconstruction(x, x̂) + β · KL( N(μ, σ²) ‖ N(0, I) ) KL = −½ Σ (1 + log σ² − μ² − σ²)

Reconstruction lossRebuilt image should match the original

KL lossKeeps the latent space smooth and gap-free, so sampling works

Key takeaways

VAE = autoencoder + randomness + KL
Latent vectors support arithmetic, e.g. face + "glasses" = face with glasses

generative-ai-ep03-vae.py

import torch, torch.nn as nn
from sklearn.datasets import load_digits

torch.manual_seed(0)
X = torch.tensor(load_digits().data / 16.0, dtype=torch.float32)   # 8x8 digits

class VAE(nn.Module):
    def __init__(self, z=2):
        super().__init__()
        self.enc = nn.Sequential(nn.Linear(64, 128), nn.ReLU(), nn.Linear(128, 2 * z))
        self.dec = nn.Sequential(nn.Linear(z, 128), nn.ReLU(), nn.Linear(128, 64), nn.Sigmoid())

    def forward(self, x):
        mu, logvar = self.enc(x).chunk(2, dim=1)
        z = mu + torch.randn_like(mu) * torch.exp(0.5 * logvar)    # reparameterization trick
        return self.dec(z), mu, logvar

vae = VAE()
opt = torch.optim.Adam(vae.parameters(), lr=1e-3)
for epoch in range(200):
    recon, mu, logvar = vae(X)
    rec = nn.functional.binary_cross_entropy(recon, X, reduction="sum") / len(X)
    kl = -0.5 * torch.sum(1 + logvar - mu ** 2 - logvar.exp()) / len(X)
    loss = rec + kl
    opt.zero_grad(); loss.backward(); opt.step()
    if epoch % 50 == 49:
        print(f"epoch {epoch + 1}  reconstruction {rec.item():.1f}  KL {kl.item():.2f}")

# Generate: decode random points from N(0, I)
with torch.no_grad():
    sample = vae.dec(torch.randn(1, 2)).view(8, 8)
for row in sample:
    print("".join(" .:-=+*#%@"[int(v * 9.99)] for v in row))

Output

epoch 50  reconstruction 29.1  KL 1.31
epoch 100  reconstruction 27.7  KL 0.97
epoch 150  reconstruction 26.7  KL 1.09
epoch 200  reconstruction 25.5  KL 1.42
..=%#+..
 :%*+#:.
.:#:-#..
..+=+#: 
 .: :#:.
  .  *-.
  +-=#-.
..-%%+..

#GenerativeAI
#DeepLearning
#PythonສຳລັບຄົນLaos
#AI
#ບອກຕໍ່ຄວາມຮູ້

Main Menu

Change Language

Currency

Variational autoencoders

Key takeaways