Lesson 08 · Six core methods

Diffusion models

Add noise little by little, then learn to remove it — the heart of today's image AI.

The forward process adds noise to the data over T steps until only noise is left. The reverse process uses a network (usually a U-Net) to predict the added noise and remove it step by step, ending with new data.

The example is a tiny 2-D DDPM: it turns noise into 1,000 points that land on the two-moons shape of the real data.

forward: xₜ = √ᾱₜ · x₀ + √(1−ᾱₜ) · ε ← jump straight to step t loss: ‖ ε − ε_θ(xₜ, t) ‖² reverse: xₜ₋₁ = (xₜ − βₜ/√(1−ᾱₜ) · ε_θ) / √αₜ + σₜ·z

Noise scheduleHow much noise each step adds (linear, cosine)

U-NetDownBlocks shrink, UpBlocks expand, joined by skip connections

DDIMDeterministic sampling in fewer steps — much faster

Key takeaways

The network learns just one thing: predict the noise
Stable Diffusion, DALL·E 2 and Imagen all build on this idea

generative-ai-ep08-diffusion.py

import torch, torch.nn as nn
from sklearn.datasets import make_moons

torch.manual_seed(0)
X = torch.tensor(make_moons(4000, noise=0.05, random_state=0)[0], dtype=torch.float32)
T = 100
betas = torch.linspace(1e-4, 0.05, T)                 # noise schedule
alphas = 1 - betas
abar = torch.cumprod(alphas, 0)

# Noise predictor eps(x_t, t): a small MLP (a U-Net does this job for images)
net = nn.Sequential(nn.Linear(3, 128), nn.SiLU(), nn.Linear(128, 128), nn.SiLU(), nn.Linear(128, 2))
opt = torch.optim.Adam(net.parameters(), lr=2e-3)

for step in range(3000):
    x0 = X[torch.randint(0, len(X), (256,))]
    t = torch.randint(0, T, (256,))
    eps = torch.randn_like(x0)
    xt = abar[t].sqrt()[:, None] * x0 + (1 - abar[t]).sqrt()[:, None] * eps   # forward: jump to step t
    loss = ((net(torch.cat([xt, t[:, None] / T], 1)) - eps) ** 2).mean()
    opt.zero_grad(); loss.backward(); opt.step()
print(f"final noise-prediction loss {loss.item():.3f}")

with torch.no_grad():                                 # reverse: denoise step by step
    x = torch.randn(1000, 2)
    for t in reversed(range(T)):
        eps = net(torch.cat([x, torch.full((1000, 1), t / T)], 1))
        x = (x - betas[t] / (1 - abar[t]).sqrt() * eps) / alphas[t].sqrt()
        if t > 0:
            x += betas[t].sqrt() * torch.randn_like(x)
dist = torch.cdist(x, X).min(1).values
print(f"generated 1000 points, median distance to real data: {dist.median():.3f}")
print(f"x range [{x[:, 0].min():.2f}, {x[:, 0].max():.2f}]  (real data: [-1.0, 2.0])")

Output

final noise-prediction loss 0.359
generated 1000 points, median distance to real data: 0.031
x range [-1.44, 2.29]  (real data: [-1.0, 2.0])

#GenerativeAI
#DeepLearning
#PythonສຳລັບຄົນLaos
#AI
#ບອກຕໍ່ຄວາມຮູ້

Main Menu

Change Language

Currency

Diffusion models

Key takeaways