Advanced GANs: StyleGAN and VQ-GAN
From ProGAN to VQ-GAN: the road to high-resolution images.
After 2017 GANs advanced fast: ProGAN trains from small to large images, StyleGAN uses a mapping network to build a style vector that controls detail at each resolution, SAGAN and BigGAN add attention, and VQ-GAN combines a GAN with a codebook and a Transformer.
The example shows the heart of VQ: snap each vector to its nearest codebook entry and use the straight-through trick so gradients still flow.
| Model | New idea |
|---|---|
| ProGAN | Progressive growing from 4×4 to 1024×1024 |
| StyleGAN / 2 | Mapping network + style injected at each resolution; v2 uses weight demodulation |
| SAGAN · BigGAN | Self-attention for long-range structure; BigGAN scales up |
| VQ-GAN | A codebook of "visual words" + a Transformer that generates code sequences |
z_q = codebook[ argminₖ ‖z − eₖ‖ ]
loss = ‖sg[z] − z_q‖² + β·‖z − sg[z_q]‖² (sg = stop-gradient)
z_q_st = z + sg[z_q − z] (straight-through)
Key takeaways
- StyleGAN separates coarse features from fine detail, giving control
- VQ turns images into tokens, so Transformers can generate them
import torch
torch.manual_seed(0)
codebook = torch.randn(16, 4, requires_grad=True) # 16 learnable "visual words"
z = torch.randn(6, 4, requires_grad=True) # encoder outputs for 6 image patches
# Quantize: replace each vector by its nearest codebook entry
idx = torch.cdist(z, codebook).argmin(1)
zq = codebook[idx]
codebook_loss = ((zq - z.detach()) ** 2).mean() # move codes toward encoder outputs
commit_loss = ((z - zq.detach()) ** 2).mean() # keep encoder close to its code
zq_st = z + (zq - z).detach() # straight-through: gradients skip argmin
(zq_st.sum() + codebook_loss + 0.25 * commit_loss).backward()
print("code indices:", idx.tolist())
print("encoder gets gradients:", z.grad is not None)
print("codebook gets gradients:", codebook.grad.abs().sum().item() > 0)Output
code indices: [12, 10, 9, 7, 4, 2] encoder gets gradients: True codebook gets gradients: True
- #GenerativeAI
- #DeepLearning
- #PythonສຳລັບຄົນLaos
- #AI
- #ບອກຕໍ່ຄວາມຮູ້