Lesson 13 · Applications

Multimodal models: DALL·E 2, Imagen, Stable Diffusion

Type text, get an image: connecting language and vision.

Multimodal models connect several kinds of data. CLIP learns to give matching images and captions nearby embeddings (contrastive learning). DALL·E 2 uses CLIP plus diffusion, Imagen uses a text-only encoder, Stable Diffusion runs diffusion in a latent space so it is light enough for your own computer, and Flamingo connects images to a language model.

The example trains a tiny CLIP: text-to-image matching goes from 0% before training to 100% after.

logits = norm(img_emb) · norm(txt_emb)ᵀ / τ loss = ½ [ CE(logits, I) + CE(logitsᵀ, I) ] ← matching pairs sit on the diagonal

Model	How it works
DALL·E 2	Text → CLIP embedding → prior → diffusion decoder
Imagen	T5 text encoder + cascaded diffusion upsamplers
Stable Diffusion	Diffusion in an autoencoder's latent space; open source
Flamingo	Feeds images/video into a pre-trained language model

Key takeaways

CLIP is the bridge between language and images
Most text-to-image systems = text encoder + diffusion

generative-ai-ep13-multimodal.py

import torch, torch.nn as nn

torch.manual_seed(0)
# Toy paired data: 64 "concepts", each seen as an image vector and a text vector
concepts = torch.randn(64, 16)
images = concepts @ torch.randn(16, 32) + 0.1 * torch.randn(64, 32)
texts = concepts @ torch.randn(16, 24) + 0.1 * torch.randn(64, 24)

img_enc, txt_enc = nn.Linear(32, 16), nn.Linear(24, 16)
opt = torch.optim.Adam([*img_enc.parameters(), *txt_enc.parameters()], lr=1e-2)

def retrieval_acc():
    with torch.no_grad():
        i = nn.functional.normalize(img_enc(images), dim=1)
        t = nn.functional.normalize(txt_enc(texts), dim=1)
        return ((i @ t.T).argmax(1) == torch.arange(64)).float().mean().item()

print(f"before training: text->image match {retrieval_acc():.0%}")
for step in range(300):
    i = nn.functional.normalize(img_enc(images), dim=1)
    t = nn.functional.normalize(txt_enc(texts), dim=1)
    logits = i @ t.T / 0.07                                # similarity of every pair
    labels = torch.arange(64)                              # matching pairs are on the diagonal
    loss = (nn.functional.cross_entropy(logits, labels) +
            nn.functional.cross_entropy(logits.T, labels)) / 2   # CLIP contrastive loss
    opt.zero_grad(); loss.backward(); opt.step()
print(f"after training:  text->image match {retrieval_acc():.0%}  (loss {loss.item():.3f})")

Output

before training: text->image match 0%
after training:  text->image match 100%  (loss 0.008)

#GenerativeAI
#DeepLearning
#PythonສຳລັບຄົນLaos
#AI
#ບອກຕໍ່ຄວາມຮູ້

Main Menu

Change Language

Currency

Multimodal models: DALL·E 2, Imagen, Stable Diffusion

Key takeaways