世界模型
让 agent 做梦：把世界学成模型，再在梦中训练。
世界模型由三部分组成：把画面压缩成潜在表示的 VAE、预测下一个潜在状态的 MDN-RNN（记忆），以及负责决策的小型控制器。当模型足够准确时，agent 就能在模型生成的“梦境”中训练策略，再迁移到真实世界。
示例从真实经验中学习 CartPole 的动力学，然后“做梦”10 步，并把杆子角度与真实情况对比。
| 部分 | 作用 |
|---|---|
| V — VAE | 视觉：把画面压缩成 z |
| M — MDN-RNN | 记忆：预测 p(zₜ₊₁ | zₜ, aₜ, hₜ) |
| C — Controller | 根据 z 和 h 决策，用 CMA-ES 训练 |
要点
- 生成模型可以充当强化学习的“模拟世界”
- 与深度强化学习系列相连：这就是深度版的基于模型的强化学习
import numpy as np, torch, torch.nn as nn
import gymnasium as gym
torch.manual_seed(0)
env = gym.make("CartPole-v1")
# 1) Collect real experience with a random policy
S, A, S2 = [], [], []
for ep in range(200):
s, _ = env.reset(seed=ep); done = False
while not done:
a = env.action_space.sample()
s2, r, term, trunc, _ = env.step(a)
S.append(s); A.append(a); S2.append(s2); s, done = s2, term or trunc
S, A, S2 = map(lambda v: torch.tensor(np.array(v), dtype=torch.float32), (S, A, S2))
# 2) Learn the dynamics: (state, action) -> next state (the "memory" model)
model = nn.Sequential(nn.Linear(5, 128), nn.ReLU(), nn.Linear(128, 4))
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
inp = torch.cat([S, A[:, None]], 1)
for step in range(2000):
loss = ((S + model(inp) - S2) ** 2).mean() # predict the change in state
opt.zero_grad(); loss.backward(); opt.step()
print(f"dynamics model loss {loss.item():.6f} ({len(S)} real transitions)")
# 3) "Dream": roll out 10 steps inside the model and compare with reality
s, _ = env.reset(seed=999); dream = torch.tensor(s, dtype=torch.float32)
for t in range(10):
a = t % 2
s, *_ = env.step(a)
with torch.no_grad():
dream = dream + model(torch.cat([dream, torch.tensor([float(a)])]))
print("real pole angle:", round(float(s[2]), 4))
print("dream pole angle:", round(float(dream[2]), 4))输出结果
dynamics model loss 0.000003 (4204 real transitions) real pole angle: 0.0641 dream pole angle: 0.0662
- #GenerativeAI
- #DeepLearning
- #PythonສຳລັບຄົນLaos
- #AI
- #ບອກຕໍ່ຄວາມຮູ້