Lesson 12 · Policy & actor-critic

Advanced actor-critic: DDPG, TD3, SAC, PPO

The four algorithms you will meet most in real projects.

DDPG and TD3 learn deterministic policies for continuous actions, SAC adds entropy to keep the policy diverse, and PPO limits how much the policy changes per update.

In the code, when the ratio is 1.8 (above 1.2), the action with a positive advantage is clipped (gradient 0), but the one with a negative advantage still gets a gradient pulling it back.

Algorithm	Policy	Learning	Highlights
DDPG	Deterministic	Off-policy	DQN for continuous actions; Gaussian noise for exploration
TD3	Deterministic	Off-policy	① Twin Q ② noise on target actions ③ delayed policy updates
SAC	Stochastic	Off-policy	Adds entropy: high reward with the most diverse policy
PPO	Stochastic	On-policy	A clipped objective limits policy change, so one batch can be reused for several updates

SAC : J(π) = E[ Σ γᵗ ( rₜ + α·H(π(·|sₜ)) ) ] PPO : rₜ(θ) = πθ(aₜ|sₜ) / πθ_old(aₜ|sₜ) L = E[ min( rₜ·Âₜ , clip(rₜ, 1−ε, 1+ε)·Âₜ ) ]

Key takeaways

Continuous actions + sample efficiency → TD3 or SAC
Want something stable and widely applicable → PPO

deep-rl-ep12-ddpg-td3-sac-ppo.py

import torch

def ppo_clip_loss(logp_new, logp_old, advantages, clip=0.2):
    ratio = torch.exp(logp_new - logp_old)                 # π_new / π_old
    unclipped = ratio * advantages
    clipped = torch.clamp(ratio, 1 - clip, 1 + clip) * advantages
    return -torch.min(unclipped, clipped).mean()           # ເລືອກຄ່າທີ່ລະມັດລະວັງກວ່າ

logp_old = torch.log(torch.tensor([0.5, 0.5]))
adv = torch.tensor([1.0, -1.0])
for p_new in (0.5, 0.6, 0.9):
    logp_new = torch.log(torch.tensor([p_new, p_new])).requires_grad_()
    loss = ppo_clip_loss(logp_new, logp_old, adv)
    loss.backward()
    print(f"π_new={p_new}: loss={loss.item():+.3f}  grad={logp_new.grad.numpy().round(3)}")

Output

π_new=0.5: loss=-0.000  grad=[-0.5  0.5]
π_new=0.6: loss=-0.000  grad=[-0.3  0.3]
π_new=0.9: loss=+0.300  grad=[0.  0.9]

#DeepRL
#ReinforcementLearning
#PythonສຳລັບຄົນLaos
#AI
#ບອກຕໍ່ຄວາມຮູ້

Main Menu

Change Language

Currency

Advanced actor-critic: DDPG, TD3, SAC, PPO

Key takeaways