Dueling DDQN and prioritized replay
An RL-aware network and replaying the experiences that matter.
The dueling architecture splits Q into V(s), shared by all actions, and A(s,a). Every experience therefore improves V(s), so learning is faster.
Prioritized experience replay (PER) samples experiences with a high |TD error| more often and corrects the bias with importance-sampling weights.
Q(s,a) = V(s) + A(s,a) − meanₐ' A(s,a')
PER: P(i) ∝ |δᵢ|^α wᵢ = (N · P(i))^(−β)
Weak spots of value-based methods
- Very sensitive to hyperparameters
- Assume the environment is Markov
- Deadly triad: bootstrapping + off-policy + function approximation can diverge
Key takeaways
- NFQ → DQN → DDQN (more stable) → Dueling + PER (more sample efficient)
import torch, torch.nn as nn
class DuelingQNet(nn.Module):
"""Q(s,a) = V(s) + A(s,a) - mean_a A(s,a)"""
def __init__(self, n_obs, n_act, hidden=128):
super().__init__()
self.body = nn.Sequential(nn.Linear(n_obs, hidden), nn.ReLU())
self.value = nn.Linear(hidden, 1) # ຫົວ V(s): ໃຊ້ຮ່ວມທຸກ action
self.advantage = nn.Linear(hidden, n_act) # ຫົວ A(s,a)
def forward(self, s):
h = self.body(s)
v, adv = self.value(h), self.advantage(h)
return v + adv - adv.mean(dim=-1, keepdim=True)
net = DuelingQNet(n_obs=4, n_act=2)
print(net(torch.zeros(3, 4)).shape) # torch.Size([3, 2])
# PER: ຄວາມສຳຄັນ ∝ |TD error|^α, ແກ້ bias ດ້ວຍ importance-sampling weight
td_errors = torch.tensor([0.1, 2.0, 0.5, 0.05])
alpha, beta = 0.6, 0.4
p = (td_errors.abs() + 1e-6) ** alpha
probs = p / p.sum()
w = (len(probs) * probs) ** (-beta)
w = w / w.max()
print("ໂອກາດຖືກສຸ່ມ:", probs.numpy().round(3))
print("IS weight :", w.numpy().round(3))Output
torch.Size([3, 2]) ໂອກາດຖືກສຸ່ມ: [0.097 0.585 0.254 0.064] IS weight : [0.847 0.413 0.575 1. ]
- #DeepRL
- #ReinforcementLearning
- #PythonສຳລັບຄົນLaos
- #AI
- #ບອກຕໍ່ຄວາມຮູ້