Dueling DDQN ແລະ PER
ອອກແບບ network ໃຫ້ເຂົ້າໃຈ RL ແລະ ເລືອກປະສົບການທີ່ມີຄ່າ.
Dueling architecture ແຍກ Q ອອກເປັນ V(s) ທີ່ໃຊ້ຮ່ວມກັນທຸກ action ແລະ A(s,a). ທຸກປະສົບການຈຶ່ງຊ່ວຍປັບ V(s) ໄດ້ ເຮັດໃຫ້ຮຽນໄວຂຶ້ນ.
Prioritized Experience Replay (PER) ສຸ່ມປະສົບການທີ່ມີ |TD error| ສູງ (ເຮັດໃຫ້ "ແປກໃຈ") ຫຼາຍກວ່າ ແລ້ວແກ້ bias ດ້ວຍ importance sampling weight.
Q(s,a) = V(s) + A(s,a) − meanₐ' A(s,a')
PER: P(i) ∝ |δᵢ|^α wᵢ = (N · P(i))^(−β)
ຈຸດອ່ອນຂອງ value-based
- ອ່ອນໄຫວກັບ hyperparameter ຫຼາຍ
- ສົມມຸດວ່າ environment ເປັນ Markov (state ມີຂໍ້ມູນຄົບ)
- Deadly triad: bootstrapping + off-policy + function approximation ອາດເຮັດໃຫ້ diverge. ໃຊ້ target network, replay buffer, double learning ແລະ learning rate ນ້ອຍພໍດີ
ສິ່ງທີ່ຄວນຈື່
- NFQ → DQN → DDQN (ສະຖຽນຂຶ້ນ) → Dueling + PER (ໃຊ້ຂໍ້ມູນຄຸ້ມຂຶ້ນ)
import torch, torch.nn as nn
class DuelingQNet(nn.Module):
"""Q(s,a) = V(s) + A(s,a) - mean_a A(s,a)"""
def __init__(self, n_obs, n_act, hidden=128):
super().__init__()
self.body = nn.Sequential(nn.Linear(n_obs, hidden), nn.ReLU())
self.value = nn.Linear(hidden, 1) # ຫົວ V(s): ໃຊ້ຮ່ວມທຸກ action
self.advantage = nn.Linear(hidden, n_act) # ຫົວ A(s,a)
def forward(self, s):
h = self.body(s)
v, adv = self.value(h), self.advantage(h)
return v + adv - adv.mean(dim=-1, keepdim=True)
net = DuelingQNet(n_obs=4, n_act=2)
print(net(torch.zeros(3, 4)).shape) # torch.Size([3, 2])
# PER: ຄວາມສຳຄັນ ∝ |TD error|^α, ແກ້ bias ດ້ວຍ importance-sampling weight
td_errors = torch.tensor([0.1, 2.0, 0.5, 0.05])
alpha, beta = 0.6, 0.4
p = (td_errors.abs() + 1e-6) ** alpha
probs = p / p.sum()
w = (len(probs) * probs) ** (-beta)
w = w / w.max()
print("ໂອກາດຖືກສຸ່ມ:", probs.numpy().round(3))
print("IS weight :", w.numpy().round(3))ຜົນລັບ
torch.Size([3, 2]) ໂອກາດຖືກສຸ່ມ: [0.097 0.585 0.254 0.064] IS weight : [0.847 0.413 0.575 1. ]
- #DeepRL
- #ReinforcementLearning
- #PythonສຳລັບຄົນLaos
- #AI
- #ບອກຕໍ່ຄວາມຮູ້