Improving agents' behaviors
SARSA, Q-learning and double Q-learning.
To improve a policy without knowing the MDP, we learn Q(s,a) instead of V and keep exploring as we go. Every method follows the GPI pattern: evaluate Q → pick better actions → repeat.
The Q-learning code learns purely by playing, yet reaches the same policy as value iteration in lesson 3, which needed the full MDP.
SARSA (on-policy): Q(s,a) ← Q(s,a) + α [ r + γ·Q(s',a') − Q(s,a) ]
Q-learning (off-policy): Q(s,a) ← Q(s,a) + α [ r + γ·maxₐ' Q(s',a') − Q(s,a) ]
- On-policy (SARSA): learns the value of the policy being played, exploration included
- Off-policy (Q-learning): plays one policy but learns the value of the best one
- Double Q-learning: two Q tables to fix the overestimation caused by max
- GLIE: explore everything infinitely often, but become greedy in the limit
Key takeaways
- Prediction = evaluate a policy; control = find the best policy
- Double learning returns in Double DQN and TD3
import numpy as np
import gymnasium as gym
env = gym.make("FrozenLake-v1", is_slippery=True)
n_s, n_a = env.observation_space.n, env.action_space.n
Q = np.zeros((n_s, n_a))
rng = np.random.default_rng(0)
gamma, alpha = 0.99, 0.1
n_episodes = 20000
for ep in range(n_episodes):
eps = max(0.01, 1.0 - ep / (0.8 * n_episodes)) # epsilon ຫຼຸດລົງເລື້ອຍໆ
s, _ = env.reset(seed=ep)
done = False
while not done:
a = rng.integers(n_a) if rng.random() < eps else int(np.argmax(Q[s]))
s2, r, term, trunc, _ = env.step(a)
target = r + gamma * Q[s2].max() * (not term) # off-policy: ໃຊ້ max
Q[s, a] += alpha * (target - Q[s, a])
s, done = s2, term or trunc
pi = Q.argmax(axis=1)
wins = 0
for ep in range(1000):
s, _ = env.reset(seed=100000 + ep)
done = False
while not done:
s, r, term, trunc, _ = env.step(pi[s])
done = term or trunc
wins += r
print("π =", "".join("←↓→↑"[a] for a in pi))
print(f"ອັດຕາສຳເລັດ: {wins / 10:.1f}%")Output
π = ←↑↑↑←←←←↑↓←←←→↓← ອັດຕາສຳເລັດ: 73.4%
- #DeepRL
- #ReinforcementLearning
- #PythonສຳລັບຄົນLaos
- #AI
- #ບອກຕໍ່ຄວາມຮູ້