Lesson 07 · Tabular RL

Achieving goals more effectively and efficiently

SARSA(λ), Q(λ) and Dyna-Q: get more out of every sample.

This lesson makes the agent more sample efficient in two ways: spreading credit faster, and learning a model to plan with.

In the Dyna-Q code, once the goal is found the first time, 20 planning updates per step propagate the reward back very quickly. By the next game the agent already walks the shortest path (6 steps).

Faster credit: SARSA(λ), Q(λ)Eligibility traces send the reward back to many recent states at once

Learn a model and plan: Dyna-QRemember "doing a in s led to s' with r", then use that model for extra Q updates without real play

Trajectory sampling is like Dyna-Q but imagines continuous paths from the current state. Model-based methods suit problems where data is expensive, such as real robots.

Key takeaways

Effective = solves the task; efficient = uses little data
Next comes "sampled" feedback: too many states to see them all

deep-rl-ep07-dyna-q.py

import numpy as np
import gymnasium as gym

env = gym.make("FrozenLake-v1", is_slippery=False)
n_s, n_a = env.observation_space.n, env.action_space.n
Q = np.zeros((n_s, n_a))
model = {}                       # (s, a) -> (r, s2, term)  ໂມເດວທີ່ຮຽນຈາກປະສົບການ
rng = np.random.default_rng(0)
gamma, alpha, eps, n_planning = 0.95, 0.5, 0.1, 20

def update(s, a, r, s2, term):
    target = r + gamma * Q[s2].max() * (not term)
    Q[s, a] += alpha * (target - Q[s, a])

for ep in range(30):
    s, _ = env.reset(seed=ep)
    done, steps = False, 0
    while not done:
        if rng.random() < eps or Q[s].max() == Q[s].min():
            a = rng.integers(n_a)
        else:
            a = int(np.argmax(Q[s]))
        s2, r, term, trunc, _ = env.step(a)
        update(s, a, r, s2, term)                  # 1) ຮຽນຈາກປະສົບການຈິງ
        model[(s, a)] = (r, s2, term)              # 2) ບັນທຶກລົງໂມເດວ
        keys = list(model)
        for _ in range(n_planning):                # 3) "ຈິນຕະນາການ" ຈາກໂມເດວ
            ps, pa = keys[rng.integers(len(keys))]
            update(ps, pa, *model[(ps, pa)])
        s, done, steps = s2, term or trunc, steps + 1
    if r == 1:
        print(f"ເກມທີ {ep:2d}: ໄປຮອດເປົ້າໝາຍໃນ {steps} ກ້າວ")

Output

ເກມທີ 20: ໄປຮອດເປົ້າໝາຍໃນ 43 ກ້າວ
ເກມທີ 21: ໄປຮອດເປົ້າໝາຍໃນ 7 ກ້າວ
ເກມທີ 22: ໄປຮອດເປົ້າໝາຍໃນ 6 ກ້າວ
ເກມທີ 23: ໄປຮອດເປົ້າໝາຍໃນ 6 ກ້າວ
ເກມທີ 24: ໄປຮອດເປົ້າໝາຍໃນ 6 ກ້າວ
ເກມທີ 25: ໄປຮອດເປົ້າໝາຍໃນ 6 ກ້າວ
ເກມທີ 26: ໄປຮອດເປົ້າໝາຍໃນ 7 ກ້າວ
ເກມທີ 28: ໄປຮອດເປົ້າໝາຍໃນ 6 ກ້າວ
ເກມທີ 29: ໄປຮອດເປົ້າໝາຍໃນ 6 ກ້າວ

#DeepRL
#ReinforcementLearning
#PythonສຳລັບຄົນLaos
#AI
#ບອກຕໍ່ຄວາມຮູ້

Main Menu

Change Language

Currency

Achieving goals more effectively and efficiently

Key takeaways