Lesson 02 · Foundations

MDPs: the engine of the environment

The maths language every RL problem is written in.

A Markov decision process (MDP) is the mathematical framework for sequential decision problems under uncertainty.

The example inspects the MDP of FrozenLake: a 4×4 frozen lake where you walk from the top-left corner to the goal at the bottom-right without falling into a hole. The ice is slippery — whichever way you press, you only go that way 1/3 of the time.

MDP = (S, A, T, R, S₀, γ, H) S set of states A set of actions T T(s'|s,a) chance of s' R R(s,a,s') reward S₀ initial state distribution γ discount factor (0..1) H horizon (episode length)

States describe the world. If the agent can't see all of it, it is a POMDP and what it sees is an observation
Markov property: the future depends only on the current state, not on the history
Transitions and rewards together form the environment's model, usually assumed not to change over time
Actions can be discrete (left/right) or continuous (a steering angle of 0–30°)

Key takeaways

Every RL problem can be written as an MDP: states, actions, transitions, rewards, γ
Lesson 3 assumes the agent knows T and R; from lesson 4 on, it doesn't

deep-rl-ep02-mdp.py

import gymnasium as gym

env = gym.make("FrozenLake-v1", is_slippery=True)
P = env.unwrapped.P          # P[state][action] = [(prob, next_state, reward, done), ...]

print("ຈຳນວນ state:", env.observation_space.n)   # 16
print("ຈຳນວນ action:", env.action_space.n)       # 4: ຊ້າຍ, ລົງ, ຂວາ, ຂຶ້ນ
for prob, s_next, r, done in P[14][2]:            # state 14, ກົດ "ຂວາ"
    print(f"p={prob:.2f} -> s'={s_next:2d}  r={r}  ຈົບ={done}")

Output

ຈຳນວນ state: 16
ຈຳນວນ action: 4
p=0.33 -> s'=14  r=0  ຈົບ=False
p=0.33 -> s'=15  r=1  ຈົບ=True
p=0.33 -> s'=10  r=0  ຈົບ=False

#DeepRL
#ReinforcementLearning
#PythonສຳລັບຄົນLaos
#AI
#ບອກຕໍ່ຄວາມຮູ້

Main Menu

Change Language

Currency

MDPs: the engine of the environment

Key takeaways