Balancing exploration and exploitation
Many slot machines: which one do you play?
Without the MDP, the agent only sees reward numbers like +1 or −4, so it never knows whether a better option is hiding somewhere.
This is the exploration vs exploitation problem. Reward lost by not picking the best option is called regret.
In the code, pure greedy gets stuck on the first machine it likes, while UCB uses uncertainty to choose and gets closer to the best possible average.
| Strategy | Idea |
|---|---|
| ε-greedy | Usually pick the best; with probability ε pick at random |
| Decaying ε-greedy | Explore a lot at first, then shrink ε |
| Optimistic initialization | Start Q high to force every option to be tried |
| Softmax | Sample in proportion to Q |
| UCB | Add an "uncertainty bonus" to rarely tried options |
| Thompson sampling | Sample from each option's Bayesian belief |
Key takeaways
- No exploration = stuck with a bad choice; too much = wasted reward
- Good strategies use uncertainty, not blind randomness
import numpy as np
rng = np.random.default_rng(0)
true_means = rng.normal(0, 1, size=10) # ເຄື່ອງ slot 10 ເຄື່ອງ (ບໍ່ຮູ້ຄ່າຈິງ)
def run(strategy, steps=1000):
Q, N, total = np.zeros(10), np.zeros(10), 0.0
for t in range(1, steps + 1):
a = strategy(Q, N, t)
r = rng.normal(true_means[a], 1)
N[a] += 1
Q[a] += (r - Q[a]) / N[a] # ຄ່າສະເລ່ຍແບບ incremental
total += r
return total / steps
def epsilon_greedy(Q, N, t, eps=0.1):
return rng.integers(10) if rng.random() < eps else int(np.argmax(Q))
def ucb(Q, N, t, c=2.0):
if (N == 0).any():
return int(np.argmin(N)) # ລອງທຸກເຄື່ອງກ່ອນ
return int(np.argmax(Q + c * np.sqrt(np.log(t) / N)))
print(f"ຄ່າສູງສຸດທີ່ເປັນໄປໄດ້ : {true_means.max():.3f}")
print(f"greedy (eps=0) : {run(lambda Q, N, t: epsilon_greedy(Q, N, t, 0)):.3f}")
print(f"epsilon-greedy (0.1) : {run(epsilon_greedy):.3f}")
print(f"UCB : {run(ucb):.3f}")Output
ຄ່າສູງສຸດທີ່ເປັນໄປໄດ້ : 1.304 greedy (eps=0) : 0.642 epsilon-greedy (0.1) : 1.039 UCB : 1.195
- #DeepRL
- #ReinforcementLearning
- #PythonສຳລັບຄົນLaos
- #AI
- #ບອກຕໍ່ຄວາມຮູ້