Introduction to value-based deep RL
When the Q-table is too big, a neural network takes over.
A Q-table breaks down when there are too many states. CartPole, for example, has 4 real-valued variables, so its states are infinite.
The fix is a neural network that approximates Q(s,a) and generalises to states it has never seen. NFQ was the first attempt: collect data, compute targets, fit the network several times, repeat.
High-dimensionalOne state has many variables, e.g. an 84×84-pixel image
ContinuousAt least one variable can take any value, e.g. an angle of 0.0137 rad
Choices to make
| Component | Common choice |
|---|---|
| What to approximate | Q(s,a) for discrete actions |
| Network shape | State in → Q for every action out |
| Target | Q-learning TD target: r + γ·max Q(s') |
| Exploration | ε-greedy |
| Loss / Optimizer | MSE, Huber · RMSprop, Adam |
Two big problems
- Data isn't IID: consecutive experiences are correlated, but optimisers expect IID data
- Moving targets: targets come from the same network that is being updated
Key takeaways
- Deep RL = everything so far + supervised learning for approximation
- Lesson 9 fixes both with a replay buffer and a target network
- #DeepRL
- #ReinforcementLearning
- #PythonສຳລັບຄົນLaos
- #AI
- #ບອກຕໍ່ຄວາມຮູ້