MuZero: Planning with a Learned Model
AlphaZero searches with the true game dynamics. MuZero instead learns an internal model designed to predict the quantities most useful for planning: rewards, values, and policies.
Three learned components
| Component | Role |
|---|---|
| Representation | Transforms the observation history into an internal state |
| Dynamics | Predicts the next internal state and immediate reward after an action |
| Prediction | Predicts a policy and value from an internal state |
observations → representation → latent state → dynamics + action → next latent state
A model for decision-making
The learned state need not reconstruct every visible detail or resemble a human description of the game. It needs to preserve information relevant to predicting rewards, values, and useful policies. MCTS can then simulate action sequences inside this learned latent model.
AlphaZero and MuZero
| Question | AlphaZero | MuZero |
|---|---|---|
| Are correct transition rules used during search? | Yes | No; dynamics are learned |
| Is a policy learned? | Yes | Yes |
| Is a value learned? | Yes | Yes |
| Is planning performed? | Yes, with MCTS | Yes, with MCTS in latent states |
Limitations and questions
- A learned dynamics model can be wrong outside familiar experience.
- Prediction errors can compound across imagined steps.
- The latent state may be difficult to interpret.
- Training and search can require substantial computation.
- Success in a game does not by itself demonstrate a general, transferable world model.
Primary source
J. Schrittwieser et al., “Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model,” Nature, 2020. A readable overview is available on the MuZero project page.