AlphaZero: Search, Learning, and Self-Play
AlphaZero combines a neural network, Monte Carlo tree search, and self-play in a repeated improvement cycle. The network guides search; search produces a stronger action distribution; self-play turns those improved decisions into training data.
The decision cycle
- The network receives a game state and predicts a policy over moves and a state value.
- MCTS uses the policy to prioritize actions and the value to evaluate newly expanded positions.
- The agent selects a move using the visit counts produced by search.
- After the game ends, its outcome supplies a training target.
- The network is updated to better predict search policies and game outcomes.
state → policy/value network → guided MCTS → improved policy → action
Why the combination matters
| Network alone | Search alone | Combined system |
|---|---|---|
| Fast generalized judgment, but can miss tactics | Examines consequences, but faces enormous branching | Learned judgment focuses and evaluates search |
Training targets
A simplified training example contains:
- the state s encountered during self-play;
- the MCTS visit distribution π, used as a policy target;
- the final result z, used as a value target.
A simplified loss combines policy error, value error, and regularization:
L = (z − v)² − π · log p + regularization
What is and is not supplied
AlphaZero is not given human opening books or hand-engineered position values, but it is given the legal actions and correct game dynamics. It learns by playing the game against itself.
Primary source
D. Silver et al., “A General Reinforcement Learning Algorithm that Masters Chess, Shogi, and Go,” Science, 2018. See also the AlphaZero overview.