Monte Carlo Tree Search: Search by Sampling
Monte Carlo tree search (MCTS) builds a game tree selectively. Instead of expanding every action to a fixed depth, it repeatedly samples play and concentrates computation on decisions that appear promising.
The four phases of MCTS
- Selection: Starting at the root, choose child nodes according to a tree policy until reaching a node that is not fully expanded.
- Expansion: Add one or more legal successor states.
- Simulation or evaluation: Estimate the value of the new state, originally through a rollout to a terminal state and increasingly through a learned value function.
- Backpropagation: Update visit counts and value estimates along the path to the root.
These four phases repeat until the time or simulation budget expires. The agent commonly chooses the root action with the largest visit count.
Why sampling helps
A uniform search spends equal effort on strong and obviously weak moves. MCTS creates an asymmetric tree: likely good variations receive many visits, while unpromising variations receive few. It is an anytime method—the answer generally improves as more simulations are allowed.
| Alpha–beta | MCTS |
|---|---|
| Usually searches to a depth limit | Expands a selective, asymmetric tree |
| Depends heavily on an evaluation function | Can begin with sampled outcomes |
| Deterministic given the same ordering | Normally stochastic |
| Strong when good move ordering and evaluation are available | Strong when sampling provides useful value estimates |
Think about it: In a game with 100 legal moves, why might MCTS be preferable to expanding every move equally to depth six? When might random rollouts be misleading?
Minimal pseudocode
while computation remains:
leaf = SELECT(root)
child = EXPAND(leaf)
value = EVALUATE(child)
BACKPROPAGATE(child, value)
return most_visited_child(root)