Exploration, Exploitation, and UCT
During selection, MCTS must choose between actions already known to be strong and actions that have not yet been tested adequately. UCT turns this into a numerical tradeoff.
UCT(i) = Q(i) / N(i) + C √(ln N(parent) / N(i))
| Term | Meaning |
|---|---|
| Q(i) / N(i) | Average value observed for child i: exploitation |
| √(ln N(parent) / N(i)) | Uncertainty bonus for less-visited children: exploration |
| C | Controls the strength of exploration |
A small example
Suppose the parent has 100 visits and two children have the following records. Let C = 1.4.
| Child | Wins / visits | Average value | Interpretation |
|---|---|---|---|
| A | 42 / 60 | 0.70 | Strong evidence, relatively certain |
| B | 6 / 10 | 0.60 | Lower average, but much less explored |
UCT may select B because its exploration bonus is larger. As B receives more visits, that bonus shrinks. If B continues to perform poorly, selection shifts toward A.
Important qualifications
- Values must be backed up from the viewpoint of the player making the decision.
- A draw, win, and loss need consistent numerical values.
- The choice of C affects behavior but does not change the game rules.
- Weak simulations can produce biased estimates.
- In modern systems, learned policies and value estimates often replace uninformed selection and random rollouts.
Check your understanding
- Why does an unvisited child require special treatment?
- What happens when C is close to zero?
- What happens when C is extremely large?
- Why is the exploration bonus not a statement about a move's true quality?