Search with Learned Policies and Values
Classical game programs rely on search plus handcrafted knowledge. Modern systems can learn which actions deserve attention and how promising a state is, while retaining search as a deliberative process.
Two learned functions
| Function | Question answered | Use in search |
|---|---|---|
| Policy p(a | s) | Which actions look promising in state s? | Orders or prioritizes branches |
| Value v(s) | How favorable is state s? | Evaluates a leaf without playing to the end |
The policy does not have to select the final action directly. It can guide search toward promising branches. Likewise, a value estimate need not be trusted blindly; search can reveal tactics that a static estimate misses.
The complementary roles
- Learning provides generalization: similar states can receive similar predictions without being explicitly enumerated.
- Search provides local computation: the agent examines consequences specific to the current decision.
- Self-play provides data: the agent can improve without a fixed collection of human demonstrations.
- Verification remains important: a learned model can be confidently wrong.
Central idea: Learning tells the search where to look and how to evaluate what it finds; search improves the decision beyond the network's immediate prediction.
From handcrafted to learned components
| Traditional component | Learned counterpart |
|---|---|
| Human-designed move ordering | Policy network |
| Hand-weighted evaluation function | Value network |
| Opening book | Patterns acquired through training or self-play |
| Complete supplied transition rules | In model-based learning, a learned predictive model |