Policy p(a|s)
Given state s, which actions a look worth considering?
Nature · 2016 An interactive paper guide
How a machine learned to see the whole board.
AlphaGo did not beat Go by searching everything. It learned which futures deserved attention—and which could be safely ignored.
- ≈30M
- expert positions
- 57%
- exact move match
- 494/495
- program games won
P
Policy prior narrows hundreds of legal moves to a promising few.
V
Value estimate judges a future position without finishing every game.
Follow the signal from human games to tree search
THE ENTIRE IDEA, IN ONE SENTENCE
A policy chooses where to look. A value judges what it finds. A search turns both into a decision.
01 The problem
Go breaks brute force.
A 19×19 board begins with 361 empty intersections. Even after illegal or nonsensical moves are removed, a typical position may offer roughly 250 choices. Looking only a few turns ahead creates an astronomical tree.
BRANCHING LAB
Every extra move multiplies the problem.
Set the number of candidate moves AlphaGo considers at each step. The naive tree grows as
bd: branching factor b, depth d.
After only 4 plies
3,906,250,000
possible paths
Policy
Value v(s)
From state s, what final outcome should the current player expect?
Search N(s,a)
Which promising actions continue to look good after simulated futures?
Corrects the networks02 The blueprint
First imitate. Then compete. Finally, search.
AlphaGo is not one model. It is a training pipeline and a live decision engine. Select each stage to see what enters, what comes out, and why the next stage needs it.
STAGE 01
INPUT
Strong human Go games from KGSLEARNS
Which move an expert chose from each boardUNLOCKS
A direct, high-quality training signalHuman games give AlphaGo a vocabulary of sensible Go before sparse win/loss rewards enter the picture.
03 Supervised policy
Give the machine human intuition.
Before self-play, AlphaGo learns a simpler skill: look at a board and predict the exact move a strong human played. This becomes the search engine’s sense of where promising branches begin.
ladder escape
liberties
19 × 19 × 48
board feature planes
5×5 conv
1×1 + softmax
MOVE PROBABILITY MAP
Hover over a signal
THE MODEL
pσ(a | s)
Probability of move a, given board state s, under weights σ.
THE LOSS
LSL = −log pσ(aexpert | s)
Make the expert move more likely. This is ordinary cross-entropy classification over legal moves.
04 Reinforcement learning
Stop predicting humans. Start optimizing wins.
The policy network keeps the same architecture and starts from the supervised weights. What changes is the objective: moves now receive credit or blame from the final result of self-play.
BEFORE
“Did I match the expert?”
maximize pσ(ahuman|s)
AFTER
“Did this sequence win?”
maximize E[z]
POLICY GRADIENT
Δρ ∝ zt ∇ρ log pρ(at | st)
The log-probability gradient points toward making the sampled move more likely. The terminal outcome
zt decides whether to follow that direction or reverse it.
Sampled move
Q16
pρ = 0.18
↑
After update
More likely
Winning actions are reinforced.
WHY PLAY PAST SELVES?
A moving opponent can make training unstable.
AlphaGo samples opponents from earlier checkpoints and adds a new snapshot to the pool every 500 iterations. The current policy must improve against a small league—not exploit one temporary self.
- Initialize
ρ = σfrom human imitation - Sample actions from
pρ(·|s) - Score only the terminal win or loss
- Push every move in the game using that outcome
05 The value network
Learn to judge a future without finishing it.
This is the conceptual leap. Instead of rolling every promising position all the way to the end, AlphaGo trains a neural network to predict the expected outcome directly from the board.
board state
s
vθ(s)
+0.60
80% win probability
TARGET
vθ(s) ≈ E[z | s, pρ]
Expected outcome if both players continue under the strong reinforcement-learned policy.
REGRESSION LOSS
L(θ) = ½(z − vθ(s))²
The error pushes winning positions toward +1 and losing positions toward −1.
GRADIENT UPDATE
Δθ ∝ (z − vθ(s)) ∇θvθ(s)
Large mistakes create large corrections; already-correct estimates move only a little.
A DATASET TRAP
Two hundred positions from one game are not two hundred independent ideas.
Adjacent positions differ by one move and share one outcome. Training on many positions from each game produced a large train–test gap. The fix: keep one position from each newly generated self-play game.
train .19
test .37
train .226
test .234
06 APV-MCTS
Where all the pieces meet.
One simulation walks from the current board to a leaf, evaluates that future, and sends the evidence back to every move along the path. Repeat thousands of times; choose the most visited root move.
s0root
AP .12 · N 18
BP .36 · N 42
CP .08 · N 6
B₁Q +.47
B₂Q +.22
sLnew leaf
PHASE 01
Walk toward value, but leave room for surprise.
At each node, select the action with the highest known value plus an exploration bonus. High-prior, under-visited moves receive more attention.
at = argmaxa[ Q(st,a) + u(st,a) ]
INTERACTIVE PUCT LAB
Which branch will search choose?
score = Q + cP · √Nparent / (1 + N)
Adjust any prior, value, or visit count. The highest Q + U wins the next simulation.
A
1.25
B
3.00
C
0.82
Next simulation selects Move B
Its low visit count creates a large exploration bonus.
LEAF EVALUATION
Two imperfect witnesses, mixed together.
The deep value network is smart but approximate. The fast rollout reaches a real terminal score, but follows a weaker policy and is noisy. Original AlphaGo gave both a voice.
VALUE NETWORK+0.62
ROLLOUT−1.00
V(sL) = (1−λ)vθ + λz
−0.19
01 What exactly is stored on each edge?
P(s,a) is the supervised policy prior. Nv, Wv track value-network evidence; Nr, Wr track rollout evidence.
Q = (1−λ) Wv/Nv + λ Wr/Nr
02 Why do signs flip during backup?
Values are from the player-to-move perspective. The player changes at every edge, so a good leaf for one player is bad for the other.
Vt = (−1)L−t VL
03 Why choose the most visited move?
A high Q can come from a few lucky evaluations. A high visit count means a move repeatedly survived the search policy, making it more robust to outliers.
a* = argmaxa N(s0,a)
04 What is virtual loss?
A thread temporarily makes its chosen edge look busier and worse. Other CPU threads spread out instead of duplicating the same path. When the simulation returns, the fake loss is removed and the real result is added.
Nr ← Nr + 3 · Wr ← Wr − 3
07 Systems engineering
The algorithm had to become a live engine.
Tree traversal is irregular and branch-heavy; neural networks thrive on dense parallel math. AlphaGo separates them, keeps both busy, and accepts slightly stale search statistics for much higher throughput.
CPUmany search threads
selectvirtual lossrolloutbackup z
positions queue →← predictions return
GPUbatched neural inference
batch leavespσ priorsvθ valuesbackup v
SINGLE MACHINE
One tightly coupled engine
40 search threads
48 CPUs
8 GPUs
DISTRIBUTED ALPHAGO
Search scaled across machines
1,202 CPUs
176 GPUs
many async workers
Never idle on a GPU.
Use a cheap placeholder tree policy, queue deep-policy evaluation, and replace priors when the result arrives.
Let evidence arrive twice.
CPU rollout results and GPU value predictions return at different times, so AlphaGo backs them up asynchronously.
Reuse yesterday’s thinking.
After a real move, the chosen child becomes the new root. Its entire searched subtree remains useful.
Accuracy per second wins.
The most accurate network is not automatically the strongest engine if it reduces the number of searches that fit the clock.
08 Results & meaning
No component wins alone.
The breakthrough was the composition: learned priors, learned evaluation, completed rollouts, statistical search, and enough systems throughput to run the loop under a real match clock.
WIN RATE VS OTHER PROGRAMS
99.8%
494 wins in 495 games
FORMAL MATCH
AlphaGo5
Fan Hui0
First computer victory over a human professional on full-sized Go without handicap.
THE MODEL TO KEEP IN YOUR HEAD
Policy
“These branches look promising.”
Value
“This future looks favorable.”
Search
“This branch survived repeated tests.”
Systems
“Run enough tests before the clock expires.”
AlphaGo turns an impossible exhaustive search into a focused conversation between learned intuition and statistical evidence.
CAN YOU EXPLAIN IT NOW?
Five questions that prove the paper has clicked.
- 01
What problem does this component solve?
- 02
What enters it, and what comes out?
- 03
How is it trained or computed?
- 04
Why is the supervised policy still used after RL?
- 05
Why does MCTS choose visits, not the largest raw Q?
+ Field notes
The notation, decoded.
sState
The complete current Go position.
aAction
A legal move from the current state.
P(s,a)Policy prior
How promising the network thinks a move looks before search evidence.
N(s,a)Visit count
How many simulations traversed an edge.
W(s,a)Total value
The accumulated outcomes backed up through an edge.
Q(s,a)Mean action value
How well an action has performed across its simulations.
zTerminal outcome
+1 for a win, −1 for a loss, from the relevant player’s perspective.
λMixing weight
How AlphaGo balances rollout and value-network evidence.