SELF-PLAY NOTES
Curriculum Mining
Training search-free board-game policies with diverse resets and sparse rewards.
Introduction
In this project I test whether pure self-play PPO, without search, can learn strong policies for Othello, Hex and Go.
The comparison is controlled. The policy uses the same network architecture as Pgx's published Gumbel AlphaZero baselines and is trained with comparable compute. The difference is the objective. The baseline uses MCTS to produce search-improved policy targets. My policy is trained with PPO from sparse terminal ±1 rewards. There is no search, teacher or reward shaping during training, and no search at inference.
The resulting raw policy beats the published baseline networks at Othello, Hex 11×11 and Go 9×9. This post is about the training ingredient I think is most important to that result: resetting games into random legal mid-game positions.
Random resets create a curriculum from the states the policy can currently learn.
Score against each published baseline over 10,240 games. Raw greedy policy on both sides.
Theory
Raw PPO struggles here because the only reward is the terminal result. If every game starts from the opening, an early action may not receive a learning signal for another 50 moves. Early in training both players are bad, so they also visit a narrow distribution of states produced by their current bad policies.
One way to shorten this credit-assignment problem is to train from positions closer to termination. The policy can learn a useful value function there first, then propagate that information toward earlier positions. In other words, solving the full game requires a sequence of smaller, currently solvable problems.
This turns credit assignment into a curriculum problem. We need intermediate states that produce useful learning signals, but we do not know which states should come next or the order in which their behaviours must be acquired. The order is also not necessarily a simple progression from late game to early game: learning one behaviour can make a qualitatively different set of positions solvable.
This is the argument in Curriculum Is Key and in Stanley and Lehman's Why Greatness Cannot Be Planned: useful stepping stones often cannot be specified from the final objective.
Random resets avoid choosing the stepping stones manually. Each sampled state is a candidate intermediate problem. PPO learns from whichever states are currently within reach; the resulting behaviours then make a different set of states learnable. This is a form of curriculum mining: an open-ended search over possible next challenges inside the finite state space of the game.
Method
The training method has two changes to standard self-play PPO:
1. Random legal resets
This was inspired directly by OmniReset. When an environment terminates, I generate a new starting state by taking a random number of random legal moves from the opening, up to 40–50 plies depending on the game. PPO then trains from that state using the same sparse terminal reward.
2. A bank of old checkpoints
Naive self-play was unstable: performance peaked and then drifted as both sides changed together. I keep a ring of 12 frozen policy checkpoints and add one every 100 iterations. Half of the environments play against a checkpoint sampled from the bank, and the loss is applied only to the learner's moves. This keeps the opponent distribution stable enough that new updates do not immediately erase earlier progress.
今 t−1t−2t−3t−4…t−12
Interactive Othello example
Choose a reset depth below to sample a legal training state, then play black against the trained Othello checkpoint. This is how the model trains: it sees a random reset, then plays the position out. The model takes one greedy policy action with no search, matching the evaluation setup.
OTHELLO RESET DISTRIBUTION
Sample a legal training state
Your move. Choose a marked cell.
Loading Othello AZNet…
Opening position · 60 empty squares
TERMINAL VERSION
Play the same checkpoint locallyThe repository also ships the original Flax weights and terminal player.
Results
Score against each published baseline over 10,240 games. Raw greedy policy on both sides; no search at inference.
How total environment steps were counted
One PPO iteration executes \(4{,}096 \times 32 = 131{,}072\) environment steps. The final policies use:
8,000 iterations × 4,096 environments × 32 steps = 1,048,576,000
For Gumbel AlphaZero, every played frame costs 32 MCTS simulations plus the played move, so each frame counts as 33 environment interactions:
Othello / Hex: 100 × 1,024 × 256 × 33 = 865,075,200
Go 9×9: 200 × 1,024 × 256 × 33 = 1,730,150,400
Within that comparison, the result is strong. Using the same 1.83M-parameter AZNet architecture, sparse terminal ±1 reward and no search in training or inference, the policy beats the three published networks. The compute-matched Othello checkpoint scores 0.579 against the baseline. On Go 9×9, the final policy beats the baseline using 0.61× its environment interactions under the symmetric accounting used here.
Conclusion
Random resets make sparse-reward PPO work by exposing the policy to short credit-assignment problems before it can solve full games from the opening. The checkpoint bank solves a separate problem: it stops self-play from drifting as both players change together.
The comparison is limited. These are published reference networks rather than state-of-the-art opponents, and both sides are evaluated as greedy raw policies. AlphaZero normally uses search at play time. The games are also fixed and finite, so random resets only approximate open-ended search over possible next challenges.
Still, the sequence of useful challenges was not specified ahead of time. It emerged from the states sampled by the reset distribution and what the policy could learn at each point in training. Random resets do not avoid curriculum; they search for one.