Pan1ni
Making a goal seeking policy with video pretraining only.
Every clip is a procedurally generated level the model has never seen, reaching a goal it is shown as an image, never told as text.
Introduction
This is the tweet that inspired this project. Pantograph recently released Pan-1, in which they show that it is possible to get goal-seeking behaviour (the kind you would normally train an RL policy for) from general, scalable pretraining. Their blog post doesn't mention their exact architecture, loss function, or design choices, which inspired me to have a go at the same challenge myself: in a simpler game (NetHack), and in only two days (I have a job now).
The Pan-1 idea
The core idea behind Pan-1 is elegant and simple. Training RL policies across many tasks is not scalable. It would be much better if we could utilise internet-scale video and still achieve the goal-seeking, intelligent policies that emerge from RL. But how can we do this given we can't "simulate" internet video (it's a static dataset)? The answer: instead of training a world/video model that takes actions and previous frames to predict the next frame, condition the model on a goal and previous frames to predict the next frame. The goal is some future frame of the same video. This way, the model is forced to construct a causal understanding of the world: "if this is the frame where I end up, what should come next?"
The payoff is that acting is now almost free: if the model can already predict what should happen next on the way to a goal, all that's missing is a small action head to translate "what happens next" into "what do I press". That head can be trained cheaply afterwards, on a tiny amount of action-labelled data.
Pantograph did not release their exact formulation, not even whether they trained a pixel-space world model or a JEPA-style one. Hence, I came up with my own formulation.
Condition prediction on where you're going, and prediction becomes planning.
Pan1ni: the architecture
In my implementation, I chose to follow a JEPA-style method based on my earlier la leWorldModel:
Unlike la leWorldModel, instead of using an inverse dynamics module/LAM to learn latent actions, we simply go ahead in the sequence of contiguous frames and grab one, up to 64 frames ahead.
The loss is nothing fancy either: a simple MSE between the predicted next latent and the actual next latent, plus SIGReg to keep the latent space from collapsing. The whole system is just two modules, an encoder that maps a frame to a latent, and a predictor that takes the previous latents plus a goal latent and outputs the next latent.
Recovering actions
The final piece is an inverse dynamics model (IDM). It takes the current frame and the predicted next frame, and predicts the action that takes you from one to the other. This is the small action head promised earlier: it requires a much smaller dataset than pretraining, because the action behind a frame-to-frame transition is normally self-evident. If the @ was here and is now one tile west, you pressed west.
At inference time this closes the loop: the predictor imagines the next latent on the way to the goal, and the IDM translates that imagined step into a keypress.
Training
Pan1ni is trained on human NetHack play from the NLD-NAO corpus: 290,331 games, each terminal frame rendered as tiles and cropped to a 9×9 player-centred window. The whole system is ≈11.2M parameters. The action head is trained afterwards on the frozen backbone, and its movement labels come for free: a single-cell tty-cursor displacement gives one of eight compass classes. For simplicity the action space is just these eight compass moves, which in NetHack also covers attacking, since you attack by moving into a monster. The full recipe is in the drop-down below.
Specifics
click for the full recipe
| Data | NLD-NAO human play (nld-nao-human-8shard), 290,331 games, 80/20 episode-level split (232,265 / 58,066) |
|---|---|
| Observation | MiniHack 16×16 tile atlas, 9×9 player-centred crop, 3×144×144 uint8 RGB |
| Encoder | ViT, 8 layers, width 320, ≈10.4M params, 64-d latent |
| Predictor | goal-conditioned causal transformer, 4 layers, 8 heads, ≈0.76M params |
| Conditioning | 8-frame history + goal frame sampled ≤64 steps ahead |
| Objective | one-step MSE on next latent + SIGReg (weight 0.2, 256 slices) |
| Optimisation | AdamW, batch 64, 400k steps, LR 3×10−4 (0–100k) → 2×10−4 (100k–200k) → 1×10−4 (200k–400k) |
| Action head | MLP, 2 hidden layers of 1024, ≈1.2M params, 5,000 steps, class-weighted cross-entropy |
The action head never sees pixels: it is trained on an inverse-dynamics feature, the concatenation of the current latent, the goal-conditioned predicted next latent, and their difference.
Results
The action head is evaluated as a stochastic goal-reaching policy in full procedural NetHack (NetHackScore-v0), rendered through the same tile pipeline. Each run drops the agent a fixed distance from a goal tile and gives it a step budget to get there; I sweep that distance from 3 to 30 tiles. Distance everywhere is Chebyshev (chessboard) distance, so a diagonal step counts as one tile, matching NetHack's 8-way movement.
The baseline, masked-random, is a uniform random walk over the eight compass moves that only skips steps into a visible wall, a closed door, or off the map (it will still bump monsters and wander into fog). Crucially the IDM policy runs under the exact same mask, so neither side can waste a move on a wall. That makes the comparison fair: the only difference between them is whether each step is chosen by the model or drawn at random.
- exact — stood on the goal tile exactly (same cell, same level) at some step.
- reached — got within one tile (Chebyshev ≤ 1: the goal or any of its 8 neighbours) at some step.
- mbd — mean best distance, the nearest it ever got, in tiles, averaged over episodes. Lower is better.
Pan1ni's success stays in the ~65–80% range all the way out to a goal 30 tiles away, while masked-random collapses to zero by about 10.
A caveat on the numbers: cells are only around 12–14 episodes (fewer at the far end, distances 25 and 30 are n=4), so read individual cells as indicative and the distance trend as the result.
Closing thoughts
I think goal-conditioned pretraining will have a role in the future of the field. However, one of the fundamental limits of this method is that it requires you to know what the goal looks like. In this case it is quite easy for us to programmatically find legitimate final states, but in reality it might not be the case.
Furthermore, only specifying a final state like this is an incomplete instruction. For example, there are many ways in which a glass could end up empty on a table: it could be because someone drank it, or because it was spilled.
Nevertheless, I'm excited to see where this all goes.