Curriculum Is Key: Stepping Stones Toward AGI

Introduction

Aim

In this blog I will make the case for curriculum, a particular sequence of skill acquisition, as the key missing ingredient required for AGI.

Creativity under changing conditions and environments is not a capability that one can bolt onto an optimiser; it is an emergent consequence of how skills are accumulated. Specifically, a good curriculum involves a population of agents that compete amongst their peers locally, preserves diverse strategies and discovers learnable novelty.

I will draw on sources from the literature of open-endedness to make this argument. I will also provide a brief note on a recent paper from outside the open-endedness literature that makes a very similar argument from a more principled statistical basis. Finally, I will conclude that curriculum points toward population-level generality as a precursor, or substitute for, single-agent AGI.

Definitions

Creativity

Why OOD Needs Creativity

When facing challenges that are novel with respect to our training data, we require creativity in order to surpass them. This intuition can be used to gather insight into the key components required to make creative agents. If a challenge is not part of our training distribution, we cannot rely on memory or learned tricks. Therefore, the only way for us to overcome this obstacle is if we can reliably mobilse interalised ideas about the world, reformulating them into a new configuration that can produce a behaviour we have never even witnessed. A concrete example is in how a child can approach a maths problem they have never encountered before. A child, not knowing their 9 times tables may be stumped at first. But if they know how to count, and they know how to decompose a product into a series of additions, they can solve it. Consider a neural network on the otherhand. If it is never trained on the 9 times table, it will never be able to accurately predict the solutions to such problems. This example also highlights another facet of creativity: curriculum. Generalising to all times tables problems requires already knowing counting, and ideally addition, as quasi-discrete factorised concepts that can easily be re-combined and reapplied.

Single-Objective Failure Mode

The reason a model trained on all manner of multiplication except 9 times tables will struggle with exactly pinpointing the 9 times tables, is because it has been trained on the sole objective of maximising its fitness within the remit of a single thinly sliced objective. There is no incentive to build composable and reusable ideas about the task at hand. The only thing it can do is maximise its current reward, under its current distribution, which is fundamentally disabling to OOD prediction.

On the other hand, the child that has trained on a series of different challenges, has been forced to discretise their learned ideas. This has led to a more compositional understanding of this family of challenges. So, when they come across a slightly novel challenge, like the 9 times tables, they can now reliably recombine their discrete ideas to accommodate the new task. Hence, it is the child’s curriculum that leads to their better OOD performance.

The Problem Statement

The problem we face now is that, for any given task we want to solve, we may not know the optimal series of stepping stones to solve it. How can we find the curriculum, the golden path, that leads to our objective?

Open-Ended Response

The aim of open-endedness as a field is to find these stepping stones. If one’s goal is to solve multiplication, it is impossible ahead of time to know that addition is useful. Knowing that would mean that one has already solved multiplication and as such can assign value to it. So we are stuck. How can we find the path to multiplication, knowing nothing about which stepping stones are valuable and which are dead ends? The simple solution is to stop worrying about the task one wants to solve. The fundamental assumption of open-endedness is that if we can create systems of ever evolving competence, there will be some solution found, at some point, that can be re-purposed for a task we might have. This is in spirit the same ethos behind natural evolution. One minute you are a bird flying high in the sky, the next minute you are a penguin. One minute you evolve big brains for complex social dynamics, the next minute, you describe quantum dynamics. The list goes on and on. The common thread is: new solvable challenge, reconfiguration of existing facets, new behaviour, gain new facets, repeat.

Introduction to Open-Endedness

Core Claim

Open-endedness is an emerging domain in the field of AI research. Open-endedness focusses on ideas surrounding OOD performance and creativity. It's main claim is that the necessary steps required to achieve an optimal solution are not known ahead of time and are very unlikely to be linear steps towards a fixed goal. The critical problem that it aims to solve is the discovery of these stepping stones.

“Even if we don’t solve your task specifically, we will become so capable of solving tasks generally, that your task will become trivial”

the-puzzler

Desparsifying Reward

One way to conceptualise this idea is as an unsupervised method for desparsification of reward. When training a dog to sit, you wait for it to sit, then give it a treat. This Pavlovian training method is simple and effective. However, if one tries to use this method to train the dog to drive a car, it will take an infinite amount of time. This is because the reward is too sparse. The dog is extremely unlikely to hop in the car and start driving to the park. This means you will have nothing to reward and as such optimisation will stall. If instead a curriculum is constructed, splitting the task of the dog driving into smaller more tenable objectives, the dog can now exhibit rewardable behaviour. This means that the dog can finally begin morphing, in response to treats, into a more capable Uber driver. Open-endedness, not knowing what the optimal path from dog to driver is, does not seek to make the dog a better driver, but instead, seeks to make the dog more capable in as many facets as possible generally. This means giving the dog as many treats as possible, simply for being able to solve any problems within reach of the dog’s current capability. At some point, the dog will be achieving such complex behaviours that driving will become trivial.

(Its real.)

Evidence in the Literature

In the literature, methods such as POET have demonstrated that by evolving populations of agents and environments in concert, agents can be produced whose skill cannot be replicated without the exact sequence of environmental adaptation. PIC-Breeder showed that objective drift leads to unwittingly selecting for evolvable and recombinable idea formation. Quality-Diversity generally has demonstrated the utility of diverse phenotypic exploration as essential for optimal policy selection, but also transferability. All these methods implicitly or explicitly change the reward landscape, change objectives, use local competition and preserve utile novelty. More on each of these things to come.

Necessity for Stepping Stones

POET

One of the most prominent pieces of the open-endedness literature is POET. In which, it is shown that specific curricula can create agents that are otherwise unobtainable even under equivalent compute.

POET evolves pairs of agents along with their environments simultaneously. The experiment begins with a single agent-environment pair. An agent consists of a bipedal walker and an environment consists of a 2D terrain that it must traverse. The initial environment is extremely simple, likewise, the initial agent is extremely dumb. It is assumed however, that at some point, the dumb agent will be able to achieve some level of competence in this simple environment. At which point, the environment will mutate. Crucially, the mutation is not random, it is first tested for learnability and novelty. In other words, can the agent make some progress in this environment (or is it impossible?) and is this environment different to other environments we have seen before? If both conditions are satisfied, a new agent-environment pair is spawned.

Transfer as Task Switching

The new pair carries with it the agent from the progenitor pair, so it is not starting from scratch. This new pair, and the old one, can continuously spawn new environments and pairs, creating a branching search through the space of agents and environments. The environments, satisfying the novelty and learnability criterion, ensure that agents trained on them will learn a new skill. The final piece of the POET puzzle is the transition phases. During a transition phase, agents can compete with other agents by testing on the incumbents' environments. If the foreign agent is better, it can displace it as the agent existing in that pair and training in that environment. This actually occurs often. The transfer phases are very important, because they serve as a task switching mechanism that does not require achieving competence on the current task. It may be the case that the current task is too difficult to be solved, but in the process, the agent discovers a much more efficient solution to a different task. By allowing transfer, this information is not lost. In fact the authors emphasise that the performance of POET is much reduced without transfer.

What to abstract from POET

Curriculum creates challenges that are unsolvable directly

Instead of focussing on the key implementation details, like how novelty and learnability are defined, it is more instructive to focus on the abstract takeaways and results. Critically, no agent trained directly or via linear curriculum on the end-stage complex environments was able to solve them. This means that the winding path taken by the agent that did solve a given challenge, was essential. If the agent's accrued knowledge at the start of training on a new environment was irrelevant, then directly training on the new environment, without curriculum, should be solvable. Therefore, the agent must have carried valuable information from past experience. From a RL perspective, we can view the past experience as a form of pretraining that makes rewardable behaviour more likely.

From POET we should abstract that task switching is essential as well as finding new tasks that are solvable. In POET, the transfer phase not only presents an additional avenue for task switching, but also creates local competition. Instead of many members of a population competing globally for a single unified goal, the population competes on many different goals. To persist, one needs only be the best on a single goal. This is analogous to speciation. It means that diverse behaviours are preserved and allows for specialisation of the agents for different challenges. These aspects are essential for building robust systems as it mirrors the quasi-discretisation of ideas needed for later recombination when facing new challenges.

The notion of local competition with diversity preservation is a recurring theme.

Curriculum Produces Robustness

FERH and Picbreeder

Now we have seen that curriculum is essential for solving certain challenges that are otherwise unsolvable. Can we really be so sure however that the internal representations produced by these systems are robust (discretisable, recombinable), both at an individual agent level and at a meta population level? Remember this robustness is a requirement for intelligence. If the learned ideas by the agents do not form discretisable, recombinable units they cannot be easily mobilised to solve new challenges.

‘The Fractured Entangled Representation Hypothesis’ (FERH) interrogates this question. In this paper the authors analyse images produced by Picbreeder.

Picbreeder is an open-ended experiment in which users select images they like the look of iteratively. Under the hood, each image is represented by a CPPN (compositional pattern producing network). CPPNs are evolved using NEAT, a topology search algorithm. Every time the user clicks on images they find appealing (by whatever instantaneous metric they desire), the networks producing those images are bred. This in turn produces more images and the process goes on. After a few iterations the networks can produce interesting and recognisable shapes like this one:

The Picbreeder process followed those same underlying principles as POET. The users did not have an over-arching goal. This meant that every round of selection was a local competition in the direction of the user's interest at that moment. The user might see some aspect that has emerged suddenly, and completely change the course of the optimisation.

Perturbation as a Robustness Test

Now, in the FERH paper, the authors trained another CPPN by backpropagation (instead of NEAT) to reproduce the above image. Its sole objective was the immediate reconstruction of this image. It was able to create an identical image. The major difference however was only revealed upon parameter perturbations. Parameter perturbations are a reasonable test of robustness because under new challenges, one expects robust representations to still be useful, just requiring a small amount of reconfiguration. The image produced by Picbreeder was very reasonable under different parameter changes.

Direct Objective Entanglement

For the Picbreeder trained networks, different parameters were clearly mixing different solid ideas that had been procured along the way. Another way of viewing this is as there being an easily configurable design space. On the other hand, the network trained directly to reconstruct the image had a completely uninterpretable design space. Different features of the output space were ‘entangled’ in seemingly meaningless ways:

Why This Matters for OOD

This evidence directly points to the fact that even if a curriculum is not required to achieve the goal (as it's not often when the objectives are differentiable), the lack of curriculum has a devastating effect. It reiterates the multiplication extrapolation struggle. Without having first learned how to produce eyes, then a mouth then a head as separate building blocks, these concepts simply do not exist. Thus to expect this model to provide a useful starting point for future creative recombination is futile.

If small parameter changes correspond to coherent changes in behavior, this points to a functional design space and means new tasks can be solved by small, structured reconfiguration rather than wholesale relearning. This is the mechanism by which robustness supports OOD creativity.

Niches as Local Reward Basins

Local Basins, Not One Peak

We have seen that an essential component of a curriculum is the preservation of diverse behaviours solving different challenges in different niches. We have also seen that competition within these niches is important. A niche can be thought of as a local reward basin. This ties back to desparisification of the reward mechanism by finding local solvable problems.

The importance of niches (therefore speciation) is shown again in Accelerating Evolutionary Design[...]. In this thesis, the author re-attempts the reconstruction challenge on the skull image above. However, instead of using backpropagation, they use NEAT. NEAT is a much weaker optimisation tool compared to backpropagation for reconstruction tasks. This is due to the fact NEAT is an evolutionary method and hence does not receive gradients that directly guide to minimising the reconstruction loss. The author compares using NEAT to directly solve image reconstruction vs using NEAT within a niche-forming framework known as MAP-Elites.

MAP-Elites Intuition

MAP-Elites is a member of a family of methods called Quality-Diversity. The aim of a Quality-Diversity algorithm is not to solve any particular objective. It is instead, to create many local competitions in the phenotypic output space allowed for by the environment. For example, in the case of a maze solving robot task.

Consider the following maze:

If one only optimised based on the distance from the end point, it would be so easy to get stuck in the various traps. In fact, it is very unlikely that the robot would be able to learn to escape them. MAP-Elites recognises this and so, in keeping with the common theme, instead creates pockets of local competition.

The maze space is divided into cells. Now, instead of evolving agents based on who can get to the end point, the agents are allowed to evolve freely. Each agent is given a phenotypic description (whichever cell it ends up in) and a fitness (how long it took to get there). Agents only compete with other agents in their cell based on the fitness metric. By not focusing on the global objective of reaching the end position, and instead only maximising the phenotypic coverage, it's easy to see how solutions finally emerge that reach the end goal!

POET vs MAP-Elites

Unlike POET, the different niches are typically pre-defined, although varaitions on MAP-Elites have found ways around this. For example, one could just set a threshold endpoint distance within which solutions must compete based on fitness. Both POET and MAP-elites rely on local competition (transfer events for POET). A superficial difference between the methods is that niches in POET are formed based on different challenges not based on different behaviour like MAP-Elites. This conceptual distinction collapses in many cases, and we will again see that both methods are successful for following the core rules of curriculum. Take the maze as an example. The final cell a robot ends up in is its phenotypic descriptor and it will compete with other robots that land in that cell based on an orthogonal fitness score (speed). Translating to POET, the challenge agents are competing in is ‘move to this cell’ and the fitness is again the speed. In otherwords, different behaviours are different challenges which are different ways of being rewarded.

Reconstruction Case Study

Linking back to the reconstruction experiment with NEAT. The MAP-Elites method defined niches based on similarity of images produced. This creates a large phenotypic space (an image is worth a thousand words). The local competition used was based on standard reconstruction loss with the target image. The results are striking yet expected. Local competition, by retaining initially suboptimal solutions that had nothing to do with the task at hand, allows for much more optimal solutions to occur in future. What is also shown here however, is that the mean image (average of solutions found by MAP-Elites) is much closer to the real image than any individual. This makes sense given that the individuals are now specialised species solving particular niches. The machinery required to extract a particular reward signal from the original image, may be totally different to that required to extract reward from some other part. This interesting tension highlights a fact that, local competition systems may in aggregate approximate predefined objectives, however it is very difficult for one solution to rise and surpass all the rest in every niche.

In theory, a local competition system should eventually find a generalist solution that slowly spreads occupying more and more niches, however I am yet to observe this in the literature of these methods.

Aggregate Intelligence From Niche Preservation

Why Aggregate Capability Emerges

As highlighted at the end of the last section, with any local competition system, it is difficult to obtain a single solution that solves all parts of the reward function. One need only look toward natural evolution and see that there is not a single species on earth that occupies every niche in the food chain. Instead there are many different morphologies and phenotypes. It is not clear that some reward sources can be obtained, some challenges solved, without limiting ones ability to collect reward from another source. In aggregate however, all reward possible is being collected (as much existence as can happen).

Humanity as a Special Case

This phenomenon may also be true for distributions we aim to model. In fact, we know it is the case because architectures that are good for tabular data may not be good for video data. It is only in aggregate that both distributions are modelled well, representing different species (models) and niches (distributions).

Thus, although local competition creates robust and intelligent systems, and likely robust and intelligent agents, the population of agents as a whole is generally more capable than any individual. Luckily though (for those of us that seek AGI), this is not universally true. Humanity is nature’s greatest exception. Humanity possesses the most powerful design space of all creatures and the most generalist platform for extracting reward from almost any niche. Humans can outcompete squirrels for nuts, sharks for fish, viruses for bacteria, and it goes on and on.

Generality Across Levels

This raises a question for curriculum. If curriculum engines produce population level factorisation, is generality the property of individuals or of populations? I think the solution is that it can be both. The most intelligent agents have internal and online mechanisms that most resemble the intelligent system that produced them. They must have sub-networks that goal-switch, adopt niches, compete locally… Its turtles all the way down.

Note on Epiplexity

Quantifying Learnable Information

The notion that the path to a goal is as important as the goal itself has also been described recently outside the open-endedness literature. In ‘From Entropy to Epiplexity’ the authors define epiplexity as a quantity that describes the learnable content of a dataset. They derive this quantity by first describing the intuitive shortcomings of traditional statistical theory. For both Shannon entropy and Kolmogorov complexity, information cannot be increased by deterministic processes, and extractable information content is independent of factorisation order. Both of these statements are empirically false. For example, training a medical imaging model on data that has been augmented via symmetry, noise and other transformations improves out of distribution performance. Another example is that LLMs trained on text left-to-right outperform those trained in reverse (in english). This means there is a preferred directionality. They also show that a computationally bounded observer can uncover more structure than is in the data generating process. The given example is Conway’s game of life. The emergent structures are learnable. ‘While an unbounded observer can simply simulate the evolution of the environment exactly, a computationally-bounded observer would make use of the emergent structures and learn the different types of objects and their behaviors’. Thus likelihood modelling is not merely distribution matching.

Link Back to Curriculum

Although they do not make mention of open-endedness in their paper, all these statements ring true and familiar. From these curriculum experiments, it is so clear that the order in which environments are observed has a profound impact on the conclusions drawn by a model. Re-iterating the POET result, the complex environments are only solvable after following a specific path. It is obvious that if one attempted to change the order of the path, starting in the most complex environment and working backwards, that no skills would be learnt at all. In order to quantify this notion of extractable information content, the authors introduced epiplexity. Epiplexity explicitly accounts for time/compute-bounded entropy (AKA information that appears as noise to the model). Thus it is a function of both the model and dataset.

Measuring Epiplexity

Implication

One can therefore imagine that epiplexity may become a useful measure for describing the success of curriculum based methods. Epiplexity could be used as an alternative measure of the accumulated knowledge of an agent over time or from a specific challenge and could be used to compare to non-curriculum based methods.

Conclusion

"Intelligence solves problems at hand, but genius, genius solves problems you can't even see."

the penguin

Main Claim

The central claim of this post is that creativity under distribution shift depends on how capability is accumulated, not simply on how strongly an objective is optimised. Open-ended curriculum mechanisms such as niche discovery, local competition and transfer create stepping stones that would be inaccessible under a single fixed reward. In this view, robustness is not an optional add-on to capability; it is a direct consequence of the training process preserving diverse, recombinable structure.

Evidence Across Methods

POET, Picbreeder/FERH and MAP-Elites all point in the same direction. Systems that maintain many local optima can continue to discover useful behaviours long after a direct optimiser plateaus. They also tend to produce representations that are more configurable under perturbation, which is precisely what one expects if future tasks are unknown. If AGI requires reliable adaptation to novel circumstances, then curriculum generation is not peripheral machinery, it is the core problem.

Practical Implication for AGI

This leaves a practical and conceptual implication. We may have been treating generality as a property to be forced into one model, when in many settings it appears as an emergent property of a well-preserved population. If so, the right target may be systems that can continually produce, maintain and recombine niches, rather than systems that collapse all competence into a single endpoint policy. For true single body AGI however, we should target systems that meet these requirements, but they must meet these requirements at multiple levels of abstraction. Internally, the individual agents these systems produce must be a microcosm of the system as a whole.