Optimiser? I Hardly Know 'Er.

Using LLM Program Search to Build Robot Controllers Without RL Optimisers

Program Search with LLMs

The Basic Formula

All LLM based program evolution systems follow this basic formula:

Shinka Evolve by Sakana, or AlphaEvolve by DeepMind, are examples of this.

Before diving in to experiment with existing solutions that can be quite complex, I decided to make my own simplified system. I would also be applying it directly to robotics/controller style tasks.

SimpleShinka: Minimal Search Loop

SimpleShinka works by maintaining an archive of past programs and their scores and feeding them to an LLM to generate new programs.

New prompts are generated by seeding 3 random past solutions, the 3 most recent past solutions and the best solution so far. The LLM also sees the score associated with each of these programs and is told how many inputs to expect and outputs to provide from its program. We generate N programs per round, in practice I do this 10 times. Each of the 10 programs are evaluated and the best of the bunch is added to the archive.

Output exactly ONE fenced Python code block and nothing else.
Do not write comments.
Define exactly:
def main(obs: list[float]) -> list[float]:

Contract:
- `obs` is a list of floats.
- Return a list of 4 floats.
- Floats will be clipped to [-1, 1].
- Imports allowed: `math` only.
- No file I/O, no network, no printing.

Goal:
Maximise two scores: `score_a` and `score_b`.
Higher is better for both.

Previous attempts to build from:
BEST: score_a=... score_b=...
def main(obs: list[float]) -> list[float]:
    ...

RANDOM_1: ...
RANDOM_2: ...
RANDOM_3: ...
LAST_1: ...
LAST_2: ...
LAST_3: ...

In practice, the evaluator defines what these scores mean for each environment. You can use any number of scores, not just two. For example, in BipedalWalker, score_a tracked distance travelled and score_b tracked speed, so the prompt nudged the model towards both progress and pace rather than a single metric.

Experiments

XOR in 2D

First, to test whether it would work at all I selected a very simple environment: 2D XOR. A simple 2D classification task where points in opposite corners share the same label, so a single straight line can’t separate the classes.

Because this problem is so trivial, I decided NOT to tell the LLM what the task actually was.

In this image, each square is the best of that iteration/generation. The red and blue dots are the actual points tested and the background colour shows the decision boundary of the generated function that round.

Within a few steps, the LLM recovered a perfect decision boundary. Note that if you try this with NEAT (traditional population based network topology search method) you can end up with something like this after many iterations:

Looking at the final solution found by the LLM, we see that is it also extremely simple and much more simple than any NEAT solution could possibly find in terms of computational graph.

def main(x0: float, x1: float) -> int:
    return 1 if (x0 > 0) != (x1 > 0) else 0

This showcases the power and expressivity of evolving within a programming language.

Bipedal Walker

Encouraged by the above result, I decided to up the difficulty. Bipedal walker is a classic environment in which a small robot with two legs and a torso must learn to traverse a 2D terrain.

The inputs include joint torques as well as observations from LIDAR-like projections on the walker's torso.

Without telling the LLM what the task was, and telling it only to maximise some score ‘A’ within 100 generations it solved it very successfully:

https://twitter.com/MozarellaPesto/status/2003025232769282433

What's more, I noticed that the LLM was capable of discovering quite interesting gaits along the way, e.g.:

Walker2d

This would be the hardest experiment by far for the black-box style SimpleShinka. Partly because this environment is much stricter than BipedalWalker when it comes to allowed joint angles and body positioning.

Despite this, SimpleShinka was able to find quite a fun hopping solution:

https://twitter.com/MozarellaPesto/status/2003892197725557010

Race Car

Finally, I wanted to really push things to the limit. For this experiment, the LLM knew what the task it was solving was. In this environment, the LLM receives only visual cues, meaning it must effectively build a function that can interpret the image it is given to make decisions on how to drive.

In this case, knowing the task at hand was extremely important, and even when I did not tell it, it was able to figure it out based on the inputs and outputs.

The best solution it found within 50 iterations this time was again, quite reasonable and simple:

def main(obs: list[float]) -> list[float]:
    steer = 0.0
    gas = 0.15
    brake = 0.0
    width = 96
    height = 96
    road_left = 0
    road_center = 0
    bottom_window = 2
    top_window = height - 20
    road_sum = 0
    road_count = 0
    for y in range(bottom_window, top_window):
        for x in range(width):
            r = obs[3 * (y * width + x) + 0]
            g = obs[3 * (y * width + x) + 1]
            b = obs[3 * (y * width + x) + 2]
            gray = 0.2989 * r + 0.587 * g + 0.114 * b
            if gray < 120:
                road_sum += x
                road_count += 1
    if road_count > 0:
        road_center = road_sum / road_count
    road_left = road_center - width / 2
    steer = road_left / 24.0
    return [steer, gas, brake]

The LLM has learnt to crop the image, grayscale it, threshold to determine where the road is, find the centre of mass of the road pixels and steer towards it.

Beyond the Minimal Loop

Shinka Evolve

Now I had done some experiments with my simplified evolution loop and had seen the success, I was excited to try the same experiments with more complex methods like ShinkaEvolve

Shinka Evolve iteratively prompts an LLM with previous solutions and their scores, asking for targeted code diffs. It uses text embeddings to deduplicate outputs and enforce novelty thresholds, and ‘meta-prompting’ to summarise what’s been learned into guidance for the next round. Over time, this builds an archive of candidate programs, from which you can pick the best-performing controller. Notice that the evolved programs stem from a single inital guess, this will be important later.

So what happens if you apply Shinka Evolve to BipedalWalker in a black box style?

In my experiments, it performs poorly. Across different models and settings, the search repeatedly converged on the same local minimum: the agent simply swan dives head first forwards. From here, it never recovers.

SimpleShinka vs Shinka Evolve

Now I wanted to understand, why did the simple algorithm I devised perform better on these robotic/agentic tasks than the more complex Shinka Evolve?

First let's quantify the differences between the algorithms:

The key difference here is that SimpleShinka stripped away a lot of the meta-learning and intelligent evolutionary features that made ShinkaEvolve sample efficient. In their place it added breadth search. Crucially this means that any program wanting to be added to the archive has to compete locally amongst N programs, instead of being added no matter what.

Lucky Guess Hypothesis

My key suspicion was that breadth search was the main driver behind the success of SimpleShinka on the controller tasks. So I developed a theory: ‘Lucky Guess’

Lucky Guess Hypothesis

When experimenting with SimpleShinka, varying the number N of programs generated per iteration and trying different environments, I noticed runs tended to bifurcate: they either found promising solutions very early and then steadily improved, or they stalled almost immediately and never recovered. In other words, most initial programs land in “dead” regions of the search space: low-reward basins where incremental optimisation can’t climb to anything better.

Yellow balls represent initial solutions. Red squares are poor regions of program space, green squares are fertile regions where optimisation can continue to improve.

Only a small minority of initial guesses fall into a region with an improvement path. Increasing N doesn’t change how optimisation behaves once you’re in a basin, it just increases the chance that at least one of the initial programs starts in (or near) a “fertile” region where optimisation can actually make progress. That reduces the risk of getting trapped in suboptimal reward wells and makes success much more likely.

To test this more systematically, I devised an experiment.

Lucky Guess Experiment

To test whether success depends on a ‘lucky first guess’ when there’s no strong prior solution, I compared SimpleShinka with SimpleIslandShinka, a variant that runs multiple independent lineages. Each island follows the same core loop as SimpleShinka but without breadth (one suggestion per lineage per iteration), and non-leading stagnant lineages are reset. This turns the search into repeated attempts to discover strong starting points in program space while preserving local exploitation inside each active lineage.

SimpleShinka vs SimpleIslandShinka flow.

With that distinction fixed, I ran a matched-compute comparison.

I ran 10 trials of each method on BipedalWalker for 20 iterations, comparing both average performance and outliers. Compute was kept roughly matched: SimpleShinka generated 10 candidates per iteration, while SimpleIslandShinka ran 10 lineages with 1 candidate each. For reference, I also ran ShinkaEvolve (10 trials, 20 generations); it uses fewer environment evaluations but took similar wall-clock time and behaved similarly to the results below.

If the ‘lucky start’ hypothesis is right, SimpleShinka should produce better best-case outliers (more exploitation around a good start) but a lower mean, while SimpleIslandShinka should improve the mean (more independent starts) but yield weaker extremes (less depth around any single promising lineage).

Lucky Guess Results

Below is a graph of results from the above experiment on BipedalWalker. Each point is the best score/program achieved in a run.

As expected, the SimpleIslandShinka runs achieved a higher average. Unexpectedly, they also produced more outliers. To investigate, the figure below shows individual lineage trajectories, recording only new best-so-far updates, on the left, and a histogram of counts on the right.

Intriguingly, if you pool results across all lineages, SimpleIslandShinka’s distribution shifts left. That is consistent with more frequent re-entry into poor regions after resets. By contrast, SimpleShinka is greedier: local selection pressure quickly prunes weak candidates, so runs spend less time in very low-quality regions.

However, both methods used roughly the same number of environment evaluations. Under that constraint, SimpleIslandShinka still produces more high-scoring programs overall, as the box-and-whisker plot shows. This supports the ‘lucky first guess’ effect: the space of awful initial programs is enormous, and while some trajectories recover, the best controllers tend to start strong and improve steadily.

This also suggests why ShinkaEvolve struggles without some breadth search. With only a single thread of search, it’s more likely to start in, and remain trapped within, a broad basin of weak solutions that’s extremely hard to escape.

For curiosity I have displayed the best SimpleShinka and SimpleIslandShinka below. ShinkaEvolve did not achieve an agent policy beyond falling over the start line as predicted.

Conclusion

In summary, LLM-driven program search can produce useful controllers without a classical RL optimiser, but only if the search allocates enough early breadth to find a fertile starting basin.

These experiments suggest you do not always need a classical RL optimiser to get useful controllers. An LLM with strong in-context learning can act as the search engine itself: it reads prior programs and scores, proposes edits, and iteratively improves policy code directly in program space.

The key constraint is initialisation. Most random programs fall into low-reward regions with no viable improvement path, while only a small minority begin in fertile regions where in-context refinement can climb. Breadth therefore becomes the critical lever when a good seed is unknown. This explains why SimpleShinka can work well despite minimal machinery, why SimpleIslandShinka improves robustness by repeatedly resampling starts, and why narrow single-thread search often converges to poor behaviour. Practically, if you cannot reliably bootstrap a policy, spend budget on breadth first; depth only helps once you have landed in the right region.

Comments