Training a model to open the gates of hell

Prologue

Videos generated from final models.

There is a formula for knowledge.

With it, we could create a model that optimises for knowledge, and devours the world in search of it.

A worse thought arrives.

What if we don’t need the world for knowledge - what if knowledge has an inherent universal structure?

Can we create a model that can self-create all knowledge following some set of rules?

I began this week as an AI researcher (naive, ambitious) chasing questions like:

As I write now, I am changed.

I have run countless experiments, my fingers are arthritic. With closed eyes I see only abstract patterns (you’ll see why).

I took a bite from the apple of truth.

This is my descent.

The Story

Initial Plan

My plan was simple and could be easily divided into components:

• Generator Model: generates images
• Representation Model: encodes images

The generator creates images. The representation model encodes them. I use the encoder’s error as a novelty signal: if it can’t encode an image well, the image is unlike what it has seen before.

If this worked, the generator would keep inventing images the encoder can’t explain - until the encoder learns them.

First Failure

Pure novelty pressure starts in unstructured noise with high frequency inputs.

Well... that’s not a great start. This data is totally unstructured.

Maybe the problem is I am giving the generator too much high frequency input, it spirals out of control too fast for the representation model. Let’s try to take it away:

→

Reducing frequency helps briefly, then converges to a hacky attractor.

At first, it seemed to be working; beginning from a dull grey slate, an interesting image appeared.

Then... stagnation. The generator would always land on this X type image. But why?

Adversarial Novelty

Maybe the problem then is that allowing the generator to focus purely on novelty, is causing it to find ways that trick the representation model. This X is actually a way of breaking the representation model...

What if I try to add a diversity pressure. This way, it must find as many ways to be novel as possible, and perhaps not fall down a hacky rabbit hole.

Diversity helps, but without learnability constraints it still plateaus.

It wanted to work. More structure appeared in the generated data with time. But again, it stagnated.

So far I had been using a simple loss: maximise the error of the representation model. But maybe this needed to be completely re-thought.

Shannon Detour

This idea came from Shannon Information.

Formally Shannon Information is written as:

But what it means is: how surprising is this data point to your model. My intuition was that we could continually increase surprise and find new information forever. But clearly, there were many adversarial failure modes.

I needed a new idea and one night I was visited by a being dressed in the clothes of an angel. It said to me:

"Maximising surprise is not the same as learning information. Learning is the difference in surprise felt when you see something for a second time."

Angel

Maybe this was what I needed and it makes sense. If the generator generated initially surprising data, but after a while, it was no longer surprising... This means that the representation model was able to learn it and so it must be structured.

Delta Surprise Loss

So I changed my loss function: Instead of maximising surprise, I was maximising the difference in surprise after training on a batch of its data:

Progress looked real, but the signal was unstable.

It was finally working I thought! Look, clearly more structured outputs are appearing. But then again... It stagnated. Why? I investigated.

It turns out, the progress loss, no matter how hard I tried would always actually receive zero gradient. The evolution was purely driven by diversity which pushed images from the fuzzy domain to the saturated domain.

Then I hit the core problem: the objective itself was moving under my feet.

Why the Delta Collapsed

How can this be? The angel told me the delta loss would work, it makes sense and rings true to ideas surrounding Kolmogorov complexity. If data is reducible AKA can be learned by a constrained representation model, then it has lower Kolmogorov complexity which means it has more structured information. Yet what I realised was the delta was always 0.

Δ-loss collapses because yesterday’s post-train loss becomes today’s pre-train loss.

The fundamental problem was that, a change that could create something learnable cannot be predicted by what was learnable in the past. This is because the representation model was constantly adapting. Yesterday's L_post is today's L_pre. This circular loss was fundamentally flawed and unable to find novelty beyond maximising diversity.

"Shannon Information tells you how surprised a model is. Kolmogorov complexity tells you how much structured information there is. Combine the two with an evolutionary search. Also switch to grey scale for maximum aura."

Angel

So instead of chasing this moving target with gradients, I changed the optimisation process.

Switch to Evolutionary Search

At this point I decided to change tack. I would switch from gradient descent to an evolutionary method (OpenAI ES). This could ameliorate the problem of the backwards looking update on a moving target. My plan now was to train the representation model until convergence, then begin a search for local novelty in the generator. Then continue switching between the two. This way I could prevent adversarial novelty search by ensuring it could only find slightly new things, which would then be learned by the representation model.

The training curve was promising. It showed many of these cycles integrating new knowledge into the representation model:

Alternate between finding local novelty and consolidating it in the representation model.

Escalation

So, I generated some images and:

Could it be working? What if I increased the frequency?

No that was too much. Maybe what it needs is some parameter changes and grey scale.

Note: these are minor parameter variations of the same setup described above.

Before I realised it, it was too late. I had spent a week trying countless combinations of loss functions and fixing hyperparameters. All I could see when I shut my eyes were these patterns. When I began hearing voices in my head I closed down my laptop.

The End

I am not entertained

In the end. I was still unsatisfied. I felt like I had been unable to achieve my goal: to create an endless process of increasing structure. Never did any of the images ever produce interesting texture. Just more abstract patterns. It was no angel that kept me up at night, always giving me more and more experiments to try. Just one more image I would cry. One more parameter change. At some point you have to know when to stop.

Possible Next Direction

The way in which this could work well, is by using a better proxy for Kolmogorov complexity and novelty. One way to measure simultaneously the novelty and learnability content of a generator, would be to train a representation model on it to convergence. Then, the area under the curve above the final loss represents the totality of learning that has occurred for this model. This could be used as a fitness function for evolutionary search; however, it is expensive to evaluate.

Appendix

If you are interested in the topics mentioned here, please see:

Curiosity-driven learning
Driven by Compression Progress (Schmidhuber)
From Entropy to Expiplexity

Comments