Self-Organising Text

Inspired by self-organising cellular automata and the broader idea that simple local rules can yield emergent global intelligence.

Here, I explore how self-organisation scales — starting from a physical basis in the Ising model. Then, we will jump into the text space and see if its possible for textual tokens, looking only at their neighbours, to organise into coherent text.

I took a lot of inspiration from this post by Francesco Sacco. If you find yourself wishing for a deeper understanding on emergence and the importance of the Ising model, I highly recommend you check it out.

Introduction: Self-organisation

Self-organisation is a key principle in complex systems: local interactions can produce surprising global order. Examples range from flocking birds to financial markets.

Cellular automata → show how local update rules can produce global patterns.
Markets → buyers/sellers act locally, prices and supply chains emerge globally.
Neural systems → synapses adapt locally, intelligence emerges globally.

The question is, is there a way to, atleast roughly, model when a system has some degree of emergent capacity... or atleast know by its topology if it doesn't?

The Ising Model

The Ising model gives a clean mathematical framework for how local interactions can lead to collective alignment and, perhaps more crucially, when they can't. In one dimension, the conventional result says no spontaneous magnetisation occurs — but with interaction ranges above a critical threshold, self-organisation is possible.

Origins

In 1925, Ising solves the 1D version of the model as a means of describing ferromagnetism: how local interactions, given the right conditions can reach a global order. In the case of magnetism this means atomic spins being "suddenly" aigned causing the overall material to behave magnetically. He discovered that it is actually impossible in 1D! The equation he used to describe the system is as follows: $$H = -J \sum_{i=1}^N s_i s_{i+1}$$ where $ s_i \in \{-1,+1\} $ are spins and $ J $ is the coupling constant.
Simply, what this means is the energy contribution from a pair of neighboring spins is $ -J $ if they are aligned and $ +J $ if they are anti-aligned.

I am summarising a bit here, but for a system, we can calculate its free energy.
The free energy tells us the balance between energy and entropy that determines the system's equilibrium.
More generally in thermodynamics, this is written as: $$ F = U - T S, $$ where $ U $ is the internal energy (the $ H $ from before) and $ S $ the entropy.
So, if we want to know whether it is likely that the system, with decreasing $ T $, will collapse into an ordered state, we must compare the free energies of the possible states.
As $ T $ decreases the entropy term $ T S $ becomes less important, so the system will lean towards order whenever the ordered state leads to a lower free energy than the disordered one.

The Kicker

Here's the kicker however: the best case ordered scenario for a 1D sequence is $$ U = - J L, $$ where $ J $ is the interaction strength and $ L $ the system size.
But introducing a single domain wall only costs a finite energy (about $ 2J $), while the number of possible locations for such a wall grows with $ L $.
In fact, the entropy is given by $$ S = k_B \ln(g), $$ where $ g $ is the degeneracy, i.e. the number of accessible microstates. For the one–domain-wall case, $ g = L - 1 $.
Therefore the entropy contribution is $$ T S = T k_B \ln(L - 1), $$ which increases without bound as $ L $ becomes large.

As a result, in 1D the system never collapses into a truly ordered state in the thermodynamic limit.
This is clear from comparing the finite energy cost of a domain wall, $ U = 2J $, with the entropy term. As the sequence length increases, $$ T k_B \ln(L - 1) \gg 2J, $$ so the entropy always dominates for any $ T > 0 $.
Hence, only at absolute zero does the perfectly ordered state survive.
See the following simulation for a practical intuition ->

1D Ising Simulation

And let the enternal battle between light and darkness commence! Even if you decrease T to near 0 to bias towards order, it will never converge!

Back to Text

Provided I didn't lose you there, what does this mean for self-organising text? It means that if we only ever allow a textual token to attend to its nearest neighbours, this is directly analogous to the 1D Ising case — and we are guaranteed to have no stable, globally ordered state.
In practice, for language models this might mean that the first tokens begin talking about one topic, but as the sequence grows the later tokens drift toward something else, since nothing enforces long-range coherence.
Modern language models use 'full attention' to solve this, all positions in a sequence can attend to all others. But this defeats the purpose of this mini project. We want to see if its possible for only local interactions to achieve global convergence a la cellular automata.

Is all hope lost?

Well hold on a minute. Sure, convergence is impossible if we only allow pair interactions, but what if we expand the interaction range? What if we allow one token to look at all of its neighbours within a certain radius?

Derivation & Critical Range

Okay so before hand we said that the energy of the system was a sum of the pairwise interactions. But now, it is a sum of interactions that each position has with other positions within a fixed distance $ m $. Mathematically: $$ U \;=\; -J \sum_{1 \le i < j \le L} \;\; \{\, |i - j| \le m \,\}\; s_i s_j \,, $$ where $ s_i \in \{\pm 1\} $, $ L $ is the sequence length, and $ m $ sets the interaction range.

In a range $ m $ there are $ N $ pairs. If we let $ d = i - j $, then for our given sequence $ L $ there are $ L - d $ pairs.
You can see this by considering: $$ N \;=\; \sum_{d=1}^{m} (L - d) $$ $$ \;=\; (L - 1) + \cdots + (L - m) $$ $$ \;=\; mL - \frac{m(m+1)}{2}. $$

Now, to figure out wether this allows us to reach a convergent state, we can pose the following question: What is the ground state energy, and what is the cost of a single barrier. If the cost of transitioning to a domain barrier state is cheaper than the ground state it will never converge (as before).
Well, given $ N $, the ground state energy is: $$ E_0 \;=\; -J\Big(mL \;-\; \frac{m(m+1)}{2}\Big). $$ Moreover, a single domain barrier flips all pairs that straddle the cut within range $ m $, i.e. $ \sum_{d=1}^{m} d = \tfrac{m(m+1)}{2} $ pairs, each costing $ 2J $. Thus the additional energy is $$ \Delta E_{\text{barrier}} \;=\; 2J \cdot \frac{m(m+1)}{2} $$ $$ \;=\; J\,m(m+1). $$

Hence $$ E_{\text{barrier}} \;=\; E_0 \;+\; J\,m(m+1). $$

Aha! So now we can just plug this in and see: $$ \Delta F \;=\; J\,m(m+1) \;-\; k_B T \ln(L-1), $$ which for $ m = 1 $ simplifies to the prior case ($ U = 2J $).
Now, all we need to do is choose an $ m $ that guarantees order even as sequence length tends to infinity! We can do this by defining $ m $ as a function of $ L $.
If $ L \to \infty $, the above simplifies to $$ \Delta F \;\sim\; J m^2 \;-\; k_B T \ln(L). $$ So for energy to dominate over entropy ($ U > TS $): $$ m \;>\; \sqrt{\tfrac{k_B T}{J} \ln L}. $$

This is great, as it means $ m $ only needs to scale with the square root of $ \ln L $, which grows extremely slowly!

Visualizing the ordering threshold

T: 1.00 J: 1.00 L range:

Curve: $m_{\min}(L)=\sqrt{\tfrac{k_B T}{J}\ln L}$ with $k_B=1$. Log-x plotted from $L=2$ to your chosen max.

So lets apply it to our simulation now, does it converge?!
->

Ising with Interaction Radius m

Now Lets Model Some Text

Overview

Now we know that global coherence is not impossible, we can start language modelling with a little confidence. To do this experiment, I used the TinyStories dataset and trained a model using strictly local attention. This means that each token in the sequence can communicate only with its nearest $ m $ neighbours. But crucially, with each layer of the model, information gets passed around the sequence, so the effective interaction radius is much larger!

Objective and Architecture

The learning objective was a very simple one. The model would see a sequence of text that had been noised to some corruption level, meaning e.g. 25% of the sequence had been replaced by nonsense tokens. Then, where ever this displacement had occured the model would have to predict the correct token (CELoss).
The model I converged to was effectivley a pre-norm transformer encoder with radial attention mask. The model had ~150M parameters and it trained over 250M text tokens (very few generally speaking). Finally I also used a pretty naive corruption level sampling and loss re-weighting as well as some other minor tricks.

Training Results

Model Training

Here are the training curves from the 28th epoch. The model was still improving but alas, my battle with the finitude of time leaves me triumphed over once again.

Experiments & Examples

I suppose now you will want to see some actual results! During training the model simultaneosuly denoises all tokens in the sequence but in practice this is ineffective. It would be like trying to paint a model that keeps changing position. To resolve this I have adopted a simple sampling scheme. Every step a random token is chosen and its logits are sampled. There are likely better methods, but for now this will do.

Play With It Yourself

You can explore the model live ->
(Simply initalise random, then live stream denoise! I recommened playing with settings.)

Reflections

Self-organisation is a general principle — whether in physics, markets, or language models. Local rules, scaled properly, can produce unexpected intelligence. This was a fun weekend project.