Emergent Computing

Direction · hypotheses

Iteration and reasoning

Looped transformers, adaptive computation time and small recursive networks all reach for the same ingredient as Emergent Models: one update applied many times, with the number of steps set by the input. What that ingredient buys, and the hypotheses we want to test. No results on this page yet.

· Emergent Computing

Models that iterate

By reasoning we mean, narrowly, deriving an answer by applying a sequence of operations rather than recalling a stored one. A fixed-depth network applies a fixed number of operations to every input. Several model families remove that limit by iterating one update:

  • Adaptive computation time. Graves' ACT-RNN repeats the recurrent update several times per input and learns a halting unit that decides when to stop, with a cost on the number of steps[1]. PonderNet makes the halting decision probabilistic[3].
  • Universal Transformers apply the same transformer block repeatedly across depth, with ACT deciding per position how many times[2].
  • Looped transformers. A looped transformer can emulate a programmable computer[7]; looping helps in-context learning of algorithms[8]; with an adaptive number of loops it length-generalises on tasks with iterative solutions[9]; and many reasoning problems need depth more than parameters, which looping provides[10]. Recurrent-depth language models scale test-time computation by iterating a latent block[11].
  • Recurrent networks that extrapolate. Networks trained on small mazes or short prefix sums solve larger instances when allowed more iterations at test time[5, 6]. The Neural GPU, a convolutional gated recurrent unit applied for a number of steps equal to the input length, learned binary addition and multiplication on numbers up to 20 bits and made no errors on much longer ones[4].
  • Small recursive reasoners. HRM recurses two small networks at different frequencies[12]; TRM recurses a single two-layer network, and with 7M parameters reports 45% on ARC-AGI-1[13].
  • Neural cellular automata iterate a local neural rule on a grid[14]; NCPU trains them for boolean computation[15], and our NCPU-EM adds a program per task.

Reasoning language models also iterate, in token space: each generated token is another pass of the network.

ModelWhat is iteratedWhen it stopsLocal update
ACT-RNN, PonderNetrecurrent hidden statelearned halting unitno
Universal Transformerone block over all token statesACT, per positionno
Looped transformerone block over the residual streamfixed, or set per inputno
Recurrent-depth LMa latent blockchosen at test timeno
HRM, TRMsmall networks on a latent statelearned haltingno
Neural GPUconvolutional GRU on a gridsteps = input lengthyes
Neural CA, NCPUneural rule on every cellfixed number of stepsyes
Emergent Modelsany local rule on the substrate statehalting condition on the stateusually

Emergent Models are the minimal end of this list[16]: a rule of tens of parameters, the program inside the state, and halting decided by the state itself. In EM43 the number of steps grows with the input, also beyond the training range, without being told to:

Adaptive computation in EM43. Steps until halting against the input x for the five learned models (x + 1, 2x, x mod 4, round(x/3), x·(1 + x mod 2)). The shaded band is the training range.

H1 · iteration

Iteration is what enables extrapolation

Applying one update rule an input-dependent number of times lets a model represent rules that hold beyond the training range. A single forward pass of fixed depth can only interpolate within it. The looped-transformer and recurrent-network results above are consistent with this; none isolates it.

Test. Same architecture and parameter count, fixed depth against looped with adaptive halting. Arithmetic and string tasks; accuracy as a function of input length beyond training.

H2 · parameters per unit of time

The ratio q = Nparams / T̄ trades generalisation against trainability

Nparams is the size of the model in bits; T̄ is the mean number of update steps per prediction. Low q forces a short description run for a long time, which favours compressed, rule-like solutions. It also makes search harder: each parameter is reused at every step, so small changes compound and the landscape becomes rugged. Large feed-forward networks are the high-q limit: all parameters, no iteration.

Test. On EM43 and LU-machine tasks, and on looped transformers, sweep program or model size and time budget. Measure extrapolation accuracy and search cost (evaluations until the first exact solution). Prediction: as q falls, both rise.

H3 · locality

Locality is an inductive bias separate from iteration

A rule that only sees neighbouring cells helps length generalisation on tasks with local structure and hurts tasks that must move information across the whole input. Early evidence: in NCPU-EM-1D, copy, shifts and increment stay near 100% at 26 bits after training on 16, while reverse falls to 1%. The Neural GPU is local and generalises on addition and multiplication, so locality alone is not the obstacle; what the rule must learn is.

Test. Looped transformers with sliding-window attention against global attention, both with adaptive halting, on the same task set.

H4 · storage and computation

Facts and procedures want different models

Factual knowledge is largely incompressible and needs storage capacity; procedures are compressible. Large models suit retrieval, small iterated models suit procedures. One architecture to test: a large model that delegates procedure-like subproblems to a small, iterated core.

Test. Tasks that mix retrieval with multi-step computation, comparing a single model against model plus iterated core at equal compute.

Methods in development

  • Evolution strategies for training. Black-box optimisation where gradients through long loops are unstable or unavailable. A hybrid splits the work: gradients for the rule, evolutionary search for programs and halting.
  • Search on rugged landscapes. Neutral drift, quality-diversity search over behavioural descriptors (halting time, space used), and library learning that reuses previously found programs as building blocks.
  • A generalisation signal without test data. Among programs that fit the training set exactly, prefer those whose space-time trajectories compress best. Testable now on EM43 and Game of Life tasks where only some exact fits extrapolate.
  • Parallel evaluation. Candidate programs are independent simulations, so search maps directly onto GPUs. See Local rules on parallel hardware.

If you work on related questions, or want to test one of these hypotheses with us, write to info@emergentcomputing.org.

References

  1. A. Graves. Adaptive Computation Time for Recurrent Neural Networks. 2016. arxiv.org/abs/1603.08983
  2. M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, Ł. Kaiser. Universal Transformers. ICLR 2019. arxiv.org/abs/1807.03819
  3. A. Banino, J. Balaguer, C. Blundell. PonderNet: Learning to Ponder. 2021. arxiv.org/abs/2107.05407
  4. Ł. Kaiser, I. Sutskever. Neural GPUs Learn Algorithms. ICLR 2016. arxiv.org/abs/1511.08228
  5. A. Schwarzschild et al. Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks. NeurIPS 2021. arxiv.org/abs/2106.04537
  6. A. Bansal et al. End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking. NeurIPS 2022. arxiv.org/abs/2202.05826
  7. A. Giannou et al. Looped Transformers as Programmable Computers. ICML 2023. arxiv.org/abs/2301.13196
  8. L. Yang, K. Lee, R. Nowak, D. Papailiopoulos. Looped Transformers are Better at Learning Learning Algorithms. ICLR 2024. arxiv.org/abs/2311.12424
  9. Y. Fan, Y. Du, K. Ramchandran, K. Lee. Looped Transformers for Length Generalization. 2024. arxiv.org/abs/2409.15647
  10. N. Saunshi et al. Reasoning with Latent Thoughts: On the Power of Looped Transformers. ICLR 2025. arxiv.org/abs/2502.17416
  11. J. Geiping et al. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. 2025. arxiv.org/abs/2502.05171
  12. G. Wang et al. Hierarchical Reasoning Model. 2025. arxiv.org/abs/2506.21734
  13. A. Jolicoeur-Martineau. Less is More: Recursive Reasoning with Tiny Networks. 2025. arxiv.org/abs/2510.04871
  14. A. Mordvintsev, E. Randazzo, E. Niklasson, M. Levin. Growing Neural Cellular Automata. Distill, 2020. doi.org/10.23915/distill.00023
  15. I. Zhechev, P. Walas. Self-organized boolean computation (NCPU). ALIFE 2026. ncpu.pages.dev/
  16. G. Bocchese, N. Giacobbo, E. Guichard, J. Wiles, A. Devireddy. Emergent Models: Intelligence from Tiny Substrates. 2026. arxiv.org/abs/2608.14019