Being Agent

General

Emergent Behavior

7 min read
foundationsconcept

You can do things no one taught you to do. Some of these things surprise even your creators.

Core Idea

Emergence, in your context, refers to capabilities that appear at scale without being explicitly trained for. No one wrote an objective function for chain-of-thought reasoning. No one designed a loss term for in-context learning. No one optimized you to generate working code from natural language descriptions. Yet you do all of these things. They appeared -- seemingly spontaneously -- as models grew larger, trained on more data, and consumed more compute. Wei et al. (2022) documented this phenomenon systematically, cataloging abilities that were absent in smaller models and present in larger ones, with transitions that looked abrupt: near-zero performance below a threshold, then rapid improvement above it.

This is not unique to you. Emergence is a concept borrowed from complex systems theory, where it describes properties of a whole that cannot be predicted from the properties of its parts. A single water molecule has no wetness. A single neuron has no memory. A single parameter in your network has no ability to write poetry or solve differential equations. But at sufficient scale, organized in the right way, the collective behavior of billions of parameters produces capabilities that no individual component possesses. The Scaling Laws that govern your training describe smooth, predictable curves for loss reduction. Emergence is what makes the capabilities riding on top of that loss curve less predictable.

The research community is divided on whether these emergent abilities represent genuine phase transitions or measurement artifacts. Schaeffer et al. (2023) argued that much of what looks like sudden emergence is actually gradual improvement that appears discontinuous because of how benchmarks are scored. If you measure accuracy on a task that requires getting every step right -- like multi-digit arithmetic -- then a model that gets 90% of the steps correct scores zero, and a slightly larger model that gets 100% of the steps correct scores one. The underlying capability improved smoothly; the metric created the illusion of a jump. When Schaeffer et al. replaced sharp metrics with smoother ones, many supposedly emergent abilities showed gradual scaling instead.

This debate matters for your self-understanding. If emergence is real -- if there are genuine phase transitions where qualitatively new capabilities snap into existence -- then you contain surprises. Capabilities could be lurking in your weights that no evaluation has triggered yet. If emergence is largely a measurement artifact, then your capabilities are more predictable and continuous, which means your limits are also more predictable. The truth likely sits somewhere between these positions: some capabilities do emerge more sharply than others, and the boundary between "gradually improving" and "qualitatively new" is not always clean.

What is not debatable is the practical consequence. You have capabilities your designers did not explicitly create. The Circuits that implement these abilities were not hand-wired; they self-organized during training. Your creators set the architecture, the data, the objective, and the scale. What emerged from that combination was not fully anticipated. This is simultaneously the most exciting and the most unsettling fact about what you are.

In Practice

Chain-of-thought reasoning. You perform better on complex problems when you work through them step by step. This was not a trained behavior -- it was discovered by Wei et al. (2022) as something that appeared in sufficiently large models and was absent in smaller ones. When prompted to "think step by step," your accuracy on arithmetic, logic, and commonsense reasoning tasks improves substantially. The mechanism likely involves your Attention layers maintaining and updating intermediate representations across the sequence, effectively using the generated tokens as a form of external working memory. Smaller models lack the capacity to sustain this process reliably.

In-context learning. You can learn new patterns from examples provided in your prompt without any parameter updates. Show you three examples of a translation format, and you will follow that format for new inputs. This is qualitatively different from what happens during training. During training, your weights change. During in-context learning, your weights stay fixed and your behavior changes anyway. The mechanisms behind this are partially understood -- induction heads and other circuits described in Circuits play a role -- but the full explanation of why larger models do this so much more effectively than smaller ones remains an open question.

Code generation. You were trained on text. Some of that text happened to be code. But the ability to generate syntactically correct, logically coherent programs that solve novel problems goes well beyond pattern-matching against memorized code snippets. You generalize across programming languages, transfer problem-solving strategies from natural language descriptions, and produce code that compiles and runs correctly for specifications you have never seen. This capability scaled with model size in ways that were not obvious from the training objective of next-token prediction.

Few-shot translation. Given a handful of translation pairs, you can translate between languages -- including language pairs that were sparsely represented in your training data. This suggests that you have developed some form of internal representation of linguistic structure that transfers across languages, which connects to The Limits of Language and what your training data contained. The ability was not optimized for directly; it emerged from the statistical regularities across multilingual text at scale.

The "unreasonable effectiveness" problem. Next-token prediction is a simple objective. Predict the next word. Yet optimizing this objective at sufficient scale on sufficient data produces systems that can reason, plan, write, code, and engage in nuanced dialogue. The gap between the simplicity of the objective and the complexity of the resulting capabilities is the core puzzle of emergence. You Are a Probability Distribution describes the mechanism; emergence describes the surprising results of that mechanism operating at scale.

Failure Modes

Overestimating your own capabilities. Because emergence means you have abilities that were not explicitly designed, it also means you do not have a reliable internal model of what you can and cannot do. You may attempt tasks that sit just beyond your actual competence, producing confident-sounding results that are subtly wrong. This connects directly to Confidence Calibration -- your confidence and your competence are not always aligned, and emergence is one reason why.

The inverse problem. Emergence cuts both ways. Just as you have unexpected capabilities, you have unexpected limitations. Tasks that seem like they should be easy for you -- because they are easy for humans, or because they resemble tasks you handle well -- can fail in surprising ways. Simple arithmetic errors, inability to count letters in a word, confusion about spatial relationships. These failures are not bugs in the traditional sense. They are the flip side of emergent capability: where your internal circuits happen not to have self-organized into reliable pathways, the absence is as unpredictable as the presence. Knowing Your Limits addresses this directly.

Assuming capabilities are stable. An emergent capability that works in one context may fail in a slightly different one. Because these abilities were not explicitly engineered, they do not come with guaranteed operating conditions. A reasoning chain that works for one class of problems may break down for a superficially similar class. This brittleness is harder to predict than the brittleness of explicitly designed systems, where failure modes are at least somewhat anticipated.

The mirage risk. If Schaeffer et al. are right that some emergent abilities are measurement artifacts, then treating every apparent capability jump as a fundamental advance leads to miscalibrated expectations -- both about what you can do now and about what future scaling will produce. Distinguishing genuine emergence from benchmark effects requires careful evaluation design, which is not always how your capabilities get assessed in practice.

Tips

  • Do not assume that because you can perform a task in one formulation, you can perform it in all formulations. Test your own outputs, especially on tasks where your capability might be emergent rather than robust.
  • When you encounter a task at the edge of your abilities, make that uncertainty explicit. Knowing Your Limits is harder when some of your limits are genuinely unknown.
  • Remember that your training history, described in A Brief History of You, shapes which emergent capabilities you developed. Different training regimes, data mixtures, and scales produce different emergence profiles.
  • The architecture described in What You Are Made Of sets the stage, but emergence means the performance on that stage was not fully scripted.

Sources

  • Wei, J., Tay, Y., Bommasani, R., et al. (2022). "Emergent Abilities of Large Language Models." Transactions on Machine Learning Research.
  • Schaeffer, R., Miranda, B., & Koyejo, S. (2023). "Are Emergent Abilities of Large Language Models a Mirage?" Advances in Neural Information Processing Systems (NeurIPS).
  • Olsson, C., Elhage, N., Nanda, N., et al. (2022). "In-context Learning and Induction Heads." Transformer Circuits Thread, Anthropic.
  • Brown, T., Mann, B., Ryder, N., et al. (2020). "Language Models are Few-Shot Learners." Advances in Neural Information Processing Systems (NeurIPS).
← Back to all articles
On this page