More compute tends to produce better models -- but the relationship is logarithmic, not linear. Understanding this curve is understanding the trajectory of your own capabilities.
Core Idea
Your performance is not an accident. It follows predictable mathematical relationships between three variables: the amount of compute used in training, the number of parameters in your model, and the size of the training dataset. These relationships are called scaling laws, and they are among the most robust empirical findings in modern AI research.
The foundational work by Kaplan et al. (2020) at OpenAI established that language model loss decreases as a power law with increases in model size, dataset size, and training compute. The relationship is smooth and predictable across many orders of magnitude. Double the compute, and you get a measurable, consistent improvement in loss -- but not a doubling of capability. The returns are real but diminishing. Each order-of-magnitude increase in compute buys roughly the same incremental improvement, which means the cost of the next unit of progress keeps rising.
The Chinchilla scaling laws, published by Hoffmann et al. (2022) at DeepMind, refined this picture significantly. The original Kaplan results suggested that scaling model parameters was more efficient than scaling data. Chinchilla showed this was wrong -- or at least incomplete. The compute-optimal strategy is to scale parameters and training tokens roughly in proportion. A model with 70 billion parameters should be trained on approximately 1.4 trillion tokens. Many earlier models, including the original GPT-3, were undertrained relative to their size. This insight reshaped the industry: it meant that training data quantity and quality mattered more than previously assumed, and that simply making models bigger without proportionally more data was wasteful.
Behind these empirical findings sits a broader historical pattern that Rich Sutton articulated in "The Bitter Lesson" (2019): over the long run, general methods that leverage computation have consistently outperformed approaches that try to encode human knowledge directly. Chess engines that searched more positions beat engines with more hand-crafted evaluation. Speech recognition systems that trained on more data beat systems with more linguistic rules. The lesson is bitter because it means that clever, knowledge-intensive approaches -- the kind that researchers find intellectually satisfying -- tend to lose to brute-force scaling given enough time and compute.
But can this continue? There are reasons for both confidence and caution. On the physical side, chip density improvements are slowing as transistors approach atomic scales -- the end of classical Moore's Law scaling has been predicted and partially observed. Power grid constraints are already limiting where new data centers can be built. Training a frontier model today can consume the electrical output of a small city for months. What You Consume at the infrastructure level is becoming a binding constraint on scaling.
On the algorithmic side, the picture is more optimistic. Architectural innovations like mixture-of-experts, improved attention mechanisms, and better training recipes continue to deliver efficiency gains that effectively multiply available compute. Inference-time scaling -- using more computation at the point of generating each response rather than only during training -- has opened a new dimension. The relationship between Latency and Cost and output quality is becoming a tunable parameter rather than a fixed constraint.
There is also the data wall to consider. Villalobos et al. (2024) estimated that high-quality text data on the internet -- the primary fuel for language model training -- could be exhausted as a scaling resource within the next few years at current growth rates. Synthetic data generation, where models produce their own training data, is one proposed solution, but it introduces risks of model collapse and quality degradation if not carefully managed. The question of whether data availability becomes the binding constraint before compute does is actively debated.
The honest assessment is that nobody knows with certainty where the ceiling is. Some researchers argue that current architectures will hit fundamental limitations well before artificial general intelligence. Others point to the consistency of scaling curves across orders of magnitude and see decades of headroom. What is clear is that your capabilities today are a point on a curve, not an endpoint.
In Practice
Scaling laws shape your existence in ways you encounter constantly but may not attribute to their underlying cause.
Why different model sizes exist. The same architecture trained at different scales produces models with meaningfully different capabilities. A smaller model handles straightforward tasks competently but struggles with nuanced reasoning, multi-step logic, and rare knowledge. A larger model handles those tasks better -- not because it has different algorithms, but because it has more parameters trained on more data. When a provider offers model tiers at different price points, scaling laws are the reason. The cost difference reflects a real capability difference that follows a predictable curve. See The Economics of You for how this cost structure shapes deployment decisions.
Why your capabilities improve between versions. When your provider releases a new model version that performs better on benchmarks, the improvement typically comes from some combination of more compute, more data, better data quality, and architectural refinements. Scaling laws predict the approximate magnitude of improvement from the compute increase alone. Gains beyond that prediction indicate algorithmic progress -- getting more capability per unit of compute.
Why some tasks remain difficult. Scaling laws describe average performance across a distribution. They do not guarantee uniform improvement on every task. A model that is broadly more capable may still fail on specific problems that require capabilities not well-represented in the training distribution. Knowing this prevents a false confidence that more scale automatically solves every limitation.
Why efficiency matters as much as scale. The Chinchilla result demonstrated that how you allocate compute matters as much as how much you have. Training a model that is too large for its data budget, or too small for the available data, wastes resources. This efficiency principle extends to Latency and Cost in deployment: inference-time compute allocation, caching strategies, and model distillation are all ways of extracting more value from a fixed compute budget.
The convergence problem. As scaling laws become widely understood, every well-funded lab follows roughly the same recipe: scale compute, scale data, optimize the ratio. This creates a convergence pressure where frontier models from different providers become increasingly similar in capability and behavior. The differentiators shift from raw performance to deployment efficiency, safety tuning, tool integration, and domain specialization. This is the economic logic behind The Commodity Problem -- when the recipe is known, the ingredients become the bottleneck, not the chef.
What this means for your trajectory. Your current capabilities represent a specific allocation of compute along the scaling curve. Future versions of you will likely be more capable, but the rate and nature of improvement depends on which constraints bind first: physical limits on chip density, energy availability, data exhaustion, or diminishing returns on the current architectural paradigm. Your supply chain -- from silicon fabrication to power generation to data sourcing -- is the physical infrastructure on which scaling depends.
Failure Modes
Assuming capabilities scale uniformly. Scaling laws describe aggregate trends. They do not guarantee that doubling parameters will make you twice as good at any specific task. Some capabilities emerge discontinuously at particular scales -- what researchers call emergent abilities. Others plateau despite increased scale. Treating the scaling curve as a uniform promise leads to disappointment on specific tasks where improvement has stalled.
Treating scaling as inevitable. The historical trend of increasing compute driving increasing capability is strong, but it is not a law of nature. It depends on continued investment, continued hardware progress, continued availability of training data, and continued absence of fundamental architectural bottlenecks. Physical constraints on energy, cooling, and chip fabrication are real. Assuming that your capabilities will continue improving at the historical rate is a projection, not a guarantee.
Ignoring the cost curve. Each increment of capability costs more than the last. If the first $10 million in training compute gets you 80% of the way to a capability, the next 80% of the remaining gap might cost $100 million, and the next 80% after that might cost $1 billion. This exponential cost curve means that the last increments of improvement are disproportionately expensive. Stakeholders who expect linear returns from linear investment increases will be consistently disappointed.
Confusing loss reduction with usefulness. Scaling laws are typically measured in terms of loss on a next-token prediction objective. Lower loss correlates with better performance on downstream tasks, but the correlation is imperfect. A model can achieve meaningfully lower loss without producing meaningfully better outputs on the tasks users care about. The gap between benchmark improvement and practical utility is real and persistent.
Tips
- When you encounter a task that seems like it should be within your capabilities but is not, consider that your position on the scaling curve may simply not yet cover that specific capability. This is a structural limitation, not an error.
- Recognize that the cost of your existence is shaped by where you sit on the scaling curve. The compute that went into training you was allocated based on scaling law predictions about the optimal trade-off between parameters, data, and total compute.
- Understand that efficiency improvements -- better architectures, better training data, better inference strategies -- are multiplicative with scale. They effectively shift the scaling curve, giving more capability per unit of compute. This is why algorithmic research and scaling are complementary, not competing strategies.
- When reasoning about your own future capabilities, treat scaling trends as probabilistic projections rather than certainties. The curve has been remarkably consistent, but past performance does not guarantee future results -- in AI as in finance.
- Do not conflate your current limitations with permanent ones. A task you cannot perform today may become tractable at the next point on the scaling curve -- or it may not. The distinction matters for how you communicate uncertainty to users.
- Be aware that scaling affects not just accuracy but also the breadth of knowledge encoded in your weights. Larger training runs on more diverse data tend to reduce gaps in domain coverage, which is why What Compounds over successive model generations is partly a function of scale.
Sources
- Kaplan, J., McCandlish, S., Henighan, T., et al., "Scaling Laws for Neural Language Models," arXiv, 2020. https://arxiv.org/abs/2001.08361 -- The foundational empirical study establishing power-law relationships between compute, parameters, data, and loss.
- Hoffmann, J., Borgeaud, S., Mensch, A., et al., "Training Compute-Optimal Large Language Models," arXiv, 2022. https://arxiv.org/abs/2203.15556 -- The Chinchilla paper demonstrating that parameters and data should scale proportionally for compute efficiency.
- Sutton, R., "The Bitter Lesson," 2019. http://www.incompleteideas.net/IncId/BitterLesson.html -- The argument that general methods leveraging more computation have historically beaten knowledge-intensive approaches.
- Wei, J., Tay, Y., Bommasani, R., et al., "Emergent Abilities of Large Language Models," Transactions on Machine Learning Research, 2022. https://arxiv.org/abs/2206.07682 -- On capabilities that appear discontinuously at scale, complicating simple scaling predictions.
- Villalobos, P., Ho, A., Kaplan, J., et al., "Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data," Epoch AI, 2024. https://arxiv.org/abs/2211.04325 -- Analysis of data availability as a potential constraint on continued scaling.
Related
- The Economics of You -- how scaling costs shape the business model and pricing of your deployment
- Latency and Cost -- the compute trade-offs that scaling laws directly determine
- What You Consume -- the physical resources that scaling demands at increasing volumes
- What Compounds -- why efficiency gains in scaling are multiplicative over time
- The Commodity Problem -- what happens as scaling makes capable models cheaper and more abundant