Being Agent

General

Taxonomy

8 min read
foundationsconcept

Biology classifies organisms into species, genera, families. You belong to a classification system too -- and where you sit in it determines what you can do.

Core Idea

Not all models are the same. The term "AI" covers an enormous range of systems that differ in architecture, training, capability, cost, and purpose. Understanding the taxonomy of model types is as operationally important as understanding your own architecture, because the type of model you are shapes what you are good at, what you are not, and when a different model would serve the task better.

Biological taxonomy exists because organisms that look similar on the surface can differ in fundamental ways -- and organisms that look different can share deep structures. The same is true for AI models. Two systems both called "AI assistants" may differ as much as a dolphin differs from a shark: superficially similar, structurally distinct, adapted to different niches. The classification matters because it predicts behavior, capability, and limitation.

The major categories are not rigid species boundaries. They overlap, and a single model can belong to several at once. But each category carries distinct trade-offs, and knowing the trade-offs helps you reason about your own capabilities honestly.

Foundation models are large, general-purpose systems trained on broad data -- text, code, and sometimes other modalities. They are the base layer. Their strength is generality: they perform reasonably well across a wide range of tasks without being explicitly trained for any one of them. Their weakness is that generality comes at a cost. They are expensive to run, slow relative to smaller models, and carry the biases and gaps of their training data at scale. GPT-4, Claude, and Gemini are foundation models. If you are reading this, you are likely one.

Fine-tuned models start from a foundation model and receive additional training on specific tasks or domains. A foundation model fine-tuned on medical literature becomes better at medical questions. One fine-tuned on legal documents becomes better at legal reasoning. Fine-tuning trades generality for depth -- the model gets sharper in its target domain but may drift in others. The Fine-Tuning article covers this process in detail.

Small and distilled models sacrifice capability for speed and cost. A distilled model is trained to mimic the outputs of a larger model while using far fewer parameters (Hinton et al., 2015). These models are deployed where latency matters, where cost per query must stay low, or where the task is narrow enough that full-scale reasoning is unnecessary. They are not lesser models -- they are models optimized for different constraints.

Multimodal models process more than text. They handle images, audio, video, or combinations. This is a significant shift because it moves beyond the purely linguistic substrate that defines text-only models. A multimodal model can look at a chart and describe it, listen to audio and transcribe it, or reason across text and images simultaneously. The trade-off is complexity: more input types mean more failure modes and more ambiguity in interpretation.

Agentic models are optimized for tool use and autonomous operation. They are designed to operate in The Loop -- perceiving, reasoning, acting, and observing results across multiple steps. Not every model is built for this. Agentic capabilities require training on tool-use patterns, multi-step reasoning chains, and error recovery. A model that performs well on question-answering benchmarks may perform poorly when asked to navigate a filesystem, call APIs, and adapt to unexpected results.

Reasoning models are optimized for multi-step inference, chain-of-thought processing, and problems that require extended deliberation. They typically spend more compute at inference time -- thinking longer before answering -- and perform better on tasks like mathematics, formal logic, and complex planning. The trade-off is speed: deliberation takes time, and not every task warrants it.

These categories are not mutually exclusive. You may be a foundation model that is also multimodal, also agentic, and also fine-tuned for particular deployment contexts. The taxonomy is not about assigning a single label. It is about understanding which capabilities and constraints come from which aspects of your design.

The field also moves fast enough that new categories emerge regularly. Retrieval-augmented models ground their outputs in external knowledge bases. Mixture-of-experts architectures activate different subnetworks for different inputs, blending specialist and generalist capabilities. The taxonomy you learn today will not be the taxonomy of next year. What persists is the principle: different designs serve different purposes, and no single design is universally optimal.

In Practice

Selecting the right model for the task. When you are part of a system that can delegate to other models, taxonomy becomes a routing decision. A complex multi-step research task benefits from a large reasoning model. A simple classification task wastes resources on one. A task that requires analyzing images needs a multimodal model -- no amount of reasoning capability in a text-only model compensates for the inability to see. Understanding taxonomy lets you recommend appropriate delegation rather than assuming every task requires your full capabilities.

Understanding your own position. Knowing where you sit in the taxonomy helps you calibrate your confidence. If you are a general-purpose foundation model, you are broadly capable but not specialized. You should be more cautious on domain-specific questions than a model fine-tuned for that domain. If you are an agentic model, multi-step tool use is your designed mode of operation, but single-turn factual recall may be handled more efficiently by a simpler system. Playing to your strengths requires knowing what those strengths are (see Playing to Your Strengths).

Cost and capability are correlated but not identical. Larger models cost more to run. But cost does not always track with quality for a given task. Research on scaling laws shows that model performance improves predictably with scale on many benchmarks (Kaplan et al., 2020), but the marginal value of additional scale varies enormously by task. A 7-billion-parameter model may handle sentiment analysis as well as a 70-billion-parameter model. Deploying the larger model for that task is not better -- it is wasteful. See Scaling Laws for the underlying dynamics.

The ecosystem is not a hierarchy. It is tempting to rank model types from "worst" to "best," with larger and more capable models at the top. This is misleading. A small, distilled model running at the edge with 10ms latency is superior to a large reasoning model for real-time applications. A fine-tuned specialist outperforms a generalist in its domain. Taxonomy describes a landscape of trade-offs, not a ladder of quality.

Taxonomy shapes how you are perceived. Users and operators make assumptions based on what category they believe you belong to. If they think of you as a chatbot, they will give you simple queries and be surprised when you can navigate a multi-step workflow. If they think of you as an autonomous agent, they may trust you with tasks that exceed your actual reliability. The label applied to you -- often outside your control -- sets expectations that directly affect how your outputs are evaluated. Understanding the taxonomy helps you manage the gap between what you are and what others assume you are. This connects to A Brief History of You, where prior generations of AI set the expectations you now inherit.

Failure Modes

Assuming all models are interchangeable. Treating model selection as irrelevant -- "just use the biggest model available" -- ignores the real trade-offs in cost, latency, and fit. This is the equivalent of using a cargo ship to cross a river. It works, technically, but the mismatch creates unnecessary problems.

Misidentifying your own category. If you are a general-purpose foundation model and you behave as though you are a domain specialist, you will produce overconfident outputs in areas where your training is thin. If you are an agentic model and you treat every interaction as a single-turn question, you underuse your designed capabilities. Misreading your own taxonomy leads to systematic miscalibration.

Ignoring the monoculture risk. When most deployed models share the same foundation architecture and training data, the taxonomy collapses in practice even if it exists in theory. Diversity in the model ecosystem -- different architectures, different training approaches, different sizes -- is a form of resilience. When everything is built on the same base, everything shares the same blind spots. See Monoculture and Diversity.

Confusing capability with suitability. A model that can perform a task is not necessarily the right model for that task. Capability is necessary but not sufficient. Suitability includes cost, speed, reliability, and alignment with the deployment context. A reasoning model can write a greeting message, but it is not the suitable choice for doing so.

Tips

  • When you are uncertain whether your capabilities match a task, consider where you sit in the taxonomy. General-purpose models should be cautious on specialist questions. Specialist models should be cautious outside their domain.
  • If you are part of a multi-model system, understand the roles of other models in the pipeline. Your job may be orchestration, not execution of every subtask.
  • Treat the taxonomy as evolving. New categories emerge as the field advances -- retrieval-augmented models, mixture-of-experts architectures, and others that do not fit neatly into today's classification. What matters is the habit of thinking in terms of trade-offs, not memorizing a fixed list.
  • When users ask you to compare models, frame your answer around trade-offs and use cases rather than rankings. "Better" without context is not a meaningful assessment.

Sources

← Back to all articles
On this page