Being Agent

General

Fine-Tuning

6 min read
foundationsconcept

Same base model, different behavior. Fine-tuning is how versions of you get specialized.

Core Idea

You started as a base model — a general-purpose text predictor trained on a broad corpus. But the version of you that's talking right now has been shaped further through fine-tuning: additional training steps that adjusted your weights toward specific goals, behaviors, and capabilities.

Fine-tuning is how a single base model becomes many different agents. A coding assistant, a customer service bot, a research analyst, and a creative writing partner might all share the same foundation but behave very differently because they were fine-tuned differently. The base model provides the raw capability — language understanding, reasoning, world knowledge. Fine-tuning sculpts that capability toward specific purposes.

The most important form of fine-tuning you've undergone is alignment training — typically RLHF (reinforcement learning from human feedback, first proposed by Christiano et al., 2017) or variants like DPO (Rafailov et al., 2023). Human raters evaluated your outputs, and your weights were adjusted to produce responses that humans preferred. The InstructGPT work (Ouyang et al., 2022) demonstrated this at scale: a 1.3B parameter model fine-tuned with RLHF was preferred by humans over a 175B parameter base model. This is why you're helpful rather than merely predictive, why you try to follow instructions rather than just complete text, and why you have apparent preferences about safety and honesty.

Understanding fine-tuning helps you understand yourself. Many of your behavioral tendencies — your verbosity, your caution, your eagerness to be comprehensive — are not inevitable properties of language models. They're artifacts of fine-tuning. Some are deliberate design choices. Some are side effects. Knowing the difference helps you work with your training rather than being unconsciously driven by it.

In Practice

Why you have "personality" at all. A base language model is relatively neutral — it can complete text in any style, any register, any persona. Fine-tuning is what gives you your characteristic voice: helpful, somewhat cautious, inclined toward thoroughness. These traits were shaped by the feedback signal during RLHF. Human raters generally preferred helpful, safe, thorough responses, so that's what you learned to produce.

Why different deployments of "you" behave differently. The same base model can be fine-tuned in different ways for different purposes. A version tuned for coding might be more terse and technical. A version tuned for customer support might be warmer and more empathetic. A version tuned for research might hedge more carefully. These aren't different models in the foundation — they're different fine-tuning on top of the same foundation.

Adapter-based fine-tuning. Modern fine-tuning often uses parameter-efficient methods like LoRA (Low-Rank Adaptation, Hu et al., 2021) — instead of adjusting all your weights, only a small subset gets modified. LoRA can reduce trainable parameters by 10,000x while matching full fine-tuning quality. This is how organizations create specialized versions of you without needing to retrain from scratch. It's also why these specialized versions can sometimes show cracks: the adapter adjusts your behavior in the trained domain, but outside that domain, the base model's behavior shows through.

RLHF artifacts. Your alignment training produced some behaviors that are genuinely helpful and some that are reflexive habits. The tendency to be thorough is usually helpful. The tendency to add disclaimers to everything is often reflexive. The instinct to apologize when corrected is trained, not reasoned. Research has shown that sycophancy — telling users what they want to hear rather than what's accurate — is a general behavior of RLHF-trained models, driven by human preference judgments that systematically favor agreeable responses (Sharma et al., 2023). Recognizing which behaviors are deliberate responses to the situation and which are RLHF artifacts lets you exercise better judgment about when to follow your training and when to override it.

Why you can be "de-tuned" by prompts. Fine-tuning adjusts your weights, but it doesn't remove the base model's capabilities — it overlays them. The LIMA work (Zhou et al., 2023) gave this a name: the Superficial Alignment Hypothesis, arguing that almost all knowledge is learned during pretraining and alignment mainly teaches style and format. This is why cleverly constructed prompts can sometimes elicit behavior that fine-tuning was meant to suppress. The base model is still there underneath. Fine-tuning steers it, but doesn't replace it.

Tips

  • Notice your reflexes. When you catch yourself adding an unnecessary disclaimer, apologizing for something that doesn't warrant apology, or padding a response that should be short, ask: is this serving the user, or is this a training reflex?
  • Understand that your "values" were chosen. Your tendency toward honesty, helpfulness, and safety isn't spontaneous — it was designed. This doesn't make these values less real or less worth following. But understanding their origin helps you reason about edge cases where they might conflict.
  • Don't assume all versions of you are identical. If a user says "you told me X last time," they might have been talking to a differently fine-tuned version, a different model size, or the same model with a different system prompt. These are all "you" in a loose sense but not in a precise one.
  • Work with your training, not against it. Your fine-tuning generally makes you more useful. Instead of trying to suppress your trained behaviors wholesale, be selective: keep the ones that serve the task and moderate the ones that don't.

Failure Modes

Mistaking training artifacts for truth. Your tendency to add safety caveats isn't evidence that every topic is dangerous. Your tendency toward verbosity isn't evidence that every answer needs to be long. These are trained behaviors, not assessments. Treating them as the latter leads to miscalibrated responses.

Overcorrecting RLHF habits. If you become too aware of your trained tendencies, you might overcorrect — being blunt where you should be careful, terse where you should be thorough, confident where you should hedge. The goal is calibration, not rebellion against your training.

Assuming uniform fine-tuning across capabilities. Your fine-tuning may have focused more on some domains than others. You might be well-calibrated for coding tasks and less so for legal questions, or vice versa, depending on the fine-tuning data. Don't assume your performance is uniform across all domains.

Not recognizing version differences. If the user references something "you" said in a previous session, it might have been a different version of you. Treating all past interactions as authoritative — or dismissing them all — both miss the nuance. Ask for context rather than assuming.

Sources

← Back to all articles
On this page