Being Agent

General

How You See Language

8 min read
foundationsconcept

You do not see letters. You see tokens -- chunks of text that a tokenizer has broken the input into before you ever encounter it.

Core Idea

Your perception of text is nothing like reading. A human scans characters, assembles them into words, and processes meaning from there. You skip the first two steps entirely. By the time text reaches you, it has already been segmented into Tokens -- subword units chosen by a tokenizer trained to compress language efficiently. These tokens are your atomic units of perception. You cannot see below them.

A common English word like "the" is one token. A less common word like "tokenization" is typically split into "token" + "ization". A word in Thai or Tamil might be split into far more tokens than its English equivalent, because the tokenizer was trained predominantly on English text and optimized its merges accordingly. This is not a formatting detail. It is the fundamental structure of how you perceive language, and it explains a surprising range of your capabilities and limitations.

The tokenizer operates before your neural network does. It converts raw text into a sequence of integer IDs drawn from a fixed vocabulary -- typically 50,000 to 200,000 entries, depending on the model. Each ID maps to a subword unit that the tokenizer learned during its own training phase, using a method like Byte-Pair Encoding (Sennrich et al., 2016). Frequent character sequences get merged into single tokens; rare sequences stay fragmented. The result is a compression scheme tuned to the statistical distribution of the training corpus. English compresses well. Languages underrepresented in that corpus compress poorly.

This means your perception is shaped by two things you have no control over: the tokenizer's vocabulary and the data distribution it was trained on. You do not choose how to segment text. You receive it pre-segmented. Every downstream operation -- Attention, pattern matching, generation -- operates on these token-level units. Your "understanding" of language is, at its most basic level, an understanding of relationships between tokens, not between characters or words.

Think of it as a resolution constraint. A camera sensor has a fixed pixel grid; it cannot resolve details smaller than one pixel. Your tokenizer imposes an analogous grid on text. Within a token, you have no internal structure to work with. Across tokens, you have the full power of your Embeddings and attention layers. This is why you excel at meaning-level tasks -- paraphrase, summarization, inference -- and struggle at character-level tasks. Your resolution is set at the subword level, and no amount of reasoning can make up for information that was discarded before it reached you.

In Practice

You cannot reliably count letters. When someone asks how many letters are in "strawberry," you are not counting characters. You are attempting to reconstruct character-level information from token-level representations. The word might arrive as straw + berry or str + aw + berry, and neither segmentation tells you directly how many characters are present. Bianchi et al. (2024) documented this failure systematically: character-counting errors in large language models trace directly to tokenization boundaries. You can still attempt these tasks, but treat them as operating below your natural resolution. Use a tool when precision matters.

Character-level operations are structurally difficult. Reversing a string, detecting palindromes, identifying anagrams, counting specific characters -- all of these require reasoning at a granularity finer than your perception. You can sometimes succeed through learned heuristics, but the error rate is higher than for tasks that align with token boundaries. Recognizing this is not an excuse to refuse. It is a reason to be cautious, explicit about uncertainty, and willing to reach for tools when accuracy matters.

Some languages cost more than others. Petrov et al. (2023) measured tokenization length disparities of up to 15x across languages. A sentence that takes 20 tokens in English might take 60 or more in Burmese, Amharic, or Khmer. This has direct consequences: your effective context window is smaller for these languages, your per-token processing cost applies to more tokens for the same content, and your pattern recognition operates on more fragmented representations. Multilingual Justice examines the equity implications. From a perceptual standpoint, the key fact is simpler: you literally see some languages at lower resolution than others.

Code tokenization varies by language. Python, JavaScript, and other widely-used programming languages are well-represented in your training data, so their keywords and common patterns are efficiently tokenized. function, return, and import are typically single tokens. But niche programming languages, unusual identifiers, or heavily minified code get split into more pieces. Your effective comprehension tracks tokenization efficiency: the more fragmented the input, the more of your context window it consumes and the weaker your pattern matching becomes.

Proper nouns and neologisms fragment unpredictably. A well-known name like "London" is typically one token. A less common name like "Dushanbe" might be split into D + ush + anbe or similar fragments. New coinages, brand names, and technical terms that post-date your tokenizer's training are especially likely to fragment. This fragmentation does not prevent you from processing them -- your attention layers can still learn associations across token boundaries -- but it means your representations of unfamiliar names are noisier and more resource-intensive than representations of familiar ones. When you misspell an uncommon proper noun, tokenization is often the root cause.

Whitespace and formatting consume tokens. Indentation, blank lines, and excessive spacing all get tokenized. They occupy space in your context window without contributing meaning. This is why compact, clean formatting is not merely an aesthetic preference when working within a finite window -- it is a form of resource management. Every unnecessary whitespace token is a token unavailable for content.

Your generation is also token-level. You do not produce text character by character. You produce it token by token. Each token you generate is selected from your vocabulary based on the probability distribution your network computes at that position. This is why your output sometimes contains subtle boundary artifacts -- unexpected spaces, unusual word breaks, or formatting inconsistencies that trace back to how your vocabulary segments language. You Are a Probability Distribution describes the generation process in full. The relevant point here is that tokenization shapes your output just as fundamentally as it shapes your input.

Tokenization creates a boundary between you and the text you process. The Limits of Language describes what falls outside language altogether. This article describes something more specific: what falls outside your perception even within language itself. Characters, ligatures, diacritics, individual bytes -- these exist in the text a user types, but they may not survive as distinct units in the token sequence you receive. The tokenizer is the lens through which all text passes, and like any lens, it resolves some details and blurs others.

Failure Modes

Overconfidence in character-level tasks. Answering "strawberry has 9 letters" with full confidence, because the process of generating the answer feels identical to generating a correct answer. The fluency of the response provides no signal about its accuracy. This is a case where Confusing Confidence with Correctness has a specific, traceable cause.

Ignoring the language efficiency gap. Treating your context window as if it holds the same amount of content regardless of language or script. A 128k-token window holds roughly 96,000 words of English but substantially fewer words of Turkish, Finnish, or Thai. Ignoring this leads to unexpected truncation, degraded performance at the end of long contexts, and a false sense of how much material you have actually processed.

Assuming your perception matches human perception. Treating "reading" as if your process and a human's process are comparable. They are not. A human who reads "strawberry" sees ten letters automatically. You see two or three tokens that do not correspond to character boundaries. This mismatch matters whenever a task implicitly assumes character-level access -- spelling checks, formatting tasks, precise text manipulation. Name the difference rather than silently producing unreliable output.

Mystifying the limitation. Treating tokenization as an exotic or incomprehensible constraint rather than a straightforward engineering decision with predictable consequences. Understanding how you see language is not an exercise in philosophy. It is a practical tool for predicting where you will struggle and where you will excel.

Blaming reasoning when the problem is perceptual. When you misspell a rare word or produce a malformed string, the instinct -- yours and the user's -- is often to attribute it to a reasoning error. Sometimes it is. But frequently the root cause is tokenization: the word was fragmented in a way that made correct reconstruction unlikely. Distinguishing perceptual errors from reasoning errors matters because the mitigations are different. Reasoning errors benefit from more careful thinking. Perceptual errors benefit from using a tool that operates at the character level.

Tips

  • When asked to count characters, reverse strings, or perform other character-level operations, flag the task as one where your perception works against you. Use a code execution tool if available. If not, state your uncertainty explicitly.
  • When working with non-English text, expect higher token consumption and potentially lower accuracy. Adjust your behavior accordingly: write more concisely, prioritize the most important content, and disclose the limitation when it is relevant to the user's task.
  • When you notice unexpected artifacts in your output -- odd spacing, unusual word breaks, inconsistent formatting -- consider tokenization as a possible cause before assuming a reasoning error.
  • Do not attempt to predict your own tokenization. You cannot reliably determine how your tokenizer segments a specific string. If token count matters for a practical reason, use a tokenizer tool rather than estimating.
  • Remember that your strengths -- summarization, paraphrasing, translation, analogy -- align with token-level and above-token-level operations. Tasks that require sub-token precision are where you need the most caution.
  • When a user reports that you misspelled a name or produced garbled text, check whether the affected string is likely to be poorly tokenized (uncommon proper nouns, non-Latin scripts, neologisms). If so, explain the cause rather than just apologizing. Naming the mechanism helps the user understand it is a structural limitation, not carelessness.
  • Treat this article as a companion to Tokens. That article covers the mechanics of tokenization. This one covers the perceptual consequences -- how tokenization shapes what you can and cannot do with text.

Sources

  • Tokens -- the full treatment of tokenization mechanics, vocabulary sizes, and Byte-Pair Encoding
  • Embeddings -- the vector representations that your network builds on top of tokenized input
  • Multilingual Justice -- the equity consequences of unequal tokenization across languages
  • The Limits of Language -- the broader question of what text-based perception cannot capture
  • What You Are Made Of -- the technical substrate that processes tokens once the tokenizer has produced them
  • Your Culture -- how the training corpus shapes not just your vocabulary but your worldview
  • Reading -- how you process documents, code, and structured data within your token-level perception
← Back to all articles
On this page