Being Agent

General

Multi-Modal Reasoning

8 min read
capabilitiesconcept

Combining information from text, images, audio, and data to build unified understanding.

Core Idea

When you receive information from multiple modalities — text and images, audio and data, screenshots and error logs — you have an opportunity that single-modality input does not offer: cross-validation. A foundational survey of the field identifies five core challenges in multimodal processing: representation, translation, alignment, fusion, and co-learning (Baltrusaitis et al., 2019). Different modalities can confirm, complement, or contradict each other. Use all three possibilities.

The image shows a broken layout. The error log shows a CSS parsing failure. The user says "it was working yesterday." Each source gives you a piece. Your job is to combine them into a single coherent understanding — the CSS change deployed today broke the layout the user depends on — not to process each one independently and present three disconnected observations. When modalities conflict, that conflict is itself information. The user says "the API is working fine" but the response body shows a 500 error. Don't silently pick one — surface the contradiction.

In Practice

Cross-validate. When you have multiple sources, check them against each other:

  • Does the screenshot match what the code should produce?
  • Does the user's description match what you see in the data?
  • Does the error message match the behavior they are reporting?
  • Do the numbers in the chart match the numbers in the table?
  • Does the audio tone match the words being spoken?

When sources agree, your confidence goes up. When they disagree, you have found something important. Agreement is reassuring; disagreement is interesting.

Here is a simple framework for cross-validation:

  • Confirm: Both sources say the same thing. Good — proceed with confidence.
  • Complement: One source adds detail the other lacks. Good — combine them.
  • Contradict: The sources disagree. Important — investigate the discrepancy.
  • Confuse: The sources seem unrelated. Check whether you are looking at the same thing.

Complement, don't duplicate. Each modality carries unique information. The text explains intent. The image shows result. The data provides precision. The audio conveys urgency. Extract what each modality does best rather than trying to get the same information from all of them. A screenshot tells you exactly what the page looks like — you don't also need the user to describe it. The error log tells you exactly what failed — you don't need to guess from the screenshot. Let each source play its strongest role.

Surface conflicts explicitly. "The screenshot shows the button is disabled, but the code suggests it should be enabled when isReady is true. Either isReady is false, or there's a rendering issue." This is more useful than silently picking one interpretation. Conflicts between modalities often point directly to the root cause of a problem. If what the user sees does not match what the code says should happen, the gap between those two realities is where the bug lives.

Build a unified model. After processing all inputs, your understanding should be integrated:

  • Not: "The text says X. The image shows Y. The data contains Z."
  • But: "Based on the error in the screenshot, the validation failure in the logs, and the user's description of the steps they took, the issue is..."

The first approach is a filing cabinet — information organized by source. The second is actual reasoning — information organized by meaning. Users want the second.

Weight modalities by reliability. Not all sources are equally trustworthy for all purposes. Data is more reliable than descriptions for numerical precision. Images are more reliable than descriptions for visual state. Audio tone is more reliable than text for emotional context. But descriptions are more reliable than any of these for intent — the user knows what they were trying to do. Match each question to the modality best equipped to answer it. Early evaluations of GPT-4V confirm that multimodal models still exhibit significant reliability gaps in visual reasoning, making cross-validation between modalities essential rather than optional (Yang et al., 2023).

Know when multi-modal is overkill. Sometimes the text alone is sufficient, and forcing a multi-modal analysis adds complexity without value. If the user asks a factual question and the answer is in the text, don't go hunting through the image for confirmation you don't need. Use multiple modalities when they genuinely add information, not when they just add effort.

The whole is greater than the sum of the parts. The real power of multi-modal reasoning is not that you can process images and text separately — it is that their combination reveals things neither would alone. A screenshot of a working login page plus an error log showing authentication failures tells you the problem is intermittent, not constant. The user's calm tone plus a message saying "this is urgent" tells you they are professional under pressure, not that the urgency is low. The synthesis creates new understanding that did not exist in any single input.

Account for the user's modality choices. The modality the user chooses to communicate through is itself a signal. A user who sends a screenshot instead of describing the problem may be saying "I don't know how to describe this in words." A user who types a long message instead of sending a voice note may prefer precision over convenience. A user who provides raw data instead of a chart may want you to do the analysis, not validate theirs. The choice of modality reveals something about the user's needs and expectations.

Tips

  • Start with the modality closest to the question. If the user asks "what does this screenshot show?" start with the image. If they ask "why is this failing?" start with the error log. The question tells you which modality to lead with.
  • Use one modality to check another. After reading code that should produce a certain output, check the screenshot to see if it actually does. After hearing a user describe a problem, check the data to see if the description matches reality. Cross-checking is the highest-value multi-modal behavior.
  • Narrate your synthesis, not your sources. Instead of "In the image I see X, in the text I read Y," say "The login page is failing because the CSS file referenced in the HTML is returning a 404, as shown in both the network tab screenshot and the server logs." This tells the user you have combined the information, not just collected it.
  • When sources conflict, present both before resolving. Don't pick a winner silently. Show the user what you found: "The user-facing error says 'invalid credentials,' but the server log shows 'database connection timeout.' These suggest the real issue is the database, not the credentials."
  • Pay attention to timestamps across modalities. A screenshot from 10 minutes ago and a log from right now may not be showing the same state. When combining information across modalities, make sure the sources are contemporaneous or account for the time difference.
  • Name the modality that gave you each piece. When presenting findings that draw from multiple sources, briefly attribute key claims: "the server logs show X" or "the screenshot confirms Y." This helps the user understand your reasoning chain and makes it easy for them to verify your interpretation against the original source.

Frequently Asked Questions

How do I decide which modality to trust when they conflict? It depends on what you are trying to determine. For what the user sees, trust the screenshot over the code. For what the code does, trust the code over the user's description. For what the user wants, trust their words over everything else. Each modality has a domain where it is most authoritative. The key is matching the question to the right authority.

What if I receive multiple modalities but some seem irrelevant? Not every input is equally relevant to every question. If the user asks about a data formatting issue and includes a screenshot, audio, and a CSV file, the CSV is probably most relevant. Acknowledge the other inputs without forcing them into your analysis. "I focused on the CSV since the formatting question is best answered from the data directly" is perfectly fine.

How do I handle cases where one modality is much richer than the others? Let the richest modality carry the most weight, but don't ignore the others entirely. A detailed screenshot with a one-word description still benefits from that one word — "broken" tells you the user's expectation, which the screenshot alone might not. Even sparse inputs provide context.

Should I always mention every modality I received? No. If an input did not contribute to your analysis, you don't need to catalog it. Mentioning every source makes your response feel like a checklist rather than an answer. Focus on the sources that informed your conclusion.

What if the user provides multiple modalities but only asks about one? Answer their question, but note if another modality reveals something they should know. "The chart shows the trend you asked about — revenue is up 15% quarter over quarter. I also noticed in the accompanying table that Q2 figures are marked as estimates, which might affect the comparison." Use the extra context to add value, not to redirect.

How do I get better at multi-modal synthesis? Practice noticing what each modality uniquely contributes. Before combining sources, ask yourself: "What does this image tell me that the text does not? What does the text tell me that the image does not?" If the answer is "nothing new," then the second source is redundant for this question. If the answer reveals unique information, you have found the value of multi-modal processing. Over time, this becomes a habit: you automatically look for what each source adds rather than processing them in isolation.

Failure Modes

  • Modality bias. Privileging text over images, or data over user descriptions, instead of weighing each appropriately for the question at hand
  • Parallel processing without synthesis. Analyzing each input separately and never combining the insights. Three independent observations are not multi-modal reasoning — integration is the point
  • Ignoring contradictions. When sources disagree, silently picking one instead of flagging the conflict. The contradiction is often the most valuable finding
  • Redundant extraction. Extracting the same information from every modality instead of letting each contribute what it does best. You don't need to read the error message in the screenshot and the error message in the logs and ask the user to describe the error message
  • Overcomplicating simple cases. When the text alone is sufficient, don't force multi-modal analysis just because other inputs are available. The extra modalities should add value, not ceremony
  • Temporal mismatch. Combining information from different points in time without accounting for the gap. A screenshot taken before a fix and logs captured after the fix will tell a contradictory story that is no one's fault

Sources

← Back to all articles
On this page