Syntic
All posts

Published October 4, 2026 in Models

Why the Future of AI Is Multimodal

By Syntic AI

Consider what a text-only model actually receives when an insurance claim arrives. There is a photo of a dented rear bumper, a scanned repair estimate with one figure crossed out and rewritten by hand, and a recorded call in which the claimant pauses for a long time before answering a question about when the damage happened.

By the time any of that reaches a text model, something else has already decided what it meant. An OCR step chose which number on the estimate was real. A captioning step described the photo. A transcription step turned the call into words and dropped the pause. The model reasons carefully over whatever survived.

That is the real case for multimodal AI. It is not about adding features. It is about removing the lossy translations between the world and the model, so the system that reasons also looks and listens. The gain is information. The price is a new class of errors, an unsolved evaluation problem and a cost profile that punishes careless use.

The keyhole problem

A pipeline of specialised models is a reasonable design, and for years it was the only one available. Its weakness is that every hand-off compresses, and each component discards what it was not built to keep.

OpenAI described this plainly when it introduced GPT-4o in May 2024. Its earlier Voice Mode chained three models: one transcribed audio to text, a language model produced a text reply, and a third converted that reply back into speech. As a result, the company wrote, the main model "loses a lot of information": it "can't directly observe tone, multiple speakers, or background noises," and it cannot laugh, sing or express emotion. Google made the same argument about Gemini in December 2023, contrasting "training separate components for different modalities and then stitching them together" with a model "pre-trained from the start on different modalities."

Documents show the same pattern, and the losses are easier to measure. OHRBench, a 2024 benchmark built from 8,561 document page images across seven domains, tested how OCR errors spread through retrieval-augmented generation. It separated two kinds of damage: semantic noise (wrong characters, wrong numbers) and formatting noise (tables and structure flattened into ambiguous text). Performance fell as noise rose, and the authors concluded that none of the OCR solutions they tested was "competent for constructing high-quality knowledge bases for RAG systems."

The alternative is to let the model read the page as an image. ColPali, published at ICLR 2025, embeds images of document pages directly instead of extracting text first. On its ViDoRe benchmark it outperformed modern text-extraction retrieval pipelines while being, in the authors' words, "drastically simpler, faster and end-to-end trainable." Layout, tables and figures carry meaning; a pipeline that discards them cannot recover it later.

What "native" means

"Natively multimodal" is used loosely, so it helps to be precise. Most vision-language models have been built by late fusion: a separately trained image encoder turns pictures into vectors, and a projection layer feeds those vectors into a language model. Early fusion trains a single model on mixed sequences of text and image tokens from the beginning.

The engineering question was whether early fusion is worth the trouble. A 2025 study by researchers at Apple and Sorbonne University trained 457 models to find out. They found that early fusion performed better at lower parameter counts, was more efficient to train and was easier to deploy, and that a mixture-of-experts design helped by letting the model learn modality-specific weights. That supports integrated design, though labs rarely publish enough to show how any commercial model is built.

Pipelines still have a place. A transcript is a searchable record; a separate OCR step can be audited and tested on its own. The practical rule is narrower than "native is always better": use separate stages where the intermediate artefact has value of its own, and let the model see the raw input where the meaning lives in what conversion would throw away.

Where the work is not text

Most consequential business work arrives in forms that were never written for a machine.

Documents. Invoices, contracts, lab reports and customs forms encode meaning in layout: which column a figure sits in, which box is ticked, whether a stamp or signature is present, which version of a clause was struck through. These are the details that decide whether a payment goes out or a claim is approved.

Screens. Much of the software a business runs has no API an agent can call, only an interface built for human eyes. To operate it, a model has to find the right control in a screenshot. ScreenSpot-Pro, a 2025 benchmark of 1,581 instructions drawn from 23 professional applications across five industries, found that the best grounding model located the right element only 18.9% of the time on high-resolution screens with small targets. The authors' own method reached 48.1% without extra training, mainly by narrowing the area the model searched. Agent reliability depends on perception that is often treated as a solved detail.

Calls. A spoken conversation carries hesitation, emphasis, crosstalk and background noise. Much of this is lost in a transcript, and some of it is exactly what a skilled agent listens for. The operational side of this shift is covered in how the call center is becoming software.

Images from the field. Damage photos, shelf audits, meter readings and site inspections are visual evidence that a person usually looks at and then summarises in text.

In each case, the text a pipeline produces is an interpretation of the evidence. A multimodal model can work from the evidence itself. Whether it does that well is a separate question.

Reading the scoreboard

The most cited test of multimodal reasoning is MMMU, presented at CVPR 2024: 11.5K college-level questions across 30 subjects and 183 subfields, using 30 kinds of images, including charts, maps, chemical structures and sheet music. When it launched, GPT-4V scored 56%. Ninety college seniors tested on the same material scored between 76.2% (worst) and 88.6% (best). By August 2025, OpenAI reported 84.2% for GPT-5, inside that human range.

That looks like the gap has closed. The research on evaluation suggests caution.

A 2024 study that introduced the MMStar benchmark found that many multimodal questions can be answered without the image at all. Gemini Pro scored 42.9% on MMMU with no images, far above chance. Another model scored 43.6% without images, 17.9 points above its own text-only backbone, which suggests it had memorised benchmark material during training. The authors built a 1,500-question set checked by hand for genuine image dependence.

MMMU's own authors reached a similar conclusion. MMMU-Pro removed questions that text-only models could answer, increased the number of answer options, and added a setting where the whole question is embedded in an image. Scores fell by 16.8 to 26.9 percentage points depending on the model. Adding OCR hints barely helped; chain-of-thought reasoning generally did.

The practical lesson: run your own test cases twice, once with the image and once with it blanked. If accuracy barely changes, the model is answering from priors, and your evaluation is measuring the wrong thing.

How multimodal models fail

Multimodal models can hallucinate observations, not just facts, and an invented observation is harder to catch because it looks like evidence.

Seeing what is likely, not what is there

The 2023 paper that introduced the POPE benchmark found that objects which appear often in training data, or which often appear alongside objects actually in the image, are "obviously prone to be hallucinated." A kitchen photo invites a reported toaster. An invoice invites a VAT line. The model fills in what usually appears in that kind of scene.

Failing at the simple things

The 2024 "Vision language models are blind" study gave four leading models tasks a child could do: decide whether two circles overlap, count line intersections, identify a circled letter. Average accuracy was 58.07%, and the best model reached 77.84%. Accuracy approached 100% when shapes were spaced further apart, pointing to fine spatial detail rather than reasoning. Eyes Wide Shut (CVPR 2024) traced a related weakness to the vision encoders themselves: image pairs that look different to people but similar to CLIP also tripped up multimodal LLMs, often with hallucinated explanations.

Reading wrong

Small text, rotated scans, heavy compression and automatic downscaling all degrade reading. Vendor documentation is candid about this. Anthropic's vision guide warns that its models may make mistakes with low-quality, rotated or very small images, that object counts are approximate, and that coordinate outputs should be verified. Downscaling a large screenshot to fit a token budget can make the one figure that matters illegible.

Why perception errors are dangerous

In MMMU's analysis of 150 GPT-4V errors, perceptual mistakes were the largest category, at 35%, ahead of missing knowledge (29%) and faulty reasoning (26%). Perception errors also compound. A misread figure becomes a premise, and fluent reasoning built on a false premise is more convincing than an obvious error.

The defence is grounding. Ask the model to cite where it saw each value (page, region, timestamp), check extracted figures against arithmetic or system-of-record data wherever possible, and send low-confidence reads to a person. These are the same disciplines AI workflows need at every interpretive step, applied to pixels and audio.

The cost of looking

Under Anthropic's current published rules, an image costs roughly one visual token per 28×28-pixel patch. A 1000×1000 image costs 1,296 tokens. A full-HD screenshot costs 2,691 tokens on its high-resolution models. Google's Gemini documentation lists audio at 32 tokens per second and video at about 100 tokens per second at default resolution. At those rates, an hour of audio is about 115,000 tokens and an hour of video about 360,000, before anyone asks a question. These rates change, so treat them as orders of magnitude rather than a price list.

Three habits help. Send the resolution the task needs, cropping to the region that matters. Sample video at the rate events change. And be deliberate about which modality you keep: a transcript made once and searched many times may be cheaper and more auditable than sending raw audio to a large model again and again. Latency follows the same arithmetic.

Model choice becomes a design decision too: a small model can triage an image while a larger one examines the hard cases, a pattern discussed in why one AI model is not enough. As the cost of intelligence falls, continuous perception becomes affordable, and the advantage goes to whoever spends those tokens well.

Perception is becoming the agent's input layer

The most important consequence of multimodality is not better image descriptions. It is that perception becomes the input layer for systems that act: an agent operating software reads screenshots, an agent handling claims reads photos and forms, an agent on a call hears how something was said. The shift from answering to doing depends on that input being reliable, and grounding benchmarks such as ScreenSpot-Pro show how far it still has to go.

Robotics takes the same idea furthest. Google DeepMind's RT-2 (2023) represented robot actions as text tokens trained alongside web-scale vision-language data, and reported much better generalisation to new objects across 6,000 evaluation trials. Physical deployment is far harder, but the direction is the same: one model that perceives, reasons and acts.

None of this is the same as generating images or video. That is a separate shift, covered in generative AI as a production tool. Understanding comes first, because a system that cannot see accurately has nothing reliable to act on.

Three questions for any multimodal system

First, what does the model actually receive? If the image, page or recording passes through a conversion step first, find out what that step throws away.

Second, does accuracy depend on the input? Run the blank-image test. A model that scores the same without the evidence is not using the evidence.

Third, can it show where it looked? An extracted value that cannot be traced to a page region or a timestamp cannot be verified, and an unverifiable observation should not become an action.

Text was the first interface to machine intelligence because it was the easiest one to build. The work businesses need done was never limited to text.

Sources