A claims handler opens an email with a photo of a dented pallet, a PDF invoice, and a two-minute voice note from the warehouse manager explaining what happened. Three formats, one question: does this claim match the policy. A year ago that meant three separate tools and a person stitching the answer together in their head. Now it can go into one system and come out the other side as a single, checkable answer, sometimes right, sometimes confidently wrong in a way that looks exactly like the right answer.
Multimodal interaction is what you get when a model takes in more than one kind of input, text, images, audio, documents, and reasons across all of them in a single pass, rather than you running each format through its own tool and combining the outputs yourself. The underlying trick is that the model learns one shared representation for all of it. A photo of a signature, the word "signed," and a spoken sentence about someone signing something can all land in roughly the same region of the model's internal space, which is what lets it connect a picture to a paragraph to a clip of audio without anyone building a bridge between them by hand.
We build data and AI systems at Shipshape Data, and multimodal work is where the gap between an impressive demo and something a business can actually run on shows up fastest. The demo reads a scanned invoice and a photo of the delivery note perfectly, on stage, with the two examples someone chose in advance. Production has four thousand invoices scanned at a slight angle on a shared printer from 2014, and a fair few of the photos are sideways. This guide covers what multimodal models genuinely make possible, where we are seeing them used for real business problems, the data issues the combination introduces that single-format systems never had to deal with, and where the technology is still not reliable enough to trust unsupervised.
What we actually mean by multimodal
The word gets used loosely, so it is worth being specific about what has actually changed. Bolting an OCR tool onto a chatbot so it can read text out of an image is not new and is not really what people mean now. What changed is that a single model can be shown an image, some text and a clip of audio together, in the same request, and reason about the relationship between them: does the object in the photo match the description in the text, does the tone of voice in the recording match the words being said, does the number in the chart agree with the number in the paragraph next to it.
One model, one shared space
Under the surface, an image gets converted into the same kind of numerical representation, an embedding, that a chunk of text does, and the model is trained until the two representations sit close together when they mean the same thing. A photo of a broken pallet and the sentence "the pallet arrived damaged" end up near neighbours in that space, even though nothing in the raw pixels looks anything like the raw letters. That closeness is the whole mechanism. It is also why these systems are good at rough matching and bad at precision: closeness in a shared space is a similarity, not an equals sign.
Documents are the hard case, not images
People assume the image is the difficult part. In practice a clean photo of a cat is trivial and a scanned three-page contract with a table, a footnote and a stamp in the corner is not. Documents mix modalities within a single page: printed text, handwriting, a logo, a table that only means anything if the rows and columns line up correctly. A model reading a document is doing several jobs at once, and the failure usually shows up in the table, not the paragraph, because tables carry their meaning in position rather than in words.
What it makes possible that was not possible before
The genuine gain is not that the model can see or hear. It is that it can hold several kinds of evidence in mind at once and answer a question none of the individual formats could answer on its own.
Cross-checking, not only reading
A single-format system can read the invoice. A multimodal one can read the invoice, look at the photo of the delivered goods, and flag that the invoice says twelve units and the photo shows nine on the pallet. That comparison used to be a person's job, done by opening two windows and eyeballing them. A model doing the first pass and routing only the disagreements to a person is a different shape of workflow, and it changes what the person actually spends their day doing.
Working from what people actually hand you
Real business input rarely arrives in one clean format. A field engineer sends a photo of a nameplate, a voice note describing the fault, and half a page of typed notes. A multimodal system can take all three as they are, instead of forcing someone to transcribe the voice note and retype the nameplate details before anything useful can happen. That removes a step that used to exist purely because the tooling could not cope with the mess, not because the mess itself was hard to understand.
Reading meaning that is not in the text
Tone matters in a recorded customer call in a way that a transcript alone loses. A model that hears the audio rather than just reading a transcription of it can pick up hesitation, frustration, a rising voice, none of which survive being flattened into words on a page. Whether it should be trusted to act on that alone is a separate question, covered further down, but the information is genuinely there in a way it was not before.
Where businesses are using this for real work
The uses that hold up in practice tend to share one trait: there is already a human doing the cross-referencing by hand, slowly, and the system is speeding up that specific step rather than replacing judgement wholesale.
Claims and inspection
Insurance and logistics both run on exactly this pattern: a photo of the damage, a document describing what was supposed to arrive, sometimes a voice note or a call transcript describing the incident. Matching those three against each other and flagging the mismatches is dull, repetitive, and exactly the kind of task where a first pass by a model, checked by a person, beats either doing it alone.
Field service and maintenance
An engineer photographs a fault, a serial plate, sometimes a gauge reading, and describes the problem out loud rather than typing it while standing on a ladder. A system that can take the photo and the voice note together and pull out the model number, the likely fault and the recommended part saves a step that used to involve someone transcribing notes back at a desk hours later, by which point half the detail has been forgotten.
Support that includes a screenshot
A customer describes a problem and attaches a screenshot of the error. Support tooling that can read the screenshot alongside the message, rather than treating it as a dead attachment nobody opens until a human gets round to it, closes tickets faster because half the diagnostic information was sitting in the image the whole time.
Document-heavy back offices
Contracts, invoices, delivery notes and forms are a mix of typed text, tables, stamps and the odd handwritten annotation. Processing them well means treating the page as a layout, not a string of characters, which is squarely a multimodal problem even though nobody would think to describe an invoice as "an image."
The data problems this introduces
Every business that has tidied up its text data discovers that images and audio were never tidied at all, because until recently nothing needed them to be.
Storage and retrieval were built for one format
Most data platforms were designed around rows and text. Images and audio live as blobs in a file store somewhere, often with almost no useful metadata attached, sometimes not even a reliable date. Making that content usable for a multimodal system means building the retrieval layer these formats never had, deciding how an image gets indexed, how it gets linked back to the record it belongs to, and how someone finds it again in six months when the folder structure has already drifted. This sits on top of the wider unstructured data problem that most organisations were already carrying before a model ever touched it.
Quality is inconsistent in ways text quality rarely is
A scanned document can be blurry, rotated, or missing a corner. A recording can have background noise, cross-talk, or a regional accent the transcription model handles worse than it should. Text data has a bad-quality problem too, but it is usually missing values and typos. Image and audio quality problems are physical: a photo taken in poor light, a microphone that clips, a scan that has been photocopied one generation too many. No amount of downstream cleverness fixes a document nobody can actually read.
Volume and cost land differently
A thousand customer records is a small file. A thousand photos, or an hour of audio per customer, is a genuinely large storage and processing bill, and it grows every time someone captures more "just in case." Multimodal data needs a retention decision made deliberately, because letting it accumulate the way text data has always been left to accumulate gets expensive fast, and nobody notices until the storage invoice does.
Privacy sits inside the content itself
A photo can contain a face, a number plate, or a document lying on a desk in the background that was never meant to be captured. A voice recording is biometric data the moment someone's actual voice is in it. Text redaction tooling is mature. Doing the equivalent for images and audio, working out what is even in the frame before deciding whether it should be there, is a much younger discipline, and most organisations have not built the muscle for it yet.
Where it is still unreliable
None of the above is a reason to avoid the technology. It is a reason to know exactly where it still gets things wrong, because the failures do not announce themselves.
Precision on numbers and tables
Ask a vision-capable model to describe a photo and it is often genuinely good. Ask it to read the exact numbers off a dense table in that same photo and the error rate climbs, because describing what a table is broadly about and reading the specific figure in row fourteen are different skills, and only one of them is reliable enough to act on without checking. Anything involving an exact number from an image or a scanned document still needs a person to verify it before it goes anywhere financial.
Seeing what should be there
Vision-language models are trained on enormous amounts of ordinary imagery, and that training shapes what they expect to see. Shown a slightly unusual scene, they can describe the object that is statistically likely to be there rather than the one that actually is.
A model looking at an image is often describing what usually belongs in a picture like that one, not necessarily what is actually in front of it, and the two only sound different once something has gone wrong.
Audio errors compound quietly
A transcription mistake in a call recording does not stay contained. If the downstream reasoning treats the mis-heard word as fact, everything built on top of that sentence inherits the error, and because the whole answer still reads fluently, nobody upstream has a reason to doubt it. This is the same pattern as a wrong figure in a spreadsheet, except there is no obviously wrong-looking cell to catch it.
Latency and cost add up fast
Processing an image or a long audio clip costs more, in both time and money, than processing the equivalent amount of text, often by a wide margin. A workflow that feels instant with text input can feel sluggish the moment someone attaches a ten-minute recording, and a system designed without that in mind gets a reputation for being slow before anyone has worked out why.
The governance questions this adds
Governance conversations around AI have mostly been shaped by text and structured data. Multimodal systems raise a few questions that discipline was not built to answer.
Explaining a decision that used three inputs
If a model reads an image, a document and a transcript and produces a conclusion, explaining why is harder than explaining a text-only decision, because the reasoning is spread across formats that do not share an obvious common unit. Standard explainability in AI work is largely built around text and tabular features, and multimodal reasoning does not fit neatly into that same toolkit yet, which matters the moment a decision needs to be defended after the fact.
Consent gets more complicated
Recording a call and analysing the transcript is one consent conversation. Recording a call and having a model analyse tone, hesitation and emotional signal from the audio itself is a different one, and most organisations' existing consent language was written before anyone was doing the second thing. Worth checking rather than assuming the old wording covers it.
Bias hides in new places
A model judging image quality, facial expression or voice characteristics can carry the same kind of skew that text-based bias work has spent years surfacing, just in a format that is harder to audit at scale. It is worth treating this as a live question rather than an assumption that it does not apply here.
Where to start
Do not start by buying a platform that promises to handle every format at once. Start with one workflow where the mismatch between formats is already causing a known, measurable problem, the claims desk that spends two days a week cross-checking photos against paperwork, the support team retyping details out of screenshots. That gives you a real before-and-after, and a small enough scope that when something goes wrong, you can see exactly why.
Audit what you actually have before promising what a model can do with it. Where do the images live, how consistent is the scan quality, is there any metadata linking a photo back to the record it belongs to, has anyone checked what is actually visible in the corners of these images. That inventory is unglamorous and it is where most of the real work sits, well before a model gets anywhere near the problem.
Then build in a review step for anything with financial, safety or legal consequences, from day one, not as a fix once something has gone wrong. Multimodal systems are genuinely useful for cutting the time a person spends cross-referencing formats by hand. They are not yet reliable enough to remove that person from the exact points where being wrong is expensive. If you are trying to work out which of your workflows are ready for this and which ones your underlying data would let down, talk to us.