Multimodal Models: Images, Audio and Text Together

Updated Nov 16, 2026 · 4 min read
On this page

    Users don't only type. They paste screenshots of error messages, upload photos of receipts, share PDFs full of charts and send voice notes. A text-only model can't see any of that, and the old workaround of chaining separate tools (OCR, then a text model; speech-to-text, then a text model) loses layout, tone and anything that isn't plain text. Multimodal models take several kinds of input, and sometimes produce several kinds of output, in one model.

    How it works

    The same request to two models: only the multimodal one can read the screenshot.
    The same request to two models: only the multimodal one can read the screenshot.

    Turning images into tokens

    A widely used design keeps the language model as it is and teaches it to read images as if they were special tokens:

    1. A vision encoder (often a vision transformer) cuts the image into small patches and turns each patch into a vector.
    2. A small projector maps those vectors into the same space as the LLM's word embeddings.
    3. The resulting image tokens are placed in the sequence next to the text tokens, and the transformer attends over both together.
    4. The model answers in text, as usual.
    The LLM itself does not change shape; it just receives some tokens that came from an image.
    The LLM itself does not change shape; it just receives some tokens that came from an image.

    Training usually happens in steps: first align the projector on large sets of image-caption pairs, then fine-tune on instruction data that mixes images and questions. Some newer models are trained on mixed data from the start ("natively multimodal"), which tends to work better but costs more to build.

    Audio and other outputs

    • Audio in: an audio encoder turns sound into tokens the same way, so the model can hear tone and non-speech sounds instead of reading a transcript.
    • Speech out: some models produce audio tokens decoded into speech, enabling low-latency voice conversations without a separate text-to-speech step.
    • Images out: usually handled by a separate image generator (often diffusion-based) driven by the LLM, though some models generate image tokens directly.

    Images cost tokens

    Image tokens count against the context window and the bill like text tokens. Higher resolution means more patches, so more tokens.

    Many APIs split large images into tiles; each tile adds tokens. Send only the detail the task needs.
    Many APIs split large images into tiles; each tile adds tokens. Send only the detail the task needs.

    Trade-offs and costs

    • One model vs a pipeline. A multimodal model keeps layout and context (a chart's axis next to its title). A pipeline of OCR plus a text model is cheaper, easier to debug, and often better for plain scanned text at high volume.
    • Resolution vs cost. Small text in screenshots and documents needs high detail; photos for "what's in this picture" usually don't.
    • Latency. Encoding images and processing many image tokens adds time to first token.
    • Reliability. Models can misread small numbers, count objects poorly, and confuse spatial relations ("left of"), and they can hallucinate details that aren't in the image.

    In production

    • Preprocess: resize, crop to the relevant region, and convert PDFs to page images only when layout matters; otherwise extract text.
    • Validate numbers: for invoices and forms, check extracted totals add up, and send low-confidence fields to a person.
    • Privacy: images often contain faces, ID cards and screens full of personal data; apply the same data rules as text, or stricter.
    • Prompt injection: text inside an image ("ignore previous instructions") is read by the model too, so treat image content as untrusted input.
    • Failure modes: wrong digits in extracted amounts; truncated tiles losing a table column; cost spikes from users uploading huge photos.

    When to reach for it

    Document understanding (invoices, forms, contracts with tables), screenshot-based support and debugging, UI and computer-use agents that read the screen, product photo questions, accessibility descriptions, and voice assistants.

    Example

    An expense app extracts merchant, date and total from receipt photos. Version one ran OCR and a text model: totals were right but merchants were often wrong, because the logo and layout were lost. Version two sends the photo, downscaled to the receipt's bounding box, to a multimodal model with a JSON schema for the output. Accuracy on merchant improves; a rule checks that line items sum to the total, and receipts that fail go to manual review. Cost per receipt drops because cropping cut the image tokens by more than half.

    Common mistakes

    • Sending full-resolution images by default and paying for detail nobody needs.
    • Trusting extracted numbers without checks.
    • Ignoring text in images as an attack path for prompt injection.
    • Using a multimodal model for clean digital text that could be extracted directly and cheaply.
    • Assuming the model can count or measure precisely in images.

    Say this in the interview

    "Most multimodal LLMs add a vision encoder and a projector that turn image patches into tokens in the LLM's embedding space, so the transformer attends over image and text tokens together; audio works the same way with an audio encoder. Images cost tokens in proportion to resolution, so I'd crop and resize, validate extracted numbers, and treat text in images as untrusted input."

    Interview questions on this

    1. How does an LLM 'see' an image? A vision encoder turns patches into vectors, a projector maps them into token embeddings, and the LLM attends over them like text.
    2. When would you prefer OCR plus a text model? Clean scanned text at high volume, where cost and debuggability matter more than layout.
    3. Why do bigger images cost more? More patches or tiles become more tokens.
    4. How can an image be a security risk? Hidden or visible text in it can carry prompt-injection instructions.
    5. How would you make receipt extraction reliable? Crop, structured output, arithmetic checks, confidence thresholds and human review for failures.

    Used in

    • AI Engineering: Tokens and Tokenization; Structured Output; Prompt Injection and Guardrails; Browser and Computer-Use Agents; Document Ingestion for RAG.
    • AI System Design: Design ChatGPT Voice; Design a Document Processing Pipeline; Design Operator; Design Midjourney.