Use Ctrl+P (or Cmd+P) to save as PDF. Back to paper
A multimodal model does not see an image. It reads a sequence of tokens.
That distinction changes how you think about memory, latency, and cost when you run vision-language models locally.
A Vision Transformer (ViT) divides an image into a grid of fixed-size patches. A 224×224 image with 16×16 patches produces 196 patch embeddings. A 336×336 image with 14×14 patches produces 576. Each patch is projected into the same vector space the language model uses for text tokens. A small connector layer — often called a projector — maps ViT output dimensions to LLM embedding dimensions.
The result: an image arrives at the LLM decoder as a flat token sequence, indistinguishable in structure from word tokens. The model attends to image tokens and text tokens with the same attention mechanism. No special pathway. No separate memory. Just more tokens.
LLaVA-1.5 encodes a standard image as 576 visual tokens. That is 576 positions consumed from your context window before the user has typed a single word.
LLaVA-NeXT, using AnyRes to handle higher resolutions, generates up to 2,880 tokens for a 672×672 image — five times more. At 3,840×1,600 resolution, later models require over 8,000 tokens per image. One high-resolution image can consume the majority of a 8K context window.
This is not a model quality problem. It is an arithmetic problem. The token count is determined by:
When you run a vision-language model locally — LLaVA, LLaMA 3.2 Vision, Qwen-VL — you are sharing context window capacity between the image, the system prompt, conversation history, and the expected response. The image does not compress. It does not share positions with text. It occupies a fixed block of the budget determined before inference starts.
At 576 tokens, a single image leaves room for a substantive conversation. At 2,880 tokens in an 8K context, you have roughly 5,100 positions for everything else. At higher resolutions, the model is context-bound before the task begins.
The second cost is KV cache. Image tokens occupy KV cache exactly as text tokens do. More image tokens means more GPU memory per request, lower concurrency, and faster context exhaustion in multi-turn conversations.
The number of visual tokens is not fixed by the image. It is determined by the resolution you pass to the encoder. Most serving frameworks default to the resolution the model was trained at. That default is not necessarily the right operating point for your workload.
If your application needs spatial detail — document parsing, chart analysis, OCR — you need the tokens. If your application needs image understanding at a coarser level — scene classification, captioning, visual QA — you can reduce resolution and cut token count significantly without meaningful accuracy loss. AVG-LLaVA demonstrated 85.3% token reduction at 2.53× inference speedup on tasks that do not require fine-grained visual information.
The question is not "how many tokens does this image produce?" The question is "how many tokens does this task require?"
Answer that first. Set the resolution accordingly. The model will work with what you give it.
1. Dosovitskiy, A., et al. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021. https://arxiv.org/abs/2010.11929
2. Liu, H., et al. (2024). LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. LLaVA Blog, January 2024. https://llava-vl.github.io/blog/2024-01-30-llava-next/
3. Kim, G. & Seo, M. (2024). On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning. EMNLP 2024. https://arxiv.org/pdf/2406.11823
4. Lan, Z., et al. (2024). AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularity. arXiv:2410.02745. https://arxiv.org/abs/2410.02745
5. Ge, C., et al. (2024). ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models. arXiv:2405.15738. https://arxiv.org/abs/2405.15738