Vision-Language Models Explained: How AI Sees Images

Hand-written rules break on real photos, and people cannot check photos fast enough. Vision language models offer a third way: software that looks at a photo and answers questions about it in words.

Picture a common setup. A snack maker pays small shops a monthly fee to keep its packets at eye level. Its 180 field sales reps photograph each shop's shelves and upload the photos from a phone app. The setup is made up, but the problem is real.

That comes to about 4,500 photos a day. An audit team of six checks each photo against four questions:

• Is the brand's packet on the shelf at all?
• How many packet fronts face the customer?
• Can you read the price tag below the packets?
• Has a rival brand taken the slot the snack maker pays for?

A careful person checks about 60 photos an hour. Over a seven-hour day, six people manage roughly 2,500, so almost half the photos are never checked. Problems show up weeks later, after the shop has been paid.

The team first tries plain code. Matching the brand's orange colour finds its packets, but also orange soap boxes and a faded poster. Every new shop's lighting, angle and clutter breaks another rule.

What the team needs is software that answers the four questions in words, the way a person would. This guide shows how a vision language model does that, where it fails and how to build around it.

Vision language models, or VLMs, are AI models that take images plus text as input and produce text as output. You show a photo, ask a question in plain words and get an answer in words.

Two terms help here. An AI model is a program that learned its behaviour from examples instead of rules a person wrote. A language model is one that learned from huge amounts of text to predict what text comes next.

A VLM gives a language model a way to see. The audit team can send one shelf photo with one question, such as whether the top shelf holds their packet, and get a sentence back.

One model can do several jobs that used to need separate systems:

• Captioning: writing a short description of a photo, such as a shelf of snack packets beside soap boxes.
• Visual question answering: answering a specific question about the photo, such as which brand sits on the top shelf.
• Reading text in images: picking out words and numbers, such as Rs 20 on a price tag.

Think of an experienced birdwatcher at Bharatpur, the bird sanctuary in Rajasthan. A bird flashes past for two seconds, and she still names it: a young painted stork, flying north.

Map it part by part. Her eyes pick out marks, such as a pink patch near the tail: that is the image encoder. Turning those marks into words is the projector. Her voice is the language model.

Where the picture breaks: the birder can turn her head and look again. The basic design in this guide cannot. It gets one fixed, often shrunken, copy of the photo. Later sections show tiles and crops that add extra views.

It cuts the photo into small squares called patches, turns each patch into a vector and passes the vectors through an image encoder. The encoder lets patches share information and outputs one vector per patch.

Start with what a photo is to a computer: a grid of pixels, tiny dots of colour. Each pixel is stored as three numbers, for red, green and blue.

A 2020 research paper showed a simple way to feed images to a transformer, the model design most language models use. It splits the image into fixed-size patches and treats them as a sequence, like words in a sentence.

Each patch is flattened into one long list of pixel numbers. A learned layer then turns that list into an embedding: a vector that captures what the patch looks like, where similar patches get similar vectors.

Here is the arithmetic for the setup in that paper. A 224 by 224 pixel image cut into 16 by 16 patches gives 14 patches per side, so 196 patches. Each flattened patch holds 16 x 16 x 3 = 768 numbers.

A patch on its own does not know where it sat in the photo. So the model adds a position embedding, a learned vector that marks each patch's place. That is how it can tell the top shelf from the bottom one.

The paper also puts one extra learned vector, the class token, at the front of the sequence. Its output stands for the whole image, so 196 patches become 197 positions.

Follow the diagram from the phone to the patch vectors. Notice that nothing in this half of the model knows any words yet.

Inside the encoder, a step called attention lets each patch take in information from every other patch. After it, the vector for a patch showing half a packet also carries information from the rest of the photo.

Many image encoders learn it from huge collections of images paired with captions, by training to pick which caption belongs to which image. This is called contrastive learning, because right pairs are contrasted with wrong ones.

A 2021 paper trained this way on 400 million image and text pairs gathered from the internet. It used two encoders, one for images and one for text, and turned each into a vector.

Take a training batch of N images and their N captions. There are N x N possible pairings. Training raises the similarity of the N real pairs and lowers it for every wrong pairing. The paper's batches held 32,768 pairs.

Because the labels are just sentences, you can write new ones without retraining. The paper calls this zero-shot transfer: using a model on a task it never trained for, by describing the classes in words.

Without using any of ImageNet's 1.28 million labelled training photos, the paper's best model scored 76.2% on that well-known photo benchmark. That matched a standard image classifier trained directly on those photos.

The same paper lists its weak spots. It reports that the model struggles with abstract, systematic tasks such as counting the objects in an image. Keep that in mind for the shelf counts.

The audit team uses this matching kind of model as a quality gate. Before any detailed question, it checks whether the photo even shows a shelf. Here is the logic, run on made-up four-number vectors instead of a real model's output.

import numpy as np

def unit(v):
    v = np.asarray(v, dtype=float)
    return v / np.linalg.norm(v)          # length 1, so dot = cosine

def check_photo(img_vec, label_vecs, labels, threshold=0.25):
    img = unit(img_vec)
    scores = [float(img @ unit(t)) for t in label_vecs]
    best = int(np.argmax(scores))
    usable = best == 0 and scores[0] > threshold
    return usable, labels[best], round(scores[best], 3)

LABELS = ["a clear photo of a shelf with snack packets",
          "a blurry or dark photo",
          "a photo of a floor, ceiling or person"]

# Made-up 4-number vectors. A real embedding model returns hundreds of numbers.
label_vecs = [[0.9, 0.1, 0.0, 0.2], [0.1, 0.9, 0.1, 0.0], [0.0, 0.2, 0.9, 0.1]]
shelf_photo = [0.8, 0.2, 0.1, 0.3]
dark_photo = [0.2, 0.7, 0.3, 0.0]

print(check_photo(shelf_photo, label_vecs, LABELS))
print(check_photo(dark_photo, label_vecs, LABELS))

The 0.25 threshold is only a placeholder. Each embedding model has its own range of scores, so tune the threshold on a few hundred of your own photos.

Matching has a limit. It scores labels you wrote and does not write sentences. It can say a photo looks like a snack shelf, but it cannot read the price on a tag. For that you need the generating kind of VLM.

It turns the patch vectors into visual tokens that a language model can read. The language model reads them along with your question and writes the answer one token at a time.

A token is a small piece of text, often a word or part of one, that a language model reads and writes. A visual token fills the same kind of slot, but it carries a piece of the image instead of a word.

The part that makes visual tokens is the projector, the bridge between seeing and speaking. A 2023 paper on visual instruction tuning used a single trainable linear layer for it, one learned multiply-and-add step. It maps each image vector into the same space as the language model's word vectors.

Here is the full path for one audit question. Follow both inputs into the language model, where the photo and the question meet for the first time.

That paper trained its model in two stages. First, the image encoder and the language model were frozen, meaning their weights, the learned numbers inside, did not change. Only the projector learned, from about 595,000 image and caption pairs.

That first stage teaches the bridge to speak the language model's dialect. In the second stage, the encoder stayed frozen while the projector and the language model trained together on instruction examples: an image, a request and a good reply.

The authors made 158,000 such examples cheaply. A text-only language model wrote conversations, detailed descriptions and reasoning questions about each image, working only from its captions and object boxes.

Designs differ in the details. A 2023 follow-up by the same lead authors swapped the single layer for a small two-layer network, which improved results. It also raised the input image to 336 by 336 pixels so the model could see more detail.

Other designs use a resampler, a part that cuts many patch vectors down to fewer visual tokens. For you as a user, the contract stays the same: images and text in, text out.

They are good at open questions about messy, real photos. They are weaker at exact counts, small text and saying confidently that something is absent.

The team measured this before trusting anything. They hand-labelled 500 shelf photos with the right answers, asked a VLM the four questions about each photo and compared.

Counting is the weakest row. The 2021 matching paper also lists counting as a weak spot. A likely reason, though not a measured one: the design has no separate counting step, so the number comes out as text, token by token.

So the team counts with an object detector, a model trained to draw a box around each item of a known kind. It is trained on the brand's own packets, and plain code counts the boxes.

Small text is the other weak row. The 2023 paper notes that its model struggled to name a product's brand, and says such detail needs high-resolution images. An engineer section below shows how little of a phone photo survives resizing.

The pattern generalises. Use the VLM for questions that need judgement and broad knowledge, and a narrow tool where an answer must be exact:

• Open questions, such as whether the shelf looks cluttered, suit a VLM.
• Exact counts suit an object detector plus code that counts its boxes.
• Long printed text suits a dedicated text-reading step, with the VLM kept for messy cases.
• Facts already in your data, such as the shop's address, belong in code, never in the model's answer.

They lean on what usually appears together. A 2023 study found that models are more likely to invent objects that are common in their training instructions, or that often appear next to the real objects. This is called object hallucination.

That study tested several such models and found that most of them suffered from severe object hallucination.

Birdwatchers know the trap. In winter at Bharatpur, a beginner who glimpses a large white wader may log a rare visitor, because that is what people expect to see there.

The audit team hit it in tea-stall photos. The rival's packets sit beside theirs so often that the model sometimes reports the rival on shelves where it is absent. One shop was flagged for a rival it never stocked.

The study also proposed a test called POPE, short for polling-based object probing evaluation. It turns hallucination checking into yes or no questions: is a given object in the image?

Half the questions name objects that are present and half name objects that are absent. Absent objects are picked three ways: at random, from the most common objects overall, and from objects that often appear with the real ones.

It scores 50%, because half the right answers are yes. That is why the study also reports the yes ratio, the share of questions a model answers yes. It found three of the tested models extremely prone to answer yes, at 95% to almost 100% of questions in the random set.

For most models, accuracy also fell from the random set to the common-object set to the appear-together set. That last set is the hardest test, and it matches the rival-brand problem exactly.

The team copies the idea. The probe below takes any ask function, so you can test the probe itself with a fake model that always says yes.

def yes_no_probe(ask, photo, present, absent):
    """Ask about items we know are, or are not, on this shelf."""
    results = []
    for item in present + absent:
        q = f"Is there any {item} in this photo? Answer yes or no."
        said_yes = ask(photo, q).strip().lower().startswith("yes")
        results.append((item, said_yes, item in present))
    accuracy = sum(said == truth for _, said, truth in results) / len(results)
    yes_ratio = sum(said for _, said, _ in results) / len(results)
    invented = [item for item, said, truth in results if said and not truth]
    return accuracy, yes_ratio, invented

def always_yes(photo, question):          # a fake model for testing the probe
    return "Yes."

print(yes_no_probe(always_yes, "shelf_0412.jpg",
                   present=["orange chips packet", "biscuit packet"],
                   absent=["green rival chips packet", "soap box"]))

On the real model, fill the absent list with items that often sit beside your brand, such as the rival's packets. Track accuracy and the yes ratio together, before and after every change.

In the 2023 recipe, one for each patch the encoder keeps. Resampler designs use fewer, and tiled designs use more. The 2020 paper gives the patch count as image height times width, divided by the patch size squared, so the cost grows with resolution.

At 224 by 224 pixels with 16-pixel patches, that is 196. The 2023 follow-up used a 336 by 336 input with 14-pixel patches, which is 24 per side, or 576 visual tokens.

Those tokens really sit in the prompt. We tested an open-source Python library for running models like these, in its latest release as of 19 October 2026. Its processor is the code that prepares images and text before the model sees them. For that recipe, the processor repeats an image placeholder token once per visual token.

We ran that processor offline on a 4,032 by 3,024 phone photo. A toy tokenizer split the prompt, a five-word question plus two chat markers that label who is speaking, into seven text tokens. The prompt came out 583 tokens long, and 576 of them were image placeholders.

If your model service bills by tokens, the image outweighs the text in that run about 80 to 1. A short question is cheap; the photo is not.

Two more details sit in that library's code. By default the model reads the encoder's second-to-last layer, not the last, and drops the class token. The 2023 paper reports that features from before the last layer scored slightly better on its science-question test.

Bigger images need care too. The 2020 paper keeps the patch size fixed, so a bigger image gives a longer sequence. The learned position embeddings are then stretched to fit by 2D interpolation, which estimates in-between values from the learned grid.

Some libraries make you ask for that stretch. In the library we tested, the image encoder used by this recipe is written to raise an error for any other input size, unless you turn on its position-interpolation setting.

A small and sometimes cropped copy. The photo is resized to the encoder's input size before it is cut into patches, so fine detail shrinks and, with some settings, the edges are cut off.

A 12-megapixel phone photo is about 4,032 by 3,024 pixels. Squeezing its short side to 336 pixels shrinks everything nine times in each direction.

In the library we tested, the image processor class for this recipe defaults to resizing the short side, then cutting out a centre square. We set the size to 336, coloured the outer eighth of each side of a 4:3 photo red and ran it.

With centre cropping on, no red pixels survived: a quarter of the photo's width was gone. With the library's pad-to-square option on instead, the red edges stayed, as 84 pixel columns in all, 42 per side.

On a shelf photo, those lost edges can hold the rival's packets. Published models ship their own processor settings, so print your processor's centre-crop and pad-to-square settings before trusting any answer.

The same arithmetic shows why price tags vanish. You can run it with plain Python:

def visual_tokens(height, width, patch):
    return (height // patch) * (width // patch)

print(visual_tokens(224, 224, 16))    # the 2020 paper's setup
print(visual_tokens(336, 336, 14))    # the 2023 follow-up's input

photo_w, photo_h, side = 4032, 3024, 336    # a 12-megapixel 4:3 phone photo
scale = side / photo_h                      # short side squeezed to 336 pixels
kept = side / (photo_w * scale)             # share of the width a centre square keeps
tag = 180 * scale                           # a price tag 180 pixels wide on the phone
print(round(scale, 3), round(kept, 2), round(tag, 1), round(tag / 14, 1))

Higher resolution helps. The 2023 follow-up also built a high-resolution variant. It splits the image into tiles the size the encoder was trained on, encodes each one and adds a shrunken copy of the whole image for context.

The authors report that hallucination dropped when they scaled the input to higher resolutions. Each tile adds its own visual tokens, though, so the team sends one small full image and a sharp crop of the price tag only.

You wrap the model in ordinary engineering. Check photo quality first, ask narrow questions with a fixed answer format, validate replies in code and send doubtful cases to people.

The diagram shows the parts. Notice that the model never writes to the results database directly. Everything passes through the audit worker, which checks it first.

A message queue is a waiting line for work. Most reps upload in the evening, so the queue holds photos until workers are free. Nothing is lost if the model service is slow for a while.

The worker asks the VLM for structured output, a reply in a fixed JSON shape with named fields. Each photo gets the same fields: brand_present, price_tag_text, rival_in_slot and a confidence of high, medium or low.

Then plain code validates every reply. The JSON must parse, every field must exist, and the price must match a pattern like Rs 20. A reply that fails goes to the review queue, not the database.

That works out to about Rs 0.40 per photo (made up), or Rs 1,800 a day for 4,500 photos. The audit team now reviews about 400 flagged photos a day (also made up) instead of 2,500 unsorted ones.

Shelf photos also catch people. The team blurs faces before storage, and the app reminds reps to photograph shelves, not people.

Explain the three parts in one breath, then spend most of your time on the system around the model: inputs, checks, cost and the human path.

1. 1

   Clarify the questions

   Ask what the images are and which decisions depend on the answers. Shelf audits that affect payments need far more checking than photo captions for search.

2. 2

   Name the three parts

   Image encoder, projector, language model. Mention that encoders are often trained on image-caption pairs first, and that matching models differ from generating ones.

3. 3

   Pick the right tool per question

   Use the VLM for judgement questions. Use a detector for exact counts and code for facts already in your data.

4. 4

   Do the token maths

   A 336 by 336 input with 14-pixel patches is 576 visual tokens. Propose a small full image plus sharp crops, and check the resize does not crop the edges.

5. 5

   Guard against hallucination

   Ask for structured output, validate it in code, and run yes or no probes with absent objects that often appear nearby. Track the yes ratio.

6. 6

   Close the loop

   Send low-confidence cases to people, keep a labelled test set, and rerun it before every change.