Design an Image Captioning System for Product Alt Text

Shoppers who use screen readers hear a file name, a vague label or nothing at all, so they cannot judge the product. Image captioning system design fixes this at scale: a pipeline that writes, checks and publishes alt text for every photo.

Picture a common setup. An online marketplace sells handloom rugs, cushion covers, bedsheets and curtains from about 3,500 sellers. It holds 2 million product photos across 4 lakh (400,000) listings. The setup is made up, but the problem is real.

The site's accessibility lead spends a week shopping with a screen reader and the screen switched off. Here is what that week sounds like.

• Photos with no alt attribute were read out as a file name, like IMG_2041 dot jpg. MDN's page on the img element warns that some screen readers do this.
• Some photos said only product image, five times in a row, for five different shots.
• Some sellers had pasted the whole product title into every photo, so the same long title played again and again.
• A size chart was a picture with no text version, so the sizes were never read out at all.

In a sample of 1,000 photos, the lead finds 62 percent with empty or useless alt text (made up). Writing good alt text by hand takes about a minute per photo. For 2 million photos, that is over 33,000 hours of work.

So the team asks for a system that writes alt text for the whole catalogue, keeps it correct and keeps up with new uploads. This guide follows that design, step by step.

Alt text is a short written stand-in for an image, stored as the image's alt attribute in the web page's code. Screen readers speak it aloud, and browsers show it when the image fails to load.

A screen reader is software that reads the screen aloud for people who are blind or have low vision. When it reaches a photo, it reads that photo's alt text. Search engines also use alt text when they judge what a page is about.

The rule behind this comes from WCAG, the Web Content Accessibility Guidelines from the W3C, one of the main web standards bodies. Success criterion 1.1.1, at the basic Level A, asks for a text alternative that serves the equivalent purpose. For a shop, a photo's purpose is to help someone decide whether to buy.

You will meet four more words in this guide. Each one gets a plain meaning here.

Caption. In this guide, a caption is the text the system writes about one photo. It becomes the photo's alt text after it passes the checks.

Captioning model. A captioning model is an AI model that takes an image, and often an instruction, and writes text about it. This guide assumes a vision-language model, one that reads both pictures and words.

Decorative image. A decorative image adds looks but no information, like a background pattern behind a sale banner. W3C's tutorial says it gets empty alt text, so screen readers skip it.

Listing facts. These are the details a seller types when listing a product: title, colour, material, size and price. They live in the catalogue database.

Image captioning system design is planning everything around the model that turns photos into text. You collect each new photo, skip photos that need no caption, write one with listing facts, check it, then publish it or send it to a person.

The model is only one box in that pipeline, a chain of steps where each step feeds the next. Most of the design sits around the model, because a model can be wrong in confident ways.

Here is the architecture. Start at the seller on the left and follow one photo. Notice that the listing database feeds the caption worker, and that every caption passes through the checks box.

A message queue is a waiting line that holds each job until a worker, a program that does one job at a time, is free. So a seller who uploads 300 photos at once does not slow the website down. The jobs wait their turn.

The alt text store keeps every version of every caption. If a new model writes worse captions, the team switches back to the old text without calling the model again.

Good product alt text says, in one short sentence, what this particular photo shows that helps a shopper decide. It is not a literal list of everything in the frame.

W3C's informative images tutorial says the text should convey the meaning the image shows, which is typically not a literal description. Its alt decision tree adds that an image repeating real text nearby gets empty alt text.

On a product page, the title already sits next to the gallery. So repeating the title in every photo wastes the shopper's time. Each photo should add what only that shot shows: the weave up close, the back side, or the cushion on a sofa for scale.

The team writes a short house style for every caption. Each rule becomes either an instruction for the model or a check in code.

• Start with the thing itself. WebAIM, a web accessibility group, advises against openings like image of, because screen readers already announce that it is a graphic.
• Say what this shot adds: the angle, a close-up detail, or the product in a room.
• Use colour, material and pattern words only when the listing confirms them.
• Keep it short. WCAG sets no character limit, so the team picks a house limit of 150 characters, about one sentence.
• No sales words like best quality or amazing, because they describe nothing.
• If a photo is mostly text, like a size chart, put that text on the page as real text.

Check one of your own pages in a minute

This check is optional, and nothing later depends on it. Open any product page, then open the browser's developer tools and their Console tab. Type or paste these three lines one at a time, pressing Enter after each. Some browsers block pasting into the Console until you type allow pasting. That warning protects you, so paste only code you understand.

document.querySelectorAll('img:not([alt])').length
document.querySelectorAll('img[alt=""]').length
Array.from(document.querySelectorAll('img[alt]'), function (i) { return i.alt; }).filter(Boolean).join(' | ')

The first line counts img elements, the page's image tags, with no alt attribute at all. The second counts images marked as decorative with empty alt text. The third prints every non-empty alt text, so you can read them as a screen reader would speak them.

On a five-image test page we got 1, then 1, then IMG_2041.jpg | Indigo cotton durrie | product image. Two of those three alt texts would fail the house style.

An image captioning model first turns the photo into a list of numbers that describe what it sees. Then it writes text one word at a time, guided by those numbers.

The first half is often called the encoder. It looks at the pixels and produces numbers for shapes, colours and objects. The second half, the decoder, writes words. Each new word depends on the image numbers and the words already written.

This idea is older than today's chat apps. A paper by Vinyals and colleagues, first posted in November 2014, joined a vision network with a recurrent network, borrowing recent advances from machine translation. A recurrent network writes a sentence one word at a time, carrying a memory of the words so far.

That paper reported results on Pascal, Flickr30k, SBU and COCO. These are public sets of photos with human-written captions, used to train and compare models.

One common design for today's vision-language models keeps the same broad plan at a much larger size. The 2023 Visual Instruction Tuning paper, for example, connects a vision encoder to a large language model, a text model trained on a huge amount of writing. Such models also take an instruction with the photo, often called a prompt, which is the main way the team steers style.

Think of a plant nursery owner who takes orders over the phone. A customer cannot see the saplings, so the owner describes them. She says what she sees: three feet tall, glossy leaves, two new shoots.

For the variety name, she does not trust her eyes. Two kinds of hibiscus can look the same as saplings. So she turns the pot over and reads the tag. In the pipeline, the tag is the listing, and reading it is the check.

Where the picture breaks: the owner chooses when to read the tag. The model does not get that choice. Every request carries the facts, and code checks every caption.

Here is the catch. A model picks each next word because it is likely, given the photo and the words so far, not because anyone checked it. Shown a soft, fluffy throw, it may write wool when the seller listed acrylic.

You stop invented details by giving the model the seller's listing facts and telling it to use only those. Then you check its caption against the same facts in code.

The model describes what it sees, like the angle, the pattern and the setting. The listing supplies what eyes cannot confirm, like the fibre and the exact size.

Below is how the caption worker builds one request. The listing arrives as a dictionary, a set of named values, from the listing database. The shot type is a label the seller picks when uploading, such as front, close-up or in-room.

def build_request(photo, listing):
    facts = {
        "colour": listing.get("colour"),
        "material": listing.get("material"),
        "size": listing.get("size"),
    }
    known = {k: v for k, v in facts.items() if v}
    instruction = (
        f"Write alt text for this {photo['shot_type']} product photo "
        "in one sentence, under 150 characters. Say what this shot adds. "
        f"For colour, material and size, use only these facts: {known}. "
        "Do not repeat the product title. Do not start with 'image of'."
    )
    return {"image": photo["bytes"], "instruction": instruction}

photo = {"bytes": b"...", "shot_type": "close-up"}
listing = {"colour": "indigo and white", "material": "", "size": "5 x 7 ft"}
print(build_request(photo, listing)["instruction"])

Running it prints one instruction. Its facts part reads {'colour': 'indigo and white', 'size': '5 x 7 ft'}, with no material, because the listing left material empty. The real request shape depends on your model provider's API.

What just happened? The known line dropped the empty material, so the instruction never shows a blank. The shot type went into the instruction itself, so a close-up gets a close-up caption.

Instructions help, but a model can still ignore them or name a fibre on its own. So every caption also goes through a check in plain code. We ran this one on Python 3.13.16.

import re

# Every spelling sellers or the model use, mapped to one material.
MATERIAL_WORDS = {
    "cotton": "cotton", "wool": "wool", "woollen": "wool", "woolen": "wool",
    "jute": "jute", "silk": "silk", "acrylic": "acrylic",
    "velvet": "velvet", "polyester": "polyester",
}
COLOUR_WORDS = {
    "red": "red", "maroon": "maroon", "indigo": "indigo", "white": "white",
    "cream": "cream", "mustard": "mustard", "grey": "grey", "gray": "grey",
    "blue": "blue", "green": "green", "black": "black",
}
BANNED_START = re.compile(r"(an? |the )?(image|photo|picture|graphic) of\b")

def words(text):
    return set(re.findall(r"[a-z]+", (text or "").lower()))

def check_caption(text, listing):
    problems = []
    lower = text.strip().lower()
    if not lower or len(lower) > 150:
        problems.append("length")
    if BANNED_START.match(lower):
        problems.append("starts with image of")
    said = {MATERIAL_WORDS[w] for w in words(lower) if w in MATERIAL_WORDS}
    listed = {MATERIAL_WORDS[w] for w in words(listing.get("material")) if w in MATERIAL_WORDS}
    if said - listed:
        problems.append(f"material not in listing: {sorted(said - listed)}")
    said = {COLOUR_WORDS[w] for w in words(lower) if w in COLOUR_WORDS}
    listed = {COLOUR_WORDS[w] for w in words(listing.get("colour")) if w in COLOUR_WORDS}
    wrong_colours = said - listed
    if wrong_colours:
        problems.append(f"colour not in listing: {sorted(wrong_colours)}")
    return problems

throw = {"material": "Acrylic", "colour": "Cream"}
print(check_caption("Soft woollen throw folded on a bed", throw))
print(check_caption("An image of a cream throw on a bed", throw))
print(check_caption("Cream acrylic throw folded at the foot of a bed", throw))
cushion = {"material": "Velvet", "colour": "Mustard"}
print(check_caption("Mustard cushion cover on a grey sofa", cushion))

It prints four results, one per caption. The first says material not in listing: ['wool']. The second says starts with image of. The third is an empty list. The fourth says colour not in listing: ['grey'].

What just happened? Woollen maps to wool, so the first caption fails against an acrylic listing. The second opens with an image of, which screen readers make redundant. The third is clean, so it can be published. Colours use the same kind of map, so grey and gray count as one colour.

The fourth is a false alarm worth knowing. The grey belongs to the sofa, not the cushion, but the check cannot tell. So in-room shots reach the review queue more often, which costs a person a minute instead of costing a shopper a return.

A listing with no material allows no material words, and a blank colour allows no colour words. So if the model writes cotton for a blank listing, the caption goes to review instead of the website.

You caption a large backlog with a one-time batch job that skips repeats and text-heavy photos. It runs on a fixed number of workers and starts with the most viewed products.

The marketplace has two kinds of work. The backlog is the 2 million old photos, done once. New uploads arrive every day, about 6,000 photos (made up), and should get alt text within an hour.

Each photo goes through a few quick decisions before any model is called. Follow the question boxes in the diagram below. Only photos that reach the caption box cost model time.

The fingerprint here is an exact file hash, a short code computed from the file's bytes with a method such as SHA-256. Two byte-for-byte copies give the same code. In the setup, about 15 percent of the backlog is exact repeats (made up).

The facts matter as much as the photo. In the setup, sellers often reuse one photo for every colour of a product. Reusing the indigo listing's caption on the red listing would read the wrong colour aloud, so reuse needs both to match.

Now the rough maths, with made-up figures. Skip about 300,000 exact repeats and about 100,000 charts and text-heavy photos. That leaves about 1.6 million photos that need a caption. The team runs 8 GPU workers, each finishing about 5 photos a second. A GPU is a chip built to do many small sums at once, which models need.

• 8 workers times 5 photos a second is 40 photos a second.
• 1.6 million photos divided by 40 is 40,000 seconds, a little over 11 hours.
• At Rs 0.08 per photo (made up), the backlog costs about Rs 1.28 lakh (128,000 rupees) in model time.
• If 8 percent fail the checks, reviewers face about 128,000 captions, which is the real bottleneck. It matches the rupee figure only because both multiply by 0.08.

That last line changes the plan. At a minute per caption, 128,000 reviews take about 2,100 hours of people's time, against about 11 hours of GPU time. So the team runs the backlog in order of page views, and the top 50,000 products get reviewed first.

New uploads take a separate, higher-priority queue, so a seller's new photo never waits behind the backlog. At 5 photos a second, one reserved worker clears 6,000 photos in about 20 minutes of work a day.

You measure caption quality with three layers: automatic checks on every caption, a fixed test set of photos with reviewed captions, and regular human review that includes screen reader users.

Research has automatic scores for captions. CIDEr, from a paper first posted in November 2014, compares a caption with several human-written captions of the same image. It rewards word runs that many of those people used, a measure of consensus.

Scores like this help compare two versions on the same photos. They do not check whether a caption is true to the listing. The engineer section shows a wrong-material caption beating a correct one.

So the team keeps a test set of 2,000 photos (made up). The review team wrote and agreed on the alt text for each one. Every new prompt or model must run on this set before release.

1. 1

   Run the candidate on the test set

   Caption all 2,000 photos with the new prompt or model. Run the same code checks used in production.

2. 2

   Count hard failures

   Count captions with a wrong material, a wrong colour or a banned opening. Any rise over the current version blocks release.

3. 3

   Compare with the reviewed captions

   Use an automatic score against the reviewed captions to spot big shifts in wording, not as the final verdict.

4. 4

   Blind human review

   Reviewers see 200 pairs of captions, old and new, without knowing which is which, and pick the more useful one.

5. 5

   Release to a slice

   Send 5 percent of new uploads to the new version for a week, and watch review rates and shopper reports.

The test set must grow from real mistakes. The next diagram shows how a reported caption becomes a test case. Follow it from the shopper's report to the release gate.

AI-generated alt text can go wrong by inventing facts, missing the point of the photo, guessing about people or going stale after edits. It can also obey text planted in a photo, or fail quietly at scale.

• Invented facts. The model says silk for a polyester curtain. Guard: listing facts in the request and the material check in code.
• Wrong colour from lighting. A warm photo makes white look cream. Guard: colour words must match the listing, or the caption goes to review.
• Missing the point. A close-up shot gets a caption about the whole bedsheet. Guard: put the shot type into the instruction.
• Guessing about people. A room photo has a person on the sofa. Guard: describe the product and setting, never the person's age, caste, religion or body.
• Stale captions. A seller swaps the photo but the old caption stays. Guard: any photo or listing change creates a new caption job.
• Planted text in photos. A photo contains written words aimed at the model, which OWASP's prompt injection page lists as a risk. Guard: give the caption worker no power except returning text.

That last guard limits the damage but does not remove it. The worst case becomes a misleading or offensive caption, and the code checks only catch the words they look for. So add one more check, not in the sample code above, that sends captions mentioning instructions, links or prices to review.

Language is another choice. The site starts with English alt text because its pages are in English. When Hindi pages arrive, add Hindi photos and reviewed Hindi captions to the test set before captioning in Hindi.

A word list catches only the exact spellings on it. An earlier version of this guide, published on 16 October 2026, used a check that matched wool but not woollen, the British and Indian spelling.

It also caught image of only at the very start, so An image of and A photo of passed. It had no colour check at all. We ran both versions on the same captions with Python 3.13.16 on 19 October 2026.

The last row is the lesson. Silky is on neither list, so both versions pass it, yet it hints at silk on an acrylic throw. A word list catches the words you thought of, nothing more.

Two habits help. Add every spelling that reviewers correct to the word lists, and keep the test set's hard-failure count as the release gate. A rise there means the lists or the prompt need work.

Not on its own. A perceptual hash is a fingerprint built to stay nearly the same when a photo is resized or re-saved. That is exactly why it can match two colours of one product.

An exact file hash changes completely when a single byte changes. A JPEG, the usual photo file format, re-saved at a different quality shows the same picture but gets a new exact hash. A perceptual hash avoids that by looking at the picture's broad shapes, not its bytes.

The ImageHash library's page says its average, perceptual, difference and wavelet hashes look at brightness without colour information. It also has a separate colour hash. Its phash function, in version 4.3.2, turns the photo grey and shrinks it to 32 by 32 pixels. It then finds the 64 broadest patterns of light and dark and keeps one bit for each: above or below their middle value. The distance between two hashes is the number of bits that differ.

What just happened? The red version sat closer than a simple re-save. Going grey threw the colour away before hashing, so the hash saw the same zigzags and stripes. Reusing captions on a near match would read indigo aloud for a red rug.

These numbers come from one image drawn in code, not a real photo, so treat them as a demonstration. Your own photos will give different distances, which is why you should test them.

So the team reuses captions automatically only on an exact hash plus matching listing facts. A perceptual match only suggests a draft, and that draft still runs through every check against the new listing's facts.

To try it on two of your own photos, install the pinned versions and run the script below. With our indigo and red test images, it printed same file: False and perceptual distance: 2 of 64 bits.

# pip install Pillow==13.0.0 ImageHash==4.3.2
import hashlib
from PIL import Image
import imagehash

def file_hash(path):
    return hashlib.sha256(open(path, "rb").read()).hexdigest()

a, b = "indigo.jpg", "red.jpg"
print("same file:", file_hash(a) == file_hash(b))
distance = imagehash.phash(Image.open(a)) - imagehash.phash(Image.open(b))
print("perceptual distance:", distance, "of 64 bits")

Because CIDEr scores shared wording, not truth. A caption that copies the reviewers' phrasing but swaps one fact can beat a caption that is correct in new words.

We used pycocoevalcap 1.2, the Python package of the COCO caption metrics. Its CIDEr code counts matching word runs of one to four words. Each run is weighted so that phrases found in many images' references count for less.

A length penalty follows, and the final average is multiplied by 10. Those weights come from the references of the photos being scored, so scores from two different test sets are not comparable.

We wrote six made-up product photos, each with three reviewed captions, and scored candidate captions for the acrylic throw on 19 October 2026. The other five captions stayed the same in every run. Running the package's own tokenizer step first gave the same per-photo scores.

What just happened? The wrong caption differed from the reviewers' wording by one word, so it kept most of its score. The correct caption shared few word runs with the references, so it scored lowest. Across all six photos, two wrong materials moved the average only from 5.940 to 5.120.

With only six photos, the weights are extreme. Words found in every photo's references, like on and a, count for nothing. A real test set has thousands of photos, so measure the gap on yours.

Two rules follow. Use CIDEr to spot big shifts in wording between versions, never as a truth check. Compare scores only on the same test set, and let the hard-failure count decide releases.

Explain it as a pipeline built around one goal, such as useful alt text, and show how each part protects that goal from a specific failure.

1. 1

   Pin down the goal

   Ask who reads the captions. Alt text for screen readers, search tags and social posts need very different styles and checks.

2. 2

   Size the work

   Split it into a one-time backlog and a daily flow of new photos. Do the throughput and cost maths out loud.

3. 3

   Draw the pipeline

   Upload, photo store, queue, prep worker, caption worker, checks, review queue and alt text store. Keep the model as one box.

4. 4

   Ground the facts

   Explain that colour, material and size come from listing data, that the request carries them, and that code checks every caption against them.

5. 5

   Plan for doubt

   Failed or unsure captions go to people. Rank that review work by page views, and reuse a caption only when the photo and the facts both match.

6. 6

   Measure and release safely

   Describe the test set, the hard-failure gate, blind human review and a small first rollout.