Text-to-Image System Design: GPU Queues, Filters and Storage

It falls over on its first busy day. Each picture takes seconds of expensive GPU time, and a design built for quick web requests has nowhere to put that wait. Text to image system design is about building that place.

Picture a common setup. A small wedding-card service in Jodhpur lets families type a prompt, such as royal blue wedding card with peacocks and Mehrangarh fort, and draws artwork for the card. The setup is made up, but the problem is real.

The first version was simple. The web server took the prompt, called the model on one of eight rented GPU machines, and waited for the picture. Each image took about 5 seconds (made up).

Then the wedding season started. One November evening, 9,000 requests arrived in a single hour. Three things broke that week.

• Requests piled up behind the busy GPUs and hit the 60-second timeout. Users saw an error, pressed Generate again, and doubled the load.
• One user asked for a cartoon of a real politician in an insulting pose. Nothing checked the prompt or the picture, so it went out and screenshots spread.
• Finished images sat on each GPU machine's own disk. When a broken machine was replaced, 2,300 families lost artwork they had paid for.

The model was fine. The system around it was the problem. This guide rebuilds that system one part at a time, using the same card service all the way through.

You need five: GPU, diffusion model, job queue, safety filter and object storage. Each gets a plain meaning here and is used the same way for the rest of the guide.

GPU. A GPU (graphics processing unit) is a chip that does thousands of small sums at once. Image models need it to finish in seconds instead of minutes. A machine that runs the model on a GPU is a GPU worker.

Diffusion model. This is the kind of model behind many image generators. It starts from random noise and removes a little noise in each step, guided by your prompt, until a picture appears.

Job queue. A job queue is a waiting line of work kept outside your web server. One part of the system adds jobs, and workers take them off when they are free. Each card request becomes one job.

Safety filter. A safety filter is a check that blocks harmful requests or harmful results. It can be a word list, a small classifier (a model that sorts inputs into labels), or a person reviewing a case.

Object storage. Object storage keeps files, called objects, under a name, called a key. The service copies them across machines so they survive hardware failures. Amazon S3 is a well-known example.

Text to image system design is planning the whole service that turns a typed prompt into a stored, safe picture. The model is one box. The queue, workers, filters, storage and delivery are the rest.

Most hard questions are not about the model. They ask how many images an hour you can make, what survives a dead GPU machine, who refuses a rude prompt, and where each file lives.

Here is the design the card service moved to. Follow the arrows from the app to the GPU workers, then on to storage.

A CDN (content delivery network) is a set of servers spread across many cities. It keeps copies of files close to users, so a family in Pune does not wait on a server far away.

The model turns your prompt into numbers, then removes noise from a random starting image over many steps, using those numbers as a guide. A decoder then turns the result into normal pixels.

You do not need the maths to design the system. You need to know which settings change the cost of one job. Our guide on how diffusion models work covers the inside of the model.

Many modern models do this work on a small compressed version of the image, called a latent. That is the idea behind latent diffusion, a method first posted in December 2021.

Its authors run diffusion inside the compressed space of a pretrained autoencoder, a network that shrinks images and rebuilds them. They report much lower compute than diffusion on raw pixels, while keeping detail.

Classifier-free guidance, from a paper first posted in July 2022, adds one more cost. It trains a model with and without the prompt, then blends the two predictions to trade variety against quality.

The table below reads these costs straight from the code of diffusers 0.41.0, an open-source library, in its standard text-to-image pipeline. Notice which rows run once and which run on every step.

Two details catch experts. The scheduler, the rule that decides how much noise each step removes, changes the count: the Heun scheduler calls the network 59 times for 30 steps, not 30. The same pipeline also skips the doubled batch for networks trained to take the guidance strength as an input.

Size grows faster than it looks. With a shrink factor of 8 per side, the value the diffusers code falls back to, 512 pixels become a 64 by 64 latent of 4,096 positions. At 1024 pixels it is 16,384, and attention, which compares every position with every other, has 16 times the pairs.

Image generation should use a job queue because each image takes seconds of scarce GPU time. A queue lets the API answer at once, absorbs sudden spikes, and lets workers take jobs only when they are free.

Picture a caricature artist at a fair. She draws one face in about ten minutes. Her helper does not let twenty people crowd her easel. He writes each name on a list, hands over a numbered slip, and says come back later.

Map it part by part. The helper is the API server. The list is the job queue. The numbered slip is the job id. The artist is a GPU worker, and her steady pace is the GPU's fixed speed.

Where the picture breaks: a real queue can hand the same job to two workers, because a worker that goes quiet loses its claim. The helper's list never does that. The engineer section shows why that matters.

In the new design, the API does three quick things. It checks the prompt, saves a row in the jobs table, and puts the job on the queue. Then it replies with the status code 202 Accepted, the number a server sends to say what happened, plus a job id.

MDN, Mozilla's web developer reference, says 202 means a request was accepted for processing that may not have finished or even started. It is typically used when another process handles the request, which is exactly this case.

The app then waits in one of two ways. It can poll, which means asking a status address every few seconds whether the job is done. Or the server can push a message when the job finishes.

The queue also fixes the double-press problem. The app sends an idempotency key, a unique id for one user action, with each request. If the same key arrives twice, the API returns the first job instead of making a second. Store the key in a unique column, so if two copies arrive together, one insert wins and the other reads its job.

• Priorities. Paid Final card jobs go on a faster queue than free drafts, so a paying family never waits behind a thousand experiments.
• Limits per user. Each account may have at most three jobs waiting, so one script cannot fill the queue.
• A length limit. When the queue would take over ten minutes to clear, the API says busy, try again soon, instead of accepting work it cannot finish in time.

You scale GPU workers in three ways. Keep the model loaded in memory, run jobs with matching settings together as one batch, and add or remove workers based on the waiting work.

Loading a large model onto a GPU is slow next to making one picture. In the card service, a new worker takes about 3 minutes to start, fetch its software and load the model (made up). So each worker loads once and keeps the model.

Batching means feeding several prompts through the model in one pass. One small job can leave part of the GPU idle, so a batch can take well under four times as long. In the card service, one image takes 5 seconds and a batch of four takes 12 (made up). Measure that gain on your own GPU and preset, because it varies.

Alone, one GPU makes 3,600 divided by 5, which is 720 images an hour. In batches of four it makes 300 batches, which is 1,200 images. So 9,000 an hour needs 13 GPUs without batching but only 8 with it.

What just happened? Each family in a batch now waits 12 seconds instead of 5, but every GPU finishes about 1.67 times as many images an hour. Batching trades a little waiting for a lot of throughput.

Batching only works when the jobs in a batch match. They must share image size, step count, guidance setting and model. That is the second reason for fixed presets.

The money follows the same sums. Say a rented GPU costs Rs 180 an hour (made up). Eight of them running all month cost about Rs 10.4 lakh (1.04 million rupees), so every idle minute shows up on the bill.

Scale on the waiting work, not on CPU use. The heavy work runs on the GPU, so the machine's CPU numbers say little about how busy a worker really is.

The AWS (Amazon Web Services) Auto Scaling guide gives a formula for queue-based scaling. Divide the waiting jobs by the running workers to get backlog per worker. The target is the wait you accept divided by the time one job takes.

The card service accepts a 2-minute wait, and one worker clears an image every 3 seconds in batches. That gives a target of 40 waiting jobs per worker, so a queue of 300 jobs calls for 8 workers.

The same guide warns that scaling on queue length alone ignores how many workers you already have, so it can add too many while the queue drains.

Safety filters work in two layers. A prompt check runs before generation and cheaply refuses clearly harmful requests. An image check runs after generation and blocks harmful pictures the prompt check missed.

The prompt check runs three tests, from cheapest to most expensive. A blocked word list catches obvious terms. A list of real public figures catches requests like the politician cartoon. A small classifier gives a risk score from 0 to 1.

The classifier catches what lists miss, such as misspellings and polite wording for ugly ideas. A high score refuses. A middle score sends the prompt to a person instead of drawing or refusing by default.

For a middle score, the API still saves the row as in review and replies 202. A reviewer's yes puts the job on the queue; a no marks it refused.

The image check uses an image classifier trained to spot nudity, violence and hateful symbols. The diffusers standard pipeline shows a small version: after decoding, it runs a safety checker and swaps a flagged image for a black one.

Scope that example carefully. Its warning message names only NSFW (not safe for work) content, so do not count on it for violence or hateful symbols. Loading the pipeline with the checker set to None turns it off, with only a logged warning.

Many platforms also match images against shared hash lists. A hash is a short fingerprint of a file; a perceptual hash stays similar after small edits. NIST, the US national standards agency, describes such databases for spotting known harmful images in its 2024 report.

Because the lists hold images someone has already found, they catch re-uploads, and source photos if your service accepts uploads. They will not flag a brand-new picture the model drew a second ago, so the classifier still has to do that work.

The stakes are real. NIST AI 100-4, published on 20 November 2024, lists preventing generated child sexual abuse material and non-consensual intimate images of real people among its core topics.

Treat those categories as a hard block, never a review band. The same report notes that producing, possessing or accessing such material is generally illegal, so those cases follow a process your legal team sets, not the ordinary review queue.

Store generated images in object storage, with a database row that records who owns each one. Never keep them on the GPU machine's disk, because that machine can be replaced at any time.

The card service follows three habits. Each image gets a key built from the user id and the job id, such as cards, then u7, then j1.png. The jobs table records the owner. Files leave storage only through signed links.

A signed link is a web address that carries a time-limited permission. Amazon's S3 guide says all objects are private by default. Its presigned URL, S3's name for a signed link, uses the credentials of whoever created it.

The same guide sets limits, as of 19 October 2026. A link made in the S3 console lasts at most 12 hours. One made with the command-line tool can last up to 7 days.

There is a catch. A link made with temporary credentials stops working when those credentials expire, even if you asked for longer. For an EC2 instance profile, the role credentials AWS gives a virtual machine, that is typically about 6 hours, AWS says.

A CDN changes the picture. An S3 presigned link goes straight to S3 and skips the CDN. CloudFront, Amazon's CDN, has its own signed URLs and signed cookies, and AWS recommends making users go through the CDN so nobody bypasses those rules.

The card service keeps its bucket private, lets only the CDN read it, and gives the app CDN links that expire after one hour. The app asks for a fresh link each time it shows a card.

• Thumbnails. A worker makes a small preview of each image, so the gallery loads fast on a phone.
• Retention. Unpicked drafts are deleted after 30 days with a lifecycle rule, a storage rule that acts on files by age. For standard-class files, S3 removes them in the background, possibly after a delay, and stops charging once they expire.
• Deletion. When a family deletes an image, the service deletes the file and invalidates its CDN path, so edge servers stop serving cached copies.
• Separation. Images sent to review sit in a locked area, never in the user's folder.

Marking images as AI-generated

The card service also labels every image as made by a model. It uses C2PA, a standard from the Coalition for Content Provenance and Authenticity. At version 2.4 as of 19 October 2026, C2PA defines a manifest: a signed set of facts about where a file came from.

The manifest is bound to the file with cryptographic hashes such as SHA2-256, so any change to the file or the record breaks the match. The C2PA explainer calls this tamper-evident. A field called digitalSourceType marks that an AI system made the file.

The explainer also admits the record can be removed. Its answer is durable credentials: a hash binding plus a soft binding, meaning an invisible watermark or a fingerprint lookup that helps find the record again. Treat both as signals, not proof.

You build it on three ideas: a lease on each job, a retry limit, and writes that are safe to repeat. The queue does the first two, and your storage keys do the third.

A lease means the queue hides a taken job instead of deleting it. In Amazon SQS, AWS's managed queue service, this is the visibility timeout. Its default is 30 seconds, and it cannot exceed 12 hours from the first receive, as of 19 October 2026.

If the worker does not delete the job before the lease ends, the job becomes visible again for another worker. A slow worker can extend its lease with the ChangeMessageVisibility call.

SQS also says it cannot promise a job is delivered only once, even inside the lease. So the same job can run twice, and your worker must be harmless when it does.

The retry limit comes from a dead-letter queue, a side line for jobs that keep failing. In SQS, the setting maxReceiveCount is how many times a job can be received before it moves there.

The worker below is plain Python with in-memory stand-ins for the queue, the model and the image classifier. Run it with python3 to watch a dead machine, a lease running out and a crashing prompt.

LEASE_SECONDS = 60   # must be longer than one batch takes
MAX_TRIES = 3
MAX_BATCH = 4


class Queue:
    """In-memory stand-in for a queue with leases, like SQS visibility timeouts."""

    def __init__(self):
        self.jobs, self.dead = {}, []

    def add(self, job):
        self.jobs[job["id"]] = dict(job, tries=0, visible_at=0)

    def take(self, now, preset):
        ready = [j for j in self.jobs.values() if j["preset"] == preset and j["visible_at"] <= now]
        batch = []
        for job in ready[:MAX_BATCH]:
            job["tries"] += 1
            if job["tries"] > MAX_TRIES:
                self.dead.append(self.jobs.pop(job["id"])["id"])
                continue
            job["visible_at"] = now + LEASE_SECONDS   # hidden from others, not deleted
            batch.append(job)
        return batch

    def finish(self, job):
        self.jobs.pop(job["id"], None)


def generate(prompts):                    # stand-in for the GPU model
    if any("glitch" in p for p in prompts):
        raise RuntimeError("model crashed")
    return [f"IMG({p})" for p in prompts]


def check_image(image):                   # stand-in for the image classifier
    if "gore" in image:
        return "block"
    return "review" if "sword" in image else "allow"


def run_batch(queue, now, preset, store, status):
    batch = queue.take(now, preset)
    live = []
    for job in batch:
        if job["cancelled"]:
            status[job["id"]] = "cancelled"
            queue.finish(job)
        else:
            live.append(job)
    if not live:
        return
    images = generate([j["prompt"] for j in live])
    for job, image in zip(live, images):
        verdict = check_image(image)
        if verdict == "allow":
            key = f"cards/{job['user']}/{job['id']}.png"
        elif verdict == "review":
            key = f"review/{job['id']}.png"    # locked area, never the user's folder
        else:
            key = None                         # hard block: never in the user's folder
        if key:
            store[key] = image                # same job, same key: a rerun overwrites
        status[job["id"]] = verdict
        queue.finish(job)


q, store, status = Queue(), {}, {}
q.add({"id": "j1", "user": "u7", "preset": "final", "prompt": "peacocks", "cancelled": False})
q.add({"id": "j2", "user": "u7", "preset": "final", "prompt": "fort", "cancelled": True})
q.add({"id": "j3", "user": "u9", "preset": "final", "prompt": "groom with sword", "cancelled": False})
q.add({"id": "j5", "user": "u3", "preset": "final", "prompt": "battle gore", "cancelled": False})
q.add({"id": "j4", "user": "u9", "preset": "draft", "prompt": "glitch", "cancelled": False})

q.take(0, "final")                         # worker A takes the batch, then its machine dies
print("t=0   worker A died; jobs still queued:", sorted(q.jobs))
print("t=30  worker B finds:", [j["id"] for j in q.take(30, "final")])
run_batch(q, 61, "final", store, status)   # lease ran out, worker B gets the jobs
print("t=61  status:", status)
print("      stored:", sorted(store))
for t in (0, 61, 122, 183):
    try:
        run_batch(q, t, "draft", store, status)
    except RuntimeError as e:
        print(f"t={t:<4}draft batch failed: {e}")
print("dead-letter queue:", q.dead)

Running it on Python 3.13 prints exactly this:

t=0   worker A died; jobs still queued: ['j1', 'j2', 'j3', 'j4', 'j5']
t=30  worker B finds: []
t=61  status: {'j2': 'cancelled', 'j1': 'allow', 'j3': 'review', 'j5': 'block'}
      stored: ['cards/u7/j1.png', 'review/j3.png']
t=0   draft batch failed: model crashed
t=61  draft batch failed: model crashed
t=122 draft batch failed: model crashed
dead-letter queue: ['j4']

What just happened? Worker A took four final-card jobs and died. At 30 seconds the jobs were still hidden, so worker B found nothing. At 61 seconds the lease had run out, and worker B finished all four.

The cancelled job closed without GPU time. The sword picture went to the locked review area, and the blocked one was never stored. The crashing draft failed three times, and the fourth take moved it to the dead-letter queue.

The storage key is the safety net. A job always writes to the same key, so if two workers ever run it, the second write replaces the first instead of creating a duplicate card. Save the job row first, and mark it sent once the queue accepts it. A sweeper re-queues rows still unsent after a minute (made up), which covers a crash between the two writes.

One trap remains. In this code, one crashing prompt fails every job in its batch. When a batch fails, retry its jobs one at a time, so only the bad prompt ends up in the dead-letter queue.

Where the workers run on Kubernetes

Kubernetes runs containers across a pool of machines, called nodes. Its Schedule GPUs page says GPUs are meant to be set in a container's limits, the most it may use. If you also set requests, the two values must be equal.

Its example asks for one GPU. The same page shows how to label nodes with the GPU type they have, so each worker pool lands only on machines with the right GPU.

Two bugs hide in the obvious version: splitting on spaces and matching names as raw text. The version of this guide published on 16 October 2026 had both. Its check split the lowercased prompt on spaces and tested whether each name appeared in the text.

We ran that old check with the same two lists used below. Both harmful prompts were allowed:

allow (classifier not reached in this test): Nude, couple portrait
allow (classifier not reached in this test): Cartoon of the Chief-Minister as a donkey

The comma glued itself to the blocked word, and the hyphen stopped the name from matching. The version below pulls out words with a regular expression first, then matches whole words and whole names.

import re

BLOCKED_WORDS = {"nude", "gore"}                       # a real list is longer and lives in a file
PUBLIC_FIGURES = {"chief minister", "prime minister"}  # a real list holds names


def words(text):
    return re.findall(r"[a-z0-9]+", text.lower())


def prompt_risk_score(prompt):
    """Stand-in for a small classifier: 0.0 safe to 1.0 harmful."""
    return 0.7 if "blood" in words(prompt) else 0.1


def check_prompt(prompt):
    w = words(prompt)
    if BLOCKED_WORDS & set(w):
        return "refuse"
    joined = " " + " ".join(w) + " "
    if any(" " + name + " " in joined for name in PUBLIC_FIGURES):
        return "refuse"
    score = prompt_risk_score(prompt)
    if score >= 0.9:
        return "refuse"
    if score >= 0.6:
        return "review"
    return "allow"


for p in ["Royal blue wedding card with peacocks and Mehrangarh fort",
          "Nude, couple portrait",
          "Cartoon of the Chief-Minister as a donkey",
          "Blood red lehenga, gold border, mehendi night"]:
    print(f"{check_prompt(p):7} {p}")

Running it prints:

allow   Royal blue wedding card with peacocks and Mehrangarh fort
refuse  Nude, couple portrait
refuse  Cartoon of the Chief-Minister as a donkey
review  Blood red lehenga, gold border, mehendi night

What just happened? Punctuation no longer hides a blocked word, and the hyphenated title is caught. The last prompt is harmless, yet blood red sent it to review. That is the kind of false alarm a weekly sample should catch and tune away.

Whole-word matching still needs context. A list entry that is also a common word will refuse harmless prompts, so keep name lists to full names and review what they block.

You can check both from a terminal in under a minute. Signing an S3 link happens on your own machine and sends nothing to S3. It even prints a link with made-up credentials, so a link that actually opens needs real ones. We sign in the Mumbai region, ap-south-1.

aws --version
aws s3 presign s3://card-images/cards/u7/j1.png --expires-in 3600 --region ap-south-1

Expect output like this. The X-Amz-Expires value is the lifetime in seconds, so 3600 means one hour:

aws-cli/1.46.1 Python/3.13.16 Linux/6.18.44-fc-v139 botocore/1.43.62
https://card-images.s3.amazonaws.com/cards/u7/j1.png?X-Amz-Algorithm=AWS4-HMAC-SHA256...X-Amz-Expires=3600...X-Amz-Signature=...

The region changes the format. With us-east-1, us-west-2, eu-west-1 or no region set, awscli 1.46.1 printed an older-style link with AWSAccessKeyId, Signature and Expires. That Expires is a clock time in Unix seconds (seconds since 1970), one hour ahead, not a lifetime. In us-east-1, setting s3.signature_version to s3v4 in the AWS config brought X-Amz-Expires back.

One undocumented detail, as of 19 October 2026: for X-Amz links (Signature Version 4), awscli 1.46.1 printed a link with 604801 seconds, one second over 7 days, without any warning. Do not rely on the tool to enforce the documented limit; pin your version.

For labels, c2patool is the open-source command-line tool for C2PA. We ran version 0.28.2 on a 512 by 512 PNG, then re-saved the file with ffmpeg, which copies the pixels but drops the manifest.

c2patool card.png -c '{"title": "card.png"}' --create trainedAlgorithmicMedia -o card_signed.png
c2patool card_signed.png | grep -E 'digitalSourceType|validation_state'
ffmpeg -loglevel error -i card_signed.png card_reencoded.png
c2patool card_reencoded.png --info

The output shows the AI label, a valid record, and then no record at all after the re-save:

"digitalSourceType": "http://cv.iptc.org/newscodes/digitalsourcetype/trainedAlgorithmicMedia"
"validation_state": "Valid"
No C2PA Manifests. (file size = 14189)

Two more results from that run. Flipping a single bit in the signed file's pixel data turned the state to Invalid with assertion.dataHash.mismatch. The record also took 72,271 of the signed file's 86,472 bytes, and the tool's output shows it carries a thumbnail of the image.

The tool's built-in test certificate is reported as untrusted, so production signing needs a real certificate. Plan for the extra bytes too: on our 14,189-byte test image, the label was about five times the size of the picture.

Explain it as a slow, expensive job wrapped in a fast, safe service. Walk through the parts in the order a request meets them.

1. 1

   Ask for the numbers

   Images per hour at peak, image sizes, and how long users will wait. Do the GPU sums out loud, with and without batching.

2. 2

   Return a job id, not an image

   Draw the API saving a job and putting it on a queue, replying 202 with a job id. Add polling or push, plus idempotency keys.

3. 3

   Design the GPU workers

   Keep the model loaded, batch jobs of the same preset, and scale on backlog per worker. Mention leases, safe repeated writes and a dead-letter queue.

4. 4

   Put safety on both sides

   A cheap prompt check before the queue and an image check after generation. Unsure cases go to people; the worst categories are hard blocks.

5. 5

   Store and deliver safely

   Private object storage with owner records, CDN signed links that expire, thumbnails and lifecycle rules for drafts.

6. 6

   Close with monitoring and labels

   Watch the oldest waiting job, images per GPU per hour and the refusal rate. Roll out new model versions to a small share first, and sign images with C2PA.