Text to Video System Design: Long Jobs, Cost and Previews

The request dies before the video exists. One clip takes minutes to make. Phones, browsers and load balancers give up on a silent connection long before that. A load balancer is the server that spreads incoming requests across an app's servers.

Text to video system design is the work of building everything around the video model so that slow, costly renders still feel quick and fair. The model is one part. Queues, storage, billing and safety checks are the rest.

Picture a common setup. A small team runs a reel maker for local shops in India. A shop owner types one line, such as a sweet shop's Diwali offer, and gets back a 15-second video ad. The setup is made up, but the problems are real.

The first version was one plain API call. The app sent the prompt, the server ran the model, and the app waited for the finished video. In testing, a 15-second reel at 720p took about 6 minutes on one GPU (made up).

Then real customers arrived, and three problems showed up in the first week.

• The load balancer closed every connection that sent no data for 60 seconds, the default idle timeout on AWS's Application Load Balancer. The video kept rendering, but nobody was waiting for it.
• Customers saw an error and tapped retry. Each tap started a fresh 6-minute render, so one reel was sometimes paid for three times.
• Many customers disliked the result at a glance. The saree was the wrong colour, or the shop looked like a mall, and the full render was already paid for.

One month's GPU bill was Rs 4.1 lakh (410,000), and customers rejected about a third of finished reels within ten seconds (made up). The rest of this guide is the redesign that fixes all three problems.

You need six words: frame, diffusion model, denoising step, latent, keyframe and job. Each one gets a plain meaning here, tied to the reel maker.

• Frame. One still picture in a video. Video plays many frames each second, written fps (frames per second), so a 15-second reel at 24 fps has 360 frames.
• Diffusion model. The kind of model most video generators use. It starts from random noise and cleans it, a little at a time, into a clip that matches the prompt.
• Denoising step. One round of that cleaning. A render might use 30 steps, and each step runs the whole model again, so steps multiply the cost.
• Latent. A small, compressed version of the video that the model works on instead of full pixels. A decoder turns it back into real frames at the end.
• Keyframe. One still image that sets the look of a shot. The reel maker shows keyframes as previews, so the customer approves the look before any motion is made.
• Job. One piece of work that runs in the background, with an id, a status and a result. Every video request becomes a job.

It means turning one short prompt into a finished video through several slow GPU stages, while the user does something else. Here is the path one reel request takes in the redesigned system.

1. 1

   Check the prompt

   Plain rules and a small classifier, a model that sorts text into labels, check the request first. Blocked requests stop here, before any GPU time is spent.

2. 2

   Plan the shots

   A text model turns the one line into a shot list. Example: three shots of 5 seconds each, with the camera move and mood for each one.

3. 3

   Draw keyframes

   An image model draws one low-resolution still per shot. These are the preview frames the customer sees and approves.

4. 4

   Add motion

   A video diffusion model turns each approved keyframe into a short clip, working on latents. Its decoder then turns the latents back into frames.

5. 5

   Upscale and smooth

   For customers who buy 1080p, an upscaler raises the 720p clip to 1080p. Frame interpolation then adds in-between frames, so the motion looks smoother.

6. 6

   Finish and encode

   Ordinary video tools add the shop name, price, logo and music. They join the shots and encode one MP4 file.

7. 7

   Store and deliver

   The file goes to object storage, cheap storage for big files. A CDN, servers near the user that keep copies, sends it quickly to the customer's phone.

Notice that the shop name and price are drawn by ordinary graphics code, not by the model. Diffusion models have long had trouble drawing exact text: a 2023 paper on image diffusion opens with exactly this problem.

It starts with random noise for every frame and removes it step by step, guided by the prompt. In the models the papers below describe, each step works on all frames of the clip together, which keeps the motion consistent.

The 2022 paper Video Diffusion Models built its video model as a natural extension of the standard image diffusion design. It trained on images and videos together, and added a sampling method for making longer, higher-resolution clips.

Working on full pixels costs too much, so latent models work on the compressed latent instead. The 2023 paper Align your Latents first trained an image model on images only. It then added a time dimension and fine-tuned the model, meaning trained it further, on videos.

The same paper also aligned its upscalers over time, turning them into video upscalers that stay consistent across frames. The reel maker uses the same idea for its 1080p reels: generate at 720p, then upscale.

Now the part that matters for system design. A 2026 survey on efficient video diffusion says video compounds cost across resolution, clip length and denoising steps. More pixels and frames mean more tokens, the small patches of the latent video that the model processes. Every step works on all of them, and every step repeats the work.

The survey names two things that dominate the running time. Attention compares each token with other tokens, and memory traffic is the time spent moving numbers in and out of GPU memory.

You handle them as background jobs. Accept the request at once, return a job id, render on a worker, and tell the user when the video is ready. No connection stays open for minutes.

HTTP has a status code for this. RFC 9110, the HTTP standard, defines 202 Accepted: the request was accepted, but processing has not finished. HTTP cannot send the final result later on that same request, so you give the client a status address to check.

Read the diagram below as a row of messages. Each box says what happens and who talks to whom.

Three details make this flow safe.

• Idempotency key. The app creates one unique key per tap of the button and sends it with every retry. A repeat with the same key returns the existing job instead of starting a second render.
• Lease and heartbeat. A worker claims a job for a few minutes and keeps renewing that claim while it works. If the worker dies, the lease runs out and another worker takes the job.
• Status updates. The worker writes progress to the job record, such as motion stage, shot 2 of 3. The app checks every few seconds, or gets a push notification at the end.

Back to the reel maker. The idempotency key ended the triple billing, because a retry now finds the render that is already running. The 60-second cut-off stopped mattering too, because no request lasts more than a second.

Save progress stage by stage

A full render has several stages, and any of them can fail. GPUs crash and drivers hang. Interruptible GPUs, spare cloud capacity sold cheaply, can be taken back at short notice.

So the worker saves each stage's output to object storage before starting the next stage. That saved output is a checkpoint. A replacement worker skips every saved stage and carries on from the first missing one.

Preview frames let the customer reject a wrong idea while it is still cheap. A few low-resolution keyframes cost a small slice of a full render and arrive in seconds, not minutes.

Think of a bridal mehendi artist. Before two hours on the full design, she dots a light outline on one palm. The bride approves it or asks for changes, and a few dots are easy to redo.

Where the picture breaks: the artist follows her outline by hand, line for line. A video model adds motion it was never shown, so the final reel can still drift from the stills.

Make the final video match the preview

A preview only helps if the final video looks like it. Diffusion starts from random noise, chosen by a seed, a number that fixes the random draw.

The noise has the shape of the latent, and that shape depends on the frame count and the resolution. Change either one and the same seed gives different starting noise, so the result can look quite different.

So the full render never re-runs the prompt from scratch. It upscales each approved still and hands it to an image-to-video stage as the first frame of its shot. The model adds movement to a picture the customer already approved.

This mode is common in open tools. The open-source diffusers library, version 0.41.0, ships more than ten image-to-video pipelines that take a starting image.

The reel maker allows five free re-rolls per reel. After that, each preview costs a few credits, the prepaid units customers buy, so nobody treats previews as a free image generator.

Give every job an idempotency key, save a checkpoint after every stage, and bill only finished stages. The program below shows all three. It is complete and runs with plain Python 3, using small dictionaries in place of real storage and databases.

# A crash-safe render worker with in-memory stand-ins for object storage,
# the job table and the credits ledger. Run it: python3 worker_demo.py
STAGES = ["motion", "decode", "upscale", "interpolate", "encode"]
CREDITS_PER_STAGE = 8                      # made up: measured GPU time per stage

saved = {}                                 # object storage: (job, stage) to output
held = {}                                  # credits ledger: job to credits on hold
jobs_by_key = {}                           # idempotency key to job id
crash_once = {"upscale"}                   # one worker will die in this stage


def submit(key, credits):
    if key in jobs_by_key:                 # a retry gets the same job back
        return jobs_by_key[key]
    job_id = f"job-{len(jobs_by_key) + 1}"
    jobs_by_key[key] = job_id
    held[job_id] = credits                 # hold credits before any GPU work
    return job_id


def render_stage(stage):
    if stage in crash_once:
        crash_once.remove(stage)
        raise RuntimeError(f"worker died during {stage}")
    return f"{stage} output"


def run_job(job_id, worker):
    for stage in STAGES:
        if (job_id, stage) in saved:       # finished before a crash
            print(f"{worker}: skip {stage}")
            continue
        saved[(job_id, stage)] = render_stage(stage)   # the checkpoint
        print(f"{worker}: did {stage}")
    used = CREDITS_PER_STAGE * sum((job_id, s) in saved for s in STAGES)  # saved stages only
    hold = held.pop(job_id)
    print(f"charged {used}, released {hold - used} of {hold} held credits")


first = submit("tap-7f3a", credits=50)
retry = submit("tap-7f3a", credits=50)     # the customer tapped again
print(first, retry)
try:
    run_job(first, "worker A")
except RuntimeError as err:
    print(f"worker A: {err}")
run_job(first, "worker B")                 # A's lease ran out, so B takes over

Run it with python3 worker_demo.py. You should see this output, tested on Python 3.13:

job-1 job-1
worker A: did motion
worker A: did decode
worker A: worker died during upscale
worker B: skip motion
worker B: skip decode
worker B: did upscale
worker B: did interpolate
worker B: did encode
charged 40, released 10 of 50 held credits

The saved-output check is what makes restarts cheap. Each stage writes to a fixed place, so running a stage twice just overwrites the same output. A real worker would also check for a cancel request at the same spot, between stages.

One gap the demo hides: two retries can arrive at the same moment. In a real database, make the idempotency key a unique column, so only one insert wins and the other reads the winner's job.

How real queues and clouds handle the same problems

Interruptions come with little warning. On AWS, a Spot Instance gets a two-minute warning before it is stopped, unless it is set to hibernate (as of 19 October 2026). Without checkpoints, every interruption restarts the render from zero.

Rented queues give you the lease for free. In Amazon SQS, a message being processed is hidden from other workers for a visibility timeout. If it is not deleted in time, another worker can receive it.

SQS suggests a heartbeat that keeps extending that timeout, up to a 12-hour limit counted from the first receipt (as of 19 October 2026). A render that might run longer needs smaller steps.

You can send the idempotency key in an Idempotency-Key request header. An IETF draft describes that header, but as of 19 October 2026 the draft has expired without becoming an RFC.

You estimate cost from the work the GPU will do: frames, pixels and denoising steps. Then you hold that amount from the customer's credits before rendering, and settle the measured amount afterwards.

A first estimate can be a simple formula. The rate comes from timing real renders on your own GPUs. A fixed charge covers loading the model, encoding and storage.

# Rough price check before a render. Rates are made up: time your own GPUs.
RATE = 0.004      # rupees per frame x megapixel x denoising step
FIXED = 1.0       # rupees per job: loading the model, encoding, storage


def estimate_cost(frames, width, height, steps, buffer=1.2):
    megapixels = width * height / 1_000_000
    work = frames * megapixels * steps
    return round((work * RATE + FIXED) * buffer, 2)


full = estimate_cost(15 * 24, 1280, 720, 30)    # 15 s at 24 fps, 720p
preview = estimate_cost(3, 640, 360, 8)         # 3 low-resolution keyframes
print(f"full reel Rs {full}, preview Rs {preview}, ratio {full / preview:.0f}x")

Run it with python3 cost_estimate.py. Expected output, in rupees:

full reel Rs 48.98, preview Rs 1.23, ratio 40x

The work line is the heart of it: frames times megapixels times steps. Every idea rejected at the preview stage costs about Rs 1.23 instead of Rs 48.98, so the preview gate pays for itself quickly.

This formula is a straight line, and it is only safe at the sizes you have timed. Longer or higher-resolution clips break it, and the next section shows why.

Why longer clips cost more than their length

Pause and predict: one 30-second clip has twice the frames of a 15-second clip. Does it cost twice as much as one 15-second clip, or more?

With full attention, where every token is compared with every other token, it costs far more. The survey gives dense attention cost as growing with the number of query tokens times the number of key tokens.

Double the tokens and the linear parts of the model double, but attention work grows four times. The script below counts both for one layer of a transformer, a network built around attention, and one denoising step.

# Why one 30-second clip can cost more than two 15-second clips.
# Rough compute for one transformer layer and one denoising step with full
# attention. Assumed: latents 4x smaller in time and 8x in width and height,
# then 2 x 2 patches, hidden size 2048. Real models differ.
D = 2048


def tokens(seconds, fps=24, width=1280, height=720):
    latent_frames = (seconds * fps - 1) // 4 + 1
    return latent_frames * (height // 16) * (width // 16)


def tflops(n):
    linear = 24 * n * D * D / 1e12        # projections and feed-forward
    attention = 4 * n * n * D / 1e12      # every token compared with every token
    return linear, attention


for s in (15, 30):
    n = tokens(s)
    lin, att = tflops(n)
    print(f"{s} s: {n:,} tokens, linear {lin:,.0f} TFLOP, attention {att:,.0f} TFLOP")

one_30 = sum(tflops(tokens(30)))
two_15 = 2 * sum(tflops(tokens(15)))
print(f"one 30 s clip costs {one_30 / two_15:.2f}x two 15 s clips")

Run it with python3 attention_scaling.py. Expected output, in TFLOP (a trillion floating-point operations):

15 s: 324,000 tokens, linear 33 TFLOP, attention 860 TFLOP
30 s: 648,000 tokens, linear 65 TFLOP, attention 3,440 TFLOP
one 30 s clip costs 1.96x two 15 s clips

At this size, attention is about 96 percent of the work. One 30-second clip costs 1.96 times as much as two 15-second clips, close to four times one 15-second clip.

The assumed compression matches the default factors in several video pipelines of the diffusers library, version 0.41.0. Real models differ, and many use cheaper attention schemes, so time your own.

Five rules that keep the real bill close to the estimate

• Hold, then settle. Reserve the estimate when the job starts, charge the measured GPU time when it ends, and release the rest.
• Refund failures automatically. A job that dies after three attempts gives back every credit it held, with no support ticket needed.
• Cap the size. The reel maker sells up to 30 seconds at up to 1080p, and bigger requests are rejected before they reach a queue.
• Reuse what you made. A preview with the same prompt, seed and settings is served from storage instead of rendered again.
• Cut steps where quality allows. The 2026 survey lists step distillation, training a model to need far fewer denoising steps, as one of four main ways to speed up video diffusion.

The table puts the two kinds of work side by side. Read the last two rows: they show why the preview gate pays for itself.

Use separate queues for quick and slow work, plus a limit on how many renders one customer runs at once. Without both, one big order can block everyone else.

One Friday, a marketing agency submitted 300 reels at once into the single shared queue. The render pool had 10 GPUs (made up).

Three hundred 6-minute renders are 1,800 GPU minutes, which is three hours for 10 GPUs. Every sweet shop that asked for a 20-second preview after them waited about three hours.

• Two lanes. Previews go to a small pool of GPUs kept ready, because someone is watching the screen. Full renders go to a larger pool that grows and shrinks with demand.
• Per-customer limits. While others are waiting, each account runs at most 2 full renders at once, and the agency's other jobs wait in its own line.
• Fair picking. Workers take the next job round-robin across customers, one customer after another, instead of strictly oldest first. In the preview lane, this stops one agency's 300 previews from crowding out a single shop's preview.
• Cheaper GPUs for patient work. Full renders run on interruptible GPUs, which checkpoints make safe. Previews stay on regular GPUs, where an interruption would annoy someone who is waiting.

Watch the trade-off in the limit. A hard cap of 2 renders stretches the agency's batch from 3 hours to 15 hours, even at night when GPUs sit idle. So the limit applies only while other customers are waiting.

Check the prompt before any GPU work, check the frames before delivery, and attach a label that says the video was made by AI. Each check is cheaper the earlier it runs.

• Prompt checks. Block requests for real people's faces, other companies' logos and unsafe content before a single keyframe is drawn.
• Keyframe checks. An image classifier scans every preview. This is cheap, because each reel has only a few stills.
• Final checks. Sample frames from the finished video, for example one each second, and scan them too. Motion can produce something the keyframes never showed.
• Labels. Attach Content Credentials to every file, following the C2PA standard.

C2PA is an open standard for recording where a piece of media came from. Its Content Credentials hold a manifest, a record of facts about the file, digitally signed with an X.509 certificate, the same kind of certificate that secures websites.

When an AI system made the content, the manifest says so in a field called digitalSourceType. The C2PA explainer calls this data tamper-evident. Change the file or the record and a hash, a short fingerprint of the data, no longer matches, so the edit shows.

The reel maker adds the credentials in the encode stage, so every delivered reel carries them. You can check all of this yourself with c2patool, the open-source C2PA command-line tool.

We ran these commands with c2patool 0.28.2 and ffmpeg 6.1.1 on a 2-second test MP4 (as of 19 October 2026). The first command signs the file with the tool's built-in test certificate.

c2patool reel.mp4 -c '{"title": "reel.mp4"}' --create trainedAlgorithmicMedia -o reel_signed.mp4
c2patool reel_signed.mp4 | grep -E 'digitalSourceType|validation_state'
c2patool reel_signed.mp4 --info
ffmpeg -loglevel error -i reel_signed.mp4 -c:v libx264 reencoded.mp4
c2patool reencoded.mp4 --info

The lines to look for in the output. File sizes vary with your ffmpeg version:

"digitalSourceType": "http://cv.iptc.org/newscodes/digitalsourcetype/trainedAlgorithmicMedia"
"validation_state": "Valid"
Validation issues:
   signingCredential.untrusted
No C2PA Manifests. (file size = 17046)

The signed file says the content came from a trained model, and the signature checks out. The untrusted warning appears because a test certificate signed it. Checkers compare the signer against trust lists such as the official C2PA trust list, so real reels need a certificate that chains to one.

Then the two weak spots showed up. Flipping one byte of video data made the --info report add assertion.bmffHash.mismatch, so the edit showed. A plain ffmpeg re-encode produced a file with no manifest at all.

Rules on labels are changing too. In India, IT Rules amendments notified on 10 February 2026 apply to intermediaries, the platforms and apps that host or pass on users' content. Those that offer tools for making synthetic content must label it prominently.

Law-firm summaries add that the rules took effect on 20 February 2026. They also say the label needs permanent metadata or a similar provenance marker, such as a unique identifier, where technically feasible. Platforms must not let anyone remove it. A C2PA manifest is one such marker, but the re-encode above showed how easily it is stripped (as of 19 October 2026).

Lead with the long job, then show how previews and stages keep cost under control. These seven steps fit a 45-minute round.

1. 1

   Clarify the size

   Ask about video length, resolution, daily volume and how long users will wait. A 15-second reel and a 2-minute film are different systems.

2. 2

   Do the GPU maths

   Estimate GPU minutes per video and multiply by daily volume. Example: 2,000 reels a day at 6 GPU minutes each is 200 GPU hours a day.

3. 3

   Make it asynchronous

   Return 202 with a job id, queue the work, and notify on completion. Mention idempotency keys, so retries never start a second render.

4. 4

   Add the preview gate

   Cheap keyframes first. Then a full render built from the approved keyframes, so the result starts from what the customer saw.

5. 5

   Split the render into stages

   Save output after each stage so restarts resume. Add leases with heartbeats, an attempt limit and a dead-letter queue.

6. 6

   Control cost and fairness

   Hold credits, settle on completion, limit running jobs per customer, and keep separate GPU pools for previews and renders. Say why long clips cost more than their length.

7. 7

   Close with safety

   Prompt checks, frame checks and signed provenance labels on every file. Say which checks run before GPU time is spent.