How Diffusion Models Generate Images, Step by Step

You cannot predict its mistakes, its speed or its randomness. You change a setting, the picture changes, and you cannot say why. Knowing how diffusion models work fixes that, because each of those behaviours comes straight from the mechanism.

Picture a common setup. A textile print studio in Surat makes designs for fabric mills: paisleys, peacocks and bandhani (tie-dye) dots. Each becomes a repeat, a tile that prints edge to edge with no visible join. The setup is made up, but the problems are real. Before each festival season a mill asks for 40 fresh designs, and one designer finishes about three a week by hand.

So the studio tries a text-to-image tool built on a diffusion model. The designer types a peacock paisley in indigo and rust, and gets ten options in a few minutes. Then three questions come up.

• Why does the same prompt give a different design each time, and how do you get a liked one back? The sampling section answers this.
• Why does the high-quality setting take minutes when a draft takes seconds? The steps section answers this.
• Why do the repeats not tile, and why do the small dots come out smudged? The latent and failure sections answer this.

Each answer comes from the mechanism, with the studio as the running example all the way through.

A diffusion model starts from random noise and removes a small amount of it at each step, until a clean picture is left. A trained neural network decides what to remove at every step.

Noise here means random speckles, like the grey fuzz on an old TV with no signal. Each pixel gets a random value drawn from a bell curve, the Gaussian distribution. A neural network is a program with millions of adjustable numbers, called parameters, tuned on examples. The 2020 DDPM paper (denoising diffusion probabilistic models), which made these models work well on images, used 35.7 million in its smallest one.

Think of the sand sculptors on Puri beach. A sculptor starts with a rough heap of wet sand. Early passes block out the big shapes, a head here and a wing there. Only the last passes carve feathers and eyes.

A diffusion model works in the same order. Early steps fix the big layout: where the peacock sits and which colours dominate. Late steps add fine detail, like the dots inside each paisley. The DDPM paper reports exactly this: large-scale features appear first and details appear last.

• The heap of sand is the starting noise.
• The sculptor's eye is the network: at each pass it judges what does not belong.
• One pass with the trowel is one step.
• The waves that wear a finished figure down are the forward process, covered next.

Where the picture breaks: a sculptor plans the figure from the start. The network has no plan. At each step it only guesses the noise, and the picture emerges from those guesses. The original sampling code also adds a little fresh noise back after every step but the last, which no sculptor would do.

So the idea has two halves. The forward process adds noise to real pictures, which is easy and needs no learning. The reverse process removes noise, which is hard and is what the network learns.

The idea is older than most people think. A 2015 paper by Sohl-Dickstein and others, inspired by physics, proposed slowly destroying the structure in data and learning a reverse process that restores it. A 2020 paper by Ho, Jain and Abbeel, the DDPM paper, made it work well on images.

The forward process is a fixed recipe that adds a little random noise to a picture, step after step, until only noise is left. Nothing in it is learned, so it is cheap and exact.

Picture a finished sand figure at high tide. Each wave softens it a little, and after enough waves it is a smooth mound again. The forward process is the waves, and you choose exactly how strong each wave is.

That choice is called the noise schedule: a list saying how much noise to add at each step. The DDPM paper used 1,000 steps. Its schedule numbers, called betas, rose in a straight line from 0.0001 at the first step to 0.02 at the last.

One handy fact makes training fast. You do not have to add noise 700 times in a row to reach step 700. A short formula jumps straight from the clean picture to any step.

The formula mixes two things: the clean picture times a picture weight, plus fresh noise times a noise weight. The table shows both weights on the DDPM schedule. The code later in this guide prints three of these rows, steps 10, 500 and 1,000, so you can check them.

The two weights are not shares that add up to 1. Their squares do: at step 500, 0.280 squared plus 0.960 squared is 1.000. So, for pixels between -1 and 1, a noisy picture never spreads wider than pure noise, at any step.

A diffusion model learns by guessing the noise. You take a clean image, add a known amount of noise, and ask the network to predict exactly which noise you added.

Because you added the noise yourself, you know the right answer. You compare the guess with the real noise and nudge the parameters so the next guess is closer. Repeat this over and over, across many images and many noise levels.

Back on the beach, it is like watching thousands of figures being taken apart by the waves. A learner who has seen enough of them can look at any half-melted shape and say which sand the last wave took away.

The network also needs to know which step it is looking at. A little fuzz needs gentle cleaning, and heavy fuzz needs bold guesses. So the step number goes in as a second input, next to the noisy image.

Scoring is simple. The guess and the real noise are compared pixel by pixel with mean squared error: the average of the squared differences. The DDPM authors tried other targets, and this plain noise-guessing score gave their best sample quality.

This guide follows that DDPM recipe. Some models are trained to predict the clean picture, or a blend of picture and noise called v-prediction, instead. As of version 0.41.0, the diffusers DDPM scheduler accepts all three, with noise as the default.

In the DDPM paper the network was a U-Net. A U-Net shrinks the image step by step to see the overall layout, then grows it back. Shortcut links carry fine detail across, which helps keep small details like dots.

A 2022 paper by Peebles and Xie replaced the U-Net with a transformer working on patches of the latent. A transformer is a network built around attention, which lets every part of the input look at every other part. As of version 0.41.0, the diffusers library ships both kinds.

It starts from a fresh grid of pure noise and runs the network once per step, from the noisiest step down to the cleanest. Each run guesses the noise, and the sampler removes part of that guess.

This is called sampling, or inference: using a trained model to make something new. Here is what happens when the studio asks for one peacock paisley.

1. 1

   Draw the starting noise

   The computer fills a grid with random numbers. A seed, a number that fixes the randomness, decides which noise you get.

2. 2

   Guess the noise at the top step

   The network looks at pure fuzz and guesses what noise covers it. At this point its guess is rough, like the first blocked-out sand shape.

3. 3

   Remove a slice of it

   The sampler, the code that runs the loop, subtracts part of the guessed noise. It does not remove it all at once, because the guess is still unsure.

4. 4

   Repeat at the next step down

   Now the grid is a little less noisy. The network guesses again, with better footing, and another slice comes off.

5. 5

   Finish at step zero

   By the final steps, the layout is fixed and only fine detail changes. What is left is the new design.

It comes out as a blur: roughly the average of every picture that could hide behind that noise. In the toy model later in this guide, one jump gives pixels within 0.013 of zero, the average of its four training tiles. Small steps let each guess build on a slightly clearer picture, so the result settles on one design.

The DDPM sampler also adds back a little fresh noise after every step except the last. In that method each reverse step is a random draw, not a fixed move. The paper notes the loop resembles Langevin dynamics, a sampling method from physics.

A text prompt controls a diffusion model by being fed into the network at every step. The network then guesses the noise hiding a picture that matches those words, not just any picture.

First, a text encoder turns the prompt into numbers. A text encoder is a separate network that reads words and outputs one list of numbers per word piece. These lists are called embeddings, and similar meanings get similar numbers.

Next, the denoising network reads those embeddings through cross-attention. Cross-attention lets each patch of the image ask which words matter to it. A patch at the edge might weigh the word border most, and a patch in the middle the word peacock.

The 2021 latent diffusion paper by Rombach and others used cross-attention for exactly this. The same layers also took other inputs, such as layouts marking where objects should go.

What is classifier-free guidance?

Classifier-free guidance makes the picture follow the prompt more strongly. At each step the network makes two noise guesses: one with the prompt and one without it.

The difference between the two guesses points towards the prompt. The sampler stretches that difference by a number called the guidance scale. Stronger guidance follows the words more closely but gives less variety.

Ho and Salimans described the method in 2022. They trained one model with the prompt dropped at random from some examples, so the same network can make both guesses. Dropping it 10 or 20 percent of the time worked best in their tests. Earlier guidance needed a separate image classifier, which is where the name comes from.

They also noticed that strongly guided samples showed saturated colours. And the method costs two network passes per step instead of one. The diffusers library, for example, sends both through as one batch of two, so the extra work runs side by side.

Latent diffusion runs all the noise steps on a small compressed version of the image, called a latent, instead of the full-size pixels. A separate decoder turns the finished latent back into a picture.

The compressor is an autoencoder, a network pair. The encoder squeezes an image into a small grid of numbers, and the decoder rebuilds the image from that grid. It is trained once, before the diffusion part, and then reused.

The Rombach paper tested shrinking each side of the image by factors from 1 to 32. Factors from 4 to 16 struck a good balance between speed and faithful detail. Too much shrinking lost information and capped the quality.

Do the maths for a 512 by 512 colour image. It holds 786,432 numbers, three colours per pixel. Shrink each side by 8 and you get a 64 by 64 grid. With 4 numbers per position, as in the paper's factor-8 autoencoders, that is 16,384 numbers.

So the latent has 64 times fewer positions but 48 times fewer numbers, because each position holds four numbers instead of three. The network runs on this small grid at every step. The full-size picture appears only once, at the end.

This diagram shows the full path the studio's prompt takes, including the designer's check before anything reaches the mill.

A diffusion model is often trained with 1,000 noise levels but rarely needs all of them when generating. Faster samplers jump across levels and often use a few dozen steps.

The best-known early fix is DDIM (denoising diffusion implicit models), from Song, Meng and Ermon in 2020. It keeps the same training as DDPM, so you reuse the trained network and change only the sampling loop. With its added noise switched off, the image depends only on the starting noise.

Skipping steps has a price, and the DDIM paper measured it. It used FID, a score of how far generated images sit from real ones, where lower is better. The tests ran on CIFAR-10, a standard set of tiny 32 by 32 photos.

Read it as a trade. DDIM at 50 steps scored 4.67 against 4.04 at 1,000 steps, for a twentieth of the network runs. The DDPM paper's own CIFAR-10 sampler settings scored 3.17 at 1,000 steps but 367.43 at 10. On these tests, at low step counts, the sampler mattered as much as the count.

The authors sum it up as quality comparable to 1,000-step models within 20 to 100 steps. They also found that, with DDIM, the same starting noise kept most high-level features at different step counts. That is what lets the studio draft cheaply and finish carefully from the same seed.

What does each step cost?

Time per image grows almost in a straight line with the step count. Here is the studio's sum with made-up numbers: 80 milliseconds per network pass, two passes per step for guidance, and 0.4 seconds to decode.

The studio's rule became simple. Make 10 options at 30 steps, let the mill pick three, then rerun those three seeds at 50 steps and compare.

Library defaults sit in the same range. In diffusers 0.41.0, as of 19 October 2026, the original latent text-to-image pipeline defaults to 50 steps and a guidance scale of 7.5. Its DDPM and DDIM scheduler classes default to the DDPM paper's schedule.

• Fewer steps: faster and cheaper, but rougher detail, and the sampler choice matters more.
• Higher guidance scale: closer to the prompt, but less variety and more saturated colours.
• Bigger image size: more detail, but more memory and time per step.
• More options per prompt: a better chance of a winner, but more time spent reviewing.

Here is one training step in Python with NumPy, a library for arrays of numbers. The model argument stands in for the network. x0 is one clean design, with pixel values scaled to between -1 and 1, as in the DDPM paper.

import numpy as np

T = 1000
betas = np.linspace(1e-4, 0.02, T)    # noise added at each step
alpha_bar = np.cumprod(1.0 - betas)   # how much picture is left by step t

def add_noise(x0, t, rng):
    noise = rng.standard_normal(x0.shape)
    xt = np.sqrt(alpha_bar[t]) * x0 + np.sqrt(1 - alpha_bar[t]) * noise
    return xt, noise

def training_loss(model, x0, rng):
    t = rng.integers(0, T)               # pick a random noise level
    xt, noise = add_noise(x0, t, rng)
    guess = model(xt, t)                 # the network guesses the noise
    return np.mean((guess - noise) ** 2) # how wrong was the guess?

rng = np.random.default_rng(0)
x0 = rng.choice([-1.0, 1.0], size=(64, 64))   # a stand-in 64 x 64 design
lazy = lambda xt, t: np.zeros_like(xt)        # always guesses "no noise"
print(round(training_loss(lazy, x0, rng), 2))
for t in [9, 499, 999]:
    print(t + 1, round(np.sqrt(alpha_bar[t]), 3), round(np.sqrt(1 - alpha_bar[t]), 3))

On NumPy 2.5.4 it prints 0.99 first: a lazy network that always guesses zero noise scores about 1, and training pushes that towards 0. The next three lines are steps 10, 500 and 1,000: 0.999 and 0.044, 0.28 and 0.96, then 0.006 and 1.0, matching the table.

Three lines do the real work. The betas line is the DDPM straight-line schedule. The xt line is the shortcut formula. The return line of training_loss is the mean squared error score. A real run would also update the network after each score, which a deep learning library handles.

Here is the sampling loop with guidance, following the DDPM sampler. It reuses T, betas and alpha_bar from the first block. The prompt argument is whatever your model takes as the prompt, such as its embedding.

def generate(model, prompt, shape, rng, scale=7.0):
    x = rng.standard_normal(shape)                 # start from pure noise
    for t in reversed(range(T)):
        with_prompt = model(x, t, prompt)
        no_prompt = model(x, t, None)
        noise = no_prompt + scale * (with_prompt - no_prompt)
        a, ab = 1.0 - betas[t], alpha_bar[t]
        x = (x - betas[t] / np.sqrt(1 - ab) * noise) / np.sqrt(a)
        if t > 0:
            x += np.sqrt(betas[t]) * rng.standard_normal(shape)
    return x

The noise line is classifier-free guidance. The x line removes a slice of the guessed noise, and the if block adds fresh noise back, except at the very end. With 1,000 steps that is 2,000 network runs for one design.

Which guidance scale does your library mean?

The paper and the library count guidance differently. Ho and Salimans write the combined guess as the prompted guess times (1 + w), minus the unprompted guess times w. So in the paper, w of 0 means no guidance.

The generate function above, like diffusers 0.41.0, uses a scale where 1 means no guidance. So scale equals w plus 1: a paper's w of 3 is a library scale of 4. Check which one a number means before you copy it from a paper.

Two more details sit in the diffusers 0.41.0 source, so pin your version (install that exact version). The pipeline runs the two-pass guidance only when the scale is above 1, so at exactly 1 it runs one pass per step. Models trained to take the guidance scale as an input always run one pass, with the scale fed in as an extra input. And its no-prompt guess is really a guess for an empty prompt, or for your negative prompt (words you want kept out) if you give one.

It shows three things you can verify in seconds: seeds repeat, one jump gives a blur, and a perfect noise guesser can only copy its training data.

The data is four tiny tiles of 2 by 2 pixels, written as four numbers each: two striped tiles and two checked tiles. For a set this small, the best possible noise guess can be worked out exactly. You weigh each training tile by how well it explains the noisy picture, blend them, and back out the noise.

Karras and others call this the ideal denoiser and give its formula. It is the guess with the lowest possible mean squared error, so it is what perfect training would reach. The code reuses the first two blocks.

tiles = {"stripes": np.array([[1, 1, -1, -1], [-1, -1, 1, 1]], float),
         "checks": np.array([[1, -1, -1, 1], [-1, 1, 1, -1]], float)}
all_tiles = np.vstack(list(tiles.values()))

def ideal_model(x, t, prompt=None):
    pics = all_tiles if prompt is None else tiles[prompt]
    ab = alpha_bar[t]
    d2 = ((x - np.sqrt(ab) * pics) ** 2).sum(axis=1)
    w = np.exp(-(d2 - d2.min()) / (2 * (1 - ab)))    # how well each tile fits
    clean_guess = (w / w.sum()) @ pics
    return (x - np.sqrt(ab) * clean_guess) / np.sqrt(1 - ab)

for seed in [7, 7, 9]:                                # 1. seeds repeat
    print(seed, generate(ideal_model, "checks", (4,), np.random.default_rng(seed)))

x = np.random.default_rng(1).standard_normal(4)       # 2. one jump
guess = ideal_model(x, T - 1)
one_jump = (x - np.sqrt(1 - alpha_bar[-1]) * guess) / np.sqrt(alpha_bar[-1])
print("one jump, biggest pixel:", round(np.abs(one_jump).max(), 3))

rng = np.random.default_rng(0)                        # 3. copies only
outs = np.array([generate(ideal_model, None, (4,), rng) for _ in range(100)])
gap = np.abs(outs[:, None, :] - all_tiles).max(axis=2).min(axis=1)
print("largest gap to a training tile:", round(gap.max(), 6))

What just happened? Seed 7 gives the checked tile minus 1, 1, 1, minus 1 both times, and seed 9 gives the other checked tile. One jump from pure noise prints a biggest pixel of 0.013: a grey blur, not a tile.

The last line prints 0.0. Across 100 runs with no prompt, every result is a training tile, to within rounding. The whole block takes a few seconds on a CPU: about 6 on our test machine with NumPy 2.5.4.

That last result is the important one. Gu and others (2023) showed that the training target has an exact best answer, and that answer can only reproduce training data. The authors note that this clashes with how well real models generalise, and that memorisation tends to show up on smaller training sets. Four tiles is the extreme case. Large models still copy sometimes, as the next section shows.

Diffusion models make pictures that look right at a glance but can break the rules a real product needs. For textile prints the usual problems are joins, colour counts, wrong placement and look-alikes.

• Broken repeats. A print must tile, so the right edge has to meet the left edge. Nothing in the standard training target asks for that, so check every join.
• Too many colours. Screen printing lays down one colour at a time, each through its own screen. An option with 40 shades has to be cut to the six the mill can print (numbers made up).
• Wrong placement and swapped colours. The GenEval benchmark (Ghosh and others, 2023) found recent models still weak at spatial relations and at attribute binding, such as giving each object its own colour.
• Look-alike designs. A model can reproduce images it was trained on, so the designer compares options with known designs before a sale.

Look-alikes are not hypothetical. Carlini and others (2023) generated 500 images for each of 350,000 heavily duplicated training captions on one popular open-source model. They found about a hundred near-copies of training images; the paper counts 109. Images duplicated many times in training were far more likely to be copied.

None of these is a reason to stop. They are reasons to keep a designer in the loop. The mill's buyers see the cleaned, tiled, colour-reduced version, never raw output.

Explain the idea in one line, then design the serving path around its biggest cost: many network runs per image.

1. 1

   Give the one-line idea

   Training teaches a network to guess the noise in noisy images. Generating starts from noise and removes it step by step, steered by the prompt.

2. 2

   Draw the request path

   Prompt, text encoder, latent noise from a seed, denoising loop, decoder, then safety and quality checks before the user sees anything.

3. 3

   Do the latency maths

   Time per image is about steps times passes per step times one pass, plus decoding. Guidance doubles the passes. Show how fewer steps and a small latent cut it.

4. 4

   Plan the GPU queue

   Requests wait in a queue and run in batches on GPUs. Show users progress, because even five seconds feels long.

5. 5

   Store what makes it repeatable

   Save prompt, seed, model version, sampler, step count and guidance scale with each image, so any result can be rebuilt or audited later.

6. 6

   Name the human checks

   Say where a person reviews output, and what the system blocks automatically, such as unsafe or look-alike images.