LoRA and QLoRA Explained: Fine-Tune a Model on One GPU

It does not fit. Full fine-tuning trains every weight, and each weight needs extra memory while it learns. LoRA/QLoRA fine-tuning fixes this by training a small add-on and freezing everything else.

Picture a common setup. A farm-advice assistant answers crop questions from farmers in Maharashtra, replying in Marathi. The setup is made up, but the problem is real.

The assistant runs on an open-weight model with 7 billion weights. Open-weight means you can download the model's numbers and run them on your own machine. It writes clean, bookish Marathi, but many farmers around Nashik speak Ahirani, and others in Vidarbha speak Varhadi.

• A farmer uses a local word for a vine disease. The model does not know it and gives advice about a different disease.
• Replies sound like a government circular. Farmers say it feels like a stranger talking down to them.
• The naive first try was long word lists inside the prompt. Answers improved a little, but each prompt grew to about 2,000 tokens (small pieces of text) and every reply got slower.

The team has 12,000 question and answer pairs across three ways of speaking, all checked by two agronomists. It has one rented GPU with 24 GB of memory. Full fine-tuning of 7 billion weights with Adam needs about 112 GB before anything else, almost five times that card.

So the real question is whether one card can teach a big model a new way of speaking. With LoRA and QLoRA, it can, as long as the change is about style and wording, not new knowledge.

You need five words: weights, fine-tuning, GPU memory, adapter and quantization. Each gets a plain meaning here, tied to the assistant.

Weights. A model is a huge stack of numbers called weights, also called parameters. Training nudges them so the model's guesses get better. They sit in big grids called matrices, such as one grid of 4,096 rows by 4,096 columns.

Fine-tuning. Fine-tuning means training an already trained model a little more, on your own examples. The team's 12,000 dialect pairs are its fine-tuning data. How a model learns in the first place is a separate topic.

GPU memory. A GPU is the chip that does model maths fast, and its own memory holds the weights while it works. If a training job does not fit in that memory, it stops with an out-of-memory error. The team's card has 24 GB.

Adapter. An adapter is a small set of new, trainable numbers added to a frozen model. Frozen means the original weights never change during training. Only the adapter learns.

Quantization. Quantization means storing each number with fewer bits, a bit like writing Rs 1,47,893 as Rs 1.48 lakh (148,000). A 16-bit weight takes 2 bytes, and a 4-bit weight takes half a byte.

LoRA, short for low-rank adaptation, freezes every original weight and trains two small matrices beside chosen layers, the steps the model's maths passes through in order. Multiplied together, the two give the change that layer learns. QLoRA does the same on a frozen copy stored in 4 bits.

Think of a phone whose camera takes poor close-ups of leaves. Full fine-tuning is buying a new phone for every job. LoRA is clipping a small macro lens over the camera you already have.

• The phone's camera is the frozen base model. It stays exactly as it was.
• The clip-on lens is the adapter. It changes what reaches the picture.
• A pouch of lenses is a set of adapters, one per job, for the same phone.
• QLoRA is the same lens on a lighter phone whose camera parts were made slightly less precise to save space.

Where the picture breaks: a lens cannot melt into the camera, but a LoRA adapter can be merged into the weights. And a lens fits many phones, while an adapter fits only the exact base model it was trained on.

The method comes from a 2021 paper by Hu and colleagues. Their bet was that the change fine-tuning makes to a big matrix can be described with far fewer numbers. Low rank is the maths name for that.

Rank, in plain words, is how many independent patterns a matrix can hold. A 4,096 by 4,096 grid can hold up to 4,096 of them. LoRA bets that a dialect change needs only a handful, say 8.

So instead of a full 4,096 by 4,096 change, LoRA learns two thin matrices. Matrix A is 8 by 4,096, and matrix B is 4,096 by 8. Their product B times A is full-size, yet it is built from just 65,536 numbers.

The full grid has 16,777,216 numbers, so the adapter is about 0.39 percent of it. Because the change is built from 8 patterns, its rank can never be more than 8.

On a 175-billion-weight model, the LoRA paper reports 10,000 times fewer trainable weights and 3 times less GPU memory than full fine-tuning with Adam. Training memory fell from 1.2 TB to 350 GB.

Storage shrinks too. The paper adapted only two attention matrices per layer, the parts that decide which earlier words matter. At rank 4, a saved task fell from about 350 GB to about 35 MB. That is the difference between a new phone and a lens in your pocket.

Inside each chosen layer, the input goes through the frozen weight matrix and also through the two small matrices. The two results are added. Only the small path learns.

Follow the input along both paths in the diagram. Notice that the frozen box sits on its own path, and nothing in training ever changes it.

Two details make training start safely. The paper starts matrix A with small random numbers and matrix B with zeros. So B times A is zero on day one, and the model behaves exactly like the original.

The paper also multiplies the small path by alpha divided by r, where r is the rank and alpha is a fixed number you choose. The authors set alpha to the first rank they tried and did not tune it. This scaling cuts how much retuning you need when you change the rank.

Here is one LoRA layer in NumPy, a Python library for maths on grids of numbers. It checks the zero start and the merge on a 4,096-wide layer. It runs in under a second on a laptop.

import numpy as np

rng = np.random.default_rng(0)
d, r, alpha = 4096, 8, 16
W0 = rng.normal(0, 0.02, (d, d))   # frozen base weight
A = rng.normal(0, 0.01, (r, d))    # small random start
B = np.zeros((d, r))               # zeros, so B @ A starts at 0
scale = alpha / r

def forward(x):
    return W0 @ x + scale * (B @ (A @ x))

x = rng.normal(size=d)
print(np.array_equal(forward(x), W0 @ x))     # True

B = rng.normal(0, 0.01, (d, r))    # pretend training moved B
merged = W0 + scale * (B @ A)
print(np.allclose(forward(x), merged @ x))    # True
print(A.size + B.size, W0.size)               # 65536 16777216

Merging matters for speed. The LoRA paper points out that a merged layer runs exactly like the original, so answers get no slower. To switch tasks, you subtract one adapter's change and add another's.

QLoRA is LoRA trained on top of a base model stored in 4 bits instead of 16. The adapters are the same, but the frozen weights take about a quarter of the memory.

QLoRA comes from a 2023 paper by Dettmers and colleagues. Its headline result was a 65-billion-weight model fine-tuned on a single 48 GB GPU. Ordinary 16-bit fine-tuning of that model needs more than 780 GB.

Quality claims need care. On models up to 3 billion weights, 4-bit adapters matched full 16-bit fine-tuning on the paper's benchmark test sets. From 7 to 65 billion, they matched 16-bit LoRA on a knowledge test.

The authors also say plainly that they did not show QLoRA matches full 16-bit fine-tuning at 33 and 65 billion weights. So the safe claim is narrower than the headline.

The trick works because frozen weights only need to be read. For each layer's maths, the 4-bit weights are turned back into 16-bit numbers, a step called dequantizing. Gradients, the signals that say which way to move, flow back through the frozen model, but only the adapters get updated.

The QLoRA paper adds three ideas to make the 4-bit copy work well. Each one removes a different memory cost:

• 4-bit NormalFloat, or NF4. Its 16 values are spaced so each covers a similar share of a bell curve, so more of them sit near zero, where most weights are. Zero itself is exact.
• Double quantization. Each block of 64 weights keeps one scale number to undo the rounding. QLoRA rounds those scales to 8 bits too, saving about 0.37 bits per weight, about 3 GB on a 65-billion-weight model.
• Paged optimizers. When a long example causes a sudden memory spike, the optimizer's notes move to the computer's main memory and come back when needed. This avoids an out-of-memory crash, and the paper saw no slowdown for its 65-billion-weight model on a 48 GB GPU at batch size 16 (16 examples processed together).

Here is one QLoRA training step on the team's card. Start at the batch of examples. Notice that the optimizer only ever sends updates to the adapters, never to the 4-bit base.

For a 7-billion-weight model, the fixed part (weights, gradients and optimizer notes) is about 112 GB for full fine-tuning, 14.3 GB for LoRA and 3.9 GB for QLoRA. Activations come on top and grow with example length and batch size.

Full fine-tuning is heavy because training keeps extra numbers for every weight. The ZeRO paper counts 16 bytes per weight for Adam with mixed-precision training, which does the maths in 16 bits but keeps a 32-bit master copy:

• The weight itself in 16 bits: 2 bytes.
• Its gradient, the direction it should move, in 16 bits: 2 bytes.
• The optimizer's notes: 12 bytes. Adam keeps a 32-bit copy of the weight plus two running averages, 4 bytes each.

With LoRA, the frozen weights need only their 2 bytes each, and the 16-byte cost applies only to the small adapter. Activations are the in-between results each layer keeps so gradients can be worked out later. LoRA does not remove them, because gradients must still flow back through the frozen layers to reach every adapter.

You can do this arithmetic in a few lines before renting anything. The shape below is a common one for 7-billion-weight models. Each linear layer multiplies its input by one weight matrix, and the code adapts all seven in each of the model's 32 repeated blocks of layers.

def lora_params(d_in, d_out, r):
    return r * (d_in + d_out)          # A is r x d_in, B is d_out x r

# A common 7-billion-weight shape: 32 blocks, width 4,096,
# four 4096x4096 attention matrices and three 4096x11008 ones
h, ff, blocks = 4096, 11008, 32
shapes = [(h, h)] * 4 + [(h, ff)] * 2 + [(ff, h)]
for r in (8, 16):
    n = blocks * sum(lora_params(i, o, r) for i, o in shapes)
    print(r, n, f"{n * 2 / 1e6:.0f} MB")      # 2 bytes each

n = 7e9
print(n * 16 / 1e9, n * 2 / 1e9)               # GB: full, 16-bit base
bits = 4 + 8 / 64 + 32 / (64 * 256)            # NF4 + double-quantized scales
print(round(bits, 3), round(n * bits / 8 / 1e9, 2))

Expect these four lines: 8 19988480 40 MB, then 16 39976960 80 MB, then 112.0 14.0, then 4.127 3.61. The units are megabytes and gigabytes of 1,000,000,000 bytes, not gibibytes.

Real measurements run higher than this arithmetic. The QLoRA paper measured its 4-bit 7-billion-weight base at 5,048 MB, against 26 MB for a small LoRA setup. So leave headroom, and measure your own peak memory over a few steps with your longest examples.

The team picks QLoRA for experiments, because the 24 GB card leaves room for longer farmer questions. It keeps LoRA on a 16-bit base as an option for the final run, to avoid 4-bit rounding in the trained result.

B starts at zero so that the adapter adds nothing on day one and the model starts from its original behaviour. The catch is that A cannot learn on the very first step.

The adapter's output is B times A times the input. A's gradient passes through B on its way back, and B is zero. B's gradient passes through A times the input, which is not zero.

import numpy as np

rng = np.random.default_rng(1)
d, r, scale = 4096, 8, 2.0
A = rng.normal(0, 0.01, (r, d))
B = np.zeros((d, r))
x = rng.normal(size=d)   # the layer's input
g = rng.normal(size=d)   # how the loss wants the output to move

grad_B = scale * np.outer(g, A @ x)
grad_A = scale * np.outer(B.T @ g, x)
print(np.abs(grad_B).max() > 0, np.abs(grad_A).max())   # True 0.0

The LoRA paper chose this start on purpose, and it costs A only one step. It explains why a fresh adapter can look half idle on step one. And if you start B at random instead, the model no longer begins at its original behaviour.

Start with rank 8 or 16 on every linear layer, and tie alpha to the rank by one fixed rule. Change one setting at a time, and judge each change on examples the model never trained on.

Rank: how many patterns the adapter can learn

Rank, or r, sets the width of A and B. The LoRA paper found rank 1 enough to adapt two attention matrices of its 175-billion-weight model, on two benchmark test sets. That shows small ranks can work, not that they always will.

A 2024 study, LoRA Learns Less and Forgets Less, tested harder changes: code and maths on a 7-billion-weight model. Ranks 16 to 64 often fell short on code, while rank 256 matched full fine-tuning in one code setting.

Alpha: how strongly the change counts

Alpha sets the strength through alpha divided by r. The 2024 study recommends alpha equal to twice the rank, and found that choice mattered at high ranks. It also found LoRA's best learning rate, the size of each training step, about ten times higher than full fine-tuning's.

A 2023 paper on rank stabilisation argues that dividing by r slows learning for higher-rank adapters. Its fix, rsLoRA, divides by the square root of r instead. If a higher rank seems to add nothing, check how your setup scales alpha before you decide the task is simple.

Target layers: which matrices get adapters

The LoRA paper adapted only attention matrices, the parts that decide which earlier words matter, and froze the rest. The QLoRA paper found that adapters on all linear layers were needed to match full 16-bit fine-tuning in its tests.

The QLoRA authors found that the total number of adapters mattered most, and that rank did not affect results in their tests. The 2024 study found low ranks fell short on code, so the answer depends on how big the change is.

Here is the recipe the team follows. The results are made up, but the order of steps is the point:

1. 1

   Hold back a test set

   Keep 1,200 of the 12,000 pairs aside. The model never trains on them, so they show honest progress.

2. 2

   Score the untouched model

   The agronomists rate the base model's answers on the test set for correct advice and natural dialect. Every later run is compared with this score.

3. 3

   Train rank 8 on all linear layers

   One QLoRA run makes three passes over the training data, about 6 hours on the rented card (made up).

4. 4

   Try rank 16 and rank 32

   Change only the rank and keep the alpha rule fixed. Rank 16 beats rank 8, and rank 32 adds nothing, so the team keeps 16 (made up).

5. 5

   Check what the model forgot

   Run 300 general farming questions in standard Marathi. Those scores must not drop compared with the untouched model.

6. 6

   Save the adapter, not the model

   Store each small adapter file next to the exact version of the base model it was trained on.

Before the run, check that the base really is stored as NF4 with double quantization. After the run, test again if you merge the adapter into a different copy of the base.

Check the 4-bit type and settings

4-bit types differ, and library defaults can surprise you. In the open-source 4-bit library we checked, version 0.50.2, the 4-bit layer's default type is FP4, another 4-bit format, not NF4. Its default block size is 64.

That is from the source code of version 0.50.2 (released 27 August 2026), the latest release as of 21 October 2026, and defaults can change between versions. Set the type, double quantization and the compute type explicitly, pin your version, and log all three with each adapter.

NF4 matters because its levels sit where weights cluster. The code below builds the 16 NF4 levels with the same recipe as the open-source reference code, matching its table to 7 decimal places. It then rounds 1.28 million bell-shaped weights, in blocks of 64, to NF4 and to 16 evenly spaced levels.

import numpy as np
from statistics import NormalDist

ppf = NormalDist().inv_cdf
pos = [ppf(p) for p in np.linspace(0.9677083, 0.5, 9)[:-1]]
neg = [-ppf(p) for p in np.linspace(0.9677083, 0.5, 8)[:-1]]
nf4 = np.sort(pos + [0.0] + neg) / max(pos)   # 16 bell-curve levels
even = np.linspace(-1, 1, 16)                 # 16 evenly spaced levels

rng = np.random.default_rng(0)
w = rng.normal(0, 0.02, (20_000, 64))          # blocks of 64 weights
absmax = np.abs(w).max(axis=1, keepdims=True)  # one scale per block

def error(levels):
    x = w / absmax
    q = levels[np.abs(x[..., None] - levels).argmin(-1)]
    return np.abs(q * absmax - w).mean()

print(len(nf4), nf4[7], nf4[0], nf4[-1])
print(f"{error(even):.5f} {error(nf4):.5f} {error(even) / error(nf4):.2f}")

Expect 16 levels, with level 7 exactly 0.0 and the ends at -1.0 and 1.0. The errors print as 0.00171 and 0.00146, and the ratio as 1.17.

Merging a QLoRA adapter

A QLoRA adapter learned its change against the rounded 4-bit weights. If you merge it into the original 16-bit weights, each merged weight differs from what training saw by that weight's rounding error.

That gap is usually small, but it is not zero. Run your held-back test set again on the exact merged model you will serve.

You load one base model once and attach a different small adapter per request. If you only ever need one adapter, merge it into the base instead.

This is the pouch of lenses from earlier. The assistant keeps one adapter per way of speaking and one copy of the base model on the GPU. Start at the farmer in the diagram below.

Each adapter is about 80 MB at rank 16, so swapping is cheap. Three full copies of the 16-bit model would need 42 GB just for weights.

Serving many adapters well needs special handling. The S-LoRA paper keeps all adapters in main memory, fetches only those that running requests need, and batches requests for different adapters together. It reports serving thousands of adapters on one GPU.

The trade-off is speed per answer. An unmerged adapter adds a little work to every layer, since the small path runs beside the big one. The LoRA paper notes that merged weights make it hard to batch requests for different tasks together.

Skip LoRA when the problem is missing or fast-changing facts, when the new skill is very large, or when good prompts already work. Fine-tuning best changes how a model answers, not what it knows today.

Mandi prices, the daily crop prices at the market yard, are a good example. They change every morning, so the team fetches them at question time and adds them to the prompt.

Research backs this up. A 2024 study tested closed-book question answering, where the model answers without looking anything up. Fine-tuning examples with new facts were learned much more slowly than familiar ones. As the model did learn them, its tendency to make things up rose.

Large new skills are the second limit. The 2024 LoRA Learns Less study reports that, in standard low-rank settings, LoRA did substantially worse than full fine-tuning at code and maths. The gap was smallest for maths instruction tuning, training on maths questions paired with worked answers. The authors advise LoRA for instruction tuning, not for continued pretraining, which feeds a model billions of tokens of plain, unlabelled text.

The same study found an upside: LoRA forgot less of the base model's other skills, and how much it forgot depended on the rank. For a change of dialect and tone, that trade suits the assistant well.

The choice between prompting, looking things up and fine-tuning has its own guide. In short:

• Prompting fits when a few examples in the prompt already give good answers.
• Looking things up at question time fits when answers depend on documents or numbers that change.
• LoRA or QLoRA fits when you need a new tone, format or dialect, have a few thousand good examples, and have one GPU.
• Full fine-tuning fits when the new skill is large, you have lots of data, and you can pay for many GPUs.

Explain them as a memory problem with a clear fix: freeze the big model, train a small change, and shrink the frozen copy if you must. Then show you know when not to fine-tune.

1. 1

   State the constraint

   One GPU, a few thousand examples, and a change of tone or dialect. Say this first, because it decides everything else.

2. 2

   Rule out cheaper options

   Try prompting first. Use lookup at question time for changing facts. Fine-tune only for behaviour that prompts cannot fix.

3. 3

   Explain the adapter

   Frozen base weights, two thin matrices per layer, B starting at zero, and alpha divided by r as the strength.

4. 4

   Do the memory maths out loud

   About 16 bytes per weight for full fine-tuning with Adam, 2 bytes per frozen weight for LoRA, and about half a byte for QLoRA, plus activations.

5. 5

   Describe the evaluation

   A held-back test set, a score for the untouched model, and a check that general skills did not drop.

6. 6

   Plan serving

   One base model loaded once, one adapter per use case, and the base version stored with each adapter.