LLM Quantization: INT8, INT4 and Quality Loss

You pay for memory you cannot avoid. A model's weights sit in GPU memory the whole time it serves users. If they do not fit on one GPU, each copy is split across two or more, and the bill grows with it. LLM quantization, storing those weights with fewer bits, is the usual way out.

Picture a common setup. A team runs a Hindi customer-support chatbot for online shops, on GPUs rented by the hour. The setup is made up, but the problem is real.

The chatbot uses an open model with 14 billion parameters. A parameter, also called a weight, is one number the model learned during training. Each weight is stored in 16 bits, which is 2 bytes.

So the weights alone need 14 billion times 2 bytes, which is 28 GB. Each rented GPU, the graphics chip that runs the model, has 24 GB of memory. The model does not fit on one card.

The team splits each copy of the model across two GPUs. It runs four copies all day to cover peak traffic, so eight GPUs. At Rs 150 an hour each, that is Rs 8.64 lakh (864,000 rupees) a month (made up).

The product team asks a fair question. Can the same model run on one GPU instead of two, without customers noticing? The honest answer is probably, but only after testing.

LLM quantization means storing a large language model's weights, and sometimes its other numbers, with fewer bits than it was trained with. The model gets smaller and often faster, at the cost of some precision.

Precision means how finely a number can be written. A 16-bit floating-point number, stored a bit like scientific notation, has tens of thousands of possible values (63,487 finite ones in the common FP16 format). An 8-bit integer, called INT8, has 256. A 4-bit integer, called INT4, has just 16.

Think of a photo that a chat app compresses before sending. The file gets much smaller, and on a small screen it looks the same. Zoom into the small digits of a shop bill, though, and they are blurred.

Map it part by part. The photo's pixels are the model's weights. The compression level is the number of bits. The blurred digits are the hard cases, such as maths reasoning, which one study found degrades fastest.

Where the picture breaks: photo compression throws away detail your eye will not miss, and you look at the result once. A model's layers feed each other, so a rounding error in one layer is passed on to every layer after it.

There are two main ways to do it. Post-training quantization (PTQ) shrinks a model that is already trained. GPTQ, one PTQ method, quantized a 175-billion-parameter model in about four GPU hours.

Quantization-aware training (QAT) lets the model practise with low precision during training, so it learns around the rounding. It costs a training run, but one QAT study reported large gains over PTQ at low bit widths. Most teams start with PTQ, and so does this one.

It picks a scale, divides each weight by that scale, and rounds the result to the nearest whole number that fits. To use the weight later, it multiplies the whole number back by the scale.

Take five made-up weights: 0.12, -0.50, 0.03, 0.91 and -0.27. INT8 holds whole numbers from -128 to 127, and the simple symmetric version used here keeps -127 to 127. The scale stretches the biggest weight, 0.91, to exactly 127.

Here is the idea in plain Python, for 8 bits and then 4 bits. Real tools do this for billions of weights and add smarter tricks, covered later. The core step is the same.

def quantize(weights, bits=8):
    top = 2 ** (bits - 1) - 1                # 127 for 8 bits, 7 for 4 bits
    scale = max(abs(w) for w in weights) / top
    q = [round(w / scale) for w in weights]  # the lossy step
    return q, scale

def dequantize(q, scale):
    return [round(x * scale, 3) for x in q]

weights = [0.12, -0.50, 0.03, 0.91, -0.27]
for bits in (8, 4):
    q, scale = quantize(weights, bits)
    print(bits, round(scale, 4), q, dequantize(q, scale))

Run it with any Python 3 (we used 3.13); it needs no libraries. It prints the bits, the scale, the whole numbers and the values you get back. Expect exactly these two lines:

• 8 0.0072 [17, -70, 4, 127, -38] [0.122, -0.502, 0.029, 0.91, -0.272]
• 4 0.13 [1, -4, 0, 7, -2] [0.13, -0.52, 0.0, 0.91, -0.26]

Real tools rarely use one scale for a whole layer. Many give each small group of weights its own scale, and the AWQ paper and a large 2024 study both used groups of 128. Each group's scale fits its own values, so the rounding stays finer.

The diagram follows one group. Notice that the scale is stored next to the small numbers. Without it, the numbers mean nothing.

INT8 stores each weight in 8 bits and halves the weight memory, usually with very little quality loss. INT4 stores each weight in 4 bits and cuts it to about a quarter, but rounds each weight far more roughly.

The second printed line shows INT4 on the same five weights. The biggest whole number is now 7, so the scale is 0.13. The weight 0.03 becomes exactly 0, and -0.27 comes back as -0.26.

That is why INT4 needs smarter methods than plain rounding. Most INT4 setups quantize only the weights, and the numbers flowing between layers stay in 16 bits. This is written W4A16: weights in 4 bits, activations in 16.

Activations are the numbers the model produces inside itself while it reads your message. Weights are fixed after training. Activations change with every message, which makes them harder to squeeze.

Here is how the options look for the chatbot's 14 billion weights. The sizes are plain arithmetic; the GPU counts and bills are made up.

The INT4 size includes the scales. One 16-bit scale per 128 weights adds an eighth of a bit to each weight. Across 14 billion weights, that is about 0.22 GB.

You can check these sizes yourself. The snippet prints them in GB (billions of bytes) and in GiB (units of 1,073,741,824 bytes), which many tools show instead.

params = 14_000_000_000
for name, bits in (("16-bit", 16), ("INT8", 8), ("INT4", 4 + 16 / 128)):
    size = params * bits / 8
    print(f"{name}: {size / 1e9:.2f} GB = {size / 2**30:.2f} GiB")

It prints 28.00 GB = 26.08 GiB, then 14.00 GB = 13.04 GiB, then 7.22 GB = 6.72 GiB. If a tool shows a 28 GB model as about 26 GiB, nothing is missing.

The freed memory is not wasted. A running model also keeps a KV cache, the saved working notes for each chat in progress. More free memory holds more chats at once, which is why the made-up INT4 setup needs three GPUs, not four.

On broad benchmarks, well-made 8-bit and 4-bit models usually lose only a little. On hard tasks and some languages, the loss can be much bigger than the averages suggest.

A 2024 study, last revised in May 2026, ran over 500,000 evaluations on one open model family. 8-bit floating point (FP8) was effectively lossless. Well-tuned INT8, with activations also in 8 bits, lost about 1 to 3%. INT4 weight-only came close to the 8-bit formats.

FP8 stores numbers a bit like scientific notation. Some bits set the size, called the exponent, and the rest set the digits, called the mantissa. One common FP8 layout uses 4 exponent bits and 3 mantissa bits.

Now zoom in on the bill. A 2024 study of multilingual models found that quantization hurts languages unevenly, with non-Latin scripts hit hardest. Maths reasoning was also strikingly affected.

The same study found a gap between test scores and people. For Japanese, automatic tests showed an average drop of 1.7%. Human judges, reading answers to realistic prompts, saw a drop of 16.0%.

Hindi is written in Devanagari, also a non-Latin script. That study scored Hindi only with automatic tests, which it found can understate the harm. So treat Hindi as a risk to measure, not a known result.

This is exactly the chatbot's risk. Users write in Hindi, often mixed with English, and answers carry order numbers, refund amounts and return dates. Those are the small digits on the bill.

The team's first INT4 try uses plain rounding. On a public English benchmark, it scores almost the same as the original. On 600 saved Hindi support chats, the trouble shows up (made up):

• Refund amounts are sometimes copied wrong, such as Rs 1,499 turning into Rs 1,449.
• Long return-policy answers drift and repeat themselves near the end.
• Rare Hindi words for product parts come back as English guesses.
• Short greetings and order-status replies are just as good as before.

The second try uses an activation-aware INT4 method, explained in the engineer section, set up with 512 real Hindi chats. Most problems shrink. But in the 100 refund chats, INT4 still gets 3 amounts wrong that the 16-bit model got right (made up).

Run the same test set through the original and the quantized model, compare answers case by case, and group the results by category. Ship only if no category you care about gets meaningfully worse.

A test set is a fixed list of real inputs with known good answers. The chatbot's set has 600 chats, each tagged with a category, and 100 of them are refund chats.

The code below takes one score per answer from each model: 1 for correct, 0 for wrong. Any grader works, such as an exact-match check on amounts. The data is the made-up result of the second try.

from collections import Counter

def worse_rate_by_category(results):
    total, worse = Counter(), Counter()
    for category, full_score, small_score in results:
        total[category] += 1
        if small_score < full_score:
            worse[category] += 1
    return {c: worse[c] / total[c] for c in total}

# (category, score of 16-bit answer, score of INT4 answer); 1 = correct
results = ([("other", 1, 1)] * 500
           + [("refund", 1, 1)] * 97 + [("refund", 1, 0)] * 3)
rates = worse_rate_by_category(results)
overall = sum(small < full for _, full, small in results) / len(results)
blocked = [c for c, r in rates.items() if r > 0.02]  # 2% limit per category
print(f"overall: {overall:.1%} worse")
print(rates)
print("Ship" if not blocked else f"Hold, worse on: {blocked}")

Run it with any Python 3. It prints three lines: overall: 0.5% worse, then a rate per category with refund at 0.03, then Hold, worse on: ['refund'].

The 2% limit is a product call, not a law of nature. Some teams set a tighter limit for anything involving money.

Passing the test set is only the first gate. The diagram shows the full path from the quantized file to all users. Notice that every gate has a way back to the original model.

A canary is a release to a small slice of real users first, so problems hit few people. Here, 5% of chats go to the INT4 model while the rest stay on 16-bit. The two groups are compared side by side.

Automatic scores can underestimate the harm, as the multilingual study showed. So the team also asks two Hindi-speaking support agents to read 100 INT4 answers each week during the canary.

Outliers are a few values far bigger than the rest, and they ruin a shared scale. When one value is huge, the small values lose most of their detail, and many become zero.

Go back to the five weights and add a sixth of 8.0. The 8-bit scale jumps from 0.0072 to 0.063. Now 0.03 rounds to 0, and 0.12 rounds to just 2.

This is not just a toy problem. The LLM.int8() paper found systematic outlier features in transformers, the network design behind language models. Each layer passes lists of numbers to the next, and a dimension is one position in those lists. In the models it studied, outliers appeared suddenly between 6 and 6.7 billion parameters, up to 20 times larger than values in other dimensions.

Almost all of them. The code below runs that test, then repeats it with one scale per group of 128. It needs NumPy, a common Python library for maths on arrays (we used NumPy 2.5.4).

import numpy as np

rng = np.random.default_rng(0)
w = rng.normal(0, 0.02, size=4096)  # 4,096 made-up weights
w[100] = 1.0                        # one outlier, 50 times the spread

def int4(w, group):
    g = w.reshape(-1, group)
    scale = np.abs(g).max(axis=1, keepdims=True) / 7
    q = np.round(g / scale)
    return q, (q * scale).ravel()

for group in (4096, 128):
    q, back = int4(w, group)
    zeros = int(np.sum(q == 0))
    err = np.mean(np.abs(back - w))
    print(f"group {group}: {zeros} zeros, mean error {err:.4f}")

It prints two lines. The first is group 4096: 4093 zeros, mean error 0.0160, and the second is group 128: 711 zeros, mean error 0.0024.

Even without the outlier, 601 of these small weights round to zero at INT4 with groups of 128. Plain rounding at 4 bits is rough, which is why the methods below exist. Four well-known ones attack the problem in different ways.

• Keep the outliers in 16 bits. LLM.int8() does over 99.9% of the multiplication in 8 bits and only the outlier dimensions in 16. Its authors report no loss in quality for models up to 175 billion parameters.
• Scale up the important weights. AWQ (2023) found that protecting about 1% of weights, chosen by looking at activations rather than the weights, greatly cuts error. It protects them by scaling up their channels, the weights that meet one input dimension, not by keeping them in 16 bits.
• Move the difficulty. SmoothQuant (2022) shifts the hard part from activations to weights with a mathematically equivalent change. That lets both run in INT8, which its authors report gave up to 1.56 times faster inference.
• Round smarter. GPTQ (2022) works one layer at a time. After rounding each weight, it adjusts the weights not yet rounded to cancel the error, and its authors report 3 or 4 bits with negligible accuracy loss.

Most of these need calibration data: sample texts run through the model to see which values are large. The sets are small. The GPTQ paper used 128 text segments of 2,048 tokens each. In the AWQ paper's own test, AWQ with 16 such sequences beat GPTQ with 192.

AWQ also held up when calibration text and test text came from different sources. Its perplexity, a score of how surprised the model is by real text, rose only 0.5 to 0.6 points. GPTQ's rose 2.3 to 4.9.

Often yes, when few requests run at once, because writing each token is limited by reading memory, not by maths. Under heavy batching, the gain from weight-only INT4 shrinks.

A model writes its answer one token at a time. The chatbot's model is dense, meaning every weight is used for every token. So the GPU reads all the weights for each new token. Chats in the same batch share that one read.

The AWQ paper calls this generation stage memory-bound: the GPU spends its time waiting for data, not doing maths. At 16 bits the chatbot reads 28 GB of weights per token. At INT4 it reads about 7.2 GB.

GPTQ's authors measured the gain at a batch of one. A 175-billion-parameter model in 3 bits ran about 3.25 times faster per token on one high-end GPU than in 16 bits on five. On cheaper GPUs, two in 3 bits against eight in 16 bits, the gain was about 4.5 times.

The picture changes when the GPU is busy. Servers handle many chats at once by batching, which means running them through the model together. With a big batch, each weight read feeds many chats, so maths becomes the limit.

Weight-only INT4 still does its maths in 16 bits, after turning each weight back. So its gain shrinks under heavy load. W8A8, with weights and activations both in 8 bits, can use faster 8-bit maths instead, on GPUs that have it.

The 2024 study measured this on a late-2024 version of a popular open-source serving engine. W4A16 was the most cost-efficient when one request runs at a time. For its 8 and 70 billion parameter models, it cut cost per query 2 to 3 times compared with 16 bits.

W8A8 did best with continuous batching, where requests are mixed in all the time. Which 8-bit type you get depends on hardware: the study used FP8 on newer GPU generations with FP8 maths units and INT8 on older ones.

Batching and the KV cache are covered in the guide on LLM inference system design. For quantization, one rule is enough: measure speed at your real traffic level, not with one test chat.

Choose INT8 when quality matters most and it already fits your memory. Choose INT4 when memory or cost is the hard limit and your own tests show the loss is acceptable.

Follow the diagram from the top. Each question is one the team asked, in this order.

The team ends up with a split. Greetings and order-status chats, most of the traffic, go to two GPUs running INT4. Refund and policy chats go to one GPU running INT8, where amounts stayed exact in testing (made up).

That is three GPUs, about Rs 3.24 lakh a month, with money answers on the safer copy. The 16-bit model stays ready as the way back.

There is one more option the team skipped for now. Instead of shrinking a big model, you can switch to a smaller model trained to be small. That is a different trade, and it needs the same testing.

Explain it as a memory and cost decision first, then show you know where quality breaks and how you would prove it did not. These six steps keep the answer in order.

1. 1

   Start with memory maths

   Parameters times bytes per weight. A 14-billion-weight model needs 28 GB in 16 bits, 14 GB in INT8 and about 7.2 GB in INT4, scales included.

2. 2

   Add the KV cache

   Say that freed memory holds more chats at once. That, not just the smaller file, is where much of the saving comes from.

3. 3

   Pick the format for the traffic

   Weight-only INT4 suits one request at a time. 8-bit weights and activations suit heavy batching. Mention that your GPU decides which 8-bit type you can use.

4. 4

   Name the quality risks

   Outliers, maths reasoning, exact numbers and non-Latin scripts. Mention methods that keep outliers in 16 bits or scale up important weights.

5. 5

   Describe the test plan

   A category-tagged test set on both models, a limit per category, human readers for your main languages, then a canary.

6. 6

   Keep a way back

   Keep the 16-bit model ready, and send risky request types to the safer copy if needed.