Labeling and Active Learning: Ask Where the Model Is Unsure

Most of the expert's time goes on examples the model already gets right. The rare and confusing cases it actually needs stay unlabelled. Active learning fixes this by letting the model choose what gets labelled next.

Picture a common setup. A crop-disease app lets grape and onion farmers around Nashik photograph a sick leaf and get a likely disease with advice. The setup is made up, but the problem is real.

The team has 2.4 lakh (240,000) leaf photos from two seasons, and only 3,000 have labels. A machine learning model learns only from labelled examples. So every new label has to come from someone who can tell the diseases apart.

That person is the team's agronomist, a crop scientist who knows plant diseases by sight. She can label about 300 photos on a good day, and her hours cost about Rs 1,200 each. At that pace, the other 237,000 photos would take 790 working days, well over two years.

So the first plan is simple: send her 300 random photos a day. After three weeks, three problems show up.

• Most random photos are clear cases, like a healthy grape leaf in good light. The model already gets these right, so their labels teach it almost nothing.
• Rare diseases barely show up. In 6,000 random photos, anthracnose on grapes appears only 40 times, so the model stays weak on it.
• The model's mistakes cluster in a few places, like early downy mildew against powdery mildew. Random picking sends very few of those to the agronomist.

The agronomist puts it plainly: she is spending her day on photos a farmer could label. The team needs a way to send her only the photos where her answer will change the model.

Active learning is a way of training a model where the model itself chooses which unlabelled examples a person should label next. It usually picks the examples it finds hardest, so each label teaches it more.

Burr Settles' 2009 survey of the field states the key hypothesis. A learning algorithm can reach better accuracy with fewer labelled examples if it may choose the data it learns from. Note the word hypothesis: it often holds, not always.

Picture a pearl grading shop in Hyderabad's old city. A junior grader sorts a heap of pearls by size, shape and shine. He handles the obvious ones himself, round and bright or clearly flawed.

Only the doubtful pearls go to the master grader, whose time is scarce and costly. Those are the ones where the junior hesitates between two grades. Each answer from the master sharpens the junior's eye for the next heap.

The model is the junior grader, and the agronomist is the master. The heap is the pile of unlabelled photos, and the master's answers are the labels. Photos the model already handles well never reach her desk.

Where the picture breaks: a junior grader knows when he is guessing. A model can be confidently wrong, and then it never asks. Later sections add checks for exactly that.

The idea is old. In 1994, David Lewis and William Gale introduced uncertainty sampling for sorting newswire stories into topics. On their task, it cut the hand-labelled data needed for the same quality by up to 500 times.

That was one result on one text task. On your own data the saving may be large, small, zero or even negative. That is why this guide measures active learning against random picking, with code you can run.

You need five words: pool, annotator, prediction probability, query strategy and round. Each gets a plain meaning and an example from the crop app.

• Pool: the large set of unlabelled examples you could send for labelling. Example: the 237,000 farmer photos with no disease name yet.
• Annotator: the person who supplies labels. Research papers often call this person the oracle. Example: the agronomist, and later two junior agronomists she trains.
• Prediction probability: a classifier, a model that sorts inputs into fixed groups, gives a probability for each group. Example: 48 percent downy mildew, 45 percent powdery mildew, 7 percent healthy.
• Query strategy: the rule that decides which pool examples to send next. Example: send the 300 photos where the top two diseases are closest.
• Round: one turn of the loop, which is pick, label, retrain and score again. Example: the team runs one round per working day.

The active learning loop trains a model on the labels you have, scores the unlabelled pool, and sends the most useful examples to a person. Then it adds their labels and repeats. Settles calls this setting pool-based active learning.

In the diagram below, start at the labelled photos and follow the arrows around. Each arrow is one step of a daily round.

Here is how the team sets it up, step by step. The first step matters more than it looks.

1. 1

   Set aside a test set first

   Pick 1,000 photos at random and have the agronomist label them. They never enter training. They tell you honestly how good the model is.

2. 2

   Start with a seed set

   Train the first model on the 3,000 labelled photos. This is the seed. A model trained on too few labels cannot judge its own doubt well.

3. 3

   Score the pool

   Run the model on all unlabelled photos overnight. Save a doubt score for each one.

4. 4

   Pick and send a batch

   Choose 300 photos that are doubtful and varied. Send them to the labelling tool, where the agronomist sees one photo at a time.

5. 5

   Retrain and measure

   Add the new labels, retrain, and check accuracy on the test set. Then start the next round.

A model measures its doubt from its prediction probabilities. The three common measures are least confidence, margin and entropy. Each reads those probabilities in a slightly different way.

• Least confidence: one minus the top probability. If the top guess is only 40 percent, the score is 0.60. Higher means more doubt.
• Margin: the top probability minus the second one. A small gap means the model is torn between two answers. Lower means more doubt.
• Entropy: a single number for how spread out all the probabilities are, measured here in bits. It is highest when every class looks equally likely. Higher means more doubt.

For a yes or no question, all three pick the same example: the one closest to 50 percent. Settles' survey states this for least confidence and entropy, and margin follows the same way. We checked all three on 10,000 random two-class trials, and they always agreed.

With more classes, they can disagree. Pause and predict: in the table below, which photo would each measure send first?

Photo A is easy, and every measure agrees to skip it. Margin sends photo B first, because the model is torn between exactly two mildews. Least confidence and entropy send photo C first, because its doubt is spread wider.

Which is better depends on what confuses your model. This model mostly mixes up pairs of diseases, so the team chooses margin. Photo B is exactly the kind of photo the agronomist can settle in seconds.

Here is the scoring code; if you do not code, skip it, because the table already shows the idea. probs is a table with one row per photo and one column per disease, produced by the model. It uses NumPy, a common Python library for working with tables of numbers.

import numpy as np

def margin_scores(probs):
    # probs: one row per photo, one column per disease
    top_two = np.sort(probs, axis=1)[:, -2:]
    return top_two[:, 1] - top_two[:, 0]   # small gap = unsure

def most_doubtful(photo_ids, probs, k=1500):
    gaps = margin_scores(probs)
    order = np.argsort(gaps)               # smallest gap first
    return [photo_ids[i] for i in order[:k]]

Two lines do the real work. The np.sort line sorts each row, so the last two columns hold the top two probabilities. np.argsort then puts the photos in order from smallest gap to largest.

Notice k is 1,500, not 300. The team takes five times more candidates than it needs, then picks a varied 300 from them. A later section explains why.

Often, but you only know by measuring both on the same budget. Here is a check you can run in under a minute. It uses scikit-learn, a widely used Python machine learning library, and its built-in digits dataset.

The dataset holds 1,797 small 8 by 8 pixel images of handwritten digits, 0 to 9. The script keeps 500 as a random test set. It starts from 20 random labels and adds 10 per round for 18 rounds, so 200 labels in all. The model is logistic regression, a simple classifier that turns weighted sums of pixel values into a probability for each digit.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X, y = load_digits(return_X_y=True)        # 1,797 tiny images of the digits 0 to 9
X_pool, X_test, y_pool, y_test = train_test_split(
    X / 16, y, test_size=500, random_state=0, stratify=y)

def fit(idx):
    return LogisticRegression(max_iter=2000).fit(X_pool[idx], y_pool[idx])

def run(strategy, seed, rounds=18, batch=10):
    rng = np.random.default_rng(seed)
    labelled = list(rng.choice(len(X_pool), 20, replace=False))   # random seed set
    for _ in range(rounds):
        model = fit(labelled)
        rest = np.setdiff1d(np.arange(len(X_pool)), labelled)
        if strategy == "random":
            pick = rng.choice(rest, batch, replace=False)
        else:
            top_two = np.sort(model.predict_proba(X_pool[rest]), axis=1)[:, -2:]
            pick = rest[np.argsort(top_two[:, 1] - top_two[:, 0])[:batch]]
        labelled += list(pick)
    return fit(labelled).score(X_test, y_test)   # accuracy on the random test set

for strategy in ["random", "margin"]:
    accs = [run(strategy, seed) for seed in range(5)]
    print(strategy, "accuracy with 200 labels:", round(np.mean(accs), 3))
print("all 1,297 pool labels:", round(fit(np.arange(len(X_pool))).score(X_test, y_test), 3))

On scikit-learn 1.9.1 and NumPy 2.5.4, the latest releases as of 19 October 2026, it prints three lines. Random picking reaches 0.919 accuracy with 200 labels, and margin sampling reaches 0.96. Training on all 1,297 pool labels gives 0.968.

What just happened? With 200 labels, margin sampling came within 0.01 of a model that saw all 1,297. Random picking was 0.04 behind margin on this split, and 0.02 to 0.03 behind on three other splits we tried. That gap is the value of choosing.

We also ran a longer test: ten repeated runs with different random starts, 40 rounds, and all three doubt measures. The table shows mean test accuracy as the number of labels grows.

Two things stand out. Random picking had still not reached margin's 200-label accuracy at 420 labels. Yet at 70 labels, least confidence was level with random and entropy was behind it. Not every doubt score helps early.

This is one small, clean dataset with one simple model. Your own gains could be larger, smaller or negative. Run the same comparison on your data before you build anything around it.

Start with uncertainty sampling, because it is simple and needs only the probabilities your model already gives. Move to a committee or a diversity-aware method when your own results show its weak spots.

Uncertainty sampling

This is what the scoring code does: one model, one doubt score per example. Settles' survey calls it perhaps the simplest and most commonly used query framework. The digits test above is uncertainty sampling with margin.

Query by committee

Query by committee trains several models on the same labels, then asks about the examples where they disagree most. Each model is a committee member that votes on the disease.

The team could train three models with different random starting weights. If all three say powdery mildew, the photo is skipped. If they split three ways, the agronomist gets it. The cost is training and running three models instead of one.

Diversity and representative picking

Diversity methods pick examples that cover the pool well, not just the doubtful ones. They need an embedding: a list of numbers that describes a photo, where similar photos get similar lists.

Sener and Savarese reported in 2017 that many existing active learning rules worked poorly for convolutional neural networks, a common model type for images, when labels came in batches. They proposed picking a core set, a small subset chosen so a model trained on it works well on the rest.

Mixing doubt and variety

BADGE, a 2019 method by Ash and colleagues, combines both ideas. It describes each example by a gradient, how the model would change if it learned that example with its own guessed label. It then picks points that are far apart and large in that space, to cover doubt and variety together.

Its authors report that it needs no hand-tuned settings to balance the two. They also report it consistently performs as well as or better than the other methods they tested, across batch sizes and model architectures.

A batch should be varied because the most doubtful examples are often near-copies of each other. Labelling all of them teaches the model the same lesson many times.

Back in the pearl shop, imagine the junior sets aside 20 doubtful pearls, all of the same odd pinkish lustre from one lot. The master's first answer settles all 20. The other 19 questions waste his time.

The crop team sees this in week four. One farmer sent 140 of the 300 picked photos. They show the same vine, shot again and again in the same evening light.

Settles' survey names the problem. Taking the top N examples by doubt often does not work well, because it ignores the overlap in what those examples teach. Batch methods add variety on purpose.

We reproduced it on the digits data. We added 40 near-copies of the single most confusing image to the pool. Then 39 of the top 50 margin picks were copies of that one image.

The team's fix is a simple filter. It walks the 1,500 doubtful candidates from most doubtful down. It keeps a photo only if its embedding is far enough from every photo already kept.

import numpy as np

def pick_varied_batch(candidates, embed, k=300, min_dist=0.3):
    kept = []                          # (photo_id, embedding) pairs
    for photo_id in candidates:        # most doubtful first
        vec = embed(photo_id)
        if all(np.linalg.norm(vec - v) >= min_dist for _, v in kept):
            kept.append((photo_id, vec))
        if len(kept) == k:
            break
    return [photo_id for photo_id, _ in kept]

embed is a stand-in for code that turns a photo into its embedding. np.linalg.norm measures the distance between two embeddings. A photo too close to one already kept is skipped, so near-copies drop out.

In the same digits test, with raw pixels as the embedding, the filter kept only 1 copy in a batch of 50. The copies sat at most 0.115 apart, while typical different images sat about 3.06 apart.

min_dist depends on your embedding, so 0.3 is not a universal value. The team tunes it by eye, looking at pairs of photos at different distances. They pick the point where pairs stop looking like the same leaf.

This one-pass filter is simpler than BADGE or core sets, and it can still keep a doubtful outlier. It does fix the one-vine problem.

A labelling pipeline with active learning has five main parts. They are a photo store, a scoring job, a labelling queue, a labelling tool for experts, and a label store that feeds training.

In the diagram below, follow a photo from the farmer's phone to the next model. The scoring job runs at night, so labelling during the day never waits on it.

Four design choices make this pipeline trustworthy. Each one fixes a problem the team actually hit.

• An unclear button. Some photos are too blurry for anyone. The agronomist marks them unclear, and they leave the pool for good, instead of returning every night.
• Two labels on a sample. One photo in ten goes to a second agronomist. If they disagree often on a disease, the label rules need fixing before the model does.
• Who and when on every label. If a labeller turns out to be wrong about one disease, the team can find and recheck every photo they labelled.
• A small random slice. One in ten photos in each batch is picked at random, not by doubt. This keeps easy and common cases in training too.

Some teams also let confident predictions label themselves, a semi-supervised trick often called pseudo-labelling. Semi-supervised means the model learns from labelled and unlabelled data together. It adds labels without anyone checking them. If the model is wrong but sure, those errors go straight into training.

This team does not let confident photos skip people entirely. The diagram below shows the rule it uses to sort each new farmer photo after the app replies. Photos that skip the queue stay in the pool, and the nightly job re-scores them with each newer model.

Active learning can go wrong by picking junk, by skewing your data, and by hiding its own failure behind bad test numbers. Each problem has a cheap guard.

• Picking outliers. The most doubtful photo may be a thumb, a tractor or a cow. Settles' survey notes the suggestion that uncertainty methods are prone to querying outliers. Guard: the unclear button, plus a filter for non-leaf photos.
• Skewed training data. Picked labels are not a random sample of real traffic. In our digits test, a full model was 0.72 sure on average on picked images, against 0.91 on the whole pool. Guard: the random slice in every batch.
• Picks tied to one model. Settles' survey warns that actively chosen labels are tied to the model that chose them. They may suit a different model less well. Guard: keep the random test set, and rerun a random baseline when you change models.
• A cold start. With too few seed labels, the model's doubt points at the wrong examples. Guard: label a large enough random seed set before turning on active picking. Covering every class alone did not fix it in our test.
• Uneven labelling cost. Settles' survey notes that labelling cost can vary a lot from one example to the next. Hard photos may take the agronomist far longer. Guard: track time per label, and use the random slice to break up hard runs.

We saw the cold start trap in our own test. On the digits data, a seed of 10 labels plus one margin batch of 50 scored 0.756 on average. Random picking scored 0.812 and won 22 of 30 trials. A 10-label seed with one image of every digit still lost: margin won only 7 of 30.

With a seed of 100 labels, the same margin batch won 25 of 30 trials, 0.925 against 0.911. Picture the new junior grader on his first morning: every pearl looks doubtful, so his doubts point nowhere useful.

Gal, Islam and Ghahramani raised a related point in 2017. Many active learning rules depend on model uncertainty, yet deep learning methods rarely represented it. They used Bayesian convolutional networks, which estimate doubt more carefully, and tested them on digits and skin lesion images.

When should you stop labelling?

Stop when new labels stop moving accuracy on your random test set. The team plots test accuracy after each round. After round 30, the line has moved less than half a point in a week.

Here is the team's result, made up for this example. With 12,000 labels in total, seed included, the model matches what 30,000 random labels gave in a side test. That saves the agronomist about 60 working days, and her daily batch drops to 100.

Explain it as a loop that spends a fixed labelling budget where the model is least sure. Back each choice with the problem it solves.

1. 1

   State the budget

   Start with the numbers: how many unlabelled examples, how many labels per day, and who supplies them. Costly expert labels make active learning worth it.

2. 2

   Draw the loop

   Seed set, train, score the pool, pick a batch, label, retrain. Say how often a round runs and where scoring runs.

3. 3

   Pick a doubt measure

   Name least confidence, margin or entropy, and say why yours fits. Margin suits models that confuse pairs of classes.

4. 4

   Make batches varied

   Explain that the top N doubtful examples are often near-copies. Add a distance filter on embeddings, or name a method that mixes doubt and variety, like BADGE.

5. 5

   Add the guards

   A random test set, a random-picking baseline, a random slice in each batch, two labels on a sample, an unclear option, and a stopping rule.