Fine-Tuning Case Study: A Model for Hospital Billing Codes

It picks codes that look right but are wrong in small, costly ways. In a 2024 Mount Sinai study, general large models given a code's description wrote back the exact code less than half the time. This fine-tuning case study follows one team that had to do better.

A billing code is a short label for one diagnosis or one procedure. Insurers pay hospitals based on these codes. A wrong code means a rejected claim, a late payment, or money the hospital must return later.

Diagnoses are often coded with ICD-10, the World Health Organization's list of diseases. For example, K80.2 means a gallstone without an inflamed gallbladder. Procedures are often billed from a hospital's or insurer's list of packages, each with its own code.

In the Mount Sinai study, models got each code's official description and had to write the code. Every model scored below 50% exact match, meaning the code it wrote was exactly the original code. The best one matched about 34% of ICD-10-CM codes, the US adaptation of ICD-10.

Many misses were near-misses: a related code that was less precise. The researchers themselves note the task was artificial, and real notes may be harder. Knowing medicine in general is not the same as coding like a trained coder.

Picture a common setup. A health-tech team in Lucknow builds software that suggests billing codes for three hospital chains. The setup is made up, but the problem is real. The chains send about 9,000 discharge notes a day.

A discharge note is the summary a doctor writes when a patient leaves hospital. It says what was wrong, what was done and which medicines to take at home. Coding one from scratch takes a trained coder about 12 minutes (made up).

At 9,000 notes a day, that is 1,800 coder-hours a day, or about 240 coders on 7.5-hour shifts. The team's job is to suggest codes, so coders mostly check instead of starting from a blank screen.

The first try: RAG over the code lists

The first version used RAG. A search step finds the most relevant documents, the program pastes them into the prompt, and the model answers from them. The prompt is the text your program sends to the model.

The team's store held every code with its official description, plus each chain's coding rules. For each note, the system found the 30 closest code descriptions and asked a large general model to pick the right ones.

The team tested it on 2,000 notes that senior coders had already coded. It got every code right on 58% of notes (made up). The lead coder read the misses one by one and found three patterns.

• Shorthand. Doctors write lap chole for keyhole gallbladder removal, and k/c/o DM for a known case of diabetes. The search matched these to the wrong descriptions, or to none.
• Order. The principal diagnosis, the main reason for the stay, must come first. The model often listed codes in the order the note happened to mention them.
• House rules. One chain bills a hernia repair with mesh as one package, and another bills the mesh separately. The rules sat in the prompt, but the model applied them unevenly.

The team kept adding rules to the prompt. It grew to about 6,000 tokens per note, where tokens are the small chunks of text that model services count and charge for. Each note cost Rs 1.20 and took 5 seconds (made up), and accuracy barely moved.

It looks like a narrow, high-volume task with a fixed list of answers and years of labelled examples. In that setting, training a model once can beat searching for context on every request.

Fine-tuning means taking a model that already understands language and training it a little more on your own examples. Each example pairs an input with the exact output you want. Here, the input is a discharge note and the output is its codes.

The team had something many teams lack: a large pile of labelled data. Labelled data means examples where an expert has already written the right answer. The chains had coded every note for four years, and senior coders had audited a share of them.

The team chose a small open-weight model, one whose weights you can download and run on your own servers. Weights are the numbers inside a model that decide how it behaves. Patient notes then never leave the team's servers.

Think of a young tabla player doing riyaz, the daily practice, under a guru. Here is how that picture maps to this case.

• The written bols, the spoken syllables of a rhythm, are the code descriptions RAG pastes into the prompt. Reading them mid-concert helps a little, but the hands still stumble.
• Years of riyaz are the thousands of coded notes the model trains on.
• The guru's corrections are the senior coders' audits, which decide which examples count as right.
• Playing without the page is the fine-tuned model coding a note from a short prompt.
• Where the picture breaks: a tabla player can read a new composition on the spot. This fine-tuned model sees only the note, so it has no reliable way to use a code its training examples never showed.

Fine-tuning beats RAG here because the hard part is a learned habit, not a missing fact. The RAG version could often already see the right code descriptions. What it lacked was the judgement the senior coders had built over years.

• The answer list is closed. Every valid code already exists, so the model must choose well, not find new information.
• The knowledge lives across examples. No single document says that lap chole plus a stone on the scan means K80.2 plus the chain's gallbladder package. Thousands of past notes show it.
• The output has a strict shape: codes in one format, principal diagnosis first. Training turns that shape into a habit, so the long prompt of rules can go.
• The volume is high. At 9,000 notes a day, a short prompt to a small model saves real money and waiting time.

What just happened? Cost and time fell the most, because the prompt shrank about seven times and a small model answers faster. Accuracy rose because habits moved from the prompt into the weights.

The last row is the catch. Fine-tuning won this case, but it has a real weakness, and much of the later design exists to cover it.

So fine-tune for skills and output formats, and keep facts that change in a lookup. In that study, accuracy on new facts rose steadily as training showed more rewordings of each fact, but the authors say this still needs more research.

Choose RAG when answers depend on facts that change often, when you must cite a source, or when you have few labelled examples. Many real systems use both, as this team ends up doing.

These signs point toward RAG:

• The answer is written in a document, such as a policy or a price list.
• The documents change weekly, and an old answer would do harm.
• Users need to see where an answer came from.
• As a rough guide, you have hundreds of labelled examples, not tens of thousands.

These signs point toward fine-tuning:

• The task is a repeated skill with a fixed set of answers, like sorting, tagging or coding.
• Experts have already labelled thousands of past cases, and you can tell which labels to trust.
• Your prompt keeps growing with rules, and accuracy stops improving.
• Volume, speed or privacy makes a small model on your own servers worth the training effort.

Always try a plain prompt first. It is the cheapest way to learn how hard the task is, and it gives you a baseline to beat. The RAG version was not wasted either: it gave the team a measured baseline and a list of failure patterns to fix.

You build it from past notes that experts already coded, cleaned so the model learns only from answers you trust. A tabla student who practises a wrong stroke ten thousand times learns the wrong stroke perfectly, and bad labels work the same way.

1. 1

   Collect past notes

   Four years at 9,000 notes a day is about 1.3 crore (13 million) coded notes, each with the codes that were billed.

2. 2

   Keep only audited labels

   Senior coders had audited about 2% of notes, roughly 260,000 (made up). Only these became the first training set. A paid claim was not treated as proof, because a claim can be paid even when a code is missing.

3. 3

   Remove patient identity

   Names, phone numbers and patient IDs became placeholders such as PATIENT_1. The model needs the medicine in the note, not the person.

4. 4

   Split by date

   Audited notes from the last three months became the test set, held back from training. The model trains on older notes and is scored on newer ones, as it will be in real use. A second score uses only patients missing from training.

5. 5

   Check the rare codes

   The lead coder listed rare but costly codes, such as some cancer surgeries. Each one either had enough examples or was marked so that any note where the model suggests it goes to a person.

The code below turns audited rows into training examples. The output file has one JSON example per line, a format often called JSON Lines. The two sample rows at the bottom let you check it.

import json

def write_training_file(rows, path):
    kept = 0
    with open(path, "w", encoding="utf-8") as f:
        for row in rows:
            if row["audit_status"] != "approved":
                continue  # never train on codes nobody checked
            example = {
                "input": row["masked_note"],
                "output": " | ".join(row["codes"]),  # principal first
            }
            f.write(json.dumps(example, ensure_ascii=False) + "\n")
            kept += 1
    return kept

rows = [
    {"audit_status": "approved", "codes": ["K80.2", "PKG-104"],
     "masked_note": "PATIENT_1, 46 F. Stone on scan. Lap chole done."},
    {"audit_status": "pending", "codes": ["K80.2"],
     "masked_note": "PATIENT_2, 51 M. Lap chole planned."},
]
print(write_training_file(rows, "train.jsonl"))
print(open("train.jsonl", encoding="utf-8").read(), end="")

Run it with Python 3 and you should see exactly these two lines: the count of kept notes, then the one example written to train.jsonl.

1
{"input": "PATIENT_1, 46 F. Stone on scan. Lap chole done.", "output": "K80.2 | PKG-104"}

Two lines carry the design. The audit check drops the second note, which no senior confirmed. The join keeps the codes in billing order, so the model learns which code goes first, not just which codes appear.

That toy is not a language model, but the trap is the same. A date split stops the model training on the future, as in real use. It does not stop near-copies, because readmissions weeks apart land on both sides of the cut.

When the toy spread each patient's notes over months, a date split scored 79%, while notes from patients it never trained on scored 56%. So report the date-split score next to an unseen-patient score.

Rare codes need extra care. A 2023 review retrained six published coding models on MIMIC-IV, a de-identified dataset from one US hospital, and found all of them struggled with rare codes.

The strongest of the six never correctly predicted 2,928 of the 5,794 ICD-10 diagnosis codes in the authors' data split. So when this model suggests a rare, costly code, a person always checks it.

The harder case is a rare code the model leaves out. If that note passes every other check, only the fast-lane coder's glance can catch it before the bill, and the 2% audit may find it later. So before routing, the team also runs a plain keyword search: notes that mention cancer or major surgery go straight to full review.

You use a method such as LoRA, short for low-rank adaptation. LoRA freezes the original weights and trains two small extra matrices beside chosen weight matrices. A matrix is a grid of numbers.

Full fine-tuning changes every weight in the model. That needs a lot of GPU memory, the memory on the chips that do the heavy maths, and a full copy of the model per version. LoRA leaves the base model untouched, and only the small add-on, called an adapter, learns.

The maths you can check

Take one weight matrix of 4,096 by 4,096 numbers. Full fine-tuning trains all 16,777,216 of them. LoRA with rank 8 trains a 4,096 by 8 matrix and an 8 by 4,096 matrix, where rank is the width of these thin matrices.

Check it: 2 times 4,096 times 8 is 65,536 numbers, which is 256 times fewer for that one matrix. Multiplying the two thin matrices gives a full-size change that is added to the frozen weights.

The paper starts one thin matrix at zero, so at the start of training the model behaves exactly like the base model. In most experiments it adapted only two attention weight matrices, query and value, where attention is the part of each layer that decides which words to look at.

After training, you can add the change into the original weights once. The merged model then runs with no extra delay. Or you can keep adapters separate and swap them, for example one per hospital chain on one base model.

What the original paper reported

The 2021 LoRA paper compared it with full fine-tuning of a 175-billion-weight model using Adam, a common optimiser, meaning the rule that updates weights during training. It reported about 10,000 times fewer trainable weights.

Training memory fell from 1.2 TB to 350 GB, and a checkpoint, the saved file of trained weights, shrank from 350 GB to 35 MB. Quality was on par or better on the four model families tested, on general language benchmarks, meaning standard public test sets, not medical coding. Test on your own notes.

For this team, one training run took about a day on two rented GPUs (made up). Each adapter file was small, so the team kept every version and could roll back in minutes. The diagram shows the monthly training loop.

When an adapter failed, the reason was usually in the data, not the training settings. Twice, the lead coder found that one chain had changed how it billed a procedure partway through a year, so the older labels had to be fixed.

You check every code in plain code against the current official lists, then send any note the model is unsure about to a human coder. The model suggests codes, and rules and people decide.

What the confidence score really measures

A model writes its answer one token at a time and gives each token a probability. The probability of the whole answer is all those token probabilities multiplied together.

Only the short one. 0.99 to the power 10 is 0.904, but 0.99 to the power 30 is 0.740. A whole-answer score punishes long answers, so notes with many codes would all land in full review.

So the team scores each code separately, multiplying only the token probabilities inside that code, and passes the lowest code score as confidence. The 0.90 cutoff was picked by testing several values on held-back notes, because these scores are only a rough signal. Codes that take more tokens still score a little lower, so check the cutoff for each kind of code.

The routing function

In the code below, valid_codes is a set built from today's ICD-10 list and each chain's package list. The function returns where the note should go, and why. The four calls at the bottom let you check it.

ALWAYS_HUMAN = {"C15.3", "C25.0"}  # rare, costly cancer codes

def route(codes, confidence, valid_codes):
    if not codes:
        return "full_review", "model gave no codes"
    unknown = [c for c in codes if c not in valid_codes]
    if unknown:
        return "full_review", f"not on any current list: {unknown}"
    if any(c in ALWAYS_HUMAN for c in codes):
        return "full_review", "rare code, always checked by a person"
    if confidence < 0.90:
        return "full_review", "model is unsure"
    return "fast_lane", "passed every check"

valid = {"K80.2", "E11.9", "C15.3", "C25.0", "PKG-104"}
print(route(["K80.2", "PKG-104"], 0.95, valid))
print(route(["K80.2", "K80.9X"], 0.97, valid))
print(route(["C25.0"], 0.99, valid))
print(route(["K80.2", "E11.9"], 0.82, valid))

Run it with Python 3.6 or later, which f-strings need, and you should see these four lines.

('fast_lane', 'passed every check')
('full_review', "not on any current list: ['K80.9X']")
('full_review', 'rare code, always checked by a person')
('full_review', 'model is unsure')

What just happened? The first note passed every gate. The second had K80.9X, which is not on the list, so an exact set lookup caught it. The third was confident but rare. The fourth was valid but under 0.90.

The order of checks matters. An invented code is caught first, by a lookup the model cannot argue with. Rare codes come next, so even a confident model never bills a rare code it suggested without a person. Confidence is the last gate, not the only one.

In the first month, about 70% of notes went to the fast lane, and coders cleared each in about a minute (made up). The other 30% still took about 12 minutes each.

Check the maths: 6,300 fast notes take 105 coder-hours, 2,700 full reviews take 540, and auditing 2% of the fast lane adds about 25. That is about 670 coder-hours a day instead of 1,800.

You keep the code lists outside the model and retrain on a schedule. The prompt here holds only the note, so the model has never seen a brand-new code used. This is where a lookup still earns its place.

Code lists change. In this setup, the chains add and retire packages every few months. Diagnosis lists change too: the WHO's ICD-11, the successor to ICD-10, came into effect on 1 January 2022, and moving to it would mean a new list.

• The code lists live in a versioned table, outside the model. The route check always uses today's version, so a retired code is caught at once.
• When a chain renames a package, a small mapping in code turns the old code into the new one until the next retraining.
• When a truly new code appears, notes that may need it go to full review. The coder screen shows the new code's description, found by a simple search.
• Retraining runs every month on the latest audited notes. Once coders have used a new code in enough audited notes, the model learns it like any other.

So the final system is not fine-tuning instead of retrieval. It is fine-tuning for the skill, plus plain lookups for the facts that change. The model never has to remember which codes exist today.

Either can work, and they fail in different ways. This team fine-tuned a model to generate codes as text. The six research models in the 2023 review are instead classifiers that score every code on a fixed list.

Each one outputs a separate probability for every code, and a cutoff decides which codes to keep. The table compares the two designs.

The team picked generation because order and house rules mattered, and the list check closes its main gap. If your task has no order and a very large code list, a classifier is worth testing on the same held-back notes.

Explain it as a choice backed by evidence: start with the cheaper option, show where it fails, then show why training fixes those failures and what guards the new risks.

1. 1

   Describe the task

   Discharge notes in, billing codes out, from a fixed list, in a fixed order, at about 9,000 notes a day.

2. 2

   Start with a baseline

   Say you would try prompting and RAG first, and measure them on notes that experts already coded.

3. 3

   Name the failure patterns

   Shorthand, code order and house rules. Explain that these are habits spread across examples, not facts in a document.

4. 4

   Show the training data plan

   Audited labels only, patient identity removed, split by date, and rare codes listed for people.

5. 5

   Add the guards

   A code list check, a per-code confidence cutoff, full review for suggested rare codes or cancer keywords, and a weekly fast-lane audit.

6. 6

   Plan for change

   Code lists outside the model, monthly retraining, and corrections from coders feeding the next training set.