Design an AI Code Review Bot: PR Comments, Noise and Cost

Developers stop reading it. When most comments are wrong or trivial, people learn to skim past all of them, and the one real bug slips through with the rest. Good AI code review system design is mostly about preventing that.

Picture a common setup. A small company sells a bot that comments on pull requests for its customers' software teams. The setup is made up, but the problem is real.

A pull request, or PR, is a request to merge a set of code changes into the main code. Teammates read it, comment on lines and approve it. The bot joins as one more reader.

The first version took six weeks to build. It sent each pull request to a model, the AI service that reads and writes text, and posted whatever came back.

Three months later, the team reads 300 of the bot's comments with two customer teams. The bot posts about 14 comments per review. Here is what they find (all numbers made up):

• About half were about style, like naming or blank lines, which the team's formatter (a tool that fixes code layout automatically) already handles.
• Many pointed at lines the author never changed, so nobody could act on them.
• Some were simply wrong, like warning about a missing null check (a test that a value exists before it is used) that was already two lines above.
• Only about 2 in 10 led the author to change any code.

Then a customer ships a bad release: a payment retry was sent twice. The bot had flagged it correctly, as comment 11 of 16. The author marked all 16 as resolved without reading them.

Each review also costs about Rs 58, and one large customer wants a discount. The goal is now clear: fewer comments, more of them right, and a lower cost per review.

You need six: diff, hunk, inline comment, false positive, precision and recall. An AI code review bot is a service that reads a pull request's changes and posts review comments. It suggests; a human still approves and merges.

Diff. A diff is the list of lines added, removed or changed in a pull request. Each block of nearby changes is called a hunk. The bot reviews the diff, not the whole codebase.

Inline comment. An inline comment is attached to one line of the diff. On GitHub, a line comment must point at a line in the pull request's diff, though a comment can also target a whole file.

False positive. A false positive is a comment that flags a problem that is not really there. A false negative is a real problem the bot missed.

Precision and recall. Precision is the share of posted comments that were right. Recall is the share of real problems the bot caught. Example: 10 comments with 6 right means 60% precision.

One picture helps for the hardest idea, why more comments can mean fewer bugs caught. Think of the quality checker at a garment export unit in Tiruppur, who inspects stitched T-shirts before packing.

If she rejects every shirt with a loose thread, her reject pile fills with good shirts. Soon the line supervisor stops looking at the pile. One day a shirt with a torn seam sits there and still gets shipped.

• The checker is the bot, and her reject pile is its list of comments.
• A loose thread is a style nit, and a torn seam is a real bug.
• The supervisor who stops looking is the author who resolves every comment unread.
• Where the picture breaks: the checker sees the whole shirt, but the bot sees only the code you send it. What you send decides what it can catch.

It is an event pipeline. The code host sends an event, a queue holds the job, and a worker builds context, asks the model and filters the result. Only the filtered comments are posted.

Start with requirements the team can test. These turn vague hopes into numbers (made up):

• Post a review within 5 minutes of a push, while the author is still looking.
• Post at most 5 inline comments per review, and only on changed lines.
• At least half of posted comments should lead to a code change.
• Keep the average cost per pull request under Rs 40.
• Never approve, merge or push code. The bot only reads and comments.

Here is the design. Follow the arrows from the code host on the left. Notice that the model is only one box in the middle.

Why the webhook goes into a queue

A webhook is an HTTP request the code host sends to your server when something happens, such as a new push. GitHub's docs ask for a 2XX reply (a success status code such as 200) within 10 seconds, or the delivery counts as failed. A review takes a minute or more.

So the receiver checks the request's signature, puts a job on the queue and replies at once. Workers take jobs at their own pace. A Monday morning burst waits in the queue instead of timing out.

GitHub does not redeliver failed deliveries automatically, as of 19 October 2026. A receiver that was down simply misses those events, so the bot should also sweep open pull requests on a timer.

Why each job is keyed by commit

The same event can arrive twice, for example when someone redelivers it by hand, and developers push several times a day. A commit is one saved set of changes, and its unique id is called a SHA.

The bot keys every job by repository, pull request number and head commit SHA, the id of the branch's latest commit. If a job with that key exists, the worker skips it, so a repeated event never posts a second set of comments.

If a newer push arrives while an old job still waits, the old job is cancelled. Nobody needs a review of code that has already been replaced.

You reduce false positives with rules in code that run after the model, not with a longer prompt (the instructions sent to the model). Keep only comments on changed lines, found in at least 2 of 3 runs, not posted or rejected before, and inside a small cap.

Back at the garment unit, the fix is not a stricter checker. It is a clear defect list: torn seam, wrong size label, stain. A loose thread gets snipped, not rejected.

The bot's filter is that defect list. The diagram shows the four questions each draft comment must pass.

Why run the model three times?

When a model service samples with some randomness, which is common, the same request can get different answers on different runs. A real bug tends to show up in most runs. A made-up problem often shows up in only one.

So the worker asks for three separate reviews and keeps an issue only if at least two agree. Three runs cost three times as much, and the cost section shows where the savings come from.

Research supports using more than one review, with a caveat. F1 is one score that balances precision and recall. A 2025 benchmark study of 1,000 real GitHub pull requests found that having a model merge several reviews and drop weak points raised F1 by up to 43.67%. The study did not test this exact voting rule.

The same study found that the review tools it tested were better at catching functional errors, meaning code that does the wrong thing, than other kinds of issues.

Why not ask the model how sure it is?

Because the number it gives is a poor guide. A 2023 study found that models asked to state their confidence in words tend to be overconfident. The same study found that checking agreement across several sampled answers helped.

Agreement across runs and real feedback from developers are the signals this design trusts.

It needs the diff, enough nearby code to understand it and the team's own rules. Sending whole files costs more and gives the model more unrelated code to comment on.

The first version pasted every changed file in full, plus files that imported them: about 45,000 tokens per review (made up). Tokens are the small pieces of text, often part of a word, that model services count and charge for.

The redesign always sends the diff with line numbers. It then adds context in this order until the context part reaches about 12,000 tokens (made up):

1. 1

   The enclosing functions

   The full function or class around each hunk. A changed line rarely makes sense alone.

2. 2

   Definitions it calls

   The signatures of functions the new code calls, found with a code search index (a lookup of where each function is defined). This step stops mistakes like the wrong null check warning.

3. 3

   The team's review rules

   A short file each team writes, such as: all money values use the Decimal type. These rules beat any general advice.

4. 4

   The pull request description

   What the author says the change is for. It helps the model spot code that does something else.

Some files skip the model entirely: lock files, generated code, test snapshots and copied-in third-party libraries. About 3 in 10 of the customers' pull requests (made up) change only such files or documentation.

It reads the diff's hunk headers. A header such as @@ -3 +3,2 @@ says where a change starts in the new file and how many lines it covers.

With git diff -U0, which prints no unchanged context lines, those numbers are exactly the added or edited lines. This parser turns a diff into a set of (file, line) pairs. It is a sketch: it assumes git's default a/ and b/ prefixes and plain file names, since git quotes unusual names.

import re, sys

HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@")

def changed_lines(diff_text):
    changed, path = set(), None
    for line in diff_text.splitlines():
        if line.startswith("+++ "):
            path = line[6:] if line.startswith("+++ b/") else None
        m = HUNK.match(line)
        if m and path:
            start, count = int(m.group(1)), int(m.group(2) or "1")
            changed |= {(path, n) for n in range(start, start + count)}
    return changed

if __name__ == "__main__":
    print(sorted(changed_lines(sys.stdin.read())))

What just happened? The +++ b/ line names the file. A deleted file shows +++ /dev/null and gives no lines, and a hunk with a count of 0 is a pure deletion, which also gives none.

Compute the diff the way GitHub shows it. Pull requests on GitHub show a three-dot diff: the changes since the merge base, the last commit the branch shares with its target. So use three dots, not two.

git diff -U0 main...HEAD | python3 changed_lines.py
# [('pay.py', 2), ('pay.py', 3)]

That is the output from a test repository where the branch added two lines after line 1 of pay.py. Swap main for the pull request's target branch in your own repository.

Store the head SHA you last reviewed, then diff from it to the new head. That keeps later reviews small, but two common kinds of push break the naive version. Both showed up in a test repository.

When the author merges main into the branch

Not zero. The raw diff from the last reviewed commit includes everything main gained. In the test, a one-line change to notes.txt on main showed up as a new line to review.

The fix is an intersection. Comment only on lines that are new since the last review and also in the pull request's three-dot diff. In the test, the intersection was empty, which is right.

When the author force-pushes

A force push replaces the branch's history, for example after squashing commits into one. The last reviewed commit is then no longer an ancestor of the new head, meaning an earlier commit in its history. So a diff from it means little.

Check with one command, and fall back to a full review when it fails. If the old commit is missing from the worker's clone (its local copy of the repository), git exits with an error, and the fallback still picks a full review.

git merge-base --is-ancestor LAST_REVIEWED_SHA HEAD && echo incremental || echo full review

In the test, a normal push printed incremental and a squash plus force push printed full review. A merge from main also printed incremental, which is why the intersection step is still needed.

If you call GitHub's compare API instead of keeping a clone, know its limits as of 19 October 2026. It lists at most 300 changed files for a comparison, and 250 commits unless you page through them.

Give every comment a fingerprint: a short id built from the file, the issue type and the code around the line, not the line number. Save the fingerprints of posted and rejected comments, and skip any draft whose fingerprint is known.

The idea comes from SARIF, an open standard format for code analysis results. It defines a fingerprint as a stable value that identifies a result over time, even when the file changes.

GitHub's code scanning, which shows alerts from code analysis tools, uses SARIF's partialFingerprints field to tell when two results are logically identical across runs, so an alert appears once. Here is a small version for review comments:

import hashlib

def fingerprint(path, category, lines, line_no, radius=2):
    lo, hi = max(0, line_no - 1 - radius), line_no + radius
    window = " ".join(" ".join(l.split()) for l in lines[lo:hi])
    return hashlib.sha256(f"{path}|{category}|{window}".encode()).hexdigest()[:16]

What just happened? In the test, inserting three lines at the top moved an issue from line 7 to line 10, and the fingerprint stayed the same. Re-indenting kept it too, because whitespace is collapsed first.

Two edits changed it, as they should: editing the flagged line, and adding a line within two lines of it. Pick the radius on purpose, since a wider window breaks on more nearby edits.

Putting the four checks together

Here is the whole filter, run on five drafts for the double payment retry example. Each draft carries a path, line, category, severity, fingerprint and the number of runs that found it.

from dataclasses import dataclass

MAX_COMMENTS = 5
SKIP = {"style", "naming"}        # the formatter and linter own these

@dataclass
class Draft:
    path: str
    line: int
    category: str
    severity: int                 # 3 = bug or security, 2 = risk, 1 = minor
    fingerprint: str
    votes: int                    # how many of the 3 passes found it

def filter_comments(drafts, changed, known):
    kept = []
    for c in drafts:
        if (c.path, c.line) not in changed:
            continue              # not a changed line
        if c.votes < 2 or c.category in SKIP:
            continue              # likely noise
        if c.fingerprint in known:
            continue              # posted before, or rejected
        kept.append(c)
    kept.sort(key=lambda c: c.severity, reverse=True)
    return kept[:MAX_COMMENTS], len(kept[MAX_COMMENTS:])

changed = {("pay.py", 2), ("pay.py", 3), ("pay.py", 7)}
drafts = [
    Draft("pay.py", 7, "bug", 3, "a1", votes=3),      # retry sent twice
    Draft("pay.py", 2, "style", 1, "b2", votes=3),    # naming nit
    Draft("pay.py", 3, "bug", 3, "c3", votes=1),      # one-off guess
    Draft("pay.py", 5, "bug", 3, "d4", votes=2),      # line not changed
    Draft("pay.py", 3, "risk", 2, "e5", votes=2),     # rejected last week
]
already_posted, rejected = set(), {"e5"}
to_post, held = filter_comments(drafts, changed, already_posted | rejected)
print([(c.line, c.category) for c in to_post], held)

What just happened? It printed [(7, 'bug')] 0. Only the double retry survived. The naming nit, the one-off guess, the comment on an unchanged line and last week's rejected comment were all dropped.

A second test with seven qualifying drafts posted the five most severe and reported 2 held back. A third test marked the double retry as already posted, and nothing was posted again. Python's sort keeps the original order for ties, so equal-severity drafts post in the order they arrived.

The held-back count goes into one short summary comment, with the lower-priority notes in a collapsed list. Nothing is lost, but nothing crowds the top five.

Measure how often a posted comment leads to a code change, how often developers mark it wrong, and how many real bugs it catches. Track time to merge too, because noise slows everyone down.

Real studies show why both matter. A 2024 industry study of an AI review tool covered 4,335 pull requests in three projects, 1,568 of them reviewed by the tool. Developers resolved 73.8% of the tool's comments.

In the same study, average pull request closure time rose from 5 hours 52 minutes to 8 hours 20 minutes, with different trends across projects.

A 2025 study compared review comments written by people and by models. Comments about readability, bugs and maintainability were resolved more often than comments about code design.

The bot collects feedback in two ways. Each comment carries a thumbs-up and a thumbs-down. The bot also checks whether a later commit changed the line it commented on, which needs no clicks and so covers far more comments.

Follow the loop in the diagram from the posted comment around to the filter.

Feedback only measures precision. For recall, the team keeps 200 old pull requests where a bug was later found and fixed (made up). Before any prompt or model change, it replays them and counts the bugs the bot flags.

Notice the recall row. In this example, the filter dropped mostly noise, and the better context let the model find bugs it missed before. A stricter filter can also drop real bugs, so the team replays the set before every filter change too.

The cost is mostly the tokens you send and receive, times the number of model runs, times the number of pushes. Shrinking context and reviewing only new changes cut it the most.

Here is the maths as a small function. The prices are made up; put in your provider's real input and output prices.

PRICE_IN = 0.0004    # rupees per input token (made up)
PRICE_OUT = 0.0016   # rupees per output token (made up)
INSTRUCTIONS = 1500  # fixed review instructions, in tokens

def review_cost(diff_tokens, context_tokens, passes=3, out_tokens=800):
    prompt = INSTRUCTIONS + diff_tokens + context_tokens
    per_pass = prompt * PRICE_IN + out_tokens * PRICE_OUT
    return round(per_pass * passes, 2)


first_push = review_cost(4000, 12000)   # 24.84
later_push = review_cost(1000, 6000)    # 14.04

What just happened? The first push sends 17,500 tokens per run and costs Rs 24.84 for three runs. Later pushes send only the new hunks with less context, 8,500 tokens per run, and cost Rs 14.04.

A typical pull request in this example gets three pushes: 24.84 plus 14.04 twice, about Rs 53. The first version cost about Rs 19.28 per single run and re-reviewed everything on each push, so about Rs 58.

Now add the skip list. About 3 in 10 pull requests never reach the model, so the average across all pull requests drops to about Rs 37. That meets the Rs 40 target.

• If your service caches repeated prompt openings (prompt caching), keep the fixed instructions first so every run starts the same way.
• Use a smaller, cheaper model for a first triage pass that only decides whether a hunk is worth a full review.
• Set a token budget per review in code, and post a summary comment instead when a huge pull request exceeds it.
• Log cost per review per customer, so one team with giant pull requests does not surprise you at month end.

Give the bot read and comment access only, treat everything in the pull request as untrusted data, and never let it approve or merge. The code it reads may come from someone you do not trust.

A pull request can contain text aimed at the model, such as a code comment telling the reviewer to approve the change. This is called indirect prompt injection.

OWASP, a non-profit that publishes security guidance, puts prompt injection first in its 2025 list for LLM applications (apps built on large language models). That was still the current page when we checked on 19 October 2026.

It describes indirect injection as a model taking in outside content, such as websites or files. Its advice includes keeping the model's access to the minimum its job needs, and marking untrusted content clearly.

The GitHub permission trap

A GitHub App is an integration with its own identity. A fine-grained token is a personal access token that can be limited to chosen repositories, and it acts as the person who created it. For either, posting review comments needs the Pull requests repository permission with write access. Creating a review needs that same permission, and a review's event can be APPROVE, REQUEST_CHANGES or COMMENT.

So permissions alone cannot stop the bot approving, per GitHub's REST docs as of 19 October 2026. Hard-code the event to COMMENT, and require a code owner's approval in the branch's protection rules (settings that block merging until conditions are met). Run the bot as a GitHub App or a machine account (a user account just for the bot). Keep it out of the code owners, so its approval never meets that rule.

Code owners are the people or teams listed in a repository's CODEOWNERS file, and they need write access. With the rule on, a pull request that touches owned files cannot merge without a code owner's approval.

Give every path an owner with a catch-all line such as * @your-team, which GitHub's own example file uses for default owners.

• Never run the pull request's code inside the review worker. Building and testing belong to the separate test pipeline.
• Keep secrets, like deploy keys, out of the worker entirely, so a tricked model has nothing to leak.
• Mark the diff as untrusted input in the instructions, and drop any draft that tries to change the bot's task.
• Set a webhook secret, and check each delivery's signature before queuing it, as below.
• Respect each customer's rules about where their code may be sent and how long it is kept.

When you set a webhook secret, GitHub signs each delivery with HMAC SHA-256, a keyed hash, in the X-Hub-Signature-256 header. Its docs say to compare signatures in constant time, never with a plain equals check. A constant-time compare takes as long whichever character differs, so timing leaks nothing:

import hashlib, hmac

def signature_ok(secret: bytes, body: bytes, header: str) -> bool:
    expected = "sha256=" + hmac.new(secret, body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, header or "")

In the test, a correct signature returned True, while a body with one extra byte and a missing header both returned False.

Explain it as an event pipeline with a strict filter, and spend most of your time on noise and cost. Those are the parts that decide whether teams keep the bot.

1. 1

   Ask about scale and goals

   How many pull requests a day, how fast a review must arrive, and what counts as a useful comment.

2. 2

   Draw the pipeline

   Webhook, queue, worker, context builder, model, filter, post. Key jobs by head commit SHA so repeated events are harmless.

3. 3

   Explain the context budget

   Diff first, then enclosing functions, called definitions and team rules, cut to a fixed token budget.

4. 4

   Spend time on false positives

   Changed lines only, agreement across runs, a cap per review and fingerprinted rejections. Say why noise hides real bugs.

5. 5

   Handle pushes properly

   Incremental review with an intersection against the pull request diff, and a full review after a force push.

6. 6

   Show how you measure

   Share of comments that led to a code change, comments marked wrong, recall on a replayed bug set, and time to merge.

7. 7

   Estimate cost and lock down access

   Do the cost per pull request maths, then say the bot only ever posts COMMENT reviews and a person approves.

If the interviewer pushes further, compare this bot with a coding agent that writes code. A reviewer only reads and comments, so its risks and costs are much smaller.