Design an AI code review agent for pull requests
PremiumLLM systemMedium
- Code review
- LLM agents
- Signal to noise
Quick answer
This LLM system interview question asks you to design an AI code review agent for pull requests. Aim for a design that uses language models to serve users well, then explain retrieval, prompting, serving, evaluation and trade-offs. Interviewers want to see how you control quality, cost and latency, and how you evaluate the results.
The problem
Design an AI code review agent that reads pull requests (PRs) on a hosted code platform and posts few but correct inline comments, like a careful senior teammate. The hard part is trust: low noise, no repeated comments, and private code that never leaks between customers.
Thousands of customer organizations install it on their private repositories. When a PR is opened or updated, the agent reads the change, pulls in the surrounding code it needs, and comments on real problems: bugs, missing error handling, security issues, broken contracts with callers and missing tests. This question has been reported at Meta in candidate interviews.
A bot that posts ten nits per PR, repeats what the linter already reported, or flags code that is actually correct gets muted within a week. The agent reads private source code for many companies, so one leak of a customer's code into another customer's PR, a log line or a training set would end the product. PR content is also untrusted input: anyone who can open a PR can write text aimed at the model.
What must an AI code review agent for pull requests do?
- Automatic review: Review every PR when it is opened and each time a new commit is pushed (a new head SHA), and on demand when a developer posts a review command.
- Inline comments: Post comments anchored to specific changed lines, each stating the problem, why it matters and, where safe, a suggested code change.
How well must it do it?
- Precision: At least 70% of posted comments are useful (accepted, fixed in a later commit, or graded useful offline), and fewer than 10% are factually wrong.
- Noise ceiling: At most 5 new inline comments per review run, and a finding already posted on the PR is never posted again.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- Code review
- LLM agents
- Signal to noise
- Prompt injection
- Tenant isolation
- Idempotency
How you’ll practise
- Problem framing
- Data and retrieval
- Prompting and orchestration
- Evaluation and safety
- Serving, cost and latency
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any LLM system design round.
- Defines the user task and what a good answer is
- Explains how relevant context reaches the model
- Plans evaluation before and after launch
- Controls cost and latency with caching and routing
- Covers safety, privacy and abuse
- Handles failures and fallbacks gracefully
Questions people ask about this interview question
- Which design type is this question?
- It is an LLM system question. You design an AI code review agent for pull requests and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Medium. Entry-level engineers can practise it, but it expects you to reason about requirements and trade-offs, not only list components.
- What should I cover first?
- Start with the user task, what a good answer looks like and how it will be measured. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team