Design spam review with AI and human labelers
PremiumML systemHard
- Human in the loop
- Spam review detection
- LLM triage agent
Quick answer
This ML system interview question asks you to design spam review with AI and human labelers. Aim for a design that turns data into useful predictions in production, then explain the data, training, serving and monitoring. Interviewers want to see how you frame the problem, measure success and keep the model healthy after launch.
The problem
Design a spam review system where a classifier, an AI triage agent and human labelers work together to keep spam out of product and seller reviews on a large marketplace. The hard part is not the classifier alone: it is routing each review to the cheapest decision maker that is accurate enough, and making human review faster without making it worse.
Today a classifier scores every review and about 150 human labelers check the unclear ones, but the queue grows every week and labelers disagree with each other. You will add an AI agent that gathers evidence and suggests a label, and redesign the loop so human labels also make the models better.
What must the system do?
- Three-stage routing: Each new review is auto-published, auto-removed, sent to the AI agent or sent to a human, based on calibrated scores and thresholds.
- Agent evidence: The agent returns a suggested label, a reason code and the evidence it used (account history, similar reviews, timing) for the human.
How well must it do it?
- Removal precision: At least 98% of automatically removed reviews must be real spam, measured on audits.
- Human throughput: The human queue must clear daily within 24 hours with the current 150 labelers.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- Human in the loop
- Spam review detection
- LLM triage agent
- Active learning
- Label quality
- Confidence thresholds
How you’ll practise
- Problem framing
- Data and labels
- Features and model
- Training and evaluation
- Serving and monitoring
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any ML system design round.
- Frames the product goal as a clear ML problem
- Defines offline and online success metrics
- Explains where the data and labels come from
- Separates training from serving and keeps features consistent
- Plans for drift, retraining and rollbacks
- Weighs model quality against latency and cost
Questions people ask about this interview question
- Which design type is this question?
- It is an ML system question. You design spam review with AI and human labelers and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Hard. It suits engineers preparing for senior interviews, where you are expected to drive the discussion and go deep on trade-offs.
- What should I cover first?
- Start with the product goal, how success is measured and what data is available. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team