Design an on-device LLM for smartphones
PremiumML systemHard
- On device inference
- Quantization
- Knowledge distillation
Quick answer
This ML system interview question asks you to design an on-device LLM for smartphones. Aim for a design that turns data into useful predictions in production, then explain the data, training, serving and monitoring. Interviewers want to see how you frame the problem, measure success and keep the model healthy after launch.
The problem
An on-device LLM for smartphones runs a small language model entirely on the phone to power smart replies, summaries, tone rewrites and proofreading, even with no signal. The hard part is fitting it in 2 GB of shared RAM, keeping it fast and cool on limited memory bandwidth, and keeping it safe and private with no server filter.
Users expect text to start appearing almost instantly, the features must work on a plane, and the model must stay polite and safe in every language it ships in. This question was reported at Google in candidate interviews for senior ML and GenAI roles.
A phone is not a server. The model shares a few gigabytes of RAM with the camera, the browser and the OS. Generation speed is limited by how fast weights can be read from memory, not by raw compute. The chip heats up and slows down, and every response costs battery. The model has to reach hundreds of millions of devices over consumer networks, and some requests are too hard for a small model and need a careful, privacy-preserving route to a larger cloud model.
What must an on-device LLM for smartphones do?
- Offline writing features: Smart reply, summarize, rewrite (tone) and proofread run fully on device with no network connection.
- Cloud fallback: Requests the phone cannot handle well, such as inputs longer than the context window, may go to a larger cloud model only when the device is online and the user has consented.
How well must it do it?
- Memory: Weights, KV cache and runtime together stay within 2.0 GB; the service releases memory when the OS signals pressure and reloads within 2 s.
- Time to first token: p50 ≤ 400 ms and p95 ≤ 800 ms for a 500-token prompt on the reference phone.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- On device inference
- Quantization
- Knowledge distillation
- KV cache
- Speculative decoding
- LoRA adapters
How you’ll practise
- Problem framing
- Data and labels
- Features and model
- Training and evaluation
- Serving and monitoring
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any ML system design round.
- Frames the product goal as a clear ML problem
- Defines offline and online success metrics
- Explains where the data and labels come from
- Separates training from serving and keeps features consistent
- Plans for drift, retraining and rollbacks
- Weighs model quality against latency and cost
Questions people ask about this interview question
- Which design type is this question?
- It is an ML system question. You design an on-device LLM for smartphones and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Hard. It suits engineers preparing for senior interviews, where you are expected to drive the discussion and go deep on trade-offs.
- What should I cover first?
- Start with the product goal, how success is measured and what data is available. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team