Design an AI coding assistant like GitHub Copilot
PremiumLLM systemHard
- Fill in the middle
- Code completion
- KV cache
Quick answer
This LLM system interview question asks you to design an AI coding assistant like GitHub Copilot. Aim for a design that uses language models to serve users well, then explain retrieval, prompting, serving, evaluation and trade-offs. Interviewers want to see how you control quality, cost and latency, and how you evaluate the results.
The problem
Design an AI coding assistant like GitHub Copilot: inline ghost-text completions and a repo-aware chat panel inside the IDE, for 1 million developers. It is hard because inline suggestions must render in 200 ms, useful context sits in other files, and enterprise code must never leak.
Developers get two features. Inline completion shows grey "ghost text" at the cursor while they type, from a single token up to a whole function body, and they press Tab to accept it. Chat is a side panel where they ask questions about selected code or the repository and get streamed answers with code blocks and file references. Individual developers and large enterprises both use the product, and enterprises connect private repositories.
This question was reported at Meta in candidate interviews for senior machine learning engineers. You own both halves: the model (base model, training objective, training data, evaluation) and the serving system (IDE plugin behaviour, context building, inference fleet, metrics and privacy). The suggestion also has to fit code that already exists after the cursor.
What must an AI coding assistant like GitHub Copilot do?
- Inline completion: As the developer types, show a single-line or multi-line suggestion at the cursor that is accepted with Tab and disappears when they keep typing something different.
- Fill in the middle: Suggestions must respect code after the cursor, for example completing a function body between its signature and the code that follows, without duplicating the following lines.
How well must it do it?
- Inline latency: p50 at most 200 ms and p95 at most 400 ms, measured from the moment the plugin sends the request (after its debounce) to the suggestion being rendered.
- Privacy: Enterprise code and prompts are never used for training and are not retained after the request completes; one tenant's code must never reach another tenant.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- Fill in the middle
- Code completion
- KV cache
- Model routing
- Speculative decoding
- Repo context retrieval
How you’ll practise
- Problem framing
- Data and retrieval
- Prompting and orchestration
- Evaluation and safety
- Serving, cost and latency
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any LLM system design round.
- Defines the user task and what a good answer is
- Explains how relevant context reaches the model
- Plans evaluation before and after launch
- Controls cost and latency with caching and routing
- Covers safety, privacy and abuse
- Handles failures and fallbacks gracefully
Questions people ask about this interview question
- Which design type is this question?
- It is an LLM system question. You design an AI coding assistant like GitHub Copilot and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Hard. It suits engineers preparing for senior interviews, where you are expected to drive the discussion and go deep on trade-offs.
- What should I cover first?
- Start with the user task, what a good answer looks like and how it will be measured. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team