Design a RAG app with changing requirements
PremiumLLM systemMedium
- RAG
- Hybrid search
- Chunking
Quick answer
This LLM system interview question asks you to design a RAG app with changing requirements. Aim for a design that uses language models to serve users well, then explain retrieval, prompting, serving, evaluation and trade-offs. Interviewers want to see how you control quality, cost and latency, and how you evaluate the results.
The problem
Design a RAG app with changing requirements: an internal question-answering assistant for 20,000 employees, grounded in the engineering wiki. The twist is that the interviewer adds new sources, permissions, freshness and citations mid-interview, so the design must absorb each change without a rebuild.
An employee types a question into a chat box, such as 'How do I rotate the payments database credentials?', and gets a short answer written by a large language model (LLM) and grounded in wiki pages. This pattern is called retrieval-augmented generation (RAG): find the relevant passages first, then ask the model to answer from them. In version 1 every employee can read every wiki page, and the index may lag the wiki by up to a day.
This question has been reported at Microsoft in candidate interviews. Roughly every seven or eight minutes a new requirement lands (listed under Context). You are judged less on your first diagram than on how much of it survives each change: which component you touch, what you re-process, and what you leave alone.
What must a RAG app with changing requirements do?
- Ask a question: An employee asks a natural-language question and receives a streamed answer of up to about 300 tokens grounded in retrieved documents.
- Ingest the wiki: All 400,000 wiki pages are ingested, split into chunks, embedded and indexed; version 1 may refresh nightly.
How well must it do it?
- Security (after change 2): No answer text, snippet or link may ever come from a document the user cannot open; when access cannot be determined, fail closed.
- Freshness (after change 3): p99 lag of at most 5 minutes from a source edit, deletion or permission change to its effect on answers.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- RAG
- Hybrid search
- Chunking
- Reranking
- Incremental indexing
- Access control
How you’ll practise
- Problem framing
- Data and retrieval
- Prompting and orchestration
- Evaluation and safety
- Serving, cost and latency
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any LLM system design round.
- Defines the user task and what a good answer is
- Explains how relevant context reaches the model
- Plans evaluation before and after launch
- Controls cost and latency with caching and routing
- Covers safety, privacy and abuse
- Handles failures and fallbacks gracefully
Questions people ask about this interview question
- Which design type is this question?
- It is an LLM system question. You design a RAG app with changing requirements and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Medium. Entry-level engineers can practise it, but it expects you to reason about requirements and trade-offs, not only list components.
- What should I cover first?
- Start with the user task, what a good answer looks like and how it will be measured. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team