Design a legal document Q&A agent for a law firm
PremiumLLM systemMedium
- Legal RAG
- Ethical walls
- Citation verification
Quick answer
This LLM system interview question asks you to design a legal document Q&A agent for a law firm. Aim for a design that uses language models to serve users well, then explain retrieval, prompting, serving, evaluation and trade-offs. Interviewers want to see how you control quality, cost and latency, and how you evaluate the results.
The problem
A legal document Q&A agent lets a law firm's attorneys ask questions over decades of internal memos and get short answers with pinpoint citations they can click and check. The hard part is trust, not traffic: every quote must be real, ethical walls must hold inside retrieval, and the agent must refuse when evidence is weak.
The memos live in the firm's document management system. Each memo belongs to a matter (one client engagement) and runs 20 to 100 pages, with numbered headings, long footnotes and hundreds of case and statute citations. An attorney might ask "Have we analysed whether a forum-selection clause survives assignment under New York law?" and expects an answer that quotes the firm's own memos with pinpoints (memo, section, page). A wrong citation in a court filing is a professional disaster.
Access is matter-based: an attorney normally sees only their own matters plus memos the knowledge-management team has cleared as firm-wide precedent, and ethical walls (screens set up to manage conflicts of interest) block named lawyers from named matters. A walled lawyer must never receive anything derived from a blocked matter, not even a snippet. Client data is privileged and confidential, some clients require their data to stay in the EU, and the general counsel needs a record of who asked what and what they were shown. This question was reported in 2026 ML system design interview loops. The system summarises what the firm's memos say; it does not give legal advice.
What must a legal document Q&A agent for a law firm do?
- Cited answer: Every factual sentence in an answer carries at least one pinpoint citation (memo, version, section heading, page, paragraph) that opens the memo at that spot.
- Access control: Results respect matter membership, firm-wide precedent visibility and every active ethical wall.
How well must it do it?
- Citation accuracy: 100% of displayed quotes verbatim and 100% of pinpoints resolvable; unsupported claims at most 2% on the firm's gold evaluation set.
- Confidentiality: Zero exposure of walled or non-entitled content; a newly created wall takes effect within 60 seconds.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- Legal RAG
- Ethical walls
- Citation verification
- Hybrid retrieval
- Document chunking
- Abstention
How you’ll practise
- Problem framing
- Data and retrieval
- Prompting and orchestration
- Evaluation and safety
- Serving, cost and latency
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any LLM system design round.
- Defines the user task and what a good answer is
- Explains how relevant context reaches the model
- Plans evaluation before and after launch
- Controls cost and latency with caching and routing
- Covers safety, privacy and abuse
- Handles failures and fallbacks gracefully
Questions people ask about this interview question
- Which design type is this question?
- It is an LLM system question. You design a legal document Q&A agent for a law firm and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Medium. Entry-level engineers can practise it, but it expects you to reason about requirements and trade-offs, not only list components.
- What should I cover first?
- Start with the user task, what a good answer looks like and how it will be measured. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team