GenAI System Design Interview: How the Round Works

You design a system that serves answers fast but cannot say whether they are right, what they cost or what stops data leaking. In a GenAI system design interview, those three gaps leave whole rows of the score empty.

Picture a common setup. A backend engineer with a few years of payment APIs (the interfaces other programs call) gets a one-line question: design a question-answering assistant over an insurance company's policy documents. The setup is made up, but the problem is real.

The answer has a clean API, a database and a cache. The feedback says it was strong on storage, but treated the model as a box that is always right. It never said how answers would be tested, and gave no cost.

This guide rebuilds that answer for a 45-minute round, phase by phase, and shows what each part earns. If retrieval and prompts are new to you, our introduction to GenAI system design covers the basics first.

A GenAI system design interview is a round where you design a product built on a large language model, or LLM. You explain the parts, the trade-offs and how you would know it works.

GenAI is short for generative AI: models that create new text, images or code. An LLM is one trained on huge amounts of text, so it can read a prompt and write a reply.

The round is usually one open question and a fixed time. This guide plans for 45 minutes; if yours is 60, give the extra time to the deep dive and testing.

A GenAI round keeps everything from a classic round, like requirements, scale and storage. It adds a model whose answers can be wrong, slow and costly, and most of the extra points sit there.

In a classic round, such as designing a URL shortener, a request either works or fails. In a GenAI round, the system can reply quickly, politely and wrongly. So you must design how you catch wrong answers, not just how you serve them.

• Correctness becomes a design topic. You describe a test set (questions with known good answers) and how you score replies, before launch and after.
• Cost moves into every request. Hosted model services usually bill per token, a short word or part of a longer one, with input and output priced separately. A long prompt costs money on every question.
• Latency, the wait for a reply, is often counted in seconds. The model writes one token at a time, so you talk about streaming the reply to the screen as it is written, and the wait before the first word appears.
• New failures appear. Hallucination means the model states something false as fact. Prompt injection means text inside an input or document tries to give the model orders.
• A document pipeline appears. Files must be parsed, split into chunks (small passages) and indexed, which means stored so they can be searched fast.
• The model is a choice you defend. A large model may be more accurate, while a small one is faster and cheaper. You pick per task, with reasons.

Some things do not change. You still ask about users and scale, draw clear boxes and plan for failures. Do not skip those basics to get to the model.

It also differs from an ML (machine learning) system design round. There, you usually train your own model from data. In a GenAI round you usually start from an existing model and spend the time on retrieval, prompts, testing and safety.

Expect to be judged row by row, not on one overall impression. Structured interviews work this way: the US Office of Personnel Management says every answer is judged against the same rating scale and standards.

A rubric is that scale written down: a few rows, each rated on its own. Rows and names vary by company, so the table below is a sample to practise with, not any company's real sheet.

Picture a kolam contest at Pongal, where judges score symmetry, colour and neatness in separate rows. A dazzling peacock in the centre cannot make up for a crooked border.

• The judges' rows are the rubric rows.
• The dazzling peacock is a clever agent, where the model picks its own next steps, or a beautiful diagram.
• The crooked border is a testing or cost row with nothing in it.

Where the picture breaks: a kolam is judged once, at the end. An interviewer scores as you go and often asks about a thin row, so you can still repair it during the round.

Level changes what is expected. Evan King of Hello Interview, an interview-prep site, writes that mid-level candidates should guide the early parts. He wants senior candidates to go deep in around two places, and staff candidates to lead almost the entire interview.

When the interviewer challenges a choice, treat it as information, not an attack. Say what the challenge changes, adjust the design, and move on.

Spend about 5 minutes clarifying, 3 estimating, 10 drawing the main design, 10 on one deep dive, 7 on testing, and the last 10 on safety, cost and wrap-up. The table shows what to cover and what to leave on the board.

Treat the minutes as a guide, not a rule. If the interviewer pushes hard on testing, follow them and shorten the deep dive.

You walk the 45-minute plan on the question, saying each choice and its reason out loud. Here is the rebuilt answer to the one-line question from the opening, phase by phase.

Minutes 0 to 5: clarify the problem

Ask short questions and write each answer on the board. These are the answers the interviewer gives in this scenario.

• Who asks? Version one is for 600 call-centre staff answering policyholders on the phone, not for customers directly.
• Which documents? About 3,000 policy wordings, add-on clauses and circulars (official notices) as PDFs, some of them scanned. Around 40 change each month.
• Can everyone see every document? No. Underwriting guidelines, the internal rules for accepting a customer, are for underwriters only.
• How good must answers be? A wrong waiting period could mislead a customer, so every answer must quote the clause it used.
• Any hard limits? No claim decisions, questions in Hindi and English, and answers within 5 seconds.
• What about personal data? Customer files stay out of version one. Only product documents are searched.

Then state the goal in one sentence. Staff ask a policy question and get a short answer with the exact clause quoted, in under 5 seconds, or a clear not found.

Minutes 5 to 8: estimate load and cost

The interviewer says staff ask about 20,000 questions a day. If most arrive in 10 busy hours, that is about 2,000 an hour, under one question a second on average in those hours. Load is small.

The number that matters is cost. Each prompt holds instructions, five passages of about 600 tokens each and the question, about 4,000 tokens, and the answer is about 300.

With made-up prices of Rs 0.02 per 1,000 input tokens and Rs 0.08 per 1,000 output tokens, that is about Rs 0.10 a question. Over a month it comes to about Rs 62,000.

The useful part is the lever. Sending 20 passages instead of 5 nearly triples the bill, so the number of passages is a design choice, not a detail. The engineer section has the script that checks this.

Minutes 8 to 18: draw the two paths

Draw two paths. The loading path turns PDFs into searchable passages ahead of time. The question path runs for every question.

Each part has one job. The ingestion worker parses PDFs, reads scanned pages with OCR (software that turns images of text into text), splits them by clause and indexes them. Keeping the clause number in each chunk makes citations possible later.

The search index offers both keyword search and search by meaning, which finds passages with similar meaning even when the words differ. Policy questions mix exact terms, like a clause number, with everyday wording. The model sits behind one internal API, so it can be swapped without touching the rest.

For more on parsing and access rules, see our guides on PDF parsing and on access control for document search. In the round, one sentence on each is enough unless the interviewer asks for more.

Minutes 18 to 28: go deep on retrieval and grounding

Pick retrieval and grounding as the deep dive, and say why. Grounding means the answer comes only from the retrieved passages, not from the model's general memory. A fluent answer about the wrong policy is the costliest failure here.

Searching first and then writing from what was found is called RAG, short for retrieval-augmented generation. The name comes from a 2020 research paper that joined a text-writing model to a search index over Wikipedia.

Explain why you will not paste all the documents into one prompt. A 2023 study tested models on answering from many documents and on finding the value for a given key in a long list. Accuracy was often highest when the needed text sat at the start or end, and dropped in the middle.

Newer models may handle long inputs better than the ones tested in 2023. The cost argument holds either way: five passages are a tiny fraction of every document, even where a repeated prompt start gets a discount.

A reranker is a second, slower model that reads the question and each passage together and scores the match. Search finds 20 likely passages fast, and the reranker keeps the best 5.

Here is the question path to draw for the deep dive. Follow the No arrow from the evidence check: saying not found is a designed outcome, not an error.

The citation check is plain code. If the answer quotes clause 4.2 but no retrieved passage is clause 4.2, the answer never reaches the staff screen. It catches invented clause numbers cheaply.

Minutes 28 to 35: show how you would test it

Testing is the row a classic design habit does not cover, so give it real time. Propose a test set of 300 real staff questions, each with the right clause and a short correct answer written by the product team.

• Retrieval check: is the right clause in the top 5 passages? This is measured without calling the model at all.
• Faithfulness check: what share of the claims in the answer do the retrieved passages support? Another model can grade this, once its grades match human grades on a sample.
• Refusal check: 50 questions with no answer in the documents. The right reply is not found.
• Release gate, a rule every change must pass: a new prompt, model or index ships only if no score drops beyond a small agreed margin. Models can answer the same question differently on different runs.
• Live checks: staff mark answers as helpful or wrong, and a person reads a weekly sample.

Using a model as a grader needs that sample check. A 2023 study found strong models agreed with human preferences over 80% of the time on open-ended chat questions. That is about as often as two human judges agreed with each other. It also found biases, such as favouring longer answers or the answer shown in a certain position.

Why does retrieval get its own score? A wrong answer has two possible causes, bad search or bad writing. Separate scores tell the team which one to fix.

Minutes 35 to 45: safety, cost and wrap-up

Cover three risks in quick order. First, access: the search runs only over documents the user's role allows, so underwriting guidelines never reach a call-centre prompt.

Second, prompt injection. A scanned circular could carry hidden text telling the model to ignore its rules. OWASP, a non-profit that publishes security guidance, ranks prompt injection first in its 2025 Top 10 for LLM applications.

OWASP published a 2026 edition of the list on 3 August 2026. As of 19 October 2026, its prompt injection page is still labelled LLM01:2025, so check the newer edition before you quote a rank.

OWASP advises marking outside content as untrusted and giving the model only the access it needs. This design gives the model no tools that change anything. An injected instruction can spoil answers that use that passage, but cannot change records. With replies shown as plain text and URLs left unclickable, a reply cannot make the browser load an image or open a link, a known way to leak data.

The citation check will not catch that, because the poisoned passage is real. The faithfulness tests and staff feedback are what catch it, which is one more reason to keep both.

Third, cost. The levers are fewer passages, a small model for easy questions, and caching (saving and reusing) answers to repeated questions until the documents change.

In the last four minutes, circle the biggest risk: an old policy wording winning over a newer circular. The version-two list adds an effective date on every chunk, customer-facing answers and Hindi voice input.

For a full build of a similar assistant, with rollout and scale, see our enterprise RAG system design case study.

Expect follow-ups that poke at quality, cost, safety and change over time. Each one tests a rubric row rather than your memory of tools.

• Why not fine-tune a model on the policies? Fine-tuning means further training a model on your own examples. A 2023 study found retrieval beat fine-tuning on raw text at adding both familiar and new facts. And these policies change monthly.
• What if answers get slower after launch? Read the logged timings for slow questions and check search, reranking and model time separately before changing anything.
• A cheaper model is released. How do you switch? Run the full test set on it and compare every score. Then send 5 percent of questions to it while watching staff feedback.
• Staff report a wrong answer. What happens next? The question, passages and answer are already logged, so the team finds whether search, ranking or writing failed. The question then joins the test set.

Each strong answer names a check, a number or a log, not a promise. It shows how you would find out what is wrong, which matters more than a perfect first design.

The costliest mistake is treating the model as a box that is always right. Most other mistakes are ways of leaving a rubric row empty, as the feedback in the opening showed.

• Drawing before asking. Fix: spend the first five minutes on users, documents and the quality bar.
• One box labelled AI. Fix: split it into search, ranking, prompt building, the model call and the checks.
• No numbers. Fix: estimate tokens per question and the monthly cost, even roughly.
• No testing plan. Fix: describe a test set, what you score and the release gate.
• Guardrails as a buzzword. Fix: name the exact check, where it runs and what happens when it fails.
• Reaching for an agent too early. Fix: start with fixed steps, and let the model decide only where a fixed step fails. Our guide on AI agents versus workflows covers that choice.
• Naming products instead of reasons. Fix: describe each part by what it must do, such as a small, fast model for sorting questions.

Practise out loud, on a timer, against a written rubric. Reading alone does not build the habit of covering every row in 45 minutes.

1. 1

   Days 1 to 4: fill the gaps

   Read our introduction to GenAI system design and skim one risk framework, such as the Generative AI Profile from NIST, the US standards body. Write one page of notes on retrieval, testing, safety and cost.

2. 2

   Days 5 to 10: four timed solo runs

   Pick four questions: a document Q&A assistant, a support bot, a meeting summariser and an internal search tool. Record yourself, then score each recording against the six rows.

3. 3

   Days 11 to 17: mock rounds with a partner

   Ask a friend to play the interviewer and interrupt with follow-ups. Swap roles once, because scoring someone else teaches you the rubric fast.

4. 4

   Days 18 to 21: fix your weakest row

   Look at your scores and rehearse your weakest row until you can say it in two minutes. If your background is backend work, check the testing row first, because classic rounds rarely ask for it.

Write the estimate as a few lines of code with the passage count as an input, then run it for 5, 10 and 20 passages. The prices are made up; swap in your provider's real per-token prices.

QUESTIONS_PER_DAY = 20_000
FIXED_TOKENS = 1_000      # instructions + the question
TOKENS_PER_PASSAGE = 600
TOKENS_OUT = 300          # a short answer with clause numbers
RS_PER_1K_IN = 0.02       # made-up prices
RS_PER_1K_OUT = 0.08


def cost_per_question(passages):
    tokens_in = FIXED_TOKENS + passages * TOKENS_PER_PASSAGE
    return tokens_in / 1000 * RS_PER_1K_IN + TOKENS_OUT / 1000 * RS_PER_1K_OUT


print(f"{QUESTIONS_PER_DAY / 10 / 3600:.2f} questions a second in 10 busy hours")
for n in (5, 10, 20):
    per_q = cost_per_question(n)
    per_month = per_q * QUESTIONS_PER_DAY * 30
    print(f"{n:>2} passages: Rs {per_q:.3f} a question, Rs {per_month:,.0f} a month")

Save it as cost.py and run python3 cost.py. On Python 3.13 it prints 0.56 questions a second, then Rs 0.104 and Rs 62,400 a month for 5 passages.

It prints Rs 0.164 and Rs 98,400 for 10 passages, and Rs 0.284 and Rs 170,400 for 20. Twenty passages cost 2.7 times as much as five, which is the nearly triple from the walkthrough.

What just happened? Input tokens dominate here: 4,000 in against 300 out, even at a quarter of the output price. That is why cutting passages moves the bill more than shortening answers.

It reads every clause number the answer quotes and blocks the answer if any of them is missing from the passages that search returned. It is a few lines of code and costs no model call.

import re

CLAUSE = re.compile(r"\bclause (\d+(?:\.\d+)*)", re.IGNORECASE)


def check_citations(answer, passages):
    cited = set(CLAUSE.findall(answer))
    if not cited:
        return "BLOCK: no clause cited"
    missing = cited - {p["clause"] for p in passages}
    if missing:
        return "BLOCK: not in passages: " + ", ".join(sorted(missing))
    return "PASS"


passages = [
    {"clause": "4.2", "text": "Cataract surgery is covered after a 24-month waiting period."},
    {"clause": "4.3", "text": "Pre-existing diseases are covered after 36 months."},
]
answers = [
    "Not in year one. Clause 4.2 sets a 24-month waiting period.",
    "Yes, it is covered from day one under clause 7.1.",
    "Yes, cataract surgery is covered.",
    "Yes, it is covered from day one under clause 4.2.",
]
for a in answers:
    print(check_citations(a, passages))

Pause and predict before you run it: four answers go in, and only the first is right about cataract cover. How many of the three wrong ones does the check block?

It prints PASS, then BLOCK: not in passages: 7.1, then BLOCK: no clause cited, then PASS. It blocks two: the second answer invented clause 7.1, and the third cited nothing.

What just happened? The fourth answer is wrong too, since clause 4.2 sets a 24-month waiting period, yet it passed. The check proves the clause was retrieved, not that the clause says what the answer claims.

That gap is the faithfulness check's job. The pattern is also strict by design. In our tests, the text clauses 4.2 and 4.3 was blocked as no clause cited. So tell the model one citation format, and test it.

Record two facts for every test question: was the right clause in the top 5 passages, and was the final answer correct? The pair tells you which step failed.

# One row per test question: the right clause, the top 5 clauses search
# returned, and whether a person graded the final answer as correct.
results = [
    ("4.2", ["4.2", "4.3", "9.1", "2.4", "4.1"], True),
    ("3.1", ["3.1", "3.2", "5.5", "1.0", "7.7"], True),
    ("6.4", ["6.1", "6.2", "6.3", "2.2", "8.0"], False),
    ("2.7", ["2.7", "2.8", "4.4", "1.1", "3.3"], False),
    ("5.0", ["5.0", "5.1", "1.2", "9.9", "6.6"], True),
    ("8.3", ["1.4", "2.5", "3.6", "4.7", "5.8"], False),
    ("7.2", ["7.2", "7.1", "0.5", "6.6", "2.1"], True),
    ("9.5", ["9.4", "9.6", "1.9", "3.8", "8.8"], True),
    ("1.3", ["1.3", "1.4", "2.0", "6.1", "5.2"], True),
    ("4.8", ["4.8", "4.9", "4.7", "3.0", "2.6"], True),
]

counts = {}
for gold, top5, correct in results:
    found = gold in top5
    if found and correct:
        verdict = "fine"
    elif found:
        verdict = "fix the writing step"
    elif correct:
        verdict = "right without evidence: check it"
    else:
        verdict = "fix search"
    counts[verdict] = counts.get(verdict, 0) + 1

hits = sum(gold in top5 for gold, top5, _ in results)
print(f"right clause in top 5: {hits} of {len(results)} ({hits / len(results):.0%})")
for verdict, n in sorted(counts.items()):
    print(f"{n}  {verdict}")

Run it with python3 search_or_writing.py. It prints right clause in top 5: 7 of 10 (70%). Then come four counts: 6 fine, 2 fix search, 1 fix the writing step and 1 right without evidence: check it.

The 70% is the retrieval hit rate at 5, the share of questions whose right clause made the top 5. It needs no model call, so you can rerun it after every change to chunking or ranking.

The last count matters most for a grounded product. A correct answer without the right clause often means the model answered from its own memory, which the design was meant to prevent. Sometimes another passage held the same fact, so read it before you decide.

Present it as a short story that follows the rubric rows, so the interviewer can tick each one as you go. These six steps fit any GenAI design question, not just the insurance one.

1. 1

   State the goal and the quality bar

   One sentence on who asks what, and what happens when an answer is wrong.

2. 2

   Give the numbers

   Questions a second at peak, tokens per question and cost per question, with the lever that moves cost most.

3. 3

   Draw the two paths

   The loading path and the question path, with the access check before search.

4. 4

   Go deep on the riskiest part

   For document Q&A, retrieval and grounding: search, reranking, the evidence check and the citation check.

5. 5

   Show the tests

   Retrieval hit rate, faithfulness, refusals, a release gate with a margin, and live feedback.

6. 6

   Close with the biggest risk

   Name the box most likely to fail, how you would watch it and what version two adds.