Enterprise RAG System Design: A Finance Team Assistant

It leaks files, quotes old versions and makes up sources. Nobody decided who may see what, which version counts, or how an answer gets checked before anyone reads it.

Picture a common setup. A lending company's finance team of 180 people works from policies, circulars and monthly reports. The setup is made up, but the problems are real.

Around every month-end, the policy desk gets about 250 questions a week by email. Analysts ask for the approval limit on a vendor payment above Rs 5 lakh (500,000), or which travel policy covers a trip that began in March. Each reply takes the desk about ten minutes.

The naive version came first. An intern pointed a chatbot at the whole shared drive and told the model never to reveal board papers. It worked well for a week.

Then a junior accountant asked about salary budgets. The bot quoted a draft board note meant only for the CFO (chief financial officer). The CFO switched it off that afternoon.

What just happened? The board note was searchable by everyone, so it landed in the model's input. An instruction cannot make the model unsee text it has already been given.

The CFO's conditions for the next version were short: correct answers, a source on every answer, and no file shown to anyone who could not already open it. This guide designs that next version, step by step.

Enterprise RAG system design is planning a document question-and-answer assistant for a whole company, end to end. It covers requirements, ingestion, permissions, search, citations, testing, cost and rollout.

RAG, short for retrieval-augmented generation, means the system first fetches passages from your own documents. Then a model, the AI that reads and writes text, answers from those passages. The term comes from a 2020 research paper that paired a search step with a text generator.

The word enterprise adds the hard parts: many users, private files, audits and a budget. Most of this guide is about those parts, not the model.

Five words this guide uses

Chunk. A chunk is a small piece of a document, a few paragraphs long, that the system searches and quotes. The travel policy becomes about 40 chunks.

Index. An index is the searchable store of all chunks, built ahead of time. A search then takes milliseconds instead of reading every file.

Metadata. Metadata is extra data stored with each chunk, such as the document name, version, fiscal year and which groups may read it.

Citation. A citation is the pointer in an answer that says exactly where a fact came from: document, version and page.

Golden set. A golden set is a fixed list of real questions with known correct answers and sources. You run it to test the system before each change.

Decide the requirements and the scale numbers first: who will ask, about which documents, how fresh and fast answers must be, and what each may cost. Every later choice depends on these.

The team spends its first week with the policy desk and the CFO. It writes down what the assistant must do:

• Answer questions about finance policies, circulars and published monthly reports, in plain English.
• Cite the document, version and page for every fact, so an analyst can check it in one click.
• Show each person only what they can already open in the document store today.
• Say clearly when it cannot find an answer, instead of guessing.
• Make a new or changed policy searchable within one hour, and remove access within 15 minutes when someone changes teams.
• Reply within 5 seconds for 95 out of 100 questions, a measure called p95 latency (the time under which 95 percent of replies arrive).
• Keep model and search charges under Rs 1 per question.

It also writes an out-of-scope list. The assistant will not read the live accounting ledger, approve payments or edit reports. A question about today's cash balance belongs to the reporting system, which already has exact figures.

Then come the scale numbers. These are estimates from the policy desk and the document store:

• Documents: about 12,000 files, roughly 90,000 pages, plus about 400 new or changed files a month.
• Chunks: about three per page, so roughly 270,000 chunks in the index.
• Users: 180 people in finance now, perhaps 600 more if other departments join later.
• Questions: about 600 per working day on average, rising to 1,500 a day during month-end close.

These numbers already settle one question. Each chunk is stored with a vector, a list of numbers that captures its meaning, so similar chunks sit close together. Say each vector has 1,024 numbers of 4 bytes each.

You can check the raw size yourself in Python:

chunks, numbers_per_vector, bytes_per_number = 270_000, 1024, 4
raw = chunks * numbers_per_vector * bytes_per_number
print(round(raw / 1e9, 2), "GB of raw vectors")

It prints 1.11 GB of raw vectors, which is 1.03 GiB if your tool counts in binary units. The table and the search index add their own overhead on top of the raw numbers.

We tested this on pgvector 0.8.1, the vector search add-on for PostgreSQL, with 3,000 rows of 1,024-number vectors and default index settings. A table of ids and vectors took 1.37 times the raw size, and the search index took 2 times. That puts the vectors plus their index near 3.7 GB. Chunk text, metadata and the keyword index add more on top.

That still fits easily on one machine, so the team does not need a search cluster, a group of machines sharing the load, on day one.

Documents get in through an ingestion pipeline: a background job that collects files, turns them into clean text, splits them into chunks, attaches metadata and writes everything to the index. It runs again whenever a file changes.

The files sit in three places: a policy library, a folder of monthly reports, and a circulars folder that compliance posts to. The team writes one small connector for each, meaning code that lists the files and notices changes.

The diagram shows the ingestion side. Look at where the company directory feeds in. Permission data joins each chunk before the chunk reaches the index.

Parsing, turning a file into clean text, is where many RAG projects quietly fail. Finance reports are full of tables. A table read as loose text loses which number belongs to which row. Our guide on PDF parsing for RAG covers tables and scanned pages in depth.

Versions matter more in finance than almost anywhere. The travel policy has three versions, each with an effective date. The team keeps all three in the index, but marks the older two as superseded, meaning replaced by a newer version.

Deletes need the same care as new files. When a circular is withdrawn, the pipeline must remove its chunks within the hour. A chunk left behind keeps answering questions long after its file is gone.

It filters every search by the asking user's permissions, so chunks they cannot open never reach the model at all. Telling the model not to share something is not a control.

Think of a library where every book has a label saying which card colours may borrow it. A good librarian checks your card before walking to the shelves, and never fetches a book your card cannot borrow.

The label is the allowed-groups metadata on each chunk. The card is the user's groups, read from their login. Checking before walking to the shelves is the filter inside the search.

The intern's bot was a librarian who fetched every book, read them all aloud to herself, and was asked to skip the secret ones. Where the picture breaks: a human librarian can keep a secret, but a model can repeat any text in its input.

The permission model has four parts. Our guide on RAG access control explains how to build each one, so here they appear at design level only.

• Copy each document's allowed groups onto every one of its chunks during ingestion.
• At question time, read the user's groups from the company directory using their login, never from anything typed in the chat.
• Put the group filter inside the search query, so blocked chunks are never ranked or returned.
• Deny by default: a document with no permission data is not searchable, and the drafts folder is never ingested.

Do this from the first day. If chunks go into the index without permission data, adding it later means a backfill over all 270,000 chunks, and until that finishes the filter has gaps.

The weak spot is timing. When someone moves from treasury to payables, the directory changes at once, but a cached copy of their groups may linger. The team caches groups for at most 15 minutes, matching the requirement.

OWASP, a non-profit that publishes security guidance, keeps a top 10 list of risks for LLM applications (apps built on large language models). As of 19 October 2026, its 2025 edition lists vector and embedding weaknesses, where an embedding is the vector made from a chunk.

That entry recommends permission-aware vector stores and strict partitioning of data by access. It also warns that users sharing one vector database can leak context to each other. The intern's bot was a simple case of it: one shared index and no filter.

Start with hybrid search, metadata filters and a reranker, and add fancier steps only when tests show a need. This stack is easy to test and handles the codes and numbers that finance text is full of.

Finance documents use exact codes, such as policy FIN-POL-017 or ledger code 4102. Vector search compares meaning, not spelling, so a rare code may not land near a question that contains it. Keyword search matches the exact characters, so the team runs both and merges the results.

That mix is called hybrid search. Our guide on hybrid search explains how the two result lists are merged.

Metadata filters narrow the pool. The permission filter always applies. A version filter keeps only current documents unless the question names a date. A fiscal-year filter applies when the question mentions one, such as 2025-26.

Next comes a reranker, a second model that reads the question and each candidate chunk together and scores how well they match. Search returns the top 40 chunks quickly, and the reranker picks the best 8 for the prompt.

The team leaves out query rewriting, multi-step agent loops and graph-based retrieval for the first release. These rephrase the question, search in several rounds, or follow links between facts. Each adds cost and new ways to fail. Our guide on naive, advanced and modular RAG explains when those upgrades earn their place.

The diagram follows one question through this stack. Notice that the user's groups are fetched before the search, and that the last check can stop an answer from going out.

Give every chunk a stable id that names its document, version and page. Ask the model to return, for each sentence, the id it used and the exact words it relied on. Then check both in code before showing the answer.

A sentence about vendor payments might carry the tag FIN-POL-017 v3 page 4. The screen turns it into a link to that exact page, with the quoted words highlighted.

Citations need checking because models get them wrong. A 2023 research paper built a benchmark for answers with citations. Even the best models it tested lacked full citation support about half the time on one of its datasets.

The team's check is small and runs without any model. Each claim carries a sentence, the chunk id it cites and the quote it copied.

from dataclasses import dataclass

@dataclass
class Chunk:
    id: str      # document, version and page
    text: str

@dataclass
class Claim:
    sentence: str
    chunk_id: str  # the id the model cited
    quote: str     # exact words the model copied from that chunk

def same_words(text):
    return " ".join(text.split())   # ignore line breaks and extra spaces

def check_citations(claims, retrieved):
    by_id = {chunk.id: chunk for chunk in retrieved}
    for claim in claims:
        if not claim.chunk_id:
            return "fail: no source for: " + claim.sentence
        chunk = by_id.get(claim.chunk_id)
        if chunk is None:
            return "fail: cites a chunk it was never given: " + claim.chunk_id
        if len(same_words(claim.quote).split()) < 4:
            return "fail: quote too short for: " + claim.sentence
        if same_words(claim.quote) not in same_words(chunk.text):
            return "fail: quote not found in " + claim.chunk_id
    return "pass"

retrieved = [Chunk("FIN-POL-017 v3 p4",
                   "Vendor payments above Rs 5,00,000 need approval\nfrom the finance controller.")]
good = [Claim("Payments above Rs 5 lakh need the controller's approval.",
              "FIN-POL-017 v3 p4", "need approval from the finance controller")]
invented = [Claim("The CFO must also sign.", "FIN-POL-099 v1 p2", "CFO must also sign")]
misquoted = [Claim("The controller approves above Rs 2 lakh.",
                   "FIN-POL-017 v3 p4", "payments above Rs 2,00,000 need approval")]
unquoted = [Claim("The CFO must also sign every payment.", "FIN-POL-017 v3 p4", "")]
print(check_citations(good, retrieved))
print(check_citations(invented, retrieved))
print(check_citations(misquoted, retrieved))
print(check_citations(unquoted, retrieved))

Running it prints four lines:

pass
fail: cites a chunk it was never given: FIN-POL-099 v1 p2
fail: quote not found in FIN-POL-017 v3 p4
fail: quote too short for: The CFO must also sign every payment.

What just happened? The first answer passes, even though the chunk had a line break inside the quote. The second cites a chunk the search never returned for this user, which catches invented ids.

The third quotes words that are not in the chunk, which catches a misquoted number. The fourth has an empty quote, which would otherwise pass, because an empty string is found in every chunk. A sentence with no id at all fails at the first check.

Scope this check exactly. A quote match proves the words exist in the chunk, not that they back the sentence. So the team adds a second check, where a small model judges each sentence against its quote. The cost table counts both checks.

When any check fails, the assistant does not show the draft. It replies that it found no clear source and lists the closest documents it did find. The CFO prefers this to a confident wrong number.

Many vector indexes are approximate, and some apply your filter after the index scan. A strict permission filter then throws away most of the candidates, and the search quietly returns fewer chunks than you asked for.

An approximate index finds near neighbours fast by checking only part of the data. A popular kind is HNSW, short for hierarchical navigable small world, a graph that links each vector to its close neighbours.

pgvector, the vector search add-on for PostgreSQL, keeps a candidate list of 40 vectors by default during an HNSW search. Its setting hnsw.ef_search controls that size. Its documentation says filtering is applied after the index is scanned.

Pause and predict. A payables analyst can read 10% of the chunks. The search asks for the top 40. How many come back?

The pgvector documentation answers: with a 10% filter and the default list of 40, only 4 rows match on average. We ran it to check.

What just happened? The index found its 40 nearest candidates, then the filter removed the 90% the analyst cannot read. No error appeared. The reranker simply got 3 or 4 chunks instead of 40.

In an enterprise assistant, that shows up as weak answers and refusals for exactly the users with the narrowest access. It looks like a model problem, but it is a search problem.

How to fix it in pgvector

Since version 0.8.0, pgvector offers iterative index scans. When too few rows survive the filter, the scan automatically searches more of the index. Turn it on with the setting hnsw.iterative_scan.

Strict order keeps results in exact distance order. Relaxed order may return them slightly out of order, with better recall, meaning it misses fewer of the true nearest chunks. Relaxed order suits RAG well, because the reranker reorders the 40 chunks anyway.

The documentation also suggests a partial index, built only over rows with one filter value, when there are few distinct values. For many values it suggests partitioning, which splits the table into separate parts by value.

A chunk readable by many groups would need a copy in many partitions. So for overlapping permission groups, we would start with iterative scans.

You can check your own table with this query. Use a row id that exists and a group your test user has. The sample uses the cosine distance operator, which suits an index built with vector_cosine_ops. The comment at the top lists the operators for vector_l2_ops and vector_ip_ops. Swap in the one that matches your index, because pgvector uses the index only when the operator matches. First run the inner SELECT alone with EXPLAIN in front. The plan must show an Index Scan on your HNSW index; a Seq Scan means the test proves nothing.

-- operator must match the index: <=> cosine, <-> L2, <#> inner product
SET hnsw.ef_search = 40;   -- the default, shown for clarity
SELECT count(*) AS rows_back FROM (
  SELECT id FROM chunks
  WHERE allowed_groups && ARRAY['payables']
  ORDER BY embedding <=> (SELECT embedding FROM chunks WHERE id = 1)
  LIMIT 40
) AS hits;

SET hnsw.iterative_scan = relaxed_order;
-- run the same SELECT again

If the group can read about 10% of the chunks, expect a small number from the first query, often well under 10. In our run on a cosine index it returned 3. After turning on relaxed order, the same query returned 40.

As of 19 October 2026, the pgvector README still says filtering is applied after the index scan. We ran these tests on 0.8.1, and iterative scans exist in every release since 0.8.0. Other vector stores filter in their own ways, so run the same count test on yours.

Test it against a golden set of real questions with known answers and sources. Score retrieval, faithfulness, citations, refusals and permission leaks, and run the whole set before every change to prompts, models, chunking or search.

The team and the policy desk write 300 questions from real month-end emails. Each has the correct answer and the document and page it comes from. They add 40 questions with no answer in the documents, where the right reply is a refusal.

They also add 30 permission probes. A probe is a question asked as a user who must not see the answer, such as a payables analyst asking about board salary papers. The pass mark for leaks is zero.

For each run, the team tracks five numbers. Our guide on RAG evaluation metrics explains how each one is calculated.

• Retrieval hit rate: how often the right chunk is in the top 8.
• Faithfulness: how often every sentence is backed by the retrieved chunks.
• Citation accuracy: how often the cited page really contains the fact.
• Refusal accuracy: how often the system declines when the documents hold no answer.
• Leaks: how many permission probes returned anything they should not. This must be zero.

A 2024 survey of RAG evaluation also keeps the two halves apart. It compares metrics for the retrieval part and for the generation part, such as relevance, accuracy and faithfulness. Keeping them apart shows which half to fix.

Add one more check from the filter trap: run the golden set as users from several groups, not only as an admin. An admin account sees everything, so it never hits the short-list problem.

The diagram shows how the golden set fits into everyday work. Follow a thumbs-down from a real user back into the set.

In this design, each question costs about Rs 0.52 in model and search charges, and a typical answer takes about 3.6 seconds. That leaves headroom under the 5-second p95 target. Your numbers will differ, so build the same breakdown with your own prices.

Most model services charge by tokens, small pieces of text, often a short word or part of a word, counted both in and out. The prompt carries the instructions, the question and 8 chunks of about 400 tokens each.

The team keeps the maths in a few lines of Python, so anyone can change one number and see the effect. The prices are made up.

PRICE_IN = 0.00008    # rupees per input token (made up)
PRICE_OUT = 0.0003    # rupees per output token (made up)
FIXED = 0.002 + 0.01 + 0.03 + 0.05   # embed, search, rerank, judge

def cost_per_question(chunks=8, chunk_tokens=400,
                      prompt_tokens=600, answer_tokens=400):
    tokens_in = prompt_tokens + chunks * chunk_tokens
    model = tokens_in * PRICE_IN + answer_tokens * PRICE_OUT
    return round(model + FIXED, 2)

print(cost_per_question())            # 0.52
print(cost_per_question(chunks=5))    # 0.42

Run it and you get 0.52 and then 0.42. The tokens_in line shows where the money goes: the 8 chunks are 3,200 of the 3,800 input tokens.

Sending 5 chunks instead of 8 cuts the cost to Rs 0.42. Make that change only if the golden set shows answers stay as good.

At about 15,000 questions a month, model and search charges come to about Rs 7,800. The index server, ingestion worker and logs cost a fixed Rs 30,000 a month (made up). At this scale, the servers cost more than the model.

Latency, the wait for an answer, is mostly the model writing. The team holds each answer until the citation check passes, so nobody sees an answer that later gets withdrawn. A status line naming the documents being read shows the analyst that work is under way.

Month-end is the real test, with 1,500 questions a day, mostly between 10 am and 1 pm. Before launch, the team runs a load test, sending fake questions at that peak rate plus a margin. It also sets a daily spend alert at twice the normal cost.

Roll it out in phases, each with a small group, a clear exit test and a way back. Widen the audience only when the numbers from the current phase are good.

1. 1

   Phase 1: shadow mode, 2 weeks

   Only the 10 people on the policy desk use it. They answer emails as usual, then compare with the assistant's reply. Exit when citation accuracy passes 95 percent on real questions.

2. 2

   Phase 2: payables pilot, 4 weeks

   The 40-person payables team gets policies only, no reports. A thumbs-down button sends bad answers into the golden set. Exit when one month-end passes with zero leaks.

3. 3

   Phase 3: the whole finance team

   All 180 people get policies and monthly reports. The team watches cost, latency, refusals and short result lists daily through the first two month-end closes.

4. 4

   Phase 4: other departments, later

   Each new department brings its own documents and permission groups. Each one starts again at phase 1, with its own golden set and permission probes.

Every phase keeps an off switch. If leaks or wrong numbers appear, one setting turns the assistant into plain search. It then shows matching documents and passages without a generated answer.

What can still go wrong

• Stale answers: an old policy version wins the search. Guard: the version filter, plus effective dates shown on every citation.
• Permission lag: someone keeps access after moving teams. Guard: the 15-minute group cache and a daily check against the directory.
• Short result lists: a narrow permission filter starves the reranker. Guard: iterative index scans and an alert on chunks returned per search.
• Poisoned documents: a file hides text meant to steer the model. Guard: ingest only trusted folders, and detect hidden text during parsing.
• Over-trust: an analyst copies a number without opening the source. Guard: show the quoted passage and the document date next to every figure.
• Model service outage at month-end. Guard: fall back to search results with highlighted passages.
• Cost creep: prompts grow as features are added. Guard: track cost per question weekly against the Rs 1 target.

OWASP's vector and embedding entry describes this poisoning risk. Its example is a resume with white text on a white background. It advises accepting data only from trusted, verified sources and keeping detailed logs of retrieval activity.

Present it as a case study: requirements and numbers first, then the two paths, then the controls that make it safe for a company.

1. 1

   Clarify users and documents

   Ask who asks, about what, how many documents there are, how fresh they must be, and what must never leak.

2. 2

   Estimate the scale

   Go from pages to chunks to gigabytes, and from users to questions per day at peak. Say whether one machine is enough.

3. 3

   Draw two paths

   Draw the ingestion path that fills the index and the question path that reads it. Keep them separate on the board.

4. 4

   Put permissions in the search

   Groups come from the login, chunks carry allowed groups, and the filter runs inside the search. Say how you make sure it still returns enough chunks.

5. 5

   Make citations checkable

   Use stable chunk ids, a citation check in code, and a clear refusal when support is missing.

6. 6

   Show how you would test

   Describe a golden set with refusal questions and permission probes, run before every change.

7. 7

   Close with cost and rollout

   Give the cost per question, the latency budget, and rollout phases with exit tests and an off switch.