Keeping a RAG Index Fresh: Updates, Deletes and Re-indexing

When a RAG index falls behind, the chatbot answers from old text even though the real document has changed. The model is not wrong. It is reading the wrong page. How closely the index keeps up with the documents is called RAG index freshness.

Picture a common setup. A hospital chain's HR team runs a policy assistant for about 4,200 staff across six hospitals. Nurses, ward staff and doctors ask it around 900 questions a day. The setup is made up, but the problem is real.

The assistant answers from 380 HR policy documents about leave, shifts, allowances and uniforms. A batch job rebuilds its whole search index every Sunday night.

On Thursday, 1 October 2026, a new leave policy starts. Staff can now carry forward 30 days of earned leave into the next year, down from 45. HR uploads the new PDF that morning.

Three things then go wrong, one after another.

• From Thursday to Sunday night, the index has never seen the new PDF. Every leave question gets the old 45-day rule.
• The new file is named Leave Policy 2026, so the old Leave Policy 2024 file still sits in the folder. Sunday's rebuild indexes both of them.
• After Sunday, some questions still pull the 2024 chunks, because their wording happens to sound closer to the old text.

By the next Friday, 37 staff have been told the old number. One ward nurse has planned 40 days of leave around it. The HR head asks the platform team a fair question: how did the assistant not know?

Here is the path the leave policy took. Look at where the old file stays alive, and at the gap between the upload and the rebuild.

Each section below closes one of these gaps on the same assistant. First, a few words you will need.

You need six words. Each is a plain idea with a technical name, and the rest of the guide uses them without stopping.

• RAG, short for retrieval-augmented generation: before answering, the system searches your documents and pastes the closest passages into the prompt it sends to the model.
• Chunk: a small piece of a document, often a few paragraphs. Documents are cut into chunks so search can return just the relevant part.
• Embedding: a list of numbers that captures the meaning of a chunk, made by an embedding model. Texts with similar meaning get similar numbers.
• Vector index: a store that finds the chunks whose embeddings sit closest to the question's embedding. It searches by meaning rather than exact words.
• Metadata: extra fields stored next to each chunk, such as document ID, version, start date and status.
• Upsert: one write that inserts a chunk if its ID is new and overwrites it if the ID already exists.

RAG index freshness is how closely the chunks in your index match your source documents as they are right now. A fresh index reflects every new, changed and deleted document soon after the change.

Freshness has two parts you can measure. Lag is how long a change takes to become searchable. Drift is any chunk in the index that no longer matches a live document version.

Picture a mobile phone shop. The owner drops a phone's price from Rs 18,999 to Rs 15,999 in the billing software, but nobody changes the paper tag on the counter. Customers read the tag and walk away.

Map it part by part. The billing software is your source documents. The paper tag is your index. The customer is the model, which trusts whatever tag it reads.

Where the picture breaks: a shop has one tag per phone. An index cuts each policy into many chunks, so half a policy can be fresh while the other half is stale.

Notice that the customer did nothing wrong, and neither did the assistant's model. A bigger model would read the same old chunk and give the same old answer.

A 2023 paper by Vu and colleagues on refreshing language models with search results points the same way. On its questions about fast-changing facts, models of every size struggled.

Adding current search results to the prompt substantially improved the answers. The catch is that retrieval only fixes old knowledge if what you retrieve is itself current.

A change reaches the index through a pipeline: notice the change, load the new text, cut it into chunks, embed each chunk, and write the chunks with their metadata. Each step adds lag.

Here is the pipeline the platform team builds after the incident. Follow a change from the HR portal into the index, then find the second, slower path through the nightly check.

The worker's main write is an upsert. Its other write is a delete, and it matters just as much.

When HR withdraws a policy, the portal sends a withdrawn event and the worker removes every chunk of that policy. Hiding the file in the portal is not enough, because the index keeps its own copy.

You detect changes in one of four ways: rebuild on a timer, poll for new timestamps, compare content hashes, or listen for change events. A sound setup combines events with a regular hash check.

A timestamp poll asks the source for every file changed since the last run. It is simple, but on its own it misses deletes. A deleted file has no new timestamp to show you.

A content hash is a short fingerprint computed from a file's bytes. The same text always gives the same hash. In our test, changing one full stop to a comma changed 60 of the 64 characters of a SHA-256 hash, a common hash function.

A change event is a message the source sends the moment something happens. When the source is a database, change data capture (CDC) produces these events by reading the database's own change log.

Debezium, an open-source CDC tool, works this way. Its version 3.7 documentation, as of 19 October 2026, says log-based capture records all data changes and can capture deletes. For MySQL or PostgreSQL, it puts the delay in the millisecond range.

The team uses change events for speed and a nightly hash check as a safety net. The nightly job is called reconciliation: comparing two lists and fixing every difference between them.

Back at the phone shop, change events are the owner calling out each price drop. Reconciliation is one person walking the counter every evening with the billing printout, checking tag by tag.

Here is the core of that nightly check. It runs on Python 3 with no extra packages. The portal side lists only live documents, and the index side keeps the hash saved at indexing time.

import hashlib

def fingerprint(text):
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

def plan_sync(source_docs, indexed_hashes):
    """source_docs: {doc_id: text}. indexed_hashes: {doc_id: hash}."""
    to_upsert, to_delete = [], []
    for doc_id, text in source_docs.items():
        if indexed_hashes.get(doc_id) != fingerprint(text):
            to_upsert.append(doc_id)
    for doc_id in indexed_hashes:
        if doc_id not in source_docs:
            to_delete.append(doc_id)
    return to_upsert, to_delete

indexed = {"LEAVE-01": fingerprint("Carry forward up to 45 days."),
           "PARKING-06": fingerprint("Staff parking is at Gate 2."),
           "UNIFORM-02": fingerprint("Blue scrubs on all wards.")}
portal = {"LEAVE-01": "Carry forward up to 30 days.",
          "UNIFORM-02": "Blue scrubs on all wards.",
          "CANTEEN-04": "Canteen open 7 am to 10 pm."}
print(plan_sync(portal, indexed))

Run it and you should see this one line:

(['LEAVE-01', 'CANTEEN-04'], ['PARKING-06'])

You update a document by writing all its new chunks, then deleting all its old chunks, found through a stable document ID and a version number. Do not edit single chunks in place.

Editing in place fails because one change near the top of a policy can shift every chunk boundary after it. Chunk 4 of the old file and chunk 4 of the new file may hold different paragraphs.

The naive first try numbers each document's chunks 0, 1, 2 and so on, and overwrites them by number on every change. It looks tidy, and it is wrong.

The script below answers it. It compares the naive overwrite with a safe replace that builds each chunk ID from document ID, version and chunk number, like LEAVE-01:v2:3. Plain dictionaries stand in for the index.

def split_into_chunks(text, size=40):
    return [text[i:i + size] for i in range(0, len(text), size)]

def overwrite_by_number(index, doc_id, text):
    for n, chunk in enumerate(split_into_chunks(text)):
        index[f"{doc_id}:{n}"] = {"doc_id": doc_id, "text": chunk}

def replace_document(index, doc_id, version, text):
    for n, chunk in enumerate(split_into_chunks(text)):
        index[f"{doc_id}:v{version}:{n}"] = {
            "doc_id": doc_id, "version": version, "text": chunk}
    # Only after the new version is in: remove every older chunk.
    old = [cid for cid, c in index.items()
           if c["doc_id"] == doc_id and c["version"] < version]
    for cid in old:
        del index[cid]

old_text = "Carry forward up to 45 days. " * 16  # 12 chunks
new_text = "Carry forward up to 30 days. " * 12  # 9 chunks

naive = {}
overwrite_by_number(naive, "LEAVE-01", old_text)
overwrite_by_number(naive, "LEAVE-01", new_text)
print("naive:", len(naive), "chunks,",
      sum("45" in c["text"] for c in naive.values()), "still say 45")

safe = {}
replace_document(safe, "LEAVE-01", 1, old_text)
replace_document(safe, "LEAVE-01", 2, new_text)
print("safe:", len(safe), "chunks,",
      sum("45" in c["text"] for c in safe.values()), "still say 45")

Run it on Python 3 and you should see:

naive: 12 chunks, 3 still say 45
safe: 9 chunks, 0 still say 45

The safe version writes the new chunks first, and only then deletes the old version. There is never a moment when the assistant has no leave policy at all.

For a few seconds both versions exist side by side. The metadata filter in the next section makes sure only the right one reaches the model.

Search engines follow the same pattern. Apache Lucene, an open-source search library, has an updateDocument call that first deletes the matching documents, then adds the new one.

Its version 10.5.2 documentation says the delete and the add are atomic as seen by a reader on the same index. A reader sees the old document or the new one, never a gap.

You stop them competing by storing status, version and start dates on every chunk, then filtering on those fields inside the search. Similarity alone should never decide which version is current.

Similarity search finds text that sounds close to the question. It knows nothing about dates, so the 2024 chunks won some searches even though that rule had ended.

Back at the phone shop, say the staff stick the new tag on the phone but leave the old tag beside it. Customers believe whichever tag they notice first.

The fix is to remove the old tag and have one rule about which tag counts. In a RAG index, that rule lives in metadata and code, not in the model.

Policies add one more twist: a policy can be announced before it starts. HR often uploads changes two weeks early. Until the start date, the old rule is still the right answer.

Here is the check the assistant applies to every candidate chunk before ranking. Start at the top and follow each question down.

The team keeps one small version table in an ordinary database, with one row per policy and the version in force today. It is the single place that says which version counts. An upcoming version joins the table on its start date.

The best place for these checks is inside the search itself, as a metadata filter. Done right, filtered-out chunks never take a place in the top results. The engineer section below shows how this can go wrong.

With this in place, a nurse asking in late September gets the 45-day rule, plus a note that 30 days applies from 1 October. After that date, she gets only the 30-day rule.

A filtered vector search comes back short when the store applies the filter after an approximate index has already picked its candidates. This is called post-filtering. It bites hardest when the filter removes the closest matches, such as withdrawn chunks.

pgvector is the PostgreSQL extension for vector search. Its HNSW index (short for hierarchical navigable small world) is a graph of vectors that it walks to find close ones quickly, without comparing the question with every row.

The pgvector README explains the catch. An HNSW query gathers a fixed list of candidates, 40 by default through the hnsw.ef_search setting, and the WHERE clause is applied after that.

So if a condition matches 10 percent of rows, the README says only 4 rows will match on average. Now picture withdrawn leave chunks sitting right next to the question.

We tested this on 19 October 2026 with pgvector 0.8.7 on PostgreSQL 18.6, run through PGlite 0.5.11. The table held 300 withdrawn chunks near the question and 300 active chunks further away.

The query asked for the 5 closest active chunks. On such a small table, PostgreSQL's planner, the part that picks how to run a query, preferred reading every row. So we turned that off to force the HNSW index, as a big table usually would.

LIMIT 5 with filter, HNSW index, default settings -> 0 rows
same query with hnsw.iterative_scan = relaxed_order -> 5 rows
exact search, no index -> 5 rows

With default settings, the query asked for 5 rows and got none. The 40 candidates were all withdrawn chunks, so the filter threw every one of them away.

Three fixes work, and you can combine them.

• Turn on iterative index scans, added in pgvector 0.8.0, with SET hnsw.iterative_scan = relaxed_order. The scan reads more of the index until enough rows pass, up to a set limit.
• Build a partial index over active chunks only, which the README suggests when you filter on a few distinct values.
• Delete withdrawn chunks from the live index instead of only flagging them. A row that is gone cannot crowd out good ones.

In our test, iterative scans brought back all 5 active chunks. The setting was off by default in 0.8.7, and the scan stops at hnsw.max_scan_tuples, 20,000 by default.

A delete often hides a record first and frees its space later. Until that cleanup, the deleted text or vector can still sit on disk, even though searches stop returning it.

Lucene stores an index as segments, the files that make up the index. Deleted documents stay inside their segment until a merge rewrites it.

Its IndexWriter documentation, version 10.5.2, covers forceMergeDeletes, which merges segments that hold deleted documents. When you call it, the default merge policy only picks segments where over 10 percent of documents are deleted.

The same page warns that forcing this merge is often very costly and rarely warranted. Plan for deleted data to linger until normal merges reach it.

Vector libraries can behave the same way. In hnswlib 0.8.0, an open-source HNSW library, mark_deleted hides a vector from search results but keeps it in the index.

We tested it with 1,000 vectors. After mark_deleted, the element count stayed at 1,000 and the search skipped the vector. Then unmark_deleted brought it straight back, so the data had never left.

A full index also stays full. Adding one more vector failed until we built the index with allow_replace_deleted and added the vector with replace_deleted=True. Together they let a new vector reuse a deleted vector's slot. Raising the limit with resize_index also worked.

Read your own store's documentation on deletes and compaction, the cleanup step that really frees the space. It matters most when you must prove that some data is gone.

Re-index everything when something changes for every chunk at once, such as the embedding model, the chunk size or the parser (the code that pulls text out of files). Ordinary document edits never need a full rebuild.

Each embedding model places meaning along its own directions, so vectors from two separately trained models cannot be meaningfully compared, even when they have the same length.

A small test shows why. We built a toy word-counting model A, then a model B that is an exact rotation of A: the same vectors, turned to face new directions. Inside each model, every similarity score came out identical.

Mixed together, they fell apart. We scored with cosine similarity, a number from minus 1 to 1 for how closely two vectors point the same way.

A model B question scored the leave chunk at minus 0.09 and ranked the uniform rules first, at 0.087. The same sentence embedded by A and B scored minus 0.059 against itself.

When the lengths differ, you get an error instead of nonsense. pgvector stopped our test query with the message: different vector dimensions 3 and 4.

So a model change means re-embedding all 380 documents. These are the usual reasons for a full rebuild, and the usual non-reasons.

• Rebuild: you switch to a new embedding model.
• Rebuild: you change chunk size or overlap, the text shared between neighbouring chunks.
• Rebuild: you adopt a better parser, for example one that reads tables properly.
• Rebuild: a bug wrote wrong metadata onto many chunks.
• No rebuild: a policy is edited, added, renamed or withdrawn. The normal update path handles all of these.

The safe way is a blue-green rebuild. You build a complete new index beside the live one, test it, then move traffic over in one step. The old index stays ready in case you need to move back.

The move uses an alias: a fixed name the app searches, which points at one real index. Elasticsearch supports this pattern, and its aliases API can run several actions in a single atomic operation.

Its documentation, as of 19 October 2026, shows an alias swapped from one target to another with no downtime. During the swap, the alias never points at both targets.

Here is the team's rebuild for a new embedding model. Notice that the assistant only ever talks to the alias, never to an index by its real name.

One detail catches many teams. Policy changes keep arriving while v2 is being built. If they reach only v1, v2 goes live already hours out of date.

So the worker writes every change to both indexes until the swap. After the swap, the team runs the reconcile job once against v2, to be sure nothing slipped through.

Measure freshness with three numbers: the lag from change to searchable, the drift the nightly check finds, and how many test questions get the current rule. Each one needs an alert.

• Freshness lag. For each change, record when it was published and when its chunks became searchable. Alert when the gap passes your target.
• Drift count. Each night, count the documents the reconcile job had to fix. Any number above zero means the event path missed something.
• Canary questions. Keep a set of questions whose answers just changed, like the carry-forward limit, and ask them every hour.
• Dates in every answer. Show the policy name and its start date in each reply, so staff can spot an old rule themselves.

The HR head and the platform team agree on a freshness SLO, short for service level objective: a target you promise and measure. Policy changes must be searchable within 15 minutes, for 99 of every 100 changes.

You can check that target with one PostgreSQL query, if the worker logs two times per change: when it was published and when its chunks became searchable.

SELECT count(*) AS changes,
       round(100.0 * avg(on_time), 2) AS pct_within_15_min,
       avg(on_time) >= 0.99 AS meets_slo
FROM (
  SELECT (coalesce(searchable_at, now()) - published_at
          <= interval '15 minutes')::int AS on_time
  FROM index_log
  WHERE published_at > now() - interval '30 days'
    AND published_at < now() - interval '15 minutes'
) AS recent;

The last WHERE line skips changes less than 15 minutes old, because they cannot have missed the target yet. On our test log of 199 changes, the query returned 199 changes, 98.99 percent and a meets_slo of false. Two late changes out of 199 is more than 1 in 100, so the target is missed.

Two details stop a falsely healthy number. One test change never became searchable, so its searchable_at was empty, and coalesce counts it as late. Without coalesce, that change drops out and the query reports 99.49 and true. Rounding hides misses too: 98.99 rounded to one decimal shows 99.0, so meets_slo compares the unrounded average.

The drift check lists chunks that have already started but whose document and version are missing from the live version table. Upcoming versions are skipped, because they wait in the index until their start date. A healthy index returns no rows.

SELECT c.doc_id, c.version, count(*) AS stray_chunks
FROM chunks c
LEFT JOIN live_versions l ON l.doc_id = c.doc_id AND l.version = c.version
WHERE l.doc_id IS NULL
  AND c.effective_from <= current_date
GROUP BY c.doc_id, c.version
ORDER BY c.doc_id;

We ran it on a smaller test set than the orphan example, plus one chunk of an upcoming shift policy. It skipped the upcoming chunk and found two leftover chunks of the old leave policy and one of a withdrawn parking policy:

doc_id     | version | stray_chunks
------------+---------+--------------
 LEAVE-01   |       1 |            2
 PARKING-06 |       1 |            1

The next policy change tests the whole setup. A new night-shift allowance of Rs 1,500 per shift is searchable 4 minutes after upload (made up). The old allowance chunks are gone in the same run, and the canary question passes.

Explain it as a data pipeline with three paths: fast updates, safe deletes and planned full rebuilds. Then show how you would measure each one.

1. 1

   Ask how documents change

   Ask how often documents change, who changes them, and whether old versions must be kept for audit. The answers set your lag target.

2. 2

   Pick change detection

   Use change events for speed and a nightly reconcile job that compares hashes, so lost events still get fixed.

3. 3

   Design stable chunk IDs

   Build IDs from document ID, version and chunk number. Replace a document by writing new chunks first, then deleting every older chunk.

4. 4

   Filter by metadata

   Store status, version and start dates on every chunk. Filter inside the search, and test that a filtered query still returns enough rows.

5. 5

   Plan full rebuilds

   For a new embedding model or chunking change, build a second index, test it, and swap an alias. Write changes to both meanwhile.

6. 6

   Measure and alert

   Track lag, drift and canary questions, each with an alert. Clear cached answers whenever a source document changes.