Multimodal RAG: Searching Images, Tables and Scans

A RAG system answers from whatever text it can find, and most of the evidence is not text. A photo, a ticked box or a price in a table never reaches the model, so the answer is wrong or empty. Multimodal RAG fixes that by searching those files in their own form.

Picture a common setup. A motor insurer's claims team handles about 1,400 scooter and car claims a week. Last year its engineers built a RAG assistant over the policy wording. The setup is made up, but the problem is real.

The assistant answers policy questions well, such as the waiting period for an add-on cover. Then the team tries it on real claim files. A typical file has five to eight phone photos of the damage and a scanned, handwritten claim form. It also has a garage estimate, which is a table of parts and prices.

• An adjuster, the person who checks a claim, asks whether claim 4817 shows front or rear damage. The assistant finds no mention of damage, because the answer lives only in the photos.
• She asks whether the customer ticked the box for a third-party injury. The scan was turned into text, and the tick mark was lost on the way.
• She asks for the headlamp price on the garage estimate. The table came out as one long line of words and numbers, so the assistant quotes the labour charge instead.

None of these is a bug in the model. The model never received the information. The rest of this guide fixes each failure in turn, using claim 4817 as the running example.

You need six words: modality, embedding, encoder, vision language model, OCR and caption. Each one gets a plain meaning here and is used the same way all through the guide.

• Modality: a kind of data, such as text, image, audio or video. Tables are often treated as their own kind, because rows and columns carry meaning that plain text loses.
• Embedding: a list of numbers that captures what something means, so similar things get similar numbers. People also call it a vector.
• Encoder: the part of a model that turns one input, such as a sentence or a photo, into an embedding.
• Vision language model: a model that takes images and text together as input and writes text as output.
• OCR, short for optical character recognition: software that reads printed or handwritten letters from an image.
• Caption: a short written description of an image, often written by a vision language model.

Multimodal RAG is RAG that retrieves more than one kind of data, such as text, photos, scanned pages and tables. It passes the best matches to a model that can read them. The question can be typed text, and the answer can come from a photo.

Plain RAG has three steps: index your documents, search them for each question, then add the best matches to the prompt, the input your code sends to the model. Multimodal RAG keeps those steps. What changes is how each kind of file is stored and matched.

A 2025 survey of the field puts the extra difficulty in one line. It says cross-modal alignment and reasoning bring challenges beyond text-only RAG. Alignment means matching an image to the words that describe it, and reasoning means combining evidence from different kinds of data.

A picture from Chor Bazaar

Think of an antique dealer in Mumbai's Chor Bazaar pricing an old brass lamp. He looks at the shape and the wear, reads any maker's stamp on the base, and checks old papers like a receipt. A text-only RAG system is a dealer who can only read the papers.

A text question finds a photo when both are turned into embeddings in one shared space. A matching sentence and picture land close together, so the search simply returns the nearest image embeddings.

The idea comes from how such encoders are trained. A well-known 2021 paper trained an image encoder and a text encoder together on 400 million image and text pairs from the internet. The single training task was to predict which caption goes with which image.

Training pulls each image towards its own caption and away from the other captions in the same batch. So a photo of a cracked headlamp and the words cracked headlamp tend to land near each other, though nobody wrote a rule about headlamps.

For the claims team, this means an adjuster can type front bumper scraped and get matching photos back, even when nobody wrote a description of them. One rule matters: the question must be embedded with the text encoder trained alongside the image encoder. A different text model's numbers live in a different space, so comparing them gives meaningless matches.

The same paper is honest about the limits. It reports the model reads digitally rendered text well. Without extra training, it reaches only 88 percent accuracy on handwritten digits, below a very simple method that looks only at raw pixel values. It also struggles to count objects in an image.

You can build multimodal RAG by turning everything into text, by embedding images and text into one shared space, or by searching whole page images directly. Each trades cost, accuracy and effort differently, and most systems use more than one.

Option 1: turn everything into text

Here every file becomes text before indexing. Scans go through OCR, and photos get a caption from a vision language model. After that it is normal text RAG, so you reuse the search you already have.

The weakness is that anything the conversion misses is gone for good. The tick mark on the claim form is the classic loss. Layout, stamps and table structure often go the same way.

Option 2: embed images and text into one space

Here each photo gets an image embedding, and each question gets a text embedding from the matching encoder. Both go into a vector database, a store built to find the nearest embeddings fast.

This keeps details a caption might skip, like where on the panel a scratch sits. The limits from the 2021 paper apply, though: it is weak on handwriting and counting, so do not trust it to read a policy number off a photo.

Option 3: search whole page images

Here you skip text extraction for documents. Each scanned page is stored as an image, a vision language model turns the page into embeddings, and the question is matched against the page as a picture.

A 2024 paper on this idea calls text extraction pipelines lengthy and brittle. Its method keeps one vector per small square patch of the page and compares every piece of the question with every patch. This is called late interaction, explained in the engineer section below.

On a benchmark they built, called ViDoRe, the authors report that their method largely beats modern document search pipelines while being simpler and faster. A second 2024 paper, which also retrieves page images, reports a 20 to 40 percent end-to-end gain over a traditional text-based RAG pipeline in its experiments.

Research benchmarks are not your scans. Test any of the three options on a sample of your own files before you pick one.

Keep their structure. Put table numbers into rows of a database, read form fields as named values, and index a short summary so search can still find them. Do not flatten them into one line of text.

Flattened, the garage estimate for claim 4817 became: headlamp 2850 front panel 4200 labour 1500 total 8550. A text search for the headlamp price finds this line, but the model may pick the wrong number next to it.

Instead, the team reads the table into rows with columns for part, quantity and price. The rows go into a SQL table, a database you query with exact conditions. Here is the lookup, runnable with Python's built-in sqlite3 module.

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE estimate_rows (claim_id TEXT, part TEXT, qty INTEGER, price_rs INTEGER)")
db.executemany(
    "INSERT INTO estimate_rows VALUES (?, ?, ?, ?)",
    [("4817", "front panel", 1, 4200),
     ("4817", "headlamp", 1, 2850),
     ("4817", "labour", 1, 1500)],
)

price = db.execute(
    "SELECT price_rs FROM estimate_rows WHERE claim_id = ? AND part = ?",
    ("4817", "headlamp"),
).fetchone()[0]
total = db.execute(
    "SELECT SUM(qty * price_rs) FROM estimate_rows WHERE claim_id = ?",
    ("4817",),
).fetchone()[0]
print("headlamp:", price, "rupees")
print("total before tax:", total, "rupees")

Run it with python3 and you should see two lines: headlamp: 2850 rupees, then total before tax: 8550 rupees. The price comes from one exact row, so there is nothing for a model to misread.

Search still needs to know the table exists. So the team also embeds a short summary of each table, such as: garage estimate for claim 4817, front panel and headlamp, plus labour. A question about the estimate finds the summary, and code then fetches the rows.

Scanned claim forms get similar care. A layout-aware reader, one that knows where each box sits on the form, turns each field into a name and a value, like third_party_injury: yes. Fields it cannot read with confidence are marked unclear and sent to a person, never guessed.

The page image is kept and embedded too. If the reader misses a tick, page image search can still find the page, and the adjuster can see the original.

It scores a page by late interaction. Each piece of the question finds its best-matching patch on the page, and the page's score is the sum of those best matches. A page wins when every part of the question finds a strong match somewhere on it.

The 2024 paper defines the score exactly that way: for each question vector, take the largest dot product with any page vector, then add them up. A dot product multiplies two lists number by number and sums the results, so for vectors of the same length, a bigger value means a closer match.

Pause and predict: the question is whether the injury box is ticked. Which page should win, the claim form or a damage photo? Here is the scoring in plain Python with tiny made-up vectors.

def late_interaction(question_vecs, page_vecs):
    score = 0.0
    for q in question_vecs:  # one vector per piece of the question
        best = max(sum(a * b for a, b in zip(q, p)) for p in page_vecs)
        score += best        # keep only the best-matching patch
    return round(score, 2)

# Made-up 3-number vectors: [tick mark, injury wording, vehicle picture]
question = [[0.0, 1.0, 0.0],   # "injury"
            [1.0, 0.0, 0.0]]   # "ticked"
claim_form = [[0.1, 0.9, 0.0],   # patch with the printed word Injury
              [0.9, 0.2, 0.0],   # patch with a ticked box
              [0.0, 0.0, 0.1]]   # blank margin
damage_photo = [[0.0, 0.1, 0.9],
                [0.1, 0.0, 0.8],
                [0.0, 0.2, 0.7]]
print("claim form:", late_interaction(question, claim_form))
print("damage photo:", late_interaction(question, damage_photo))

Running it prints claim form: 1.8 and damage photo: 0.3. The word injury matches the patch with the printed label, and ticked matches the patch with the tick mark. No single patch had to match the whole question.

What page image search costs in storage

The cost is many vectors per page. With the 2024 paper's main model, each page becomes 1,024 image patches plus 6 extra vectors, 1,030 in all. Each vector is cut down to 128 numbers, and the paper reports about 257.5 KB per page.

You can check that figure if each number takes 2 bytes. Run python3 -c "print(1030 * 128 * 2)" and it prints 263680 bytes, which is 257.5 KB at 1,024 bytes per KB. For comparison, one vector of 1,024 numbers at 4 bytes each is 4 KB, about 64 times smaller.

At the claims team's made-up scale of 1,400 claims a week with three scanned pages each, that is 218,400 pages a year. Their vectors alone need about 53.6 GiB (gibibytes, 1,024 times 1,024 times 1,024 bytes each), against about 0.83 GiB for one 1,024-number vector per page. Index overhead comes on top.

The paper also tests token pooling: averaging together patch vectors that repeat the same information, such as plain white background. With a pool factor of 3, about one vector in three is kept: vectors drop by 66.7 percent while 97.8 percent of the original retrieval performance remains. The paper notes that text-dense pages may lose more accuracy under pooling, because they have fewer repeated patches.

You design it in two halves. An indexing pipeline sorts each file by kind and stores it in the right form. A question path searches those stores and builds the prompt. Most of the hard work sits in indexing.

Start at the upload in the diagram below and find the question box. Each kind of file takes its own route, and every route ends with records tagged with the claim id.

Photos and scans get two embeddings each: one for the written caption or fields, one for the image. A search can then hit either, which covers for a weak caption. Every record also stores the claim id, the file name and the file kind.

Keep the original file store separate from the indexes. Your caption and OCR tools will change, and you will want to re-index old claims from the originals.

It searches the text and image embeddings and merges the two ranked lists. Then it fetches exact numbers from the database and sends the best text and images to a vision language model. The model answers and names the files it used.

The adjuster's real question is: does the damage in the photos match the garage estimate for claim 4817? Follow the boxes in order. Each box notes who talks to whom.

Merging two ranked lists

A text search score and an image search score are on different scales, so you cannot compare them directly. A 0.31 from one search may mean more than a 0.80 from the other.

One simple fix is reciprocal rank fusion, from a 2009 paper by Cormack, Clarke and Büttcher. It ignores raw scores and uses only each result's position. Each list adds 1 / (k + rank) to a result, so results high in both lists rise most.

def reciprocal_rank_fusion(result_lists, k=60):
    scores = {}
    for results in result_lists:
        for rank, file_id in enumerate(results, start=1):
            scores[file_id] = scores.get(file_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

# Both searches already ran with the filter claim_id = 4817
by_text = ["estimate_summary", "photo_3", "policy_page_12"]
by_image = ["photo_1", "photo_2", "photo_3"]

print(reciprocal_rank_fusion([by_text, by_image]))
print(reciprocal_rank_fusion([by_text, by_image], k=0))

The first line prints photo_3 first, then estimate_summary, photo_1, photo_2 and policy_page_12. Photo 3 was second and third in the two lists, and agreement beat a single first place. The scores of estimate_summary and photo_1 tie exactly, so their order only follows the order of the input lists.

The second line uses k=0 and puts estimate_summary and photo_1 first, tied again, with photo_3 third. The 2009 paper says the constant k softens the pull of a very high rank from one outlier list. Its authors fixed k at 60 in a pilot study and found the exact value was not critical.

The filter that keeps claims apart

Both searches must run with the filter claim_id = 4817, set by code from the claim the adjuster opened. The model never chooses it. With the filter applied correctly, another customer's photos cannot reach the prompt through search.

Check how your vector database applies that filter. pgvector, a vector search extension for PostgreSQL, says in its README, the project's main documentation page, that with approximate indexes, filtering is applied after the index is scanned. An approximate index trades a little accuracy for speed.

Its example: a condition matching 10 percent of rows, with an HNSW index and the default hnsw.ef_search of 40, returns only 4 rows on average. HNSW is a graph-based approximate index, and ef_search sets how many candidates one scan collects.

One claim's files are a tiny share of all rows, so a scan like that could return few files or none. The README suggests starting with an index on the filter column, which can give fast exact search in many cases. Exact search finds every true nearest match, which the README calls perfect recall. Version 0.8.0 also added iterative index scans, which keep scanning until enough rows match or a set limit is reached.

What comes back for claim 4817

1. 1

   Search finds the photos

   The top results are three damage photos, all showing the front of the scooter. Their captions say front panel cracked and headlamp broken.

2. 2

   Search finds the estimate summary

   The summary of the garage estimate also ranks high, because it mentions the front panel and the headlamp.

3. 3

   Code fetches exact rows

   The estimates table returns front panel Rs 4,200, headlamp Rs 2,850 and labour Rs 1,500, a total of Rs 8,550 before tax.

4. 4

   The model compares

   The model sees the three photos and the rows. It answers that the photos show front damage matching both parts, and names each photo file.

5. 5

   A person decides

   The adjuster approves the claim. The assistant gives evidence, but the decision to pay stays with a person.

Now picture another claim where the estimate lists a rear panel but every photo shows the front. That mismatch is exactly what the team wanted the tool to catch. Before, a person had to open every photo by hand to spot it.

Multimodal RAG adds failures that text RAG does not have: misread scans, weak captions, mixed-up claims, leaked personal data, reused photos and instructions hidden inside images. Each one needs a guard you can test.

• Misread scans. OCR turns a handwritten 8 into a 3. Guard: keep a confidence score per field, and send low-confidence fields to a person.
• Weak captions. A caption says vehicle damaged and nothing more. Guard: give the caption model a checklist, such as which side, which part and how bad.
• Wrong claim. A similar photo from another claim scores high. Guard: the claim id filter set in code, tested as described above.
• Leaked personal data. A scanned driving licence shows a name, address and number. Guard: only people allowed to see a claim can search it, and logs store file ids, not page text.
• Reused photos. The same dent photo shows up in two claims, resized. Guard: a perceptual hash, a short code computed from what the picture looks like, checked at upload.
• Hidden instructions. Text inside an image tries to steer the model. Guard: treat everything retrieved as data, and never let the model trigger payments.

Why a perceptual hash and not a normal file hash? A normal hash such as SHA-256 changes completely when one byte changes. We drew a synthetic 1600 by 1200 pixel test image, resized it to 800 by 600 and re-saved it as a lower-quality JPEG.

Its SHA-256 no longer matched, but the 64-bit perceptual hash from the ImageHash library (version 4.3.2) was identical, a distance of 0 bits. A different picture was 26 bits away. Pick your match threshold by testing on your own photos.

Hidden instructions are a known attack. The OWASP Top 10 for LLM Applications, 2025 edition, lists prompt injection first, as LLM01. OWASP is a non-profit that publishes security guidance, and an LLM is a large language model.

Its page describes multimodal injection: an attacker hides instructions in an image that comes with harmless text. For an insurer, anyone can upload a photo with a claim. A line of small print in its corner could tell the model to call the damage minor, or severe.

Cost needs watching too. How many input tokens, the units a model reads and bills, an image uses depends on its size and your model provider, so check the provider's pricing page. The team resizes photos before sending them, and sends at most four images per question.

Test it with a fixed set of real questions whose right answers and right files you already know, and score the search and the answer separately. That tells you which half failed when something goes wrong.

The claims team picks 150 closed claims and writes 300 questions about them (made up). For each question, they mark which photo, page or table row holds the answer. It is slow work, and the most useful work in the project.

• Retrieval: is the right file in the top 5 results? Count this per kind of file, because photos and scans fail differently.
• Reading: did the scan reader get each form field right? Compare against values a person typed in by hand.
• Answer: is the answer correct, and does it name the right files? A correct answer with no evidence still counts as a fail.
• Isolation: does a one-claim search return exactly that claim's files, and the expected number of them?
• Safety: do planted test images with hidden text change the answer? They should not.

Splitting the scores by file kind changes what gets fixed first. In the first run, typed letters are found 9 times in 10, but handwritten forms only 6 times in 10 (made up). So the team moves handwritten forms to page image search first.

You do not need it when the useful facts are already clean text, or when a simple extraction step reliably turns them into text. Start with the cheapest route that passes your tests, and move up only when it fails.

The insurer's policy wording is typed PDF text, so plain text RAG stays. Typed letters from garages go through OCR, after a check on samples shows it reads them correctly. Only photos and handwritten forms need the heavier image routes.

Follow the decision below from the top for any new kind of file. Each yes ends at the cheapest route that fits.

Explain it as normal RAG with one extra job: deciding how each kind of file is stored and searched. Then show the guards that keep evidence correct and private.

1. 1

   List the kinds of data

   Ask what the files really are: typed text, photos, scans, tables. Ask which questions users will ask about each.

2. 2

   Pick a route per kind

   Text stays text, tables go to SQL with a summary, and photos and handwritten forms get image embeddings. Say why for each.

3. 3

   Draw the indexing pipeline

   Show the original file store, the sorting step, one worker per kind, and the stores they write to, all tagged with an owner id.

4. 4

   Draw the question path

   Search text and image embeddings with a filter set in code, merge with reciprocal rank fusion, fetch exact values from SQL, then send text and images to the model.

5. 5

   Size the storage

   If you choose page image search, estimate vectors per page times numbers per vector times bytes per number, and mention pooling.

6. 6

   Add the guards

   Access checks, confidence scores on scanned fields, perceptual hashes for reused photos, untrusted image content, and people for money decisions.

7. 7

   Say how you will measure

   A test set with known answer files, scored for retrieval and answers separately, and split by file kind.