ColPali and ColQwen2 for visual document retrieval: GPU, index size and vector database

Page images instead of OCR text

ColPali retrieves scanned, table-heavy or chart-heavy PDF pages by their image, without OCR, layout detection or chunking: a vision-language model turns every page into a set of small vectors, one per image patch, and a query is matched against all of them by late interaction. Indexing needs one forward pass of a 2B to 3B model per page, which an L4 or L40S handles. What grows is the index: about 264 GB per million pages for ColPali in 16-bit, against about 9 GB for one text vector per chunk.

The ColPali paper (arXiv 2407.01449, version 6 of 28 February 2025) builds on PaliGemma-3B. Its model card shows the shape of one page's embedding as (1030, 128): 1,024 image patches, which a 448 × 448 input split into 14 × 14-pixel patches gives, plus 6 text tokens, each projected to 128 dimensions. The paper puts this at "a memory footprint of 257.5 KB per page", counting 1,024 bytes to the kilobyte; in decimal units it is 263.7 KB, the figure our tables use.

Scoring follows ColBERT: for every query vector, the most similar page vector is found, and these maxima are summed into the page score, which Qdrant and Milvus call MaxSim. Our guide to embedding and reranker servers for RAG sizes the text route, and vision-language models for document AI sizes OCR and extraction with VLMs. This article covers the retrieval of page images, which neither of them sizes.

Visual retrievers and their licences

Hugging Face model cards of vidore/colpali-v1.3, vidore/colqwen2-v1.0, vidore/colqwen2.5-v0.2 and nvidia/llama-nemotron-colembed-vl-3b-v2, and the Qwen2.5-VL-3B-Instruct LICENSE file, read 24 October 2026; NVIDIA's tile count is a maximum (8 tiles plus a thumbnail), the product 2,304 is our arithmetic.

For company use the backbone licence decides. ColQwen2 v1.0 states that its backbone "is under apache2.0 license" and its adapters under MIT, so it has no commercial restriction. ColPali v1.3 says its backbone "is under gemma license as specified in its model card". ColQwen2.5's backbone is under the Qwen Research License Agreement of 19 September 2024, which grants rights "FOR NON-COMMERCIAL PURPOSES ONLY" and states "If you are commercially using the Materials, you shall request a license from us." NVIDIA's llama-nemotron-colembed-vl-3b-v2, released on 26 January 2026, "is for non-commercial/research use only". Whether a given use falls under one of these licences is a question for your legal department.

ColQwen2 and ColQwen2.5 take dynamic resolutions, and their cards set the maximum so that "768 image patches are created at most"; the Qdrant tutorial puts a typical page at about 700. Both cards note that the models focus on "PDF-type documents and high-ressources languages", so test them on German, Polish or Czech pages of your own before you size an index.

Index size per million pages

Our arithmetic: vectors × dimensions × bytes, GB = 10^9 bytes; 2.1 chunks per page from the ColPali paper; pool factor 3 reduces vectors by 66.7 per cent in the paper; index structures, metadata and replicas excluded.

The paper reports that token pooling with a factor of 3 "maintains 97.8% of the original performance", with a text-dense dataset degrading more. Storing 1 bit per dimension is how binary quantisation stores a vector, as our vector database comparison describes for Qdrant; check that your database applies it to multi-vector fields before you plan on the 16.5 GB row. The NVIDIA row shows how fast the index grows with dimensions: at 3,072 dimensions per token a full page is larger than Qdrant's limit of "number_of_sub_vectors * vector_size < 1,048,576" per point, which ColPali's 131,840 values stay well inside, by our arithmetic.

An index of a few hundred GB lives on NVMe, with the part the first stage searches in RAM. Our guide to NVMe storage for an LLM server covers the drives. We supply the GPUs and the servers for such an index; send us your page count, growth per year and query load, and our team replies with a sizing of storage, memory and GPUs.

GPU for indexing: one pass per page

Indexing renders each page to an image on the processor and runs the encoder once. There is no text generation, which is what makes OCR with a VLM slow: an OCR model writes hundreds or thousands of tokens per page one after another, as our VLM guide shows. ColQwen2 v1.0 has 4.42 GB of BF16 weights, about 2.2B parameters, so it fits any card we supply with room for large batches.

By the standard count, a forward pass costs at least 2 operations per parameter per token, so a ColQwen2 page of 768 tokens costs at least about 3.4 × 10^12 operations; the vision encoder's work on the image patches adds to this. Dividing StorageReview's measured BF16 matrix rates by that figure gives an upper bound by our estimate.

Our estimate for pages at ColQwen2's maximum of 768 patches: 2 × 2.2B parameters × 768 tokens per page, divided by StorageReview's BF16 MAMF results of 14 October 2026; pages with fewer patches cost less; page rendering, the vision encoder's extra work and database writes are not included, so measured rates are lower.

The ColPali paper measured indexing on an NVIDIA L4, averaged over 100 pages, and shows in its Figure 2 that skipping OCR and layout detection speeds it up considerably; the per-page times appear in the figure, not in the text, so we do not quote them. OpenSearch's own benchmark of 21 April 2026, on OpenSearch 3.5.0 with ColPali on a remote GPU endpoint, reports about 9 seconds per document to ingest, a figure that includes its whole pipeline and document size. Measure on a sample of your own documents: a page of 768 patches and a page of 300 differ by more than half in cost.

ColPali v1.3 with 1,030 tokens and a model of about 3B parameters costs at least about 6 × 10^12 operations per page by the same count, so plan roughly half the ColQwen2 rate. A re-index after a model change repeats the whole run, so size the card for that window as well as for daily growth.

GPU and memory at query time

A query passes through the same model: the paper gives about 30 ms on an L4 to encode a query, against about 22 ms for a text embedding model, at batch size 1. The query then becomes about 25 to 30 vectors, as the model cards' examples show with shapes of (25, 128) and (28, 128). The encoder has to stay loaded for the search service, on a small card of its own or a MIG instance on a larger one.

Scoring is cheap in arithmetic but reads every candidate page in full. The paper gives "≈1 ms per 1000 pages in the corpus" for the scoring. At ColPali's 263.7 KB per page, a full scan of a million pages reads 264 GB, which takes at least 0.9 seconds at the L4's 300 GB/s and 0.3 seconds at the L40S's 864 GB/s, by our arithmetic; that matches the paper's rate. For one query per second over a million pages, a brute-force scan therefore occupies most of an L4 and a third of an L40S.

Two-stage retrieval avoids this. Qdrant's tutorial on PDF retrieval at scale applies mean pooling to the ColPali or ColQwen2 output for a first-stage search, switches off HNSW for the original multi-vectors (m=0) and uses them only "to rerank the results retrieved in the first stage", with indexing "faster by an order of magnitude" and "Retrieval quality comparable to the original model". A text reranker on OCR text of the top pages, as in our reranker guide, can follow as a further stage.

Vector databases with multi-vector support

Of the four databases in our comparison, two document late interaction directly. Qdrant stores "a variable amount of same-shaped dense vectors in a single point" when a dense vector is created with a multivector comparator; its max_sim comparator is "defined as a sum of maximum similarities between each pair of vectors in the matrices", and "MaxSim returns a single combined score per point". Milvus 2.6.4, announced on 5 December 2025, added Array of Structs fields with the MAX_SIM metric type, with MAX_SIM_IP in its ColPali example and an HNSW index on the vector field of the struct.

pgvector's README, version 0.8.7, describes no multi-vector type, late interaction or MaxSim; the page vectors would have to be stored as rows and scored by the application, which loses the database's index for the second stage. OpenSearch published its benchmark of three multimodal approaches on 21 April 2026, with ColPali at about 1,683 ms per query on its test cluster, six to seven times slower than the other two approaches there. Our engineering partner Vixen.UNO builds RAG platforms in our Private AI/ML service; describe your documents and where they are stored, and we propose the retrieval design.

ColPali, OCR or both

ColPali finds the page; it does not turn the page into text. A RAG answer then needs a model that reads the retrieved page images, which means a VLM in the answer step with its image tokens per page, or OCR text of the top pages for a text LLM. The VLM guide's figures apply to that step, but only to the handful of pages per answer, not to the whole corpus.

Our recommendations from the figures above and from our embedding, VLM and vector database guides.

Keep the OCR route where answers must quote exact figures or where users search for part numbers and codes, since keyword matching needs text. ColPali's strength is the page whose meaning sits in its layout: a chart, a filled form, a table without clean text. Our article on fine-tuning embedding models for RAG covers adapting a retriever to your own documents, and RAG on company data covers permissions, which apply to page vectors as to text chunks. Page images of invoices or personnel files contain personal data, so the index needs the same data-protection assessment as the documents; your data-protection officer confirms the details.

Configurations by archive size

Our estimates from the tables above; GPU times are lower bounds at the upper-bound page rates; database servers, replicas and NVMe sized separately.

A retrieval card often sits next to the LLM cards. In a server with RTX PRO 6000 Server Edition cards, one card split into MIG instances of 24 GB holds the retriever, a reranker and the embedding model, as our reranker guide describes; the L4 and L40S have no MIG. We build these servers with L4, L40S or RTX PRO cards for retrieval next to the LLM, sized by page count and query load.

What we supply

We supply the L4, the L40S and the RTX PRO cards for retrieval models, as cards for your servers or in AI servers built to order, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component. The retrieval platform itself, with page-image search, permissions and an answer model that cites its source, is our Private AI/ML service, with engineering by our partner Vixen.UNO.

FAQ