TurboVec on 100M RAG vectors

I used the MS MARCO Web Search vector dataset to test TurboVec at 1M, 10M, and 100M vectors. I wanted to measure how much storage compression saved and what happened to recall and search time as the index grew.

At full scale, the 310GB float32 payload became a 37GB 4-bit index. That index reached 0.914 recall@10, with a median search time of 5.67 seconds.

Setup

I ran the benchmark on AWS EC2.

Full-scale results

The 2-bit index needed the least storage:

18.9GB for 100M vectors

Its recall@10 was 0.608. I'd start with the 4-bit index for this dataset. It stayed under 40GB and reached 91.4% recall@10. Whether that recall is sufficient depends on the retrieval task.

Scaling

I also measured the same setup at 1M and 10M checkpoints.

For 4-bit TurboVec:

Index size and search time both grew roughly with the corpus. Moving from 10M to 101M vectors increased the 4-bit index from 3.7GB to 37.4GB and median search time from 562ms to 5.67s. The compressed index saved storage, but searches still took seconds at full scale.

Measurement notes

I measured latency with 300 sampled queries, repeated 3 times. Recall@10 used 1000 sampled queries against the provided ground truth.

The full 4-bit index took about 66 minutes to build. After loading and preparing it, the benchmark process used about 73 GiB RSS, compared with 37.4GB for the persisted index file. The file size alone would understate the memory this query process needed.

I saw a similar difference in the large-JSON Python benchmark. Files of similar size used very different amounts of memory depending on whether Python loaded the whole array or streamed records.

Choosing between 2-bit and 4-bit

The 4-bit index used about twice the storage of 2-bit, with recall@10 increasing from 0.608 to 0.914. It also took longer to search, 5.67s at p50 compared with 3.19s for 2-bit. I'd choose which one to test further based on the application's recall and latency requirements.

Code

The benchmark code is available on GitHub ↗.

Limits of this benchmark

I compared 2-bit and 4-bit TurboVec on this dataset. I didn't benchmark FAISS or test a full RAG application with reranking, filtering, batching, caching, or query-specific routing. These p50 values describe the benchmark, not a production serving system, and other workloads may produce different recall.

I'd test the 4-bit index further for its 0.914 recall@10 at 37.4GB. Before using it in an application, I'd need to account for the 73 GiB prepared-process memory use and decide how to handle searches that took seconds in this test.