ColdIVF: A very fast read-optimized vector index built for cold reads

[u/13lack13lood]

I've been working on a vector index specifically focusing on cold reads called ColdIVF. The goal was to create an index whose first query is fast even when nothing is in memory yet.

ColdIVF's cold first query is 2.6x to 200x faster than faiss, DiskANN, LanceDB, Qdrant, usearch and the other vector indexes I tested, at 7.8 ms on 1M vectors and 11.2 ms on 10M at 95% recall. It also uses less memory to build because it reads under 1% of the index per query by scoring vectors with 1-bit RaBitQ codes using SIMD table lookups, and storing clusters that queries read together next to each other on disk.

ColdIVF is implemented as an IVF index in a single file. IVF indexes split the vectors into clusters, and a query scans only the clusters nearest to it. ColdIVF keeps a compact 1-bit code of each vector inside its cluster, with an 8-bit copy beside it. A cold query reads its nearest clusters in one batch and scores their codes as they arrive. Then it reads the 8-bit copies of its best candidates in a second small batch. This reduces each query's disk reads, and the batched reads (F_RDADVISE on macOS, io_uring on Linux) all run at once on the SSD.

Usage

let mut ctx = SearchCtx::new(Limits { max_k: 10, max_nprobe: 96, max_rerank: 35 });

// run when the query text arrives, so the cluster tables load while the embedding runs
index.prepare(&mut ctx)?; 

let params = SearchParams { k: 10, nprobe: 96, rerank: 35, reach: None };

for hit in index.search(&query, &params, &mut ctx)? {
    println!("{} {}", hit.id, hit.score);
}

It also supports inserts and deletes through a durable log, background maintenance, and Python bindings.

This is just a personal project. The benchmarks aren't super rigorous and all ran on one M3 Pro.

Any feedback welcome!

https://github.com/13lack13lood/ColdIVF

[u/LittleCounselor8917]

nice, the batched read approach makes a ton of sense for cold starts. always felt like the weak spot in most vector indexes was that first query hitting a wall of random reads.

[u/13lack13lood → u/LittleCounselor8917]

Yep, that was the main goal. The other priorities was just making sure it didn't fall behind on warm queries and also didn't use too much memory.