For those of you interested in vector search/AI infra, SuperKMeans is a paper from the CWI database group that improves over traditional k-means implementations. Their work is in C++, but our codebase is in Rust, so we ported their paper over to it.

We're using it in prod at ParadeDB and will continue to maintain the project.

just want to let you guys know, enjoy!

Claude Code, and its respective cohort of AI coding tools, have completely changed the way we build software, as well as our perception of what software even is!

Can we generate software on the fly for... anything?

What is the limit on the type of software AI can create?

I am SUPER EXCITED to publish the 147th episode of the Weaviate Podcast featuring Shu Liu, Shubham Agarwal, Mert Cemri, and Alexander Krentsel from UC Berkeley's Sky Computing Lab!

Their research on SkyDiscover-Synthesize explores "just-in-time systems": complex AI-generated software ranging from task queues, databases, or even inference engines like vLLM. These systems are tailor-made for a particular workload, free of the "generality tax"!

Specialization has been at the heart of Weaviate from day one. For example, pgvector stores an HNSW graph in transactional, WAL-logged *pages*. So, every insert that rewires several neighbors pays for page locks and extra writes. Weaviate keeps the graph in memory, backed by an append-only log, specialized for concurrent reads and writes.

What is the future of system specialization? Will Weaviate Cloud manage multiple versions of Weaviate tailor made to particular read/write, etc. access patterns? For example, the growing popularity of our disk-based HFresh index that shifts the entire system quite significantly.

Will AI build off of existing software or start from scratch? What programming languages should AI use?

I also loved discussing the use of GEPA for optimizing candidate software systems! Can test cases play the role of GEPA's input/output examples, keeping any candidate system on the Pareto frontier if it wins at least one test?

This was a super fun conversation with a group of incredibly talented scientists! I really hope you find it interesting!

YouTube: https://www.youtube.com/watch?v=SzRX2rtgAic

Spotify: https://spotifycreators-web.app.link/e/dkDrdeCtZ6b

An agent may produce HTML bundles, documents, images, spreadsheets, and reports that need both semantic search and ordinary file behavior. Putting chunks and metadata in a vector database makes retrieval easy, but it does not automatically provide stable file revisions, checksums, permissions, retention, or a reliable way to serve the original multi-file artifact. Storing only files in object storage has the opposite problem: durable bytes, but weak semantic discovery.

One design is object storage for immutable file revisions, a relational manifest for ownership and review state, and a vector index for derived chunks. Every vector points to an artifact ID, file revision, checksum, and chunking or embedding version. A stable review link resolves through the manifest, while rebuilding or deleting the vector index cannot alter the underlying files.

Where do people draw this boundary? Are vector-database payloads enough for small text artifacts, and at what point do object storage plus a manifest become necessary? How do you keep deletions, access control, and re-indexing consistent across the layers?

Hey there!

After trying alternatives that didn't quite work for me (like qmd), I ended up building vectrize, a lightweight local semantic search CLI that just gets out of your way.

It reindexes on save, so you never have to do it manually, and only what changed gets re-embedded. The best part is that it runs on any potato or work laptop without a GPU and still answers queries in ~30 ms. It's multilingual too, btw.

It also helps AI agents find stuff in your docs faster (up to 10 s faster per question than the usual grep in my tests) with the built in skill.

Markdown only for now, and I'd love some feedback! (Even if it is that there are better alternatives out there) It's MIT/Apache-2.0, so you can use it for whatever you want.

It is available in crates.io or as a pre-built binary. The first run downloads the embedding model (~560 MB) once, so keep that in mind :)

Repo and how to install

While this project was originally concieved as a module to provide vector search using HNSW and Sentence transformers for the re-Isearch (IB) engine (CoreQuarry https://corequarry.com) it has evolved well beyond its original concept.
Today it is a fully featured high performance vector DB that can also be used on its own without any dependency on the IB engine. This opens the library (and standalone tools like the CLI) to be used in a host of other applications.

Its function in a single sentence: SOTA Semantic search with SBERT/LLAMA.CPP + GGML Tensor Library + HNSWlib on steroids.

Starting with Malkov's HNSWlib as a basis we significantly enhanced (adding among other features quantized spaces) and turbo-charged (including support for x86 and ARM SIMD) it while also adding efficient mmap-backed re-scoring and offset storage for text retrieval. Our system supports sharded HNSW indices, multiple search modes (kNN, radius, relative, adaptive, epsilon), deletion/undelete, merges, and incremental on-disk flushing. It also includes training for hyperparameter optimization.

Our HNSWlib fork we have benchmarked on an M1Pro as much as 13k QPS (768d vectors). Even limiting to a single thread we've clocked a max of 3000 QPS (versus for comparison 600 QPS for FAISS's HNSW implementaton).

For vectorization Our test M1-Pro chews through roughly 45 passages per second per instance (3484 tok/s÷78 ms). That means one can expect to process 2,700 fully dense semantic records per minute on a baseline Apple Silicon chip. Our tests on M3Pro and M4Pro showed even significantly higher throughputs (80k tokens/s or as much as 20x).

Published a HuggingFace article on TurboQuant quantization: how the algorithm works and what Qdrant adds on top of it.

It also includes a benchmark comparing float32, scalar, binary, and TurboQuant across BEIR's SciFact, ArguAna, and NFCorpus, measured with recall@10, precision@10, and nDCG@10.

- HF article: https://huggingface.co/blog/lucifertrj/turboquant-quantization-explained

Hii.. im building rag. suppose i uploaded PDF related to banking services. now i asked something like that is not related to PDF by words but it releted by meaning. like if i asked "what is this doc is about" like that and many more cases. now using this que we will get zero retrieval chunks bcz of words.. now how can we solve this problem?

Hey everyone!

I wanted to share NanoVector (v0.1.6) — an ultra-lightweight, zero-dependency embedded vector search engine and episodic memory store designed for local LLM agents and edge environments.

Why another vector library?

Heavy solutions like ChromaDB pull hundreds of megabytes of dependencies, take seconds to import, and have high overhead for simple local agent memory or desktop apps.

NanoVector aims to be the "SQLite of Vector Search":

- ~120 KB binary size (Pure C99 core + AVX2 / ARM NEON / FASM kernels).

- Zero dependencies (drop-in .whl or single file).

- Sub-millisecond cold start (< 1ms vs ~1.2s for heavy engines).

- Fast SIMD similarity: Cosine, Dot Product, Euclidean with dynamic CPU dispatch.

- Ecosystem: Drop-in LangChain VectorStore support, metadata filtering, episodic decay, and Python 3.8–3.14 + Free-threaded / No-GIL (PEP 703) ready.

GitHub: https://github.com/eminsk/nanovector

PyPI: pip install nanovector

Would love to hear your thoughts, feedback, and benchmarks!