Vector Databases Explained in 3 Levels of Difficulty MachineLearningMastery

Izzo Your search bar works great for finding exact matches. It breaks completely when you need to find things that are other things.

Izzo You're listening to Exploring Next, I'm Izzo, and this is episode two-fifty-two with Boone. Today we're diving into vector databases — the tech behind every AI search feature you've used in the last year.

Boone And trust me, this is way more interesting than it sounds. We're talking about the difference between asking 'does this exact thing exist?' versus 'what's similar to this thing I'm thinking about?'

Izzo Right, and that shift is huge. Every time you search through your photos for 'cats' without tagging them, or ask ChatGPT to find relevant docs — vector databases are doing the heavy lifting.

Boone The core insight is brilliant actually. You take unstructured data — text, images, whatever — and use embedding models to convert it into vectors. Arrays of floating-point numbers where geometric distance equals semantic similarity.

Izzo Boone, break that down for me. What does 'geometric distance equals semantic similarity' actually mean?

Boone Think of it like this: the words 'dog' and 'puppy' end up close together in this high-dimensional space. A photo of a cat and a drawing of a cat also cluster near each other. The math doesn't care about the format — just the meaning.

Izzo Okay, so the magic is in the embedding models. OpenAI's text-embedding-3-small, vision models for images — they're creating these numerical representations where similar things naturally group together.

Boone Exactly. And we're talking 256 to 4096 dimensions typically. The specific numbers don't mean anything to humans, but the geometry is everything.

Izzo But here's where I start thinking about product reality — how do you actually search through millions of these vectors without melting your servers?

Boone Right. Brute force comparison is linear with dataset size. At 10 million vectors with 1536 dimensions each, you're doing billions of floating-point operations per query. That's way too slow for real-time.

Izzo So approximate nearest neighbor algorithms. You're trading perfect accuracy for massive speed gains.

Boone And the trade-off is surprisingly good. You can get 95% recall — meaning you find 95% of the truly best matches — while being orders of magnitude faster than exhaustive search.

Izzo Now I'm curious about the production angle. Users don't just want semantic similarity — they want 'find similar documents that I own, created this month.' That's filtering plus similarity.

Boone Hybrid retrieval, yeah. You've got pre-filtering where you apply the attribute filter first, then run ANN on the subset. Or post-filtering where you run ANN first, then filter results.

Izzo Which performs better?

Boone Pre-filtering is more accurate but expensive for selective queries. Most production systems use smart indexing to make pre-filtering fast enough to be practical.

Izzo And then there's the keyword precision problem. Pure semantic search for 'GPT-5 release date' might drift toward general AI content instead of finding the exact document with that phrase.

Boone That's where hybrid search shines — dense vector search plus sparse retrieval like BM25. You run both in parallel and merge results using reciprocal rank fusion. Best of both worlds.

Izzo Alright, but let's get into the real engineering. How do these ANN algorithms actually work under the hood?

Boone Three big ones: HNSW, IVF, and Product Quantization. Each hits a different point on the speed-memory-accuracy triangle.

Boone HNSW builds a multi-layer graph. Each vector is a node with edges to similar neighbors. Higher layers are sparse for long-range hops, lower layers are dense for local precision.

Izzo So query time is like graph traversal?

Boone Exactly. You hop through the graph toward your target. HNSW is fast, delivers excellent recall, but it's memory-hungry. It's the default in most modern systems for good reason.

Boone IVF takes a different approach — clusters vectors using k-means, builds an inverted index mapping clusters to members, then only searches the nearest clusters at query time.

Izzo Memory efficient?

Boone Much more than HNSW, but often slower and you need a training step to build those clusters. Trade-offs everywhere.

Boone Then there's Product Quantization — pure compression play. Divides vectors into subvectors and quantizes each to a codebook. Can reduce memory by 4 to 32x. That's how you get to billion-scale datasets? Yep. Often combined with IVF as IVF-PQ. Faiss uses this approach extensively. You're compressing aggressively but can still search effectively. I'm giving these algorithms a solid A-minus for elegance. But configuring them sounds like an art form. Oh it absolutely is. HNSW has