talk-data.com
Beyond hybrid search: Creating wormhole vectors to bridge disjoint vector spaces
Description
Modern search systems increasingly rely on various vector representations—dense embeddings for semantic similarity, sparse vectors for lexical matching, and behavioral vectors from collaborative filtering. But these vector spaces typically operate in isolation, forcing us to choose between different notions of relevance. This talk introduces the concept of "wormhole vectors"—a technique for dynamically connecting and traversing between heterogeneous vector spaces using retrieved documents as a bridge. Rather than running parallel searches and merging results (the most common hybrid search approach), we can start in one vector space (e.g., using sparse embeddings for keyword search), retrieve documents, and then derive wormhole vectors to instantly transport us to another space (e.g., using dense embeddings for semantic search) to continue searching with a fundamentally different notion of similarity. We'll explore practical uses of this approach across three vector spaces: dense semantic embeddings, sparse lexical/keyword vectors, and behavioral vectors from matrix factorization. We’ll demonstrate how document sets can dynamically materialize bridges between these vector spaces, enabling novel retrieval patterns like semantic → behavioral → lexical traversal that surface user intent patterns invisible to single-space approaches.