Back in May, I submitted a paper on arXiv titled "OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries". In this post, I'll give a rundown of how it came to be, what it says, and why I think it's hinting at something much bigger.
It all started in October of last year. I had just finished my Reasoning-Intensive Regression (RiR) paper and was looking for my next project. Information retrieval is a space I've always been interested in. At its core, retrieval is simple: given a corpus C = {d1, ..., dn} and a query q, a retrieval system returns a candidate D(q) ⊂ C with |D(q)| ≪ |C|, so that a human or downstream algorithm can reason over the right evidence. But that simple formulation hides a much harder question: what makes some retrieval problems harder than others? I started by pulling on that thread, asking small, concrete questions about what makes retrieval difficult and what it really means for retrieval to be reasoning intensive.
With the help of my advisors, we developed some intuition around the (spectral) geometry of information retrieval. Unfortunately, when it came to evaluating our ideas, we hit a few snags and decided to postpone that project. While that was disappointing, during this foray into more theoretical questions, one consistent footgun was the lack of public benchmarks to evaluate the types of queries we considered difficult.
After many research conversations (and some lamentations), we started formulating ideas about the kind of retrieval benchmark we wanted. But none of them quite went anywhere, and I began (heavily) procrastinating by scrolling through Twitter. Then, one morning, somewhere in the middle of that procrastination, it clicked. Tweets, and search over those tweets, fitted neatly into all of our previous research conversations. With the API recently out, this seemed like the perfect time to scrape some data and start toying around with it. So, that's what I did. Within a few days, I had v0 of the Twitter task. And from there, the race was on. Twitter gave me a starting point, and suddenly it became much easier to see what the next task could be, and then the one after that. Furthermore, it became clear that these tasks were all instances of the same underlying problem.
For retrieval where the query shares words or concepts with the documents it needs, there is a growing sense that the problem is largely solved: modern systems score well on the benchmarks the field has long optimized for (see BRIGHT). In our paper, we argue that there is an important class of queries that fall outside of standard regimes. Here, the user knows the pattern they want but not the surface form it will take. Take, for example, a researcher attempting to recover a fuzzy recollection of a research conversation, an analyst scanning logs for the early signs of a cyber-attack that is never named as such, or an engineer searching model traces for a failure mode demonstrated implicitly by a sequence of model actions. In these cases, the target documents are not semantically aligned with the words of the query. I call such a query oblique: relevance is determined by a latent relation with each document, with little direct surface expression in the document's contents.
OBLIQ-Bench contains five tasks, split up into three non-exhaustive categories where this obliqueness property manifests. Descriptive queries are those that seek a latent property that must be inferred from document content such as tweets that subtly imply a stance or an implicit failure mode during a human-AI multi-turn conversation. Analogue queries are those that attempt to instantiate a shared archetype across multiple documents. The analogue tasks we created are about retrieving math problems that share the same underlying reasoning pattern despite being from different fields and retrieving writings from the same author spanning multiple domains. Lastly, Tip-of-Tongue (ToT) queries are those which seek to match fuzzy recollections to obscure passages such as transcripts of a congress hearing.
One cool diagnostic we introduced when evaluating the benchmark is the retrieval-verification gap. This gap measures the difference between a state-of-the-art (SOTA) retrieval system and SOTA reasoning model acting as a verifier. In most settings, given a reasoning model and the right candidate set, it should be able to identify the gold documents. The trouble is getting those documents into the candidate set in the first place. In essence, the gap points to a clear asymmetry between "generation" (top K retrieval) and "verification" (reranking gold infused documents). Figure 1 below encapsulates this idea by showing how this gap manifests in the various OBLIQ-Bench tasks, as well as tasks like BRIGHT's Art of Problem Solving (AoPS) split.
I do not think it necessary to expound on details such as how we generated the benchmark data, what models we evaluated on, metrics we used, in this post. If you want to know more, please check out the paper, play around with the dataset on HuggingFace and stay tuned for a new and improved v2!
So, we've talked a bit about how OBLIQ came to be and what it says, but I promised a third and final thing: why I believe it points to something bigger. Oblique queries are pervasive. Every time someone begrudgingly writes a report, sends an email, or conducts deep research, they crop up. While we can readily agree that current retrieval methods fall short of capturing the pertinent documents for these types of queries, a natural question is, why not just wait for reasoning models to become cheap (in cost and latency) enough to use as retrievers? Or wait for them to have infinite context whereby we can then fit our entire corpus into it? Is retrieval solely a question of token economics? I believe that developing methodologies for oblique search is useful today and for the future.
Retrieval is not solely token economics and infinite context does not "solve" it. Even in the regime where throughput is most optimized for, some (oblique) queries are not answerable by scanning, within a reasonable time frame. Cheap tokens can shrink the constant, but they don't change how the problem scales. To do that, you'd need some ingenuity in how the data is processed and structured.
If we can agree that exposing structure matters, then whether a query is actually difficult to retrieve depends enormously on what the system knows about the search space. If I tell you every request in advance, retrieval is trivial since you can simply precompute the answers. Even if I only tell you the precise shape of future queries, many seemingly difficult retrieval problems become indexing problems. If I know that users will repeatedly ask about political stance, for example, I can spend arbitrary effort up front inferring stance for every document and construct an index around that property (indeed this was the idea many people converged to for the Twitter task).
The more interesting question for oblique search is which latent relevance relations can be surfaced from a massive corpus at sublinear cost when we do not know in advance which relations people will care about. This makes obliqueness somewhat relative, meaning a query can be oblique to a system today and ordinary to another system that happened to anticipate the right latent representation. The difficulty lies in the interactions between the corpus, the query distribution, and what computation we are willing to perform before and after observing that distribution. One of the most pertinent questions is which properties of a corpus can ever be made efficiently retrievable without knowing in advance what someone will ask. OBLIQ-Bench helps expose this idea, but now is the time to flesh it out, understand its boundaries, and figure out how far we can push it!