These are the prompts and qrels used for the experiments in Thomas et al., "System Comparison using Automated Generation of Relevance Judgements in Multiple Languages", SIGIR 2025.
ORCAS is a click-based dataset associated with the TREC Deep Learning Track. It covers 1.4 million of the TREC DL documents, providing 18 million connections to 10 million distinct queries.
The TREC Deep Learning Track studies information retrieval in a large training data regime. This is the case where the number of training queries with at least one positive label is at least in the tens of thousands, if not…
The Tip of the Tongue (ToT) dataset is from the paper Tip of the Tongue Known-Item Retrieval: A Case Study in Movie Identification (opens in new tab). It is comprised of 758 question/answer pairs scraped from the website iRememberThisMovie.com between…
This is a quick start guide for the document ranking task in the TREC Deep Learning (TREC-DL) (opens in new tab) benchmark. If you are new to TREC-DL, then this repository may make it more convenient for you to download…
MS MARCO is a collection of datasets focused on deep learning in search. The first dataset was a question answering dataset featuring 100,000 real Bing questions and a human generated answer. Since then, we released a 1,000,000-question dataset, a natural…
This is a small library for implementing several standard "test collection" or "offline" evaluation measures for search systems. See: https://github.com/Microsoft/irmetrics-r (opens in new tab)
This data is being released for research purposes only. The DESM Word Embeddings dataset may include terms that some may consider offensive, indecent or otherwise objectionable. Microsoft has not reviewed or modified the content of the dataset. Microsoft is providing…
Principal Architect
Senior applied scientist
