LinkSO: A Benchmark for Learning to Retrieve Similar Question-Answer Pairs on Software Development Forums

We present LinkSO, a dataset for learning to rank similar questions
on Stack Overflow. Stack Overflow contains a massive amount of
crowd-sourced question links of high quality, which provides a great
opportunity for evaluating retrieval algorithms for community-based
question answer (cQA) archives and for learning to retrieve
similar questions. However, due to the existence of missing links,
one question is whether question links can be readily used as the
relevance judgment for evaluation. We study this question by measuring
the closeness between question links and the relevance judgment,
and we find their agreement rates range from 80% to 88%. We
conduct an empirical study that evaluates existing retrieval models’
performance on LinkSO. While existing work focuses on non-learning
approaches, our preliminary exploration that assembles
simple learning models shows great potential for further improving
the retrieval performance with machine learning.