FRank: A Ranking Method with Fidelity Loss

MSR-TR-2006-155 |

Ranking problem is becoming increasingly important in many applications, especially in information retrieval. Many machine learning technologies have been proposed to solve this problem, such as RankSVM, RankBoost and RankNet. Among them, RankNet, which is based on the probabilistic ranking framework, is leading to promising results and has been applied to commercial Web search engine. Inspired by the success of RankNet, we conduct further study on the probabilistic ranking framework in this paper, and propose a novel loss function named Fidelity to measure loss of ranking. This new loss function not only inherits effective properties of the loss function used in RankNet, it also possesses several new properties that are helpful for ranking. This includes Fidelity loss obtaining zero for each desired document pair, and having a finite upper bound that is necessary for conducting query-level normalization. In these aspects, the Fidelity loss is more consistent with widely-used query-level evaluation criteria for information retrieval. To efficiently minimize Fidelity loss and learn an effective ranking function, we propose an algorithm named FRank based on a generalized additive model. We provide experiments and significance test on a large-scale dataset collected from a commercial Web search engine showing that the proposed FRank algorithm provides significantly better results than RankSVM, RankBoost, and RankNet.