Social features are increasingly integrated within the search results page of the main commercial search engines. There is, however, little understanding of the utility of social features in traditional search. In this paper, we study utility in the context of social annotations, which are markings indicating that a person in the social network of the user has liked or shared a result document. We introduce a taxonomy of social relevance aspects that influence the utility of social annotations in search, spanning query classes, the social network, and content relevance. We present the results of a user study quantifying the utility of social annotations and the interplay between social relevance aspects. Through the user study we gain insights on conditions under which social annotations are most useful to a user. Finally, we present machine learned models for predicting the utility of a social annotation using the user study judgments as an optimization criterion. We model the learning task with features drawn from web usage logs, and show empirical evidence over real-world head and tail queries that the problem
is learnable and that in many cases we can predict the utility of a social annotation.