{"id":265,"date":"2013-04-02T09:00:00","date_gmt":"2013-04-02T09:00:00","guid":{"rendered":"https:\/\/blogs.technet.microsoft.com\/inside_microsoft_research\/2013\/04\/02\/exploring-the-biases-of-big-data\/"},"modified":"2016-07-20T07:31:52","modified_gmt":"2016-07-20T14:31:52","slug":"exploring-the-biases-of-big-data","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/exploring-the-biases-of-big-data\/","title":{"rendered":"Exploring the Biases of Big Data"},"content":{"rendered":"

Posted by Rob Knies<\/span><\/p>\n

\"Kate (opens in new tab)<\/span><\/a><\/p>\n

On Feb. 28, at the Santa Clara (Calif.) Convention Center, Kate Crawford (opens in new tab)<\/span><\/a>, principal researcher at Microsoft Research New England (opens in new tab)<\/span><\/a>, took the stage during the Strata Conference (opens in new tab)<\/span><\/a> to deliver an illuminating, 17-minute talk entitled Algorithmic Illusions: Hidden Biases of Big Data.<\/p>\n

During that presentation, she cautioned that data and collections of data are not objective. They are created and shaped by human beings, and understanding the unavoidable hidden biases people bring to data collection and analysis can be as significant as the data themselves.<\/p>\n

Now, on the heels of that appearance, Crawford is bringing a similar message to a different audience, that of the Harvard Business Review<\/em>, which has just published her contributed article, The Hidden Biases in Big Data<\/em>” href=”http:\/\/blogs.hbr.org\/cs\/2013\/04\/the_hidden_biases_in_big_data.html” target=”_blank”>The Hidden Biases in Big Data (opens in new tab)<\/span><\/a><\/em>, that underscores the concepts she discussed during Strata 2013.<\/p>\n

In this brief but compelling article, Crawford raises questions that need to be raised when examining a big data set: “… which people are excluded? Which places are less visible? What happens if you live in the shadow of big data sets?”<\/p>\n

I had to know more, so I contacted her. The first thing I wondered about was how she began investigating the biases inherent in what increasingly is being invoked by the term “big data.” As it turns out, her investigation had been prompted by one of the world’s biggest natural disasters in recent memory.<\/p>\n

“I became fascinated with the insights and limits of big data when I started working with large sets of social-media data,” Crawford says, “most particularly while working on crisis communications projects back in 2010 and 2011, when Australia was experiencing the worst flooding on record. Collaborating with a team of social scientists, we were tracking tweets about the floods, seeking to understand communications patterns.<\/p>\n

“People were using Twitter to share information and to squash rumors and to thank emergency services. The majority of tweets came from Queensland’s capital city of Brisbane, but the most substantial damage and loss of life was in smaller towns and rural areas.”<\/p>\n

The preponderance of city dwellers’ tweets meant that their observations were overshadowing the experiences of those most directly affected. The data set reflected a sort of bias.<\/p>\n