{"id":171486,"date":"2015-07-07T10:11:03","date_gmt":"2015-07-07T10:11:03","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/project\/image-classification\/"},"modified":"2026-04-03T09:42:36","modified_gmt":"2026-04-03T16:42:36","slug":"image-classification","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/image-classification\/","title":{"rendered":"Image Classification"},"content":{"rendered":"

Instructions<\/h1>\n
\n
\n

This web page provides detailed instructions and examples to aid you in successfully completing our Mechanical Turk task for labeling images that have been embedded in public tweets. Note that we have randomly selected these images from public Twitter messages \u2013 they were not pre-screened for content \u2013 it is possible that some content may be offensive or mature in nature.<\/p>\n

Here’s a key in regards to parts of a tweet:<\/p>\n<\/div>\n

\"twitter_websiteexample\"<\/span><\/span><\/div>\n\t
\n\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\tExpand all\t\t\t\t\t<\/button>\n\t\t\t\t\t | <\/span>\n\t\t\t\t\t\n\t\t\t\t\t\tCollapse all\t\t\t\t\t<\/button>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t\t\t\t\t
  • \n\t\t
    \n\t\t\t\n\t\t\t\tQuestion 1\t\t\t<\/button>\n\t\t<\/div>\n\t\t\n\t\t\t
    \n\t\t\t\t
    \n

    Question 1:<\/h1>\n

    Please categorize the picture in one of the following categories:<\/p>\n

      \n
    • Unofficial Retweets<\/li>\n
    • Memes<\/li>\n
    • Inspirational Quotes<\/li>\n
    • Graphs\/Charts<\/li>\n
    • Screenshots\/Screen Captures<\/li>\n
    • Informational Pictures<\/li>\n
    • Pictures of Text\/Mostly Text<\/li>\n
    • Drawings\/Graphics<\/li>\n
    • Photographs of Real-Life Objects<\/li>\n
    • Broken Links<\/li>\n
    • None of the Above<\/li>\n<\/ul>\n<\/div>\n

      <\/p>\n

      \n

      Unofficial Retweets<\/h2>\n

      Images that are screenshots of another tweet that could\u2019ve been retweeted but instead was posted as a picture with the tweet\u2014e.g. Person A tweets a picture; that picture is a picture of a tweet by person B. Person A probably could\u2019ve retweeted the tweet by person B, but instead took a screenshot and shared it that way instead.<\/p>\n<\/div>\n

      \n

      Unofficial Retweet Example 1 (including original tweet):<\/h4>\n

      \"twitter_retweet2\"<\/p>\n

      Unofficial Retweet Example 2 (including original tweet):<\/h4>\n

      \"twitter_retweet1\"<\/span><\/span><\/p>\n

      Unofficial Retweet Example 3:<\/h4>\n

      \"twitter_retweet3\"<\/p>\n

      Unofficial Retweet Example 4:<\/h4>\n

      \"twitter_retweet4\"<\/span><\/span><\/p>\n<\/div>\n

      \n

      Memes<\/h2>\n

      Generally a picture with blocky, white-with-a-thick-black-outline text relevant to the picture or a picture created using stock photos and has relevant text in the font Comic Sans<\/p>\n<\/div>\n

      \n

      Meme Example\u00a01 (including original\u00a0tweet):<\/h4>\n

      \"twitter_meme2\"\u00a0<\/span><\/span><\/p>\n

      Meme Example 2 (including original tweet):<\/h4>\n

      \"twitter_meme1\"<\/span><\/span><\/p>\n

      Meme Example 3:<\/h4>\n

      \"twitter_meme3\"<\/span><\/span><\/p>\n

      Meme Example 4:<\/h4>\n

      \"twitter_meme4\"<\/p>\n<\/div>\n

      <\/div>\n
      \n

      Inspirational Quotes<\/h2>\n

      Pictures with a usually inspirational, motivational, encouraging, or deep quote involved\u2014e.g. a quote about dreaming with a picture of the sky as the background, a quote said by Disney and a picture of Walt Disney next to it<\/p>\n<\/div>\n

      \n

      Inspirational Quote Example 1 (including original tweet):<\/h4>\n

      \"twitter_quotes2\"<\/p>\n

      Inspirational Quote Example 2 (including original tweet):<\/h4>\n

      \"twitter_quotes1\"<\/p>\n

      Inspirational Quotes Example 3:<\/h4>\n

      \"twitter_quotes3\"<\/span><\/span><\/p>\n

      Inspirational Quotes Example 4:<\/h4>\n

      \"twitter_quotes4\"<\/span><\/span><\/p>\n

      Inspirational Quotes Example 5:<\/h4>\n

      \"twitter_quotes5\"<\/span><\/span><\/p>\n

      Inspirational Quotes Example 6:<\/h4>\n

      \"twitter_quotes6.png\"<\/span><\/span><\/p>\n<\/div>\n

      \n

      Graphs\/Charts<\/h2>\n

      Pictures of graphs, charts, infographics; essentially any visual representation of data<\/p>\n<\/div>\n

      \n

      Graph Example 1 (including original tweet):<\/h4>\n

      \"twitter_graph2\"<\/p>\n

      Graph Example 2 (including original tweet):<\/h4>\n

      \"twitter_graph1\"<\/span><\/span><\/p>\n

      Graph Example 3:<\/h3>\n

      \"twitter_graph3\"<\/span><\/span><\/p>\n

      Graph Example 4:<\/h4>\n

      \"twitter_graph4\"<\/span><\/span><\/p>\n

      Graph Example\u00a05:<\/h4>\n

      \"twitter_graph5\"<\/span><\/span><\/p>\n<\/div>\n

      \n

      Screenshots\/Screen Captures<\/h2>\n

      Pictures that have text as part of the picture and are screenshots of a phone\u2019s screen (may include the notification bar at the top of the screen), app, game, or TV screen; screenshot of part of a computer screen\u2014e.g. a screenshot of a phone\u2019s screen with Instagram open, a screenshot of part of a computer window that only shows a Facebook post with some comments<\/p>\n

      Screenshot Example 1 (including original tweet):<\/h4>\n<\/div>\n
      \n

      \"First<\/span><\/span><\/p>\n

      Screenshot Example 2 (including original tweet):<\/h4>\n

      \"Second<\/span><\/span><\/p>\n

      Screenshot Example 3:<\/h4>\n

      \"Third<\/span><\/span><\/p>\n

      Screenshot Example 4:<\/h4>\n

      \"Fourth<\/span><\/span><\/p>\n

      Screenshot Example 5:<\/h4>\n

      \"Fifth<\/span><\/span><\/p>\n

      Screenshot Example 6:<\/h4>\n

      \"Sixth<\/span><\/span><\/p>\n

      Screenshot Example 7:<\/h4>\n

      \"Seventh<\/span><\/span><\/p>\n

      Screenshot Example 8:<\/h4>\n

      \"Eighth<\/span><\/span><\/p>\n<\/div>\n

      \n

      Informational Pictures<\/h2>\n

      Some form of graphics that go along with and is important to the text in the photo; text is usually overlaid on the picture; picture with text explaining the photo\u2013e.g. movie posters, screenshot\/still frame with subtitles, advertisement with text.<\/p>\n<\/div>\n

      \n

      Informational Picture Example 1 (including original tweet):<\/h4>\n

      \"First<\/span><\/span><\/p>\n

      Informational Picture Example 2:<\/h4>\n

      \"Second<\/span><\/span><\/p>\n

      Informational Picture Example 3:<\/h4>\n

      \"Third<\/span><\/span><\/p>\n

      Informational Picture Example 4:<\/h4>\n

      \"Fourth<\/span><\/span><\/p>\n

      Informational Picture Example 5:<\/h4>\n

      \"\"<\/span><\/span><\/p>\n<\/div>\n

      \n

      Pictures of Text\/Mostly Text<\/h2>\n

      Any image that mostly contains text\u2014e.g. pictures of pure text, screenshots of conversations\/SMS\/IM\u2019s, pictures of book pages, picture with mostly text and an accompanying photo (e.g. Tumblr photo + commentary)<\/p>\n<\/div>\n

      \n

      Picture of Text\u00a0Example 1 (including original tweet):<\/h4>\n

      \"First<\/span><\/span><\/span><\/p>\n

      Picture of Text\u00a0Example 2 (including original tweet):<\/h4>\n

      \"Second<\/span><\/span><\/p>\n

      Picture of Text\u00a0Example 3:<\/h4>\n

      \"Third<\/span><\/span><\/p>\n

      Picture of Text\u00a0Example 4:<\/h4>\n

      \"Fourth<\/span><\/span><\/p>\n

      Picture of Text\u00a0Example 5:<\/h4>\n

      \"Fifth<\/span><\/span><\/p>\n

      Picture of Text\u00a0Example 6:<\/h4>\n

      \"Sixth<\/span><\/span><\/p>\n<\/div>\n

      \n

      Drawings\/Graphics<\/h2>\n

      Picture that was created through digital or physical means (Photoshop or hand drawn)\u2014e.g. cartoon, hand-drawn art, digital artwork, etc.<\/p>\n

      Drawing Example 1 (including original tweet):<\/h4>\n<\/div>\n
      \n

      \"First<\/span><\/span><\/p>\n

      Drawing Example 2:<\/h4>\n

      \"Second<\/span><\/span><\/p>\n

      Drawing Example 3:<\/h4>\n

      \"Third<\/span><\/span><\/p>\n

      Drawing Example 4:<\/h4>\n

      \"Fourth<\/span><\/span><\/p>\n

      Drawing Example 5:<\/h4>\n

      \"Fifth<\/span><\/span><\/p>\n<\/div>\n

      \n

      Photographs of Real-Life Objects<\/h2>\n

      Picture of something in real life\u2014e.g. selfies, a person\u2019s facial reaction, animals, nature, real locations, etc.; Snapchat screenshots included\u2014the text in a snapchat is negligible (characteristics of Snapchat: translucent grey text bar across the picture, small circular or square timer\/countdown at the top right corner)<\/p>\n

      Photograph Example 1 (including original tweet):<\/h4>\n<\/div>\n
      \n

      \"First<\/span><\/span><\/p>\n

      Photograph Example 2 (including original tweet):<\/h4>\n

      \"Second<\/span><\/span><\/p>\n

      Photograph Example 3:<\/h4>\n

      \"Third<\/span><\/span><\/p>\n

      Photograph Example 4:<\/h4>\n

      \"Fourth<\/span><\/span><\/p>\n

      Photograph Example 5:<\/h4>\n

      \"Fifth<\/span><\/span><\/p>\n<\/div>\n

      \n

      Broken Links<\/h2>\n

      The link to the image is broken; there is no picture that shows up.<\/p>\n<\/div>\n

      \n

      None of the Above<\/h2>\n

      The image is either a combination of the categories above or doesn’t fall under any of the categories above.<\/p>\n<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t<\/li>\n\t\t

    • \n\t\t
      \n\t\t\t\n\t\t\t\tQuestion 2\t\t\t<\/button>\n\t\t<\/div>\n\t\t\n\t\t\t
      \n\t\t\t\t
      \n

      Question 2:<\/h1>\n

      Is there any text in the picture? If yes, would the text serve as a good caption for the picture?<\/p>\n

        \n
      • Yes, there is text in the picture; yes, the text would be a good caption<\/li>\n
      • Yes, there is text in the picture; no, the text would not be a good caption<\/li>\n
      • No, there is no text in the picture<\/li>\n<\/ul>\n<\/div>\n

        <\/p>\n

        \n

        Yes, there is text in the picture; yes, the text would be a good caption<\/h2>\n<\/div>\n
        \n

        Example: The text in the picture “Raspberry vanilla \u00e9clair” would be a good caption for the picture.<\/p>\n

        \"Picture<\/span><\/span><\/p>\n<\/div>\n

        \n

        Yes, there is text in the picture; no, the text would not be a good caption<\/h2>\n<\/div>\n
        \n

        Example: The text in picture “Guess Who?” would not be a good caption for the picture.<\/p>\n

        \"Man<\/span><\/span><\/p>\n<\/div>\n

        \n

        No, there is no text in the picture<\/h2>\n<\/div>\n
        \"Satellite<\/span><\/span><\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t<\/li>\n\t\t
      • \n\t\t
        \n\t\t\t\n\t\t\t\tQuestion 3\t\t\t<\/button>\n\t\t<\/div>\n\t\t\n\t\t\t
        \n\t\t\t\t

        <\/p>\n

        \n

        Question 3:<\/h1>\n

        How well does the content\/text of the tweet serve as a caption for the picture?<\/p>\n

          \n
        • Very Well<\/li>\n
        • Well<\/li>\n
        • Average<\/li>\n
        • Poor<\/li>\n
        • Very Poor<\/li>\n
        • There is no text in the tweet<\/li>\n<\/ul>\n<\/div>\n

          <\/p>\n

          \n

          Very Well<\/h2>\n<\/div>\n
          \n

          Example 1 of “very well”:<\/h4>\n

          \"Tweet<\/span><\/span><\/p>\n

          Example 2 of “very well”:<\/h4>\n

          \"Tweet<\/span><\/span><\/p>\n<\/div>\n

          \n

          Well<\/h2>\n<\/div>\n
          \n

          Example of “well”:<\/h4>\n

          \"Tweet<\/span><\/span><\/p>\n<\/div>\n

          \n

          Average<\/h2>\n<\/div>\n
          \n

          Example of “average”:<\/h4>\n

          \"Tweet<\/span><\/span><\/p>\n<\/div>\n

          \n

          Poor<\/h2>\n<\/div>\n
          \n

          Example of “poor”:<\/h4>\n

          \"Tweet<\/span><\/span><\/p>\n<\/div>\n

          \n

          Very Poor<\/h2>\n<\/div>\n
          \n

          Example of “very poor”:<\/h4>\n

          \"Tweet<\/span><\/span><\/p>\n<\/div>\n

          \n

          There is no text in the tweet<\/h2>\n<\/div>\n
          \n

          Example of “there is no text in the tweet”:<\/h4>\n

          \"Selfie<\/span><\/span><\/p>\n<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t<\/li>\n\t\t

        • \n\t\t
          \n\t\t\t\n\t\t\t\tQuestion 4\t\t\t<\/button>\n\t\t<\/div>\n\t\t\n\t\t\t
          \n\t\t\t\t
          \n

          Question 4:<\/h1>\n

          How important is the photo to the intent of the tweet? In other words, can the tweet stand on its own and not lose any meaning without the photo?<\/p>\n

            \n
          • The photo is very important<\/li>\n
          • The photo is somewhat important<\/li>\n
          • The photo is nice to have but not necessary to fully appreciate the tweet<\/li>\n
          • The photo is nonessential to the tweet and the tweet will have lost nothing without the picture<\/li>\n<\/ul>\n<\/div>\n

            <\/p>\n

            \n

            The photo is very important<\/h2>\n<\/div>\n
            \n

            Example of “very important”:<\/h4>\n

            \"Tweet<\/span><\/span><\/p>\n<\/div>\n

            \n

            The photo is somewhat important<\/h2>\n<\/div>\n
            \n

            Example of “important”:<\/h4>\n

            \"Tweet<\/span><\/span><\/p>\n<\/div>\n

            \n

            The photo is nice to have but not necessary to fully appreciate the tweet<\/h2>\n<\/div>\n
            \n

            Example of “not necessary”:<\/h4>\n

            \"Tweet<\/span><\/span><\/p>\n<\/div>\n

            \n

            The photo is nonessential to the tweet and the tweet will have lost nothing without the picture<\/h2>\n<\/div>\n
            \n

            Example of “non-essential”:<\/h4>\n

            \"Tweet<\/span><\/span><\/p>\n<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t<\/li>\n\t\t\t\t\t\t<\/ul>\n\t<\/div>\n\t\n<\/div>\n","protected":false},"excerpt":{"rendered":"

            Instructions This web page provides detailed instructions and examples to aid you in successfully completing our Mechanical Turk task for labeling images that have been embedded in public tweets. Note that we have randomly selected these images from public Twitter messages \u2013 they were not pre-screened for content \u2013 it is possible that some content […]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"research-area":[13554],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-171486","msr-project","type-msr-project","status-publish","hentry","msr-research-area-human-computer-interaction","msr-locale-en_us","msr-archive-status-complete"],"msr_project_start":"7\/7\/2015","related-publications":[],"related-downloads":[],"related-videos":[],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[],"related-articles":[],"tab-content":[],"slides":[],"related-researchers":[],"msr_research_lab":[],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/171486","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":3,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/171486\/revisions"}],"predecessor-version":[{"id":1167653,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/171486\/revisions\/1167653"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=171486"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=171486"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=171486"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=171486"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=171486"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}