{"id":4131,"date":"2015-09-11T09:00:00","date_gmt":"2015-09-11T16:00:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/msr_er\/2015\/09\/11\/meeting-the-data-challenges-of-urban-computing\/"},"modified":"2016-07-20T07:28:59","modified_gmt":"2016-07-20T14:28:59","slug":"meeting-the-data-challenges-of-urban-computing","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/meeting-the-data-challenges-of-urban-computing\/","title":{"rendered":"Meeting the data challenges of urban computing"},"content":{"rendered":"<p><span style=\"font-family: verdana,geneva;font-size: medium\">Many people living in the People\u2019s Republic of China and Hong Kong have a new habit: they check the air pollution index before venturing outside. Air quality has deteriorated rapidly in China, with nitrogen dioxide and particulate matter levels frequently<sub>\u00a0<\/sub>exceeding safety guidelines set by the World Health Organization.<\/span><\/p>\n<p><span style=\"font-family: verdana,geneva;font-size: medium\">While poor air quality clearly impacts public health, many cities have a dearth of air-quality monitoring stations: there are just 35 in Beijing and 15 in Hong Kong, for example. This lack of monitoring stations hinders evidence-based decision-making and leads to harsh criticisms of the transparency and public relevance of China\u2019s official air pollution index.<\/span><\/p>\n<p style=\"text-align: center\"><img decoding=\"async\" title=\"Researchers proposed that they could estimate air quality at locations not covered by monitoring stations by analyzing the relationship between urban dynamics data.\" src=\"https:\/\/msdnshared.blob.core.windows.net\/media\/MSDNBlogsFS\/prod.evol.blogs.msdn.com\/CommunityServer.Blogs.Components.WeblogFiles\/00\/00\/01\/32\/81\/6765.urban-computing-1-496px.png\" alt=\"Researchers proposed that they could estimate air quality at locations not covered by monitoring stations by analyzing the relationship between urban dynamics data.\" \/><\/p>\n<p style=\"text-align: center\"><span style=\"font-family: verdana,geneva;font-size: small;color: #808080\">Researchers proposed that they could estimate air quality at locations not covered by monitoring stations b<span style=\"font-family: verdana,geneva;font-size: small;color: #808080\">y analyzing the relationship between urban dynamics data<\/span>.<\/span><\/p>\n<p><span style=\"font-family: verdana,geneva;font-size: medium\">Since air pollution is highly location-dependent, a citywide air-quality monitoring system would require building many additional monitoring stations, a solution that is prohibitively expensive. And so there is a grim reality: people may diligently check the air pollution index, but they cannot really know the air quality in their specific locale.<\/span><\/p>\n<p><span style=\"font-family: verdana,geneva;font-size: medium\">\u201cDo we have sufficient data to produce reliable air-quality metrics by using urban computing?\u201d <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"http:\/\/www.eee.hku.hk\/people\/vli.html\" target=\"_blank\">Professor Victor Li<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, chairman of information engineering and head of the department of Electrical and Electronic Engineering at the University of Hong Kong, posed this question to his research team. As luck would have it, while Li was challenging his researchers in Hong Kong, Microsoft Research Asia was launching a Call for Research Proposals dealing with urban computing. (Urban computing is a process of acquisition, integration, and analysis of big and heterogeneous data generated by a diversity of sources in urban spaces to address issues that major cities face, such as air pollution, excessive energy consumption, and traffic congestion.) Li promptly submitted a proposal, which came to the attention of Microsoft researcher <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"http:\/\/research.microsoft.com\/en-us\/people\/yuzheng\/\" target=\"_blank\">Yu Zheng<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, who has conducted extensive research on urban computing and was also looking into the issues surrounding urban air quality.<\/span><\/p>\n<p><span style=\"font-family: verdana,geneva;font-size: medium\">As Li and his researchers were detailing the correlations between air pollution and various urban dynamics\u2014noting, for instance, that air quality and temperature are spatially correlated\u2014Yu and his team had concluded that a variety of urban big data could be used compensate for the lack of monitoring stations. Li and Yu decided to collaborate and drive more in-depth research.<\/span><\/p>\n<div class=\"yt-consent-placeholder\" role=\"region\" aria-label=\"Video playback requires cookie consent\" data-video-id=\"CwfQvTeftbM\" data-poster=\"https:\/\/img.youtube.com\/vi\/CwfQvTeftbM\/maxresdefault.jpg\"><iframe aria-hidden=\"true\" tabindex=\"-1\" title=\"map demo with audio\" width=\"500\" height=\"281\" data-src=\"https:\/\/www.youtube-nocookie.com\/embed\/CwfQvTeftbM?feature=oembed&rel=0&enablejsapi=1\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p>\n<div class=\"yt-consent-placeholder__overlay\"><button class=\"yt-consent-placeholder__play\"><svg width=\"42\" height=\"42\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" aria-hidden=\"true\" focusable=\"false\"><g fill=\"none\" fill-rule=\"evenodd\"><circle fill=\"#000\" opacity=\".556\" cx=\"21\" cy=\"21\" r=\"21\"\/><path stroke=\"#FFF\" d=\"M27.5 22l-12 8.5v-17z\"\/><\/g><\/svg><span class=\"yt-consent-placeholder__label\">Video playback requires cookie consent<\/span><\/button><\/div>\n<\/div>\n<p><span style=\"font-family: verdana,geneva;font-size: medium\">The joint team proposed that by analyzing the relationship between urban dynamics data, such as vehicular traffic and measured air quality, they could estimate air quality at locations not covered by monitoring stations. \u201cWe can infer real-time, fine-grained air quality information throughout a city based on the historical and real-time air quality data reported by existing monitoring stations, combined with a variety of data sources we observe in the city,\u201d predicted Yu.<\/span><\/p>\n<p><span style=\"font-family: verdana,geneva;font-size: medium\">However, the researchers now grappled with the challenge of processing the massive volume of human dynamics data. A challenge initially imposed by a lack of data had become a problem of having too much data\u2014a 180-degree swing from one extreme to the other!<\/span><\/p>\n<p><span style=\"font-family: verdana,geneva;font-size: medium\">But a solution to the problem of crunching the big data soon arrived, when Li\u2019s project received a <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"http:\/\/research.microsoft.com\/en-us\/projects\/azure\/\" target=\"_blank\">Microsoft Azure for Research Award<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. As Julie Zhu, a doctoral student on the project team noted, \u201cThe program arrived at exactly the right time. We were just looking into building multi-node clusters.\u201d<\/span><\/p>\n<p><span style=\"font-family: verdana,geneva;font-size: medium\">Zhu and her colleagues attended Microsoft Azure training and quickly set up the new computing environment. Citing the enormous amounts of data provided by just one city, Shenzhen, Zhu described the value of Azure. \u201cWe need to collect and process about 1 terabyte per month of urban data on air quality, meteorological data, and traffic information, and so on,\u201d she said. \u201cWhat\u2019s great about Microsoft Azure is that it goes way beyond the data storage. It integrates all the functionalities we need for data crawling, indexing, training, and visualization. Microsoft Azure truly enables us to do the real-time and scalable data processing.\u201d<\/span><\/p>\n<p style=\"text-align: center\"><img decoding=\"async\" title=\" Professor Li presenting the project at the 2014 Microsoft Research Asia Faculty Summit.\" src=\"https:\/\/msdnshared.blob.core.windows.net\/media\/MSDNBlogsFS\/prod.evol.blogs.msdn.com\/CommunityServer.Blogs.Components.WeblogFiles\/00\/00\/01\/32\/81\/2021.urban-computing-2_496px.jpg\" alt=\" Professor Li presenting the project at the 2014 Microsoft Research Asia Faculty Summit.\" \/><\/p>\n<p style=\"text-align: center\"><span style=\"font-family: verdana,geneva;font-size: small;color: #808080\">Professor Li presenting the project at the 2014 Microsoft Research Asia Faculty Summit.<\/span><\/p>\n<p><span style=\"font-family: verdana,geneva;font-size: medium\">Within months, the team had arrived at their initial results for Shenzhen. When Li and Yu presented their work at the 2014 Microsoft Research Asia Faculty Summit in Beijing, their novel approach generated great excitement, not only for how it processed the massive volumes of data, but also for finding the data sources in the first place. As Li commented, \u201cAll the data we used are from public channels. There are tons of data out there.\u201d Li and Yu are now working together to build a predictive model, which would address the initial dilemma of the paucity of air-quality monitoring stations.<\/span><\/p>\n<p><span style=\"font-family: verdana,geneva;font-size: medium\">Yu mused on how the challenge had morphed from insufficient data into a surplus of data. \u201cWe now know that the lack of data and big data do not conflict with each other. They co-exist in the same problem. What\u2019s important is to identify the dataset from various channels that are key contributors,\u201d Yu explained. To which Zhu, now a Microsoft Research intern, added, \u201cYou just need a good platform and tools to handle them.\u201d<\/span><\/p>\n<p><span style=\"font-family: verdana,geneva;font-size: medium\"><em>\u2014Winnie Cui, Senior Research Program Manager, Microsoft Research<\/em><\/span><\/p>\n<p><span style=\"font-family: verdana,geneva;font-size: medium\"><strong>Learn more<\/strong><\/span><\/p>\n<ul>\n<li><span style=\"font-family: verdana,geneva;font-size: small\"><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"http:\/\/research.microsoft.com\/en-us\/projects\/azure\/\" target=\"_blank\">Microsoft Azure for Research<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/span><\/li>\n<li><span style=\"font-family: verdana,geneva;font-size: small\"><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"https:\/\/azure.microsoft.com\/en-us\/\" target=\"_blank\">Microsoft Azure<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><br \/>\n<\/span><\/li>\n<li><span style=\"font-family: verdana,geneva;font-size: small\"><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"http:\/\/research.microsoft.com\/en-us\/events\/asiafacsum2014\/\" target=\"_blank\">2014 Microsoft Research Asia Faculty Summit<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/span><\/li>\n<li><span style=\"font-family: verdana,geneva;font-size: small\"><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"http:\/\/www.scmp.com\/news\/hong-kong\/article\/1433803\/air-pollution-worst-18-years-shopping-areas\" target=\"_blank\">News article about air pollution in Hong Kong<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/span><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Many people living in the People\u2019s Republic of China and Hong Kong have a new habit: they check the air pollution index before venturing outside. Air quality has deteriorated rapidly in China, with nitrogen dioxide and particulate matter levels frequently\u00a0exceeding safety guidelines set by the World Health Organization. While poor air quality clearly impacts public [&hellip;]<\/p>\n","protected":false},"author":32627,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[],"msr_hide_image_in_river":0,"footnotes":""},"categories":[194474,194477,194478],"tags":[194606,193658,186831,194979,186889,195821,196084,193659,197091,197594,197653,187389,197656,197695,197790,197875],"research-area":[],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-4131","post","type-post","status-publish","format-standard","hentry","category-data-visulalization","category-distributed-systems","category-ecology-and-environment","tag-air-pollution-index","tag-azure-for-research","tag-big-data","tag-china","tag-cloud-computing","tag-hong-kong","tag-julie-zhu","tag-microsoft-azure","tag-scalable-data-processing","tag-university-of-hong-kong","tag-urban-air-quality","tag-urban-computing","tag-urban-dynamics-data","tag-victor-li","tag-winnie-cui","tag-yu-zheng","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[],"related-events":[],"related-researchers":[],"msr_type":"Post","byline":"","formattedDate":"September 11, 2015","formattedExcerpt":"Many people living in the People\u2019s Republic of China and Hong Kong have a new habit: they check the air pollution index before venturing outside. Air quality has deteriorated rapidly in China, with nitrogen dioxide and particulate matter levels frequently\u00a0exceeding safety guidelines set by the&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/4131","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/32627"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=4131"}],"version-history":[{"count":1,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/4131\/revisions"}],"predecessor-version":[{"id":260688,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/4131\/revisions\/260688"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=4131"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=4131"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=4131"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=4131"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=4131"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=4131"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=4131"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=4131"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=4131"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=4131"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=4131"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}