Datacenter Networking & Performance Optimization of Cloud Services
We are pursuing a multi-year cross-lab research program that focusses on producing the next generation data center networking and services. We are experimenting with radical new designs in network architecture, programming abstractions, and performance management tools. We care about inexpensive future-proof networking inside the data centers, between globally distributed data centers and to the data centers. Our research includes several projects the cut across various systems and networking research areas that are being pursued in collaboration with Microsoft’s Global Foundation Services Team, Windows Azure Team, Bing Team, and the Management Solutions Division.
Enterprise Network Management & Services
We are pursuing several different projects in this area. In particular, NetHealth is a network management research program in which end-hosts cooperatively detect, diagnose, and recover from network faults. Unlike existing products we take an end-host centric approach to gathering, aggregating, and analyzing data at all layers of the networking stack for determining the root cause of the problems. NetHealth includes several on-going projects in the wireless and wired space that are being pursued in collaboration with Microsoft’s Management Solutions Division and Microsoft’s Unified Communications Group.
Cognitive Wireless Networking
The next generation of wireless networks will include software defined radios, cognitive radios, and multi-radio systems which will co-exist harmoniously while operating over a very wide range of frequencies. We are revisiting “classical” wireless networking problems and designing new solutions that incorporate and build upon recent advances in software and hardware technologies. Of interest lately has been our research solutions to problems in white space networking (the KNOWS project). We are working with policy makers, business units and academia to address the societal needs for providing inexpensive broadband connectivity everywhere.
Mobile Computing & Software Services
Mobile devices, especially smartphones, are being adopted at a phenomenal pace. We are pursuing a variety of mobility-related projects: studying how the cloud can enhance the user experience on mobile devices (HAWAII); understanding how people use smartphones and the performance characteristics of 3G networks (3GTest & Diversity Studies); building systems to enhance smartphone performance, functionality, and battery lifetime using code offload (MAUI); building infrastructure to enable mobile social applications (Virtual Compass); and enhancing mobile device sensors by making their sensor readings trustworthy. In the software services arena, we are pursuing a variety of systems to simplify building scalable and geo-distributed services (PRS/Centrifuge, Volley, and Stout). Another area of emphasis is home networks, where we are pursuing network diagnosis services for the home (NetMedic & NetClinic), as well as new services and abstractions for building networked applications for the home (HomeOS).
CloudCmp: CloudCmp is a systematic comparator of the performance and cost of cloud providers that helps customers pick a cloud that fits their needs. CloudCmp measures the elastic computing, persistent storage, and networking services offered by a cloud along metrics that directly reflect their impact on the performance of customer applications.
3GTest: is a tool for systematic comparison of smartphone application performance along several important dimensions such as carrier networks, device capabilities, and server configurations. This cross-platform measurement tool is useful for controlled experiments. As of Summer of 2010, 3GTest had been downloaded and executed by more than 70,000 users from all over the world.
Entact: is a technique that jointly optimizes the cost and performance of delivering traffic from an online service provider (OSP) network to its users. It uses a route-injection mechanism to measure the performance of alternative paths that are not being currently used, without disturbing current traffic. Based on the cost, performance, traffic, and link capacity information, it computes the optimal cost vs. performance curve for the OSP.
WebProphet: Web service providers have many options for improving service performance, varying greatly in feasibility, cost and benefit, but have few tools to predict the impact of these options. WebProphet is a system that automates performance prediction for web services. It employs a novel technique based on timing perturbation to extract web object dependencies, and then uses these dependencies to predict the performance impact of changes to the handling of the objects.
Orion: While the knowledge of dependencies is invaluable for ensuring the stability and efficiency of network applications, thus far the only proven way to discover these complex dependencies is by exploiting human expert knowledge, which does not scale with the number of applications in large enterprises. Orion discovers network dependencies by inspecting packet headers and timing information. The dependency extraction techniques are based on delay spike based analysis.
ELDA: ELDA, formerly known as SureMail, addresses the problem of e-mail delay and loss. Over 1 percent of all mail is lost due to infrastructure failures and aggressive spam filtering, and because e-mail loss is a silent problem, you generally aren’t even aware of it! We have built an Outlook 2007 add-in that alerts you of any e-mail sent to you which has become delayed or lost.
NetPolice: Enables detection of content- and routing-based differentiations in backbone ISPs. We have used NetPolice to study 18 large ISPs spanning 3 major continents over a 10 week period in 2008. The study was published in IMC 2008.
Netdiff: is a system that enables detailed performance comparisons among ISP networks. It helps customers and applications determine which ISP offers the best performance for their specific workload. Using Netdiff, we find that the relative performance of ISPs depends on many factors, including the geographic properties of traffic and the popularity of destinations.
NetMedic: NetMedic helps operators perform detailed diagnosis in enterprise networks. It diagnoses not only generic faults (e.g., performance-related) but also application specific faults (e.g., error codes). It identifies culprits at a fine granularity such as a process or firewall configuration. Our work focuses on both the algorithmic aspects of detailed diagnosis as well as the important task of explaining diagnostic reasoning to the operator.
VanLan: Our goal is to enable cheap and high-throughput wireless connectivity to moving vehicles in urban areas. The available options for such connectivity today fall short in significant ways. Cellular networks are expensive and have low throughput. While some existing WiFi basestations can provide opportunistic connectivity to passing vehicles, they are unable to support longer periods of connectivity. However, WiFi deployment is becoming denser and in many cases, entire cities are being covered. But given the short range of WiFi and the presence of many interfering sources, can such deployments enable continuous, cheap, high-throughput connectivity, by themselves or in conjunction with cellular networks?
Measurement-based models of wireless networks: Based on detailed measurements of wireless behavior in the wild, we build practical models that aid in understanding and predicting network performance. Our goal is to have a level of performance predictability that is similar to that of wired networks.
Bunker: Bunker is a network tracing system that offers strong privacy while simplifying the development of network tracing software. Bunker uses virtualization, encryption, and restricted I/O interfaces to protect the raw network traces and the tracing software, exporting only an anonymized trace. With Bunker, all sensitive data is stored in a buffer on disk that is “locked down” along with the tracing software. In this way, no raw data can be lost, leaked, or stolen.
BlueMonarch: BlueMonarch is a system for evaluating Bluetooth applications in the wild. BlueMonarch emulates a Bluetooth transfer to any device responding to Bluetooth Service Discovery requests; because many cell-phones, laptops, and PDAs in the wild respond to such probes, BlueMonarch enables quick prototyping of Bluetooth applications in the wild, to hundreds of unmodified Bluetooth devices.
Lockr: Lockr is an access control system that incorporates social networking abstractions. Lockr was designed to simplify content sharing on the Internet. It decouples the management of social information from online sharing systems by letting users exchange application-independent attestations. This decoupling facilitates the integration of Lockr’s access control with any online application, such as Web 2.0 sites and P2P file sharing.
WiFiAds: Delivering location-sensitive content to clients over Wi-Fi networks without requiring the clients to connect to the network. The WiFiAds project is a joint collaboration between the Networking Research Group and the Mobility, Network and Systems Group at Microsoft Research, India.
Mesh A multi-hop wireless network for residential broadband Internet access in urban and rural communities. We worked on problems related to: range & capacity enhancement; self-management; multi-hop routing; privacy and security, and spectrum management. We pursued these problems in collaboration with Microsoft’s Advanced Strategy & Technology Group.
CoopNet: Coopnet (Cooperative Networking) is a distributed system that improves the performance and functionality of client-server applications by employing selective use of peer-to-peer (p2p) communications. Our focus was on alleviation of web flash crowds, and streaming media content distribution.
VirtualWiFi: VirtualWiFi is a virtualization architecture for wireless LAN (WLAN) cards. It enables the user to connect his/her machine to multiple wireless networks using just one WLAN card. VirtualWiFi works over Windows XP and the software is available for download to the research community.
UCom: UCom (or Universal Communicator) is the first multi-radio wireless system to show that wireless system performance and functionality improves significantly when multiple radios work in conjunction in the same network. It proved that such systems are more dependable, more flexible, and allow more innovation than traditional single-radio wireless systems.
PeerMetric: The goal of PeerMetric is to understand and characterize the network performance of broadband hosts in the Internet. Our work focuses on both characterizing and investigating implications of issues like asymmetric links, non-FIFO packet scheduling, and rate shaping on bandwidth estimation, overlay multicast, etc.
Yoda: We studied the characteristics of notification and browse services provided by a large commercial web site: MSN Mobile designed specifically for users who access it via their cell-phones and PDAs. The purpose of this, first of its kind, study was to gain insights required to design fast and effective web content for battery and bandwidth constraint devices.
Dynamics of Large Internet Servers: We analyzed the dynamics of large internet servers, such as the MSNBC News Site, identifying key characteristics that can be exploited in the design of efficient algorithms for serving content.
Network Tomography: Inferring packet loss characteristics of internet links using server-based measurements. We made inferences based on passive observation of existing end-to-end traffic. We developed three techniques to identify lossy links in the network based on random sampling, linear optimization, and Bayesian inference using gibbs sampling.
IPv6: Our implementation formed the basis of Microsoft’s first IPv6 product stack, which shipped in Windows XP and Windows Server 2003. We authored several standards-track Internet Engineering Task Force (IETF) RFCs and made our source code available to the research community.
Choice: To the best of our knowledge, Choice is the first WiFi based public-area hot-spot network in the world. Our unique edge-server based architecture included support for network discovery, global authentication, user mobility, differentiated services, first-hop security, and location/context services. The underlying techniques are the basis of many commercially deployed hot-spot networks.
Radar: Radar is the first WiFi signal-strength based indoor positioning system. It proved that RF fingerprinting and environmental profiling with commodity wireless LAN hardware can be used to determine user and machine location inside buildings, thereby enabling indoor location-aware applications.
WiLib: As part of our “wireless is not Ethernet” project, we developed a user-level library to program wireless network cards dynamically. This level of programming abstraction was previously not available. NDIS WLAN extensions and the more recent Native WiFi programming framework in Windows are based on WiLib. A subset of the original library is still available from UCSD as WRAPI.
Important Note: The software provided by our group is not supported by Microsoft. Its use of is subject to the respective research license agreements accompanying the software.
Academic Resource Toolkit
During 2005-2009 we made available to faculty members and researchers at accredited academic institutions our Networking Academic Resource Toolkit — a research and teaching resource for exploring core technologies in wireless networks. Approximately 1000 kits in total were distributed worldwide.
This kit is no longer available. Many of the contents remain available as downloads from this web site. We continue to explore alternative ways to engage with the academic community in networking research.
Here are some additional downloads
- ELDA: ELDA, formerly known as SureMail, addresses the problem of e-mail delay and loss. Over 1 percent of all mail is lost due to infrastructure failures and aggressive spam filtering, and because e-mail loss is a silent problem, you generally aren’t even aware of it! We have built an Outlook 2007 add-in that alerts you of any e-mail sent to you which has become delayed or lost.
- Mesh Connectivity Layer: The Mesh Connectivity Layer (MCL) driver with Link-Quality Source Routing (LQSR). LQSR is a routing protocol for ad-hoc networks. MCL implements a virtual network adapter, so that to the rest of the system the ad-hoc network appears as an additional (virtual) network link.
- Virtual WiFi: Virtual WiFi allows connections to multiple IEEE 802.11 networks with one WiFi card. It virtualizes the WiFi card, and connects each virtual adapter to a different wireless network. It switches the WiFi card across all virtual interfaces to give an illusion of simultaneous connectivity on multiple wireless networks.
- NS-2 for Windows: This is a Windows port of NS-2 Simulator version 2.1b9
- Simulation Code and Documentation for SSCH: The Slotted Seeded Channel Hopping (SSCH) protocol was developed to increase the capacity of single-radio multi-hop wireless networks. These files allow you to simulate SSCH in QualNet. The software is produced by Cornell
- IPv6 Protocol Stack: We wrote an IPv6 stack for Windows NT and Windows 2000. Our implementation went on to form the basis of Microsoft’s first IPv6 product stack, which shipped in Windows XP and Windows Server 2003. The source code is also available for download.
We funded some software development efforts. Links to some of these are provided as a service, we are not responsible for the content on these web site:
- Wireless Research API: WRAPI provides an Application Programming Interface to wireless network interfaces on Windows XP. We developed the original version of WRAPI, the current version is produced by UCSD. (Note: WRAPI is not a Microsoft offering, we only provide this link as a reference, it is likely that these interfaces may not work as Microsoft’s software evolves)
- XORP – eXtensible Open Router Platform: XORP’s goal is a open router platform that is stable and fully featured enough for production use, and flexible and extensible enough to enable network research. XORP implements routing protocols for IPv4 and IPv6 and a unified means to configure them. IT will be available for Windows Server 2003 shortly.
MSR Redmond PhD Fellows (Networking Research)
Microsoft awards a two-year fellowship to outstanding Ph.D. students. The full description of the award is available on our Graduate Fellowship Program page. Past recipients with Ph.D.s in networking are:
- Shravan Rayanchu, University of Wisconsin (2009-11)
- Rohan Narayana Murty, Harvard University (2008-10)
- Aruna Balasubramanian, University of Massachusetts Amherst (2008-10)
- Karthik Lakshminarayanan, University of California Berkeley (2005-07)
- Jinyang Li, Massachusetts Institute of Technology (2004-05)
- Qiang Huang, Princeton University (2004-05)
- Magdelena Balazinska, Massachusetts Institute of Technology (2003-05)
- Ratul Mahajan, University of Washington (2003-05)
- Ranveer Chandra, Cornell University (2002-05)
- David Andersen, Massachusetts Institute of Technology (2002-04)
Interns in the Networking Group
We have been lucky to have had the pleasure of mentoring several excellent interns at Microsoft Research. Many are now pursuing promising careers in academic and industrial research labs.
Sara Alspaugh, Graduate Student, University of California BerkeleyWorked on video catalog service for Windows Mobile Phone
Colin Dixon, Ph.D. Candidate, University of Washington SeattleWorked on HomeOS and MarketPlace for home
Daniel Halperin, Ph.D. Candidate, University of Washington SeattleWorked on 60 GHz wireless & optical flyways in datacenters
Srinivas Krishnan, Ph.D. Candidate, Univ of North Carolina at Chapel HillWorked on low cost backbone for data centers
He Liu, Ph.D. Candidate, University of California San DiegoWorked on trusted sensors prototypes
Justin Manweiler, Ph.D. Candidate, Duke UniversityWorked on mobile multi-player gaming on SmartPhones
George P. Nychis, Ph.D. Candidate, Carnegie Mellon UniversityWorked on problems related to wireless networking and XBox
Radhika Niranjan Mysore, Ph.D. Candidate, University of California San DiegoWorked on diagnosis of large distributed systems with a focus on Unified Communications
Zhiyun Qian, Ph.D. Candidate, University of MichiganWorked on request routing in cloud services
Eric Rozner, Ph.D. Candidate, University of Texas AustinWorked on problems related to audio-video streaming in XBox Games
Bo Tan, Ph.D. Candidate, University of Illinois Urbana-ChampaignWorked on multi-path TCP inside datacenters
Ashok Anand, Ph.D. Candidate, University of WisconsinWorked on traffic engineering for bandwidth-sensitive applications
Mohammad Reza Alizadeh Attar, Ph.D. Candidate, Stanford UniversityWorked on datacenter TCP
Aruna Balasubramanian, Ph.D. Candidate, University of Massachusetts AmherstWorking on Decision Engine Logic for mobile code offload to Cloud
Dae-ki Cho, Graduate Student, University of California Los AngelesWorked on augmented reality applications on SmartPhones
Hossein Falaki, Ph.D. Candidate, University of California Los AngelesWorked on studying the usage patterns of SmartPhone users
Ang Li, Ph.D. Candidate, Duke UniversityWorked on WAN performance diagnosis for web search
Rohan Narayana Murty, Ph.D. Candidate, Harvard UniversityWorked on channel occupancy database service
Radhika Niranjan Mysore, Ph.D. Candidate, University of California San DiegoWorked on scalable datacenter storage
Anh M. Nguyen, Ph.D. Candidate, University of Illinois – Urbana ChampaignWorked on decreasing the trusted computing base of infrastructure for mobile services
George P. Nychis, Ph.D. Candidate, Carnegie Mellon UniversityWorked on techniques to demonstrate co-existence of mics & ws devices
Ki-Woong Park, Ph.D. Candidate, KAISTWorked on personal web server for SmartPhones
Abhinav Pathak, Ph.D. Candidate, Purdue UniversityWorked on dynamic partitioning and offload of applications running on a SmartPhone
Swapnil Patil, Ph.D. Candidate, Carnegie Mellon UniversityWorked on the interactions between storage systems and datacenter networks
Hariharan Rahul, Ph.D. Candidate, Massachusetts Institute of TechnologyWorked on techniques for harmonious existence of hetrogenoeous white space networks
Shravan Rayanchu, Ph.D. Candidate, University of WisconsinWorked on improving the security of infrastructure for mobile services
Joshua Reich, Ph.D. Candidate, Columbia UniversityWorked on the design and deployment of sleepserver technology for the Enterprise
Alan C. Shieh, Ph.D. Candidate, Cornel UniversityWorking on Decision Engine Logic for mobile code offload to Cloud
Janani Sriram, M.S Student, Dartmouth UniversityWorked on improving resolution of Indoor location technologies for the retail sector
Eeyore Wang, Ph.D. Candidate, Carnegie Mellon UniversityWorked on modifying the Windows networking stack to support white space networking
Nilanjan Banerjee, Ph.D. Candidate, University of Massachusetts AmherstWorked on wireless relative localization
Mudit Jain, Undergraduate, Indian Institute of TechnologyWorked on indoor localization
Zhichun Li, Ph.D. Candidate, Northwestern UniversityWorked on WedProphet for web service performance prediction
Murtaza Motiwala, Ph.D. Candidate, Georgia Institute of TechnologyWorked on traffic analysis in content provider network
Patrick Verkaik, Ph.D. Candidate, University of California San DiegoWorked on diagnosis in enterprise networks
Ying Zhang, Ph.D. Candidate, University of MichiganWorked on Wisdom for system capacity planning
Zheng Zhang, Ph.D. Candidate, Purdue UniversityWorked on Entact for traffic engineering in content provider network
Aruna Balasubramanian, Ph.D. Candidate, University of Massachusetts AmherstWorked on vehicular Wi-Fi networking
Dhiman Barman, Ph.D. Candidate, University of California RiversideWorked on IPTV management
Xu Chen, Ph.D. Candidate, University of MichiganWorked on dependency graph analysis in enterprise networks
Brent Couvrette, Woodinville High SchoolWorked on game bandwidth estimation
Abhinav Jain, Undergraduate, Indian Institute of Technology KanpurWorked on providing Internet connectivity in Microsoft’s shuttle system
Vaishnav Janardhan, Ph.D. Candidate, Columbia UniversityWorked on Datagram Congestion Control Protocol (DCCP) implementation on Windows
YongChul Kwon, Ph.D. Candidate, University of WashingtonWorked on measuring and monitoring tools in Windows Live platform
YonugKi Lee, Korea Advanced Institute for Science & Technology (KAIST)Worked on Game topological analysis
Lindsey Poole, Ph.D. Candidate, Princeton UniversityWorked on Netdiff for ISP performance comparison
Ramya Raghavendra, Ph.D. Candidate, University of California San DiegoWorked on pushing the envelope on software smarts in Wi-Fi chips
Nilendu Sekhar, Undergraduate, Indian Institute of Technology KanpurWorked on enterprise network management
Francisco Alvarez Cavazos, Ph.D. Candidate, ITESM – Monterrey, MexicoWorked on network performance tools
Krishna Ramachandran, Ph.D. Candidate, University of California Santa BarbaraWorked on enabling group communications in wireless mesh networks
Lun Li, Ph.D. Candidate, California Institute of TechnologyWorked on generalizing fault detection in enterprise networked applications
Nikitas Liogkas, Ph.D. Candidate, University of California Los AngelesWorked on self-diagnosing faults in web browsers
Rohan Murty, Ph.D. Candidate, Harvard University, BostonWorked on a providing wireless access point functionality using desktop machines
Srikanth Kandula, Ph.D. Candidate, Massachusetts Institute of Technology, BostonWorked on diagnosing faults in enterprise networks
Tulika Garg, Undergraduate, Indian Institute of Technology Roorkie, IndiaWorked on implementing the Mesh Connectivity Layer (MCL) in QualNet
Vladimir Brik, Ph.D. Candidate, University of WisconsinWorked on self-diagnosing network faults in WiFi clients
Yuvraj Agarwal , Ph.D. Candidate, University of California San DiegoWorked on lowering energy consumption in VoIP applications
Jakob Eriksson, Ph.D. Candidate, University of California RiversideWorked on all-wireless offices
Dilip Antony Joseph, Ph.D. Candidate, University of California BerkeleyWorked on email reliability
Manish Anand, Ph.D. Candidate, University of MichiganWorked on a comparison study of 802.11 and 802.16
Ranveer Chandra, Ph.D. Candidate, Cornell UniversityWorked on fault diagnosis in infrastructure wireless networks
Pradeep Kyasanur, Ph.D. Candidate, University of Illinois, Urbana ChampionWorked on enhancing wireless mesh networks by using a separate control channel
Ananth Rajagopala-Rao, Ph.D. Candidate, University of California, BerkeleyWorked on measuring wireless interference
Sriram Ramabhadran, Ph.D. Candidate, University of California, San DiegoWorked on Internet measurement
Sreedhar Veeravalli, Undergraduate, Indian Institute of Technology, IndiaWorked on TCPScope, a tool to analyze performance of TCP flows
Can Vuran, Ph.D. Candidate, Georgia TechWorked on measurements of directional antennas
Ashwin Baharambe, Ph.D. Candidate, Carnegie Mellon UniversityWorked on Internet gaming
Ranveer Chandra, Ph.D. Candidate, Cornell UniversityWorked on Native WiFi and fault diagnosis in infrastructure wireless networks
Dejan Kostic, Ph.D. Candidate, Duke UniversityWorked on Fuse, light-weight distributed failure notifications
Ananth Rajagopala-Rao, Ph.D. Candidate, University of California, BerkeleyWorked on trouble-shooting wireless mesh networks
Amit Saha, Ph.D. Candidate, Rice UniversityWorked on developing tools for measuring corporate wireless networks
Maneesh Varshney, Ph. D. Candidate, University of California, Los AngelesWorked on neighbor location determination and MAC with directional antennas
Ranveer Chandra, Ph. D. Candidate, Cornell UniversityWorked on MultiNet and placement of Internet TAPs in wireless mesh networks
Kyle Jamieson, Ph. D. Candidate, Massachusetts Institute of TechnologyWorked on Bandwidth Sharing in Neighborhood Meshes
Karthik Lakshminarayanan, Ph.D. Candidate, University of California, BerkeleyWorked on measurement of broadband networks
Eugene Shih, Ph. D. Candidate in Electrical Engineering, Massachusetts Institute of TechnologyWorked on hardware and systems aspects of the Universal Communicator
Kunwadee Sripanidulchai, Ph.D. Candidate, Carnegie Mellon UniversityWorked on a peer-to-peer system for sharing Web content and on-demand streaming media content
Anand Balachandran, Ph. D. Candidate, University of California San Diego (now @ Microsoft)Worked on The CHOICE Network and RADAR
Li Li, Ph. D. Candidate, Cornell University (now @ Bell Labs, Lucent Technologies)Worked on power conserving algorithms in ad hoc sensor networks
Allen Miu, Ph. D. Candidate, Massachusetts Institute of TechnologyWorked on mobility management within The CHOICE Network and its deployment
Shoamin Wang, Ph. D. Candidate, Massachusetts Institute of TechnologyWorked on Location Determination (WISH) and StudioMIT