Datacenter Networking & Performance Optimization of Cloud Services
We are pursuing a multi-year cross-lab research program that focusses on producing the next generation data center networking and services. We are experimenting with radical new designs in network architecture, programming abstractions, and performance management tools. We care about inexpensive future-proof networking inside the data centers, between globally distributed data centers and to the data centers. Our research includes several projects the cut across various systems and networking research areas that are being pursued in collaboration with Microsoft’s Global Foundation Services Team, Windows Azure Team, Bing Team, and the Management Solutions Division.
Enterprise Network Management & Services
We are pursuing several different projects in this area. In particular, NetHealth is a network management research program in which end-hosts cooperatively detect, diagnose, and recover from network faults. Unlike existing products we take an end-host centric approach to gathering, aggregating, and analyzing data at all layers of the networking stack for determining the root cause of the problems. NetHealth includes several on-going projects in the wireless and wired space that are being pursued in collaboration with Microsoft’s Management Solutions Division and Microsoft’s Unified Communications Group.
Cognitive Wireless Networking
The next generation of wireless networks will include software defined radios, cognitive radios, and multi-radio systems which will co-exist harmoniously while operating over a very wide range of frequencies. We are revisiting “classical” wireless networking problems and designing new solutions that incorporate and build upon recent advances in software and hardware technologies. Of interest lately has been our research solutions to problems in white space networking (the KNOWS project). We are working with policy makers, business units and academia to address the societal needs for providing inexpensive broadband connectivity everywhere.
Mobile Computing & Software Services
Mobile devices, especially smartphones, are being adopted at a phenomenal pace. We are pursuing a variety of mobility-related projects: studying how the cloud can enhance the user experience on mobile devices (HAWAII); understanding how people use smartphones and the performance characteristics of 3G networks (3GTest & Diversity Studies); building systems to enhance smartphone performance, functionality, and battery lifetime using code offload (MAUI); building infrastructure to enable mobile social applications (Virtual Compass); and enhancing mobile device sensors by making their sensor readings trustworthy. In the software services arena, we are pursuing a variety of systems to simplify building scalable and geo-distributed services (PRS/Centrifuge, Volley, and Stout). Another area of emphasis is home networks, where we are pursuing network diagnosis services for the home (NetMedic NetClinic), as well as new services and abstractions for building networked applications for the home (HomeOS).
CloudCmp: CloudCmp is a systematic comparator of the performance and cost of cloud providers that helps customers pick a cloud that fits their needs. CloudCmp measures the elastic computing, persistent storage, and networking services offered by a cloud along metrics that directly reflect their impact on the performance of customer applications.
3GTest: is a tool for systematic comparison of smartphone application performance along several important dimensions such as carrier networks, device capabilities, and server configurations. This cross-platform measurement tool is useful for controlled experiments. As of Summer of 2010, 3GTest had been downloaded and executed by more than 70,000 users from all over the world.
Entact: is a technique that jointly optimizes the cost and performance of delivering traffic from an online service provider (OSP) network to its users. It uses a route-injection mechanism to measure the performance of alternative paths that are not being currently used, without disturbing current traffic. Based on the cost, performance, traffic, and link capacity information, it computes the optimal cost vs. performance curve for the OSP.
WebProphet: Web service providers have many options for improving service performance, varying greatly in feasibility, cost and benefit, but have few tools to predict the impact of these options. WebProphet is a system that automates performance prediction for web services. It employs a novel technique based on timing perturbation to extract web object dependencies, and then uses these dependencies to predict the performance impact of changes to the handling of the objects.
Orion: While the knowledge of dependencies is invaluable for ensuring the stability and efficiency of network applications, thus far the only proven way to discover these complex dependencies is by exploiting human expert knowledge, which does not scale with the number of applications in large enterprises. Orion discovers network dependencies by inspecting packet headers and timing information. The dependency extraction techniques are based on delay spike based analysis.
ELDA: ELDA, formerly known as SureMail, addresses the problem of e-mail delay and loss. Over 1 percent of all mail is lost due to infrastructure failures and aggressive spam filtering, and because e-mail loss is a silent problem, you generally aren’t even aware of it! We have built an Outlook 2007 add-in that alerts you of any e-mail sent to you which has become delayed or lost.
NetPolice: Enables detection of content- and routing-based differentiations in backbone ISPs. We have used NetPolice to study 18 large ISPs spanning 3 major continents over a 10 week period in 2008. The study was published in IMC 2008.
Netdiff: is a system that enables detailed performance comparisons among ISP networks. It helps customers and applications determine which ISP offers the best performance for their specific workload. Using Netdiff, we find that the relative performance of ISPs depends on many factors, including the geographic properties of traffic and the popularity of destinations.
NetMedic: NetMedic helps operators perform detailed diagnosis in enterprise networks. It diagnoses not only generic faults (e.g., performance-related) but also application specific faults (e.g., error codes). It identifies culprits at a fine granularity such as a process or firewall configuration. Our work focuses on both the algorithmic aspects of detailed diagnosis as well as the important task of explaining diagnostic reasoning to the operator.
VanLan: Our goal is to enable cheap and high-throughput wireless connectivity to moving vehicles in urban areas. The available options for such connectivity today fall short in significant ways. Cellular networks are expensive and have low throughput. While some existing WiFi basestations can provide opportunistic connectivity to passing vehicles, they are unable to support longer periods of connectivity. However, WiFi deployment is becoming denser and in many cases, entire cities are being covered. But given the short range of WiFi and the presence of many interfering sources, can such deployments enable continuous, cheap, high-throughput connectivity, by themselves or in conjunction with cellular networks?
Measurement-based models of wireless networks: Based on detailed measurements of wireless behavior in the wild, we build practical models that aid in understanding and predicting network performance. Our goal is to have a level of performance predictability that is similar to that of wired networks.
Bunker: Bunker is a network tracing system that offers strong privacy while simplifying the development of network tracing software. Bunker uses virtualization, encryption, and restricted I/O interfaces to protect the raw network traces and the tracing software, exporting only an anonymized trace. With Bunker, all sensitive data is stored in a buffer on disk that is “locked down” along with the tracing software. In this way, no raw data can be lost, leaked, or stolen.
BlueMonarch: BlueMonarch is a system for evaluating Bluetooth applications in the wild. BlueMonarch emulates a Bluetooth transfer to any device responding to Bluetooth Service Discovery requests; because many cell-phones, laptops, and PDAs in the wild respond to such probes, BlueMonarch enables quick prototyping of Bluetooth applications in the wild, to hundreds of unmodified Bluetooth devices.
Lockr: Lockr is an access control system that incorporates social networking abstractions. Lockr was designed to simplify content sharing on the Internet. It decouples the management of social information from online sharing systems by letting users exchange application-independent attestations. This decoupling facilitates the integration of Lockr’s access control with any online application, such as Web 2.0 sites and P2P file sharing.
WiFiAds: Delivering location-sensitive content to clients over Wi-Fi networks without requiring the clients to connect to the network. The WiFiAds project is a joint collaboration between the Networking Research Group and the Mobility, Network and Systems Group at Microsoft Research, India.
Mesh: A multi-hop wireless network for residential broadband Internet access in urban and rural communities. We worked on problems related to: range & capacity enhancement; self-management; multi-hop routing; privacy and security, and spectrum management. We pursued these problems in collaboration with Microsoft’s Advanced Strategy & Technology Group.
CoopNet: Coopnet (Cooperative Networking) is a distributed system that improves the performance and functionality of client-server applications by employing selective use of peer-to-peer (p2p) communications. Our focus was on alleviation of web flash crowds, and streaming media content distribution.
VirtualWiFi: VirtualWiFi is a virtualization architecture for wireless LAN (WLAN) cards. It enables the user to connect his/her machine to multiple wireless networks using just one WLAN card. VirtualWiFi works over Windows XP and the software is available for download to the research community.
UCom: UCom (or Universal Communicator) is the first multi-radio wireless system to show that wireless system performance and functionality improves significantly when multiple radios work in conjunction in the same network. It proved that such systems are more dependable, more flexible, and allow more innovation than traditional single-radio wireless systems.
PeerMetric: The goal of PeerMetric is to understand and characterize the network performance of broadband hosts in the Internet. Our work focuses on both characterizing and investigating implications of issues like asymmetric links, non-FIFO packet scheduling, and rate shaping on bandwidth estimation, overlay multicast, etc.
Yoda: We studied the characteristics of notification and browse services provided by a large commercial web site: MSN Mobile designed specifically for users who access it via their cell-phones and PDAs. The purpose of this, first of its kind, study was to gain insights required to design fast and effective web content for battery and bandwidth constraint devices.
Dynamics of Large Internet Servers: We analyzed the dynamics of large internet servers, such as the MSNBC News Site, identifying key characteristics that can be exploited in the design of efficient algorithms for serving content.
Network Tomography: Inferring packet loss characteristics of internet links using server-based measurements. We made inferences based on passive observation of existing end-to-end traffic. We developed three techniques to identify lossy links in the network based on random sampling, linear optimization, and Bayesian inference using gibbs sampling.
IPv6: Our implementation formed the basis of Microsoft’s first IPv6 product stack, which shipped in Windows XP and Windows Server 2003. We authored several standards-track Internet Engineering Task Force (IETF) RFCs and made our source code available to the research community.
Choice: To the best of our knowledge, Choice is the first WiFi based public-area hot-spot network in the world. Our unique edge-server based architecture included support for network discovery, global authentication, user mobility, differentiated services, first-hop security, and location/context services. The underlying techniques are the basis of many commercially deployed hot-spot networks.
Radar: Radar is the first WiFi signal-strength based indoor positioning system. It proved that RF fingerprinting and environmental profiling with commodity wireless LAN hardware can be used to determine user and machine location inside buildings, thereby enabling indoor location-aware applications.
WiLib: As part of our “wireless is not Ethernet” project, we developed a user-level library to program wireless network cards dynamically. This level of programming abstraction was previously not available. NDIS WLAN extensions and the more recent Native WiFi programming framework in Windows are based on WiLib. A subset of the original library is still available from UCSD as WRAPI.
Important Note: The software provided by our group is not supported by Microsoft. Its use of is subject to the respective research license agreements accompanying the software.
Academic Resource Toolkit
During 2005-2009 we made available to faculty members and researchers at accredited academic institutions our Networking Academic Resource Toolkit — a research and teaching resource for exploring core technologies in wireless networks. Approximately 1000 kits in total were distributed worldwide.
This kit is no longer available. Many of the contents remain available as downloads from this web site. We continue to explore alternative ways to engage with the academic community in networking research.
Here are some additional downloads
- ELDA: ELDA, formerly known as SureMail, addresses the problem of e-mail delay and loss. Over 1 percent of all mail is lost due to infrastructure failures and aggressive spam filtering, and because e-mail loss is a silent problem, you generally aren’t even aware of it! We have built an Outlook 2007 add-in that alerts you of any e-mail sent to you which has become delayed or lost.
- Mesh Connectivity Layer: The Mesh Connectivity Layer (MCL) driver with Link-Quality Source Routing (LQSR). LQSR is a routing protocol for ad-hoc networks. MCL implements a virtual network adapter, so that to the rest of the system the ad-hoc network appears as an additional (virtual) network link.
- Virtual WiFi: Virtual WiFi allows connections to multiple IEEE 802.11 networks with one WiFi card. It virtualizes the WiFi card, and connects each virtual adapter to a different wireless network. It switches the WiFi card across all virtual interfaces to give an illusion of simultaneous connectivity on multiple wireless networks.
- NS-2 for Windows: This is a Windows port of NS-2 Simulator version 2.1b9
- Simulation Code and Documentation for SSCH: The Slotted Seeded Channel Hopping (SSCH) protocol was developed to increase the capacity of single-radio multi-hop wireless networks. These files allow you to simulate SSCH in QualNet. The software is produced by Cornell
- IPv6 Protocol Stack: We wrote an IPv6 stack for Windows NT and Windows 2000. Our implementation went on to form the basis of Microsoft’s first IPv6 product stack, which shipped in Windows XP and Windows Server 2003. The source code is also available for download.
We funded some software development efforts. Links to some of these are provided as a service, we are not responsible for the content on these web site:
- Wireless Research API: WRAPI provides an Application Programming Interface to wireless network interfaces on Windows XP. We developed the original version of WRAPI, the current version is produced by UCSD. (Note: WRAPI is not a Microsoft offering, we only provide this link as a reference, it is likely that these interfaces may not work as Microsoft’s software evolves)
- XORP – eXtensible Open Router Platform: XORP’s goal is a open router platform that is stable and fully featured enough for production use, and flexible and extensible enough to enable network research. XORP implements routing protocols for IPv4 and IPv6 and a unified means to configure them. IT will be available for Windows Server 2003 shortly.