Cloud Computing and Storage



Microsoft Research blog


Lead by Jin Li, the Cloud Computing and Storage (CCS) group focuses on cloud infrastructure and algorithm research. In addition to pursuing original research and publishing papers, this team devotes time to working with Microsoft product groups on the technology transfer of their research.

CCS’s recent projects include Deep Learning Workspace (DL Workspace) and storage improvements.

Deep Learning Workspace

DL Workspace is an open-source toolkit that empowers AI scientists to quickly create a cloud AI infrastructure (either in a public cloud, such as Azure, or in an on-premises cluster) to manage AI training, interactive exploration, inference, and analytics. DL Workspace supports all major deep-learning toolkits out of the box, including TensorFlow, CNTK, Caffe, MXNet, and others. It also supports big data analytics frameworks such as Apache Hadoop and Apache Spark. It is used in daily production by multiple Microsoft teams such as Microsoft Cognitive Services, SwiftKeys, and Bing Relevance.

Other notable research efforts

CCS has assisted the Azure team to architect and implement the local reconstruction code (LRC) used in Windows Azure Storage. This is a new family of erasure codes that provide significant reduction in storage overhead and has reduced the minimum number of fragments that need to be read to reconstruct a data fragment. This effort has led to Microsoft saving hundreds of millions of dollars, a Best Paper Award at USENIX ATC 2012, and a 2013 Microsoft Technical Community Network Storage Technical Achievement Award. CCS also architected the erasure code used in Storage Spaces in Windows 8.1 and Windows Server 2012 R2. They have also architected and implemented the erasure coding used in Lync, Xbox, and RemoteFX.

CCS assisted the Windows File Server group in architecting and implementing the primary data deduplication feature in this Windows Server 2012 Paper and end-to-end deduplication for storage virtualization in Windows Server 2012 R2. Key contributions include a new data-chunking algorithm, a low-RAM footprint indexing data structure to detect duplicate data (based on ChunkStash), and a data partitioning and reconciliation technique.  The last two contributions both enable scaling index resource usage with data size. It has led to 20-82 percent data savings for customers and it was among the top three features for Windows File Server that were introduced in Windows Server 2012. After receiving rave reviews, there is evidence that some customers are upgrading to Windows Server 2012 specifically to obtain this primary data deduplication feature.

In another project, CCS leveraged the benefits of Solid State Drive (SSD) for storage applications. FlashStore implements as an SSD-optimized, low-RAM footprint key-value store that organizes storage on flash, in a log-structured manner. It is now in use in Bing and SkimpyStash implements an ultra-low RAM footprint key-value store. The storage design of SkimpyStash has been incorporated into Bw-Tree, a joint project of CCS, MSR Database group, and the Azure DocumentDB team. Bw-Tree has been shipped to SQL Server 2014 (Hekaton) and Azure DocumentDB.

Partnering with the Remote Desktop Virtualization (RDV) team, CCS has also assisted in the architecture and implementation of the RemoteFX for WAN feature in Windows 8 and Windows Server 2012, which provides a fast and fluid user experience for remote sessions running over any WAN and wireless networks. Read Optimizing FEC Transmission Strategy for Minimizing Delay in Lossless Sequential Streaming (2011, IEEE).

In the news