Optics for the Cloud

Optics for the Cloud

Publications

Events

Opportunities

Overview

Optics for the Cloud is a programme of research to advance and enable the adoption of optical technologies in the rapidly growing field of cloud computing. The scope of this opportunity includes storage, network, compute, memory and specialized accelerators. The research also spans the entire vertical stack, ranging from optical device fabrication, to optical system and sub-system design, to co-design and integration with the rest of the data center infrastructure and will include new applications enabled by optical technologies.

The world is moving to the cloud at an ever-increasing rate. Microsoft is at the core of this transformation and is continuously building new data centres to keep up with demand. However, most of the core technologies used in these centres today were designed or conceived before the cloud existed. Many of those technologies represent compromises, encumbered by historic design choices pertinent to an antiquated application, and encrusted with features appealing to the idiosyncrasies of many different scenarios. At the cloud scale, there is the scope to create cloud-specific technologies that free us from decades of legacy thinking.

At Microsoft Research Cambridge we are building a team to explore Optics for the Cloud. We believe that by taking a holistic view of the needs of the cloud we can create disruptive technologies for the cloud. To achieve this, we are assembling a highly inter-disciplinary team to ask the simple question – how can optics for the cloud change the cloud? We hope to invent optical end-to-end systems, across the three primary resources of storage, network and compute, that will underpin the next generation of the cloud.

Explore careers in optics research

Read the full story >

Projects

Silica glass

Project Silica

Project Silica, a collaboration with the University of Southampton, is a ground-breaking cloud storage system that uses femtosecond lasers to write data into quartz glass which, due to its durable properties, provides long-lasting, easily-readable storage.

Sirius illustration

Project Sirius

This project aims to develop an all-optical, data-center-wide network that is completely flat, in contrast to the hierarchy of electrical switches used today.

Iris illustration

Project Iris

This project explores novel designs of regional and wide-area cloud networks, from the ground up.

People

Team

Interns and Collaborators

Storage

Research background: Storage

The amount of data being generated worldwide is growing exponentially.  The volume is expected to exceed 100 zettabytes by 2023. Simply put, that’s 10 terabytes for every person on the planet.  The challenge is that storing all that data is expensive.  Soon, we could be unable to afford to store it all.  At Microsoft Research, Project Silica is tackling this problem by developing the world’s first data storage technology that has been designed and built from the media up, and specifically for the cloud.

Currently, data storage in the cloud is deployed in multiple tiers based on the type of customer workload that it is best suited for. There are usually at least three or four tiers, and the important workload characteristics are the frequency of datum access (equivalent to the ratio between operations and stored capacity), the latency (delay) permissible for providing data when requested, and the size of the individual data units. Tiers are given names which draw parallels with temperature, as shown below:

TierHot
Access frequencyhigh
Latencysub-millisecond
Current mediaflash
TierStandard and cool
Access frequencymedium
Latencyup to tens of milliseconds
Current mediahard disk drive
TierCold
Access frequencylow
Latencyminutes to hours
Current mediatape

Cloud providers present pricing models based on abstracted marginal costs, which enable customers to engage in informed rational behaviour when selecting the appropriate tier. Due to the extremely large data collections used in modern cloud applications, data storage is a very cost-sensitive application.

The rapid expansion of cloud computing, and the corresponding data volumes, is also driving the demand for storage. To date, for both HDD and tape, this demand has been met by advances in the achieved information storage density of the magnetic material. However, the storage density is now very close to fundamental physical limits imposed by the size of the magnetic domains that can be manipulated and successfully read. In the Cold tier, the limited storage lifetime of tape (3-5 years) is also problematic, as data must be regularly transferred from old media to new to ensure data integrity.

To keep up with the increasing demand for archival storage, it is necessary to investigate new approaches to storage that provide properties that are better than or equal to today’s technology, provide greatly extended data lifetimes and reduce the cost per byte. The read/write throughput should be around 100M-1GByte/s to be competitive with existing technology and support the demand. In the Standard and Cool tiers, HDD-based systems are sometimes limited by the number of operations that can be performed per second, so approaches to storage that can improve this rate will be advantageous.

Project Silica chip

Project Silica

Project Silica, a collaboration with the University of Southampton, is a ground-breaking cloud storage system that uses femtosecond lasers to write data into quartz glass which, due to its durable properties, provides long-lasting, easily-readable storage.

Learn more about Project Silica >

Inside Azure datacenter architecture

Join Mark Russinovich, Azure CTO, to learn how Microsoft’s Azure enables intelligent, modern and innovative applications at scale in the cloud, on-premises and on the edge. (watch from 0:39:32 – 0:41:52).

Project Silica video

In September 2017, Mark Russinovich, CTO of Azure, announced a number of new collaborations with MSR Cambridge including Project Silica. See Mark present the impressive early results (watch from 1:01:13 – 1:03:18).

Network

Research background: Network

Network requirements, both in terms of throughput and latency, are expected to increase significantly in the next few years, driven by new emerging workloads such as large-scale machine learning and resource disaggregation.  These will demand throughputs beyond 100 GB/s and latency below 100 ns.  Hitherto, the industry has relied on exponential scaling of silicon to keep up with increasing traffic demands, but with the expected slowdown of Moore’s Law, novel network technologies with new growth curves that can match the requirements of future workloads must be investigated. Our research aims to re-invent the network that will underpin the cloud in the year 2025 and will meet these key demands via emerging optical technologies. Optics already plays a major role in our technical network infrastructure as the underlying technology used to interconnect racks and data centres across the globe. Our goal is to further improve optics technology by innovating across the whole stack, from new network architectures to new transceivers and optical fibers to deliver higher performance at a lower cost as well as extending its use to new areas such as optical switches and board interconnects to take advantage of their ultra-low and predictable latency, very high bandwidth, and low cost. This effort requires a cross-disciplinary approach combining skills and expertise across the hardware, physics, network, and software domains. If successful, however, this has the potential to revolutionize the cloud infrastructure by providing predictable and uniform high performance (bandwidth and latency) within and across data centres, thus breaking today’s silos (e.g. a single server or a rack) with significant benefits in terms of fault tolerance, resource management and application performance.

Project Iris

This project explores novel designs of regional and wide-area cloud networks, from the ground up.

Project Sirius

This project aims to develop an all-optical, data-center-wide network that is completely flat, in contrast to the hierarchy of electrical switches used today.

Compute

Research background: Compute

With the exponential growth of digital data available today, research in artificial intelligence has sparked the explosion of computational power to drive advanced algorithms that are capable to communicate and perform with humanlike ability and cleverness.

Will the demand for such high-performance but high-power computing disrupt the models of today’s Cloud operators? In that case what is the next technological breakthrough that will allow us to continue enjoying the exponential growth of computational power? Can we use light to achieve faster and more energy efficient information processing for specific information processing in artificial intelligence?

For example, the optics community has long known that lenses apply a spatial Fourier transform to the light waves travelling through them.  Thus, a lens computes a function over its input data.  In a traditional von Neumann computer, such a Fourier transform would be computationally expensive to calculate yet is almost instantaneous with light: the advantages to computing with optics are clear.

Optical computing, in a limited form, is close to reality: as a matter of fact, all-optical matrix multiplication was first demonstrated in the 1970s.  More recently, in the field of machine learning and artificial intelligence, deep neural networks (DNNs) are becoming widely adopted.  These link simple processing steps together into layered networks: could these restricted yet popular AI kernels be computed using light?  Even non-linear functions required for DNN’s such as ReLU and tanh may have optical analogues which would allow DNNs to scale. Computing with light is naturally a parallel processing model, and of course heat dissipation is much less of an issue.  While many hard challenges need to be addressed to get these whiteboard ideas closer to reality, it is also important to think about other functions.

We seek to spur cross-disciplinary research between the optics community and computer scientists at this exciting interface between physics and computing.

Optical Accelerators for AI

In this seedling project, the aim is to develop an optical hardware that can do, in an energy-efficient manner, some restricted yet popular computations specifically for AI. In order to achieve a potential 2-3 order performance improvement in optics as compared to state of the art CMOS-based solutions, radical and key innovations have to be carried out both: i) in the optical space with fJ/bit optoelectronics and high radix optical interconnects to accommodate typical AI requirements; ii) in the machine learning space with the co-design of ad-hoc computational models for AI with the optics. The latter prerequisite involves the development of specialized algorithms for low bit resolution or for new kernels that are more amenable to implementation using optics.

While extremely ambitious, if successful, light-powered computing has the potential to revolutionize the cloud infrastructure by providing next generation high-performance, low latency and low-energy computing platform for specific applications in AI.

Talks and Workshops

Talks

November 2019 | Envisioning Tomorrow Keynote: innovating for an alternative future at Microsoft Ignite 2019 | Mitra Azizirad, Antony Rowstron (at 22:10), Asta Roseway, Mark Hammond, Sara-Jane Dunn

November 2019 | Vision Keynote Highlights at Microsoft Ignite 2019 | Satya Nadella

September 2019 | Transnet Conference Optical Networks-the next 25 years, on “Optics for the Cloud” at University College London |Benn Thomsen

September 2019 | “Optics for the Cloud: Challenges and Opportunities in the next-generation cloud infrastructure” at DTU Fotonik, Department of Photonics Engineering, Copenhagen |Fotini Karinou

June 2019 | “Optics for the Cloud – A New Approach To Data Centre Technology” at EPSRC CDT in Applied Photonics, Edinburgh |Tom Empson

June 2019 | “Microsoft Optics for the Cloud – A New Approach to Data Centre Technology” at PASC19 |Scarlet Schwiderski-Grosche

May 2019 | “Inside Azure datacenter architecture” at Microsoft Build 2019 |Mark Russinovich, Azure CTO

September 2018 | “Challenges and opportunities for photonics in the cloud” at University of Southampton, Future Photonics Hub Industry Day |Benn Thomsen

July 2018, August 2018, and September 2018 | “Optics for the Cloud: Opportunities and Challenges” at OSA Advanced Photonics Congress, Microsoft Faculty Summit, and ECOC |Hitesh Ballani

July 2018 | “Glass: A New Media for a New Era?” at Usenix HotStorage |Ioan Stefanovici

July 2018 | “Rethinking Data Storage for the Zettabyte Cloud Era: The Journey from Metal to Glass” at OSA Advanced Photonics Congress |Ant Rowstron

June 2018 | “How would you store a Zetta-Byte of Cold Data?” at vETC |Austin Donnelly

May 2018 | “Optics for the Cloud” at Downing CollegeAriel Gomez Diaz

January 2018 and March 2018 | “Bridging the Last Mile for Optical Switching in Data Centers” at Photonics West and OFC |Paolo Costa

June 2017 | “Optics for the cloud: Opportunities for Light” at UCL Optics Summit |Benn Thomsen

Optics for the Cloud Research Alliance

Microsoft Research Cambridge has also brought together its collaborations with several university research groups in Europe and the US to create the Optics for the Cloud Research Alliance. This Alliance will investigate and develop optical technologies for the cloud. The Alliance will take a multi-disciplinary approach to tackle big and open challenges, with an aim to enable greater adoption of optical technologies in cloud infrastructure, including cloud storage, network and compute.

The research will span a significant part of the cloud infrastructure ecosystem, ranging from photonics fabrication, via optical system and sub-system design, to co-design and integration with the rest of the data centre stack. The Alliance will provide a vibrant collaborative research environment, involving experts across optical and cloud research disciplines, with an open publications policy. The Alliance funds a cohort of PhD scholars. The three-year PhDs target long-term problems and aim to advance the state of the art through forward-looking fundamental research that will be published in top technical venues. Each PhD student is hosted and supervised by a university faculty member and co-supervised by a Microsoft researcher. The students will also benefit from discussion, insights and feedback from practitioners at Microsoft and, more broadly, through greater interaction with partners in industry and academia.

Goals of the Research Alliance

The primary goals of the alliance are to:

  • Leverage and enable true cross-disciplinary research
  • Support high impact pre-competitive research
  • Aspire to generate disruptive, not sustaining, ideas

The secondary goals are to:

  • Help to expand the ecosystem
  • Communicate the value of optics and photonics research in UK and Europe
  • Send a message that we view our members as world leaders in this space

Founder Members of the Research Alliance

University of Cambridge logo
EPFL logo
Eindhoven University of Technology logo
University of Southampton logo
UCL logo
University of California Santa Barbara logo

Blogs & news

Microsoft’s Project Silica offers robust thousand-year storage

Ars spoke Tuesday with Dr. Ant Rowstron, a principal researcher at Microsoft Research in Cambridge, UK, about an innovative cold storage project called Silica. Silica aims to replace both tape and optical archival discs as the media of choice for large-scale, (very) long duration cold storage. Microsoft Research is partnering with film giant Warner Bros., which is directly interested in reducing costs and increasing reliability in its own cold storage programs.

ars Technica | November 7, 2019

Optics for the cloud: storage in the zettabyte era with Dr. Ant Rowstron and Mark Russinovich

Remember when a hard drive that could hold a terabyte of data was a big deal? Well, we’re now in an era where peta-, exa- and even zetta-bytes are the bytes of the day, and it turns out it’s hard to fit that many zeroes on a hard drive. That’s where Dr. Ant Rowstron, Deputy Lab Director of Microsoft Research Cambridge, and Mark Russinovich, Chief Technical Officer of Azure, come in. Their respective teams are working on paradigm-breaking solutions to give us phenomenal storage power in an itty-bitty living space.

Microsoft Research Podcast | November 6, 2019

Microsoft Highlights Azure Management, Office Improvements, and More at Ignite

Microsoft has the broadest portfolio of software of any company, and at the firm’s annual Ignite conference, it proved it by introducing a raft of new products and services in cloud, development, security and management, and productivity. These include Azure Arc, a multi-cloud management solution; Azure Synapse, a much faster data warehouse and big data platform; Power Automate with robotic process automation capabilities…

PCMag | November 6, 2019


Superman: The Movie Gets Archived on Futuristic Glass Disc by Microsoft

Superman: The Movie can add another accomplishment to its already truly impressive legacy. Microsoft and Warner Bros. have partnered to preserve the iconic superhero movie on a piece of glass roughly the size of a coaster. Yes, a piece of glass. The idea, hopefully, is that this type of storage technology could help to preserve classic movies for decades to come…

MOVIEWEB | November 6, 2019

Microsoft Ignite 2019: Not Your Grandfather's Microsoft

If you thought Microsoft was simply a productivity company, think again. CEO Satya Nadella kicked off this year’s Microsoft Ignite 2019 Conference in Orlando, Fla. with a big sky vision for the company: “How do we power every person and every organization on the planet to achieve more? It starts with one. One employer. One developer. One organization. From one employee to a team collaborating with another team.

CMS WiRE | November 6, 2019

From glass to DNA: technologies that could store our data for billions of years

When Elon Musk sent his Tesla Roadster into space aboard SpaceX’s Falcon Heavy rocket last year, the spacesuit-clad mannequin placed in the driver’s seat had a few essentials packed. A copy of Douglas Adams’ The Hitchhiker’s Guide to the Galaxy was stored in the glovebox, and David Bowie’s Space Oddity was programmed to play repeatedly on the car stereo. Less noticed was a coin-sized, transparent disc secured in a case…

The Telegraph | November 6, 2019


Microsoft: Here's why we just stored the whole Superman movie inside a bit of glass

The world has been hard at work trying to find solutions to store the 100 zettabytes of data that are expected to hit the cloud by 2023. Microsoft and Warner Bros, for their part, have just demonstrated why they think they are ahead of the game – in both protecting past archives, and in storing those to come.

ZDNet | November 5, 2019

Microsoft to send Superman to the Phantom Zone - will archive movie on glass

That small glass disc may not look too impressive or high-tech. But, in fact, it could be a way of storing data for hundreds of years. Storage is often a problem for film and TV studios – normally done on celluloid, that option isn’t exactly foolproof. For one, it’s flammable. And secondly, it doesn’t last – it can very easily warp and decay, meaning archival footage is at risk of being lost forever.

Short List | November 5, 2019

Project Silica proof of concept stores Warner Bros. ‘Superman’ movie on quartz glass

Microsoft and Warner Bros. have collaborated to successfully store and retrieve the entire 1978 iconic “Superman” movie on a piece of glass roughly the size of a drink coaster, 75 by 75 by 2 millimeters thick. It was the first proof of concept test for Project Silica, a Microsoft Research project that uses recent discoveries in ultrafast laser optics and artificial intelligence to store data in quartz glass…

Microsoft Innovation Stories | November 4, 2019


Why Microsoft and Warner Bros. Archived the Original ‘Superman’ Movie on a Futuristic Glass Disc (EXCLUSIVE)

Microsoft began to investigate glass as a storage medium in 2016 in partnership with the University of Southampton Optoelectonics Research Centre. The goal of these efforts, dubbed “Project Silica,” is to find a new storage medium optimized for what industry insiders like to call cold data — the type of data you likely won’t need to access for months, years, or even decade…

Variety | November 4, 2019

Microsoft puts classic Superman on a piece of glass to keep it safe

It’s a bird, it’s a plane, it’s… the latest concept test for Project Silica, Microsoft’s research project that uses ultrafast laser optics and artificial intelligence to store data on quartz glass. In collaboration with Warner Bros. Entertainment, the team was able to store and retrieve the entire 1978 Superman film on a piece of glass that’s about the size of a drink coaster. The silica glass is hard and can withstand being boiled…

CNET | November 4, 2019

Microsoft's 'Project Silica' targets glass as the archival storage medium of the future

In contrast to Microsoft’s new Edge browser, natural-language querying in Excel and other near-term Microsoft innovations, one of the more interesting announcements you won’t see anytime soon is storing data within glass: Project Silica. At Microsoft Ignite, the company showed off how it developed (with help from the Warner Bros. film studio) a three-dimensional recording system to store the movie Superman within a pane of glass…

PCWorld | November 4, 2019


DNA, laser-etched glass, and beyond: A peek into the future of data storage

For most of us, data storage has only gotten easier. Rather than messing around with computer hard drives with limited storage, floppy disks, and rewritable CDs or DVDs, today we just hit “save” on whatever file we’re working on and let it be whisked off to the cloud. It’s a no muss, no fuss approach to data storage that not only means we don’t run out of space, but that we can also access our files wherever we go. What’s not to love?

Digital Trends | November 4, 2019

Microsoft CEO Satya Nadella’s 5 Biggest Statements At Ignite 2019

With its latest wave of product announcements, Microsoft is focused squarely on turning every company into a tech company, CEO Satya Nadella said Monday during the Microsoft Ignite 2019 conference. Ignite, Microsoft’s conference for IT professionals and developers, is being held this week in Orlando.

CRN | November 4, 2019

Next-Gen Cloud Computing

UC Santa Barbara selected as charter member of Microsoft’s Optics for the Cloud Research Alliance. At an ever-increasing rate, people around the world are turning to the cloud — a globally distributed computer network connected by the internet — to store, access and process their data. The cloud enables users to operate beyond the bounds of local computers and hard drives…

UC Santa Barbara | August 27, 2019


Optics for the Cloud Research Alliance establishes collaborative research approach to improving cloud technology

The cloud is an enormous success story. Microsoft Azure, the company’s cloud offering, was launched in 2010. It has grown by more than ten times in the past five years and is now operating in 54 regions around the world. Azure Chief Technology Officer, Mark Russinovich, asserts that “the fundamental disruption that the cloud is creating is based on instant access to compute and infinite storage”…

Microsoft Research Blog | August 7, 2019