The Cosmos Big Data Platform at Microsoft: Over a Decade of Progress and a Decade to Look Forward
- Conor Power ,
- Hiren Patel ,
- Alekh Jindal ,
- Jyoti Leeka ,
- Bob Jenkins ,
- Michael Rys ,
- Ed Triou ,
- Dexin Zhu ,
- Lucky Katahanas ,
- Chakrapani Bhat Talapady ,
- Joshua Rowe ,
- Fan Zhang ,
- Rich Draves ,
- Marc Friedman ,
- Ivan Santa Maria Filho ,
- Amrish Kumar
VLDB 2021 |
The twenty-first century has been dominated by the need for large scale data processing, marking the birth of big data platforms such as Cosmos. This paper describes the evolution of the exabyte-scale
Cosmos big data platform at Microsoft; our journey right from scale and reliability all the way to efficiency and usability, and our next steps towards improving security, compliance, and support for
heterogeneous analytics scenarios. We discuss how the evolution of Cosmos parallels the evolution of the big data field, and how the changes in the Cosmos workloads over time parallel the changing requirements of users across industry.