POLARIS: the distributed SQL engine in azure synapse
- Josep Aguilar-Saborit ,
- Raghu Ramakrishnan ,
- Krish Srinivasan ,
- Kevin Bocksrocker ,
- Ioannis Alagiannis ,
- Mahadevan Sankara ,
- Moe Shafiei ,
- Jose Blakeley ,
- Girish Dasarathy ,
- Sumeet Dash ,
- Lazar Davidovic ,
- Maja Damjanic ,
- Slobodan Djunic ,
- Nemanja Djurkic ,
- Charles Feddersen ,
- Cesar Galindo-Legaria ,
- Alan Halverson ,
- Milana Kovacevic ,
- Nikola Kicovic ,
- Goran Lukic ,
- Djordje Maksimovic ,
- Ana Manic ,
- Nikola Markovic ,
- Bosko Mihic ,
- Ugljesa Milic ,
- Marko Milojevic ,
- Tapas Nayak ,
- Milan Potocnik ,
- Milos Radic ,
- Bozidar Radivojevic ,
- Srikumar Rangarajan ,
- Milan Ruzic ,
- Milan Simic ,
- Marko Sosic ,
- Igor Stanko ,
- Maja Stikic ,
- Sasa Stanojkov ,
- Vukasin Stefanovic ,
- Milos Sukovic ,
- Aleksandar Tomic ,
- Dragan Tomic ,
- Steve Toscano ,
- Djordje Trifunovic ,
- Veljko Vasic ,
- Tomer Verona ,
- Aleksandar Vujic ,
- Nikola Vujic ,
- Marko Vukovic ,
- Marko Zivanovic
2020 VLDB: Very Large Data Bases |
Published by VLDB Endowment
In this paper, we describe the Polaris distributed SQL query engine in Azure Synapse. It is the result of a multi-year project to re-architect the query processing framework in the SQL DW parallel data warehouse service, and addresses two main goals: (i) converge data warehousing and big data workloads, and (ii) separate compute and state for cloud-native execution.
From a customer perspective, these goals translate into many useful features, including the ability to resize live workloads, deliver predictable performance at scale, and to efficiently handle both relational and unstructured data. Achieving these goals required many innovations, including a novel “cell” data abstraction, and flexible, fine-grained, task monitoring and scheduling capable of handling partial query restarts and PB-scale execution. Most importantly, while we develop a completely new scale-out framework, it is fully compatible with T-SQL and leverages decades of investment in the SQL Server single-node runtime and query optimizer. The scalability of the system is highlighted by a 1PB scale run of all 22 TPC-H queries; to our knowledge, this is the first reported run with scale larger than 100TB.