DIFF: A Relational Interface for Large-Scale Data Explanation

A range of explanation engines assist data analysts by performing feature selection over increasingly high-volume and high-dimensional data, grouping and highlighting commonalities among data points. While useful in diverse tasks such as user behavior analytics, operational event processing, and root cause analysis, today’s explanation engines are designed as standalone data processing tools that do not interoperate with traditional, SQL-based analytics workﬂows; this limits the applicability and extensibility of these engines. In response, we propose the DIFF operator, a relational aggregation operator that uniﬁes the core functionality of these engines with declarative relational query processing. We implement both single-node and distributed versions of the DIFF operator in MB SQL, an extension of MacroBase, and demonstrate how DIFF can provide the same semantics as existing explanation engines while capturing a broad set of production use cases in industry, including at Microsoft and Facebook. Additionally, we illustrate how this declarative approach to data explanation enables new logical and physical query optimizations. We evaluate these optimizations on several real-world production applications, and ﬁnd that DIFF in MB SQL can outperform state-of-the-art engines by up to an order of magnitude.

This is joint work with Peter Kraft, Sahaana Suri, Edward Gan, Eric Xu, Atul Shenoy†, Asvin Ananthanarayan†, John Sheu†, Erik Meijer‡, Xi Wu§, Jeff Naughton§, Peter Bailis, Matei Zaharia at Stanford, Facebook (‡), Google (§), Microsoft (†).

[SLIDES]

Speaker Details

Firas Abuzaid is a 4th-year PhD student from Stanford, working with Profs. Peter Bailis and Matei Zaharia. His research primarily focuses on the intersection of systems and machine learning: how to take high-level machine learning tasks and build software systems to improve the efficiency and usability of these tasks without sacrificing their accuracy; his work has appeared at NIPS, VLDB, and ICDE. This summer he’s working with Srikanth Kandula on optimizing network traffic across Microsoft’s datacenter WANs. When he’s not working, you can find him playing basketball, hiking, going for a run, or solving puzzles.