Logan: Logfile Analysis

Logan: Logfile Analysis

Established: October 12, 2015

Understanding Techniques and Tools for More Effective Telemetry and Log Data Analysis.

Increasingly, business processes require data-driven real-time feedback based on large quantities of log data and customer telemetry from multiple sources. The Logan Project takes a broad approach to understanding the specific needs of consumers of telemetry and log data, focusing on giving them better support for extracting the data they need, cleaning it, and creating queries against it.

To understand the needs of real users, we conduct both broad-based email surveys and in-person interviews to determine the major challenges that data scientists and consumers of telemetry data face. We work closely with internal users to understand requirements that tool-based automation need to support.

Based on these findings, we have implemented a proof-of-concept for visually building queries involving sequences of events, phrased as visual regular expressions. Our most recent research focuses on cohort comparison and data cleaning.

We are also exploring programming by example, with a prototype for filtering data streams and forming nested queries. We are just beginning to scratch the surface of ways to allow both programmers and non-programmers to extract, filter, and query telemetry data sets comfortably.

Learn more about event-oriented sequential analysis at the IEEE Vis 2016 Event Event.

People

Publications

Videos

Supporting Materials

Paper Data

These are supporting materials for the Bones of the System, a paper about practices of logging and telemetry at .

Event Sequence Datasets

It can be helpful to use event-driven datasets to test these systems; we have made a sample dataset available to test and compare your own system.