Interacting with storage – be it main memory, local storage, or cloud storage – is one of the hardest challenges faced by application and platform developers. We have a “kitchen sink” of solutions available today, each optimized for a specific workload. The SimpleStore project aims at simplifying the use of storage for modern cloud, edge, serverless, and big data applications. Our recent presentation at HPTS overviews the project. We tackle the problem under two broad umbrellas:
- SimpleStore-Tx aims to simplify individual object access, update, and read-modify-write, for embedded edge and cloud applications, streaming, and auto-scaling serverless and actor-oriented compute frameworks. Towards this vision, FASTER aims to provide embedded key-value, cache, and log abstractions over tiered storage, at very high performance. We are currently working on making it easier to use our log (FasterLog) and index/cache (FasterKV) in (1) distributed client-server and (2) auto-scaling embdedded serverless and actor environments.
- SimpleStore-OLAP aims to support storage for rich analytical data sources with fixed or flexible schemas (e.g., JSON). It supports fast time-based ingestion of data in and allows users to impose a complex workload on storage, ideally with no a priori index or data layout selection necessary. As a first step, we have build and open-sourced a fast ingestion, storage, and retrieval system called FishStore. We are currently developing techniques to better leverage workload information to optimize data layouts, as well as generalize FishStore to arbitrary types of queries over rapidly ingested logs.