The Quantcast File System
- Michael Ovsiannikov ,
- Silvius Rus ,
- Damian Reeves ,
- Paul Sutter ,
- Sriram Rao ,
- Jim Kelly
39th International Conference on Very Large Data Bases (VLDB'13) |
The Quantcast File System (QFS) is an efficient alternative
to the Hadoop Distributed File System (HDFS). QFS is
written in C++, is plugin compatible with Hadoop MapReduce,
and offers several efficiency improvements relative to
HDFS: 50% disk space savings through erasure coding instead
of replication, a resulting doubling of write throughput,
a faster name node, support for faster sorting and logging
through a concurrent append feature, a native command
line client much faster than hadoop fs, and global
feedback-directed I/O device management. As QFS works
out of the box with Hadoop, migrating data from HDFS
to QFS involves simply executing hadoop distcp. QFS is
being developed fully open source and is available under an
Apache license from https://github.com/quantcast/qfs.
Multi-petabyte QFS instances have been in heavy production
use since 2011.