And Eat It Too: High Read Performance in Write-Optimized HPC I/O Middleware File Formats
CARNEGIE-MELLON UNIV PITTSBURGH PA PARALLEL DATA LABORATORY
Pagination or Media Count:
The two IO middleware layer examined in this paper are the Adaptable IO System ADIOS, a library-based approach developed at Oak Ridge National Laboratory to provide a high-level IO API that can be used in place of netCDF or HDF5 to do much more aggressive write-behind and efficient reordering of data locations within the file and the Parallel Log-structured Filesystem PLFS, a stackable FUSE filesystem approach developed at Los Alamos National Laboratory that decouples concurrent writes to improve the speed of checkpoints. We observe that not only can write-optimized IO middleware be built to not penalize read speeds, but for important workloads, techniques that improve write performance can, perhaps counterintuitively, improve read speeds over reading from a contiguously organized file format. In the remainder of this paper, we investigate this further through case studies of PLFS and ADIOS on simulation checkpoint restarts.
- Computer Programming and Software