Accession Number:

ADA512866

Title:

And Eat It Too: High Read Performance in Write-Optimized HPC I/O Middleware File Formats

Descriptive Note:

Conference paper

Corporate Author:

CARNEGIE-MELLON UNIV PITTSBURGH PA PARALLEL DATA LABORATORY

Report Date:

2009-11-15

Pagination or Media Count:

6.0

Abstract:

The two IO middleware layer examined in this paper are the Adaptable IO System ADIOS, a library-based approach developed at Oak Ridge National Laboratory to provide a high-level IO API that can be used in place of netCDF or HDF5 to do much more aggressive write-behind and efficient reordering of data locations within the file and the Parallel Log-structured Filesystem PLFS, a stackable FUSE filesystem approach developed at Los Alamos National Laboratory that decouples concurrent writes to improve the speed of checkpoints. We observe that not only can write-optimized IO middleware be built to not penalize read speeds, but for important workloads, techniques that improve write performance can, perhaps counterintuitively, improve read speeds over reading from a contiguously organized file format. In the remainder of this paper, we investigate this further through case studies of PLFS and ADIOS on simulation checkpoint restarts.

Subject Categories:

  • Computer Programming and Software

Distribution Statement:

APPROVED FOR PUBLIC RELEASE