DataStager: scalable data staging services for petascale applications

• DOI: 10.1007/s10586-010-0135-6
• Publication Date: 2010-06-14T18:53:27Z
• Authors: Hasan Abbasi; Matthew Wolf; Greg Eisenhauer; Scott Klasky; Karsten Schwan; Fang Zheng
• Keywords: 0202 electrical engineering, electronic engineering, information engineering; 02 engineering and technology

Known challenges for petascale machines are that (1) the costs of I/O for high performance applications can be substantial, especially for output tasks like checkpointing, and (2) noise from I/O actions can inject undesirable delays into the runtimes of such codes on individual compute nodes. This paper introduces the flexible 'DataStager' framework for data staging and alternative services within that jointly address (1) and (2). Data staging services moving output data from compute nodes to staging or I/O nodes prior to storage are used to reduce I/O overheads on applications' total processing times, and explicit management of data staging offers reduced perturbation when extracting output data from a petascale machine's compute partition. Experimental evaluations of DataStager on the Cray XT machine at Oak Ridge National Laboratory establish both the necessity of intelligent data staging and the high performance of our approach, using the GTC fusion modeling code and benchmarks running on 1000+ processors.