Lessons learned from moving earth system grid data sets over a 20 Gbps wide-area network

• DOI: 10.1145/1851476.1851519
• Publication Date: 2010-09-22T12:04:33Z
• Authors: Raj Kettimuthu; Alex Sim; Dan Gunter; Bill Allcock; Peer-Timo Bremer; John Bresnahan; Andrew Cherry; Lisa Childers; Eli Dart; Ian Foster; Kevin Harms; Jason Hick; Jason Lee; Michael Link; Jeff Long; Keith Miller; Vijaya Natarajan; Valerio Pascucci; Ken Raffenetti; David Ressman; Dean Williams; Loren Wilson; Linda Winkler
• Keywords: Petascale computing; Petabyte; Earth system science; Implementation; National laboratory; Exascale computing

In preparation for the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report, climate community will run Coupled Model Intercomparison Project phase 5 (CMIP-5) experiments, which are designed to answer crucial questions about future regional change and results of carbon feedback different mitigation scenarios. The CMIP-5 experiments generate petabytes data that must be replicated seamlessly, reliably, quickly hundreds research teams around globe. As an end-to-end test technologies used perform this task, a multi-disciplinary team researchers moved small portion (10 TB) multimodel Project, Phase 3 set in IPCC Fourth Report from three sources---the Argonne Leadership Computing Facility (ALCF), Lawrence Livermore National Laboratory (LLNL) Energy Research Scientific Center (NERSC)---to 2009 Supercomputing conference (SC09) show floor Portland, Oregon, over circuits provided by DOE's ESnet. achieved sustained rate 15 Gb/s 20 network. More important, effort critical how deploy, tune, monitor middleware replicate upcoming petascale datasets. We report obstacles overcome key lessons learned successful bandwidth challenge effort.