Towards a Dynamic Composability Approach for Using Heterogeneous Systems in Remote Sensing

• DOI: 10.5281/zenodo.7159278
• Publication Date: 2022-10-01
• Authors: Altintas, Ilkay; Perez, Ismael; Mishin, Dmitry; Trouillaud, Adrien; Irving, Christopher; Graham, John; Tatineni, Mahidhar; DeFanti, Thomas; Strande, Shawn; Smarr, Larry; Norman, Michael L.
• Keywords: FOS: Computer and information sciences; Composable Systems; Paper-presentation; Computer Science - Artificial Intelligence; Multi-Cluster Federation; 0211 other engineering and technologies; 02 engineering and technology; 01 natural sciences; Remote Sensing; Artificial Intelligence (cs.AI); Computer Science - Distributed, Parallel, and Cluster Computing; Artificial Intelligence; 0103 physical sciences; Distributed, Parallel, and Cluster Computing (cs.DC); Kubernetes

None<br/>Influenced by the advances in data and computing, the scientific practice increasingly involves machine learning and artificial intelligence driven methods which requires specialized capabilities at the system-, science- and service-level in addition to the conventional large-capacity supercomputing approaches. The latest distributed architectures built around the composability of data-centric applications led to the emergence of a new ecosystem for container coordination and integration. However, there is still a divide between the application development pipelines of existing supercomputing environments, and these new dynamic environments that disaggregate fluid resource pools through accessible, portable and re-programmable interfaces. New approaches for dynamic composability of heterogeneous systems are needed to further advance the data-driven scientific practice for the purpose of more efficient computing and usable tools for specific scientific domains. In this paper, we present a novel approach for using composable systems in the intersection between scientific computing, artificial intelligence (AI), and remote sensing domain. We describe the architecture of a first working example of a composable infrastructure that federates Expanse, an NSF-funded supercomputer, with Nautilus, a Kubernetes- based GPU geo-distributed cluster. We also summarize a case study in wildfire modeling, that demonstrates the application of this new infrastructure in scientific workflows: a composed system that bridges the insights from edge sensing, AI and computing capabilities with a physics-driven simulation.<br/>