kNNDM: k-fold Nearest Neighbour Distance Matching Cross-Validation for map accuracy estimation

• DOI: 10.5194/egusphere-2023-1308
• Publication Date: 2023-07-05T05:54:13Z
• Authors: Jan Linnenbrink; Carles Milà; Marvin Ludwig; Hanna Meyer
• Keywords: Loo; Cross-validation; Nearest neighbour

Abstract. Random and spatial Cross-Validation (CV) methods are commonly used to evaluate machine learning-based prediction models, the obtained performance values often interpreted as map accuracy estimates. However, appropriateness of such approaches is currently subject controversy. For common case where no probability sample for validation purposes available, in Milà et al. (2022) we proposed Nearest Neighbour Distance Matching (NNDM) Leave-One-Out (LOO) CV method. This method produces a distribution geographical Distances (NND) between test train locations during that matches NND training locations. Hence, it creates predictive conditions comparable what required when predicting defined area. Although NNDM LOO produced largely reliable estimates our analysis, LOO-based method, cannot be applied large datasets found many studies. Here, propose novel k-fold strategy estimation inspired by concepts CV: (kNNDM) CV. The kNNDM algorithm tries find configuration Empirical Cumulative Distribution Function (ECDF) matched ECDF We tested simulation study with different sampling distributions compared other including performed similarly reasonably across patterns strong reductions computation time sizes. Furthermore, positive linear association quality match two ECDFs reliability provided advantages original while bypassing its size limitations.