Degrees of freedom for nonlinear least squares estimation

• DOI: 10.48550/arxiv.1402.2997
• Publication Date: 2014-01-01
• Authors: Hansen, Niels Ric...; Sokol, Alexander
• Keywords: FOS: Mathematics; Mathematics - Statistics Theory; Statistics Theory (math.ST); 0101 mathematics; 01 natural sciences

We give a general result on the effective degrees of freedom for nonlinear least squares estimation, which relates the degrees of freedom to the divergence of the estimator. We show that in a general framework, the divergence of the least squares estimator is a well defined but potentially negatively biased estimate of the degrees of freedom, and we give an exact representation of the bias. This implies that if we use the divergence as a plug-in estimate of the degrees of freedom in Stein's unbiased risk estimate (SURE), we generally underestimate the true risk. Our result applies, for instance, to model searching problems, yielding a finite sample characterization of how much the search contributes to the degrees of freedom. Motivated by the problem of fitting ODE models in systems biology, the general results are illustrated by the estimation of systems of linear ODEs. In this example the divergence turns out to be a useful estimate of degrees of freedom for $\ell_1$-constrained models.<br/>26 pages, 6 figures, 10 pages supplement<br/>