trasolix.blogg.se

Our results thus indicate that in general “no spurious local minima” is a property limited to deep linear networks, and insights obtained from linear networks may not be robust. We prove that even for one-hidden-layer networks with “slightest” nonlinearity, the empirical risks have spurious local minima in most cases. We investigate the loss surface of neural networks. These observations identify a couple of potential causes for problems in the optimization of neural networks such as no guaranteed convergence, explosion of parameters, and very slow convergence.ħth International Conference on Learning Representations, ICLR 2019. In other words, if $f_1,f_2$ are two functions realized by neural networks that are very close in the sense that $\|f_1−f_2∥_ \leq \epsilon$, it is usually not possible to find weights $w_1,w_2$ close together such that each $f_i$ is realized by a neural network with weights $w_i$. Finally, the function that maps a family of weights to an associated network is not inverse stable, for every practically used activation function. Moreover, the set is not closed with respect to $L^p$-norms, $0latter result also holds without the assumption of local affine linearity and without any conditions on the hidden layers.

It is further shown that the appearance of the spurious local minima in the considered training problems is a direct consequence of the universal approximation theorem and that the underlying mechanisms also cause, e.g., Lp-best approximation problems to be ill-posed in the sense of Hadamard for all networks that do not have a dense image. In contrast to previous works, our analysis covers all sampling and parameterization regimes, general differentiable loss functions, arbitrary continuous nonpolynomial activation functions, and both the finite- and infinite-dimensional setting. It is shown that such problems possess a continuum of spurious (i.e., not globally optimal) local minima for all target functions that are not affine. We study the loss landscape of training problems for deep artificial neural networks with a one-dimensional real output whose activation functions contain an affine segment and whose hidden layers have width at least two.