Skip to Main content Skip to Navigation
Journal articles

Analysis of one-hidden-layer Neural Networks via the Resolvent Method

Abstract : In this work, we investigate the asymptotic spectral density of the random feature matrix M = Y Y * with Y = f (W X) generated by a single-hidden-layer neural network, where W and X are random rectangular matrices with i.i.d. centred entries and f is a non-linear smooth function which is applied entry-wise. We prove that the Stieltjes transform of the limiting spectral distribution approximately satisfies a quartic self-consistent equation, which is exactly the equation obtained by Pennington and Worah [22] and Benigni and Péché [6] with the moment method. We extend the previous results to the case of additive bias Y = f (W X + B) with B being an independent rank-one Gaussian random matrix, closer modelling the neural network infrastructures encountered in practice. Our key finding is that in the case of additive bias it is impossible to choose an activation function preserving the layer-to-layer singular value distribution, in sharp contrast to the bias-free case where a simple integral constraint is sufficient to achieve isospectrality. To obtain the asymptotics for the empirical spectral density we follow the resolvent method from random matrix theory via the cumulant expansion. We find that this approach is more robust and less combinatorial than the moment method and expect that it will apply also for models where the combinatorics of the former become intractable. The resolvent method has been widely employed, but compared to previous works, it is applied here to non-linear random matrices.
Document type :
Journal articles
Complete list of metadata

https://hal-ens-lyon.archives-ouvertes.fr/ensl-03475949
Contributor : Vanessa Piccolo Connect in order to contact the contributor
Submitted on : Saturday, December 11, 2021 - 7:18:23 PM
Last modification on : Tuesday, January 4, 2022 - 5:51:53 AM

File

neurips_2021_paper.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : ensl-03475949, version 1

Collections

Citation

Vanessa Piccolo, Dominik Schröder. Analysis of one-hidden-layer Neural Networks via the Resolvent Method. Advances in Neural Information Processing Systems, 2021. ⟨ensl-03475949⟩

Share

Metrics

Les métriques sont temporairement indisponibles