Analysis of one-hidden-layer Neural Networks via the Resolvent Method - Archive ouverte HAL Access content directly
Journal Articles Advances in Neural Information Processing Systems Year : 2021

Analysis of one-hidden-layer Neural Networks via the Resolvent Method

(1) , (2)
1
2

Abstract

In this work, we investigate the asymptotic spectral density of the random feature matrix M = Y Y * with Y = f (W X) generated by a single-hidden-layer neural network, where W and X are random rectangular matrices with i.i.d. centred entries and f is a non-linear smooth function which is applied entry-wise. We prove that the Stieltjes transform of the limiting spectral distribution approximately satisfies a quartic self-consistent equation, which is exactly the equation obtained by Pennington and Worah [22] and Benigni and Péché [6] with the moment method. We extend the previous results to the case of additive bias Y = f (W X + B) with B being an independent rank-one Gaussian random matrix, closer modelling the neural network infrastructures encountered in practice. Our key finding is that in the case of additive bias it is impossible to choose an activation function preserving the layer-to-layer singular value distribution, in sharp contrast to the bias-free case where a simple integral constraint is sufficient to achieve isospectrality. To obtain the asymptotics for the empirical spectral density we follow the resolvent method from random matrix theory via the cumulant expansion. We find that this approach is more robust and less combinatorial than the moment method and expect that it will apply also for models where the combinatorics of the former become intractable. The resolvent method has been widely employed, but compared to previous works, it is applied here to non-linear random matrices.
Fichier principal
Vignette du fichier
neurips_2021_paper.pdf (449.39 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

ensl-03475949 , version 1 (11-12-2021)

Identifiers

  • HAL Id : ensl-03475949 , version 1

Cite

Vanessa Piccolo, Dominik Schröder. Analysis of one-hidden-layer Neural Networks via the Resolvent Method. Advances in Neural Information Processing Systems, 2021. ⟨ensl-03475949⟩
20 View
15 Download

Share

Gmail Facebook Twitter LinkedIn More