Universität zu Köln

Segadlo, Kai, Epping, Bastian, van Meegen, Alexander, Dahmen, David, Kraemer, Michael and Helias, Moritz (2022). Unified field theoretical approach to deep and recurrent neuronal networks. J. Stat. Mech.-Theory Exp., 2022 (10). BRISTOL: IOP Publishing Ltd. ISSN 1742-5468

Full text not available from this repository.

Abstract

Understanding capabilities and limitations of different network architectures is of fundamental importance to machine learning. Bayesian inference on Gaussian processes has proven to be a viable approach for studying recurrent and deep networks in the limit of infinite layer width, n -> infinity. Here we present a unified and systematic derivation of the mean-field theory for both architectures that starts from first principles by employing established methods from statistical physics of disordered systems. The theory elucidates that while the mean-field equations are different with regard to their temporal structure, they yet yield identical Gaussian kernels when readouts are taken at a single time point or layer, respectively. Bayesian inference applied to classification then predicts identical performance and capabilities for the two architectures. Numerically, we find that convergence towards the mean-field theory is typically slower for recurrent networks than for deep networks and the convergence speed depends non-trivially on the parameters of the weight prior as well as the depth or number of time steps, respectively. Our method exposes that Gaussian processes are but the lowest order of a systematic expansion in 1/n and we compute next-to-leading-order corrections which turn out to be architecture-specific. The formalism thus paves the way to investigate the fundamental differences between recurrent and deep architectures at finite widths n.

Item Type:	Journal Article
Creators:	Creators Email ORCID ORCID Put Code Segadlo, Kai UNSPECIFIED UNSPECIFIED UNSPECIFIED Epping, Bastian UNSPECIFIED UNSPECIFIED UNSPECIFIED van Meegen, Alexander UNSPECIFIED UNSPECIFIED UNSPECIFIED Dahmen, David UNSPECIFIED UNSPECIFIED UNSPECIFIED Kraemer, Michael UNSPECIFIED UNSPECIFIED UNSPECIFIED Helias, Moritz UNSPECIFIED UNSPECIFIED UNSPECIFIED
URN:	urn:nbn:de:hbz:38-675010
DOI:	10.1088/1742-5468/ac8e57
Journal or Publication Title:	J. Stat. Mech.-Theory Exp.
Volume:	2022
Number:	10
Date:	2022
Publisher:	IOP Publishing Ltd
Place of Publication:	BRISTOL
ISSN:	1742-5468
Language:	English
Faculty:	Unspecified
Divisions:	Unspecified
Subjects:	no entry
Uncontrolled Keywords:	Keywords Language CHAOS; SPACE Multiple languages Mechanics; Physics, Mathematical Multiple languages
URI:	http://kups.ub.uni-koeln.de/id/eprint/67501