Regularization of Neural Networks using DropConnect
Li WanCIMS - Courant Institute of Mathematical Sciences [New York] (251, Mercier Street, New York, NY 10012 - United States)Matthew D. ZeilerCIMS - Courant Institute of Mathematical Sciences [New York] (251, Mercier Street, New York, NY 10012 - United States)Sixin ZhangInstitut de Recherche en Informatique de ToulouseYann LecunCIMS - Courant Institute of Mathematical Sciences [New York] (251, Mercier Street, New York, NY 10012 - United States)Rob FergusCIMS - Courant Institute of Mathematical Sciences [New York] (251, Mercier Street, New York, NY 10012 - United States)
2013en
ABI
Abstract
We introduce DropConnect, a generalization of Dropout (Hinton et al., 2012), for regular-izing large fully-connected layers within neu-ral networks. When training with Dropout, a randomly selected subset of activations are set to zero within each layer. DropCon-nect instead sets a randomly selected sub-set of weights within the network to zero. Each unit thus receives input from a ran-dom subset of units in the previous layer. We derive a bound on the generalization per-formance of both Dropout and DropCon-nect. We then evaluate DropConnect on a range of datasets, comparing to Dropout, and show state-of-the-art results on several image recognition benchmarks by aggregating mul-tiple DropConnect-trained models. 1.
Identifiers
Citations and references
Cited by 20 references