Empirical Evaluation of Rectified Activations in Convolutional Network

Bing Xu, Naiyan Wang, Tianqi Chen, Mu Li · arXiv.org · 2015

Summary

The experiments suggest that incorporating a non-zero slope for negative part in rectified activation units could consistently improve the results, and are negative on the common belief that sparsity is the key of good performance in ReLU.

Type: preprint
Published: 2015-05-05
Cited by: 3,210
References: 17
Access: Open access
DOI: https://doi.org/10.48550/arxiv.1505.00853
arXiv: https://arxiv.org/abs/1505.00853
OpenAlex: https://openalex.org/W1921523184
Semantic Scholar: https://api.semanticscholar.org/CorpusID:14083350

Keywords

Computer science, Artificial intelligence

References

Transferring Rich Feature Hierarchies for Robust Visual TrackingNaiyan Wang, Siyi Li, Abhinav Gupta — 2015
Rectified Linear Units Improve Restricted Boltzmann MachinesVinod Nair, Geoffrey E. Hinton — 2010
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet ClassificationKaiming He, Xiangyu Zhang, Shaoqing Ren — 2015
Deeply learned face representations are sparse, selective, and robustYi Sun, Xiaogang Wang, Xiaoou Tang — 2014
Dropout: a simple way to prevent neural networks from overfittingNitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky — 2014
Going deeper with convolutionsChristian Szegedy, Wei Liu, Yangqing Jia — 2014
Rich Feature Hierarchies for Accurate Object Detection and Semantic SegmentationRoss Girshick, Jeff Donahue, Trevor Darrell — 2013
Spatial Pyramid Pooling in Deep Convolutional Networks for Visual RecognitionKaiming He, Xiangyu Zhang, Shaoqing Ren — 2014
ImageNet Large Scale Visual Recognition ChallengeOlga Russakovsky, Jia Deng, Hao Su — 2014
ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton — 2012
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftSergey Ioffe, Christian Szegedy — 2015
Network In NetworkMin Lin, Qiang Chen, Shuicheng Yan — 2013
Learning Multiple Layers of Features from Tiny ImagesAlex Krizhevsky — 2024
Rectifier Nonlinearities Improve Neural Network Acoustic ModelsAndrew L. Maas — 2013
Network In NetworkMin Lin, Qiang Chen, Shuicheng Yan — 2013
Learning Multiple Layers of Features from Tiny ImagesA. Krizhevsky — 2009

Cited by

Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in VideoLionel Pigou, Aäron van den Oord, Sander Dieleman — 2015
Environmental sound classification with convolutional neural networksKarol J. Piczak — 2015
A Lightened CNN for Deep Face RepresentationXiang Wu, Ran He, Zhenan Sun — 2015
FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute ClustersForrest Iandola, Matthew W. Moskewicz, Khalid Ashraf — 2015
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)Djork-Arné Clevert, Thomas Unterthiner, Sepp Hochreiter — 2015
GradNets: Dynamic Interpolation Between Neural ArchitecturesDiogo Almeida, Nate Sauder — 2015
The Influence of the Amount of Parameters in Different Layers on the Performance of Deep Learning Models喜斌岳 — 2015
Robust visual track using an ensemble cascade of convolutional neural networksDan Hu, Xingshe Zhou, Xiaohao Yu — 2015
Revise Saturated Activation FunctionsBing Xu, Ruitong Huang, Mu Li — 2016
Lung Pattern Classification for Interstitial Lung Diseases Using a Deep Convolutional Neural NetworkMarios Anthimopoulos, Stergios Christodoulidis, Lukas Ebner — 2016
Landmark perturbation-based data augmentation for unconstrained face recognitionJiang-Jing Lv, Cheng Cheng, Guodong Tian — 2016
Multi-Bias Non-linear Activation in Deep Neural NetworksHongyang Li, Wanli Ouyang, Xiaogang Wang — 2016
Multi-crop Convolutional Neural Networks for lung nodule malignancy suspiciousness classificationWei Shen, Mu Zhou, Feng Yang — 2017
No bad local minima: Data independent training error guarantees for multilayer neural networksDaniel Soudry, Yair Carmon — 2016
Image Analysis and Deep Learning for Applications in MicroscopyO. Ishaq — 2016
Noise robust speech recognition using recent developments in neural networks for computer visionTakuya Yoshioka, Katsunori Ohnishi, Fuming Fang — 2016
Deep Convolutional Neural Networks for Predominant Instrument Recognition in Polyphonic MusicYoonchang Han, Jaehun Kim, Kyogu Lee — 2016
Parametric Exponential Linear Unit for Deep Convolutional Neural NetworksLudovic Trottier, Philippe Giguère, Brahim Chaib-draa — 2016
Systematic evaluation of convolution neural network advances on the ImagenetDmytro Mishkin, Nikolay Sergievskiy, Jiřı́ Matas — 2017
Deep Columnar Convolutional Neural NetworkSomshubra Majumdar, Ishaan Jain — 2016

Empirical Evaluation of Rectified Activations in Convolutional Network

Summary

Keywords

References

Cited by

Related papers