Skip to main content

Table 1 Some recent attempts to improve different aspects of CNNs

From: Enlarging smaller images before inputting into convolutional neural network: zero-padding vs. interpolation

Network architectureConvolutional layersNetwork in networkLin et al. [30]
Inception and improved Inception modelsSzegedy et al. [46, 47]
Doubly convolutionZhai et al. [60]
Pooling layersLp poolingSermanet et al. [40]
Stochastic poolingZeiler and Fergus [57]
Fractional max poolingGraham [11]
Mixed poolingYu et al. [55]
Gated poolingLee et al. [28]
Tree poolingLee et al. [28]
Spectral poolingRippel et al. [38]
Spatial pyramid poolingGrauman and Darrell [12], He et al. [18], Lazebnik et al. [27], Yang et al. [54]
Multiscale orderless poolingGong et al. [9]
Transformation invariant poolingLaptev et al. [26]
Nonlinear activation functionsRectified linear unit (ReLU) Nair and Hinton [35]
Leaky rectified linear unit (LReLU) Maas et al. [32]
Parametric rectified linear unit (PReLU) He et al. [19]
Adaptive piecewise linear (APL) activation functions Agostinelli et al. [1]
Randomized rectified linear unit (RReLU) National Data Science Bowl |Kaggle [36]
Exponential linear unit (ELU) Clevert et al. [4]
S-shaped rectified linear unit (SReLU) Jin et al. [23]
Maxout activations Goodfellow et al. [10]
Probout activations Springenberg and Riedmiller [42]
Loss functionSoftmax loss Liu et al. [31]
Contrastive and triplet losses Liu et al. [31]
Large margin loss Liu et al. [31]
L2-SVM loss Collobert and Bengio [5], Nagi et al. [34]
Regularization mechanismsDropout Hinton et al. [21], Srivastava et al. [43]
Fast dropout Wang and Manning [51]
Adaptive dropout Ba and Frey [2]
Multinomial dropout and evolutional dropout Li et al. [29]
Spatial dropout Tompson et al. [48]
Nested dropout Rippel et al. [37]
Max pooling dropout Wu and Gu [53]
DropConnect Wan et al. [49]
Optimization techniquesEnhanced initialization schemesXavier initializationGlorot and Bengio [8]
Theoretically derived adaptable initializationHe et al. [19]
Standard fixed initializationKrizhevsky et al. [25]
Layer sequential unit variance initializationMishkin and Matas [33]
Skip connectionsHighway networksSrivastava et al. [44, 45]
Residual networksHe et al. [17]
Improved residual networksHe et al. [20]
Densely connected convolutional networksHuang et al. [22]