On the Layer Selection in Small-Scale Deep Networks

Abstract

Deep learning algorithms (in particular Convolutional Neural Networks, or CNNs) have shown their superiority in computer vision tasks and continue to push the state of the art in the most difficult problems of the field. However, deep models frequently lack interpretability. Current research efforts are often focused on increasingly complex and computationally expensive structures. These can be either handcrafted or generated by an algorithm, but in either case the specific choices of individual structural elements are hard to justify. This paper aims to analyze statistical properties of a large sample of small deep networks, where the choice of layer types is randomized. The limited representational power of such models forces them to specialize rather than generalize, resulting in several distinct structural patterns. Observing the empirical performance of structurally diverse weaker models thus allows for some practical insight into the connection between the data and the choice of suitable CNN architectures.