TBD: Benchmarking and Analyzing Deep Neural Network Training
International Symposium on Workload Characterization (IISWC 2018) |
The recent popularity of deep neural networks (DNNs) has generated considerable research interest in performing DNN-related computation efﬁciently. However, the primary focus is usually very narrow and limited to (i) inference – i.e. how to efﬁciently execute already trained models and (ii) image classiﬁcation networks as the primary benchmark for evaluation. Our primary goal in this work is to break this myopic view by (i) proposing a new benchmark suite for DNN training, called TBD1, which comprises a representative set of eight DNN models and covers six major machine learning applications: image classiﬁcation, machine translation, speech recognition, object detection, adversarial networks, reinforcement learning, and (ii) performing an extensive performance analysis of these models on three major deep learning frameworks (TensorFlow, MXNet, CNTK) across different hardware conﬁgurations (single-GPU, multi-GPU, and multi-machine). We present a new toolchain for performance analysis for these models that combines the targeted usage of existing performance analysis tools, careful selection of performance metrics, and methodologies to analyze the results. We also build a new set of tools for memory proﬁling in three major frameworks. These tools can shed light on precisely how much memory is consumed by different data structures (weights, activations, gradients, workspace) in DNN training. Using our tools and methodologies, we make several important observations and recommendations on where future DNN training research and optimization should be focused.