Parallel Deep Convolutional Neural Network Training by Exploiting the Overlapping of Computation and Communication

Sunwoo Lee, Dipendra Jha, Ankit Agrawal, Alok Choudhary, Wei-keng Liao

HiPC, 2017

This paper presents a parallel CNN training strategy that uses a per-node communication thread to overlap gradient communication with backpropagation computation layer-by-layer, and shows on VGG-A/ImageNet that this achieves 62.97x and 77.97x speedups on 128 compute nodes with mini-batch sizes of 256 and 512, respectively.

BibTeX

@inproceedings{lee2017parallel,
  title={Parallel deep convolutional neural network training by exploiting the overlapping of computation and communication},
  author={Lee, Sunwoo and Jha, Dipendra and Agrawal, Ankit and Choudhary, Alok and Liao, Wei-keng},
  booktitle={2017 IEEE 24th international conference on high performance computing (HiPC)},
  pages={183--192},
  year={2017},
  organization={IEEE}
}