Achieving Small-Batch Accuracy with Large-Batch Scalability via Hessian-Aware Learning Rate Adjustment

Sunwoo Lee, Chaoyang He, Salman Avestimehr

Neural Networks, 2023

This work proposes a Hessian-aware learning rate adjustment framework that steers large-batch data-parallel neural network training toward flatter minima, closing most of the generalization gap versus small-batch training (e.g., 92.31% vs. 92.83% accuracy on CIFAR-10/ResNet20 at 16,384 vs. 128 batch size) at negligible extra cost.

BibTeX

@article{lee2023achieving,
  title={Achieving small-batch accuracy with large-batch scalability via Hessian-aware learning rate adjustment},
  author={Lee, Sunwoo and He, Chaoyang and Avestimehr, Salman},
  journal={Neural Networks},
  volume={158},
  pages={1--14},
  year={2023},
  publisher={Elsevier}
}