Improving MPI Collective I/O for High Volume Non-contiguous Requests with Intra-node Aggregation

Qiao Kang, Sunwoo Lee, Kaiyuan Hou, Robert Ross, Ankit Agrawal, Alok Choudhary, Wei-keng Liao

IEEE Transactions on Parallel and Distributed Systems, 2020

This paper presents a two-level collective I/O design that adds an intra-node request-aggregation layer to reduce inter-node communication contention in two-phase I/O’s request redistribution, and shows on Theta and Cori using production application I/O patterns and a benchmark that it significantly cuts communication cost and preserves scalability at high process counts.

BibTeX

@article{kang2020improving,
  title={Improving mpi collective i/o for high volume non-contiguous requests with intra-node aggregation},
  author={Kang, Qiao and Lee, Sunwoo and Hou, Kaiyuan and Ross, Robert and Agrawal, Ankit and Choudhary, Alok and Liao, Wei-keng},
  journal={IEEE Transactions on Parallel and Distributed Systems},
  volume={31},
  number={11},
  pages={2682--2695},
  year={2020},
  publisher={IEEE}
}