Supporting Data Compression in PnetCDF

Kaiyuan Hou, Qiao Kang, Sunwoo Lee, Ankit Agrawal, Alok Choudhary, Wei-keng Liao

BigData, 2021

This paper presents a variable compression feature for the Parallel NetCDF library that adopts HDF5-style chunking while enabling I/O aggregation across multiple requests, and shows through real-world scientific I/O kernels that handling multiple requests at once significantly improves parallel I/O performance on chunked and compressed data.

BibTeX

@inproceedings{hou2021supporting,
  title={Supporting data compression in PnetCDF},
  author={Hou, Kaiyuan and Kang, Qiao and Lee, Sunwoo and Agrawal, Ankit and Choudhary, Alok and Liao, Wei-keng},
  booktitle={2021 IEEE International Conference on Big Data (Big Data)},
  pages={86--97},
  year={2021},
  organization={IEEE}
}