Weight Concentration Regularization for Improving Pruning Robustness Under High Sparsity

Vincent-Daniel Yun, Junhyuk Jo, Sunwoo Lee

ICML Workshop, 2026

arXiv

This work proposes a Weight Concentration Regularizer (WCR) that, unlike prior uniform-shrinkage or scale-invariant sparsity regularizers, concentrates weight energy onto a small subset of informative parameters during training so that one-shot magnitude pruning removes mainly functionally negligible weights, yielding consistent pruning-robustness gains across LLM fine-tuning, image classification, and medical segmentation while remaining compatible with existing pruning-robust optimizers.

BibTeX

@article{yun2025weight,
  title={Weight Concentration Regularization for Improving Pruning Robustness Under High Sparsity},
  author={Yun, Vincent-Daniel and Jo, Junhyuk and Lee, Sunwoo},
  journal={arXiv preprint arXiv:2511.14282},
  year={2025}
}