Overcoming Resource Constraints in Federated Learning: Large Models Can Be Trained with only Weak Clients

Yue Niu, Saurav Prakash, Souvik Kundu, Sunwoo Lee, Salman Avestimehr

Transactions on Machine Learning Research, 2023

arXiv

This work proposes PriSM, a federated learning method that assigns each resource-constrained client a small, importance-sampled low-rank sub-model of a full server model—derived via principal kernel analysis and low-rank intermediate representations, without any client training the full model or sharing intermediate information—achieving accuracy close to full-model training while improving over prior heterogeneous-client approaches across CNNs, LSTMs, and Transformers.

BibTeX

@article{niu2023overcoming,
  title={Overcoming resource constraints in federated learning: Large models can be trained with only weak clients},
  author={Niu, Yue and Prakash, Saurav and Kundu, Souvik and Lee, Sunwoo and Avestimehr, Salman},
  journal={Transactions on Machine Learning Research},
  year={2023}
}