Selective Pre-training for Private Fine-tuning - Yu, Da; Gopi, Sivakanth; Kulkarni, Janardhan (Jana); Lin, Zinan; Naik, Saurabh; Religa, Tomasz Lukasz; Yin, Jian; Zhang, Huishuai (TMLR, 2024)
Suppose we want to train text prediction models in email clients or word processors. The models must preserve the privacy of user data and adhere to a specific fixed size to meet memory and inference time requirements. We introduce a generic framework to solve this problem. Specifically, we are given a public dataset and a private dataset corresponding to a downstream task . How should we pre-train a fixed-size model on and fine-tune it on such that performance of with respect to is maximized and satisfies differential privacy with respect to ? We show that pre-training on a subset of dataset that brings the public distribution closer to the private distribution is a crucial ingredient to maximize the transfer learning abilities of after pre-training, especially in the regimes where model sizes are relatively small. Besides performance improvements, our framework also shows that with careful pre-training and private fine-tuning, smaller models can match the performance of much larger models, highlighting the promise of differentially private training as a tool for model compression and efficiency.