One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation Publication date: 09 June 2024 Topic: Image Classification Paper: https://arxiv.org/pdf/2410.07170v1.pdf GitHub: https://github.com/ml-jku/EVA Description: We propose to enhance LoRA by initializing the new weights in a data-driven manner by computing singular value decomposition on minibatches of activation vectors. Then, we initialize the LoRA matrices with the obtained right-singular vectors and re-distribute ranks among all weight matrices to explain the maximal amount of variance and continue the standard LoRA fine-tuning procedure. This results in our new method Explained Variance Adaptation (EVA). We apply EVA to a variety of fine-tuning tasks ranging from language generation and understanding to image classification and reinforcement learning. EVA exhibits faster convergence than competitors and attains the highest average score across a multitude of tasks per domain.