The article presents LoraRetriever, a new framework that adaptively retrieves and composes multiple Low-Rank Adaptations (LoRAs) according to the input prompts in large language models (LLMs). It aims to enhance the capabilities of LLMs by integrating diverse domain-specific LoRAs. The LoraRetriever framework consists of three main components: identifying and retrieving relevant LoRAs, formulating strategies for integrating the retrieved LoRAs, and developing efficient batch inference. The experimental results indicate that LoraRetriever outperforms the baseline models, demonstrating its practical effectiveness and versatility.
Publication date: 16 Feb 2024
Project Page: Not Provided
Paper: https://arxiv.org/pdf/2402.09997
