The multiplicity of adaptations of a base language model (LM) via parameter-efficient adapters calls for studying whether reusing such trained adapters can improve performance for new tasks or new inputs. In this paper, we study how to best build a library of adapters given multi-task data and study techniques for zero-shot inference and effective task adaptation through routing in such library. We benchmark existing approaches to build this library and introduce a clustering-based method MBC (“model-based clustering”) which groups tasks based on the similarity of their adapter weights, indirectly optimizing for transfer across the multi-task dataset. To re-use this library of adapters, we present a novel zero-shot routing mechanism, Arrow, which enables dynamic selection of the most relevant adapters for new inputs without the need for retraining. We make steps towards creating modular, adaptable LMs that can outperform traditional full finetuning, paving the way for efficient and flexible utilization of LMs across a wide array of tasks. We will release code, models and dataset upon acceptance.