In contrast to close-set scenarios that restore images from a predefined set of degradations, open-set image restoration aims to handle the unknown degradations that were unforeseen during the pretraining phase, which is less-touched as far as we know. In this work, we explicitly study this challenging problem and reveal its essence, \textit{i.e.}, the unidentified distribution shifts between test and training data. In recent, test-time adaptation emerges as a fundamental method to address this inherent disparities. Inspired by this, we propose a test-time degradation adaption framework for open-set image restoration, which involves three components, \textit{i.e.}, i) a pre-trained and degradation-agnostic diffusion model to generate clean images, ii) a test-time degradation adapter adapts the unknown degradations based on the input image during the testing phase, and iii) the adapter-guided image restoration guides the model through the adapter to produce the corresponding clean image. Through experiments on multiple degradations absent from the training data, we show that our method achieves comparable even better performance than those task-specific methods.