We propose a Bayesian framework for fine-tuning large diffusion models with a novel network structure called Bayesian Power Steering (BPS). We clarify the meaning behind adaptation from a large probability space to a small probability space and explore the task of fine-tuning pre-trained models using learnable modules from a Bayesian perspective. BPS extracts task-specific knowledge from a pre-trained model's learned prior distribution. It efficiently leverages large diffusion models, differentially intervening different hidden features with a head-heavy and foot-light configuration. By incorporating well-designed conditions for the target domain, BPS achieves high generation quality across various tasks, e.g., layout-to-image and artistic drawing, etc., even with limited amount of data. Our experiments demonstrate that BPS outperforms several existing advanced models and achieve a new state-of-the-art benchmark with a FID score of 10.49 using the sketch condition in the COCO17 dataset.