Abstract: Dynamic convolution learns a linear mixture of $n$ static kernels weighted with their input-dependent attentions, demonstrating superior performance than normal convolution. However, it increases the number of convolutional parameters by $n$ times. This leads to no research progress that can allow researchers to explore the setting $n>100$ (an order of magnitude larger than the typical setting $n<10$) for pushing forward the performance boundary of dynamic convolution while enjoying parameter efficiency. To fill this gap, in this paper, we propose KernelWarehouse, a more general form of dynamic convolution, which redefines the basic concepts of "kernels", "assembling kernels" and "attention function" through the lens of exploiting convolutional parameter dependencies within the same layer and across neighboring layers of a ConvNet. We testify the effectiveness of KernelWarehouse on ImageNet and MS-COCO datasets using various types of ConvNet architectures. Thanks to its flexible design, KernelWarehouse can even reduce the model size of a ConvNet while improving the model accuracy, and it is also applicable to vision transformers. Code is provided for results reproduction.