Few-shot action recognition aims at quickly adapt- ing a pre-trained model to the novel data with a distribution shift using only a limited number of samples. Key challenges include how to identity and leverage the transferable knowledge learned by the pre-trained model. Our central hypoth- esis is that temporal invariance in the dynamic system between latent variables lends itself to transferability (domain-invariance). We therefore propose DITeD, or Domain-Invariant Temporal Dynamics for knowledge transfer. To detect the temporal invariance part, we propose a genera- tive framework with two-stage training strategy during pre-training. Specifically, we explicitly model invariant dynamics including temporal dy- namic generation and transitions, and the vari- ant visual and domain encoders. Then we pre- train the model with the self-supervised signals to learn the representation. After that, we fix the whole representation model and tune the classifier. During adaptation, we fix the transferable invari- ant dynamics and update the perception encoders. The efficacy of our approach is revealed by the superior accuracy of DITeD over leading alterna- tives across standard few-shot action recognition datasets. Moreover, we validate that the learned perception and temporal invariance modules pos- sess transferable qualities.