To solve safety-critical decision-making problems, Inverse Constrained Reinforcement Learning (ICRL) infers constraints from expert demonstrations and seeks to imitate expert preference by utilizing these constraints. While prior ICRL research commonly overlooks the discrepancy between the training and deploying environments, we demonstrate that such a discrepancy can substantially compromise the reliability of the inferred constraints and thus induce unsafe movements. Motivated by this finding, we propose the Robust Constraint Inference (RCI) problem and propose an Adaptively Robust ICRL algorithm to efficiently solve RCI. Specifically, we model the impact of misspecified dynamics with an opponent policy and learn robust policies to facilitate safe control in a Markov Game. Subsequently, we adjust our constraint model to align these learned policies to expert demonstrations, accommodating both soft and hard optimality in our behavioral models. Empirical results demonstrate the significance of robust constraints and the effectiveness of the proposed robust ICRL algorithm under continuous and discrete domains.