In non-convex sampling problems, replica exchange stochastic gradient Langevin dynamics (reSGLD) is one of the main workhorses. However, its effectiveness is limited by the production of unnatural samples, a result of over-exploration in high-temperature chains. To address this, we propose reflected reSGLD (r2SGLD): an algorithm tailored for constrained non-convex exploration. The analysis not only bridges the theoretical gap in both continuous and discrete-time dynamics but also uncovers a crucial finding: a smaller diameter accelerates the mixing rate, which is characterized as quadratic. Empirically, we test its performance through extensive experiments, including identifying dynamical systems with physical constraints, sampling from constrained multi-modal distributions, and image classification tasks. The theoretical and empirical findings highlight the crucial role of r2SGLD in improving constrained non-convex exploration.