This paper tackled the challenging Fourier phase retrieval problem, the \emph{absolute uniqueness} of which does not hold. The existence of \emph{equivalent solution} (a.k.a. trivial solution ambiguity) hinders the successful recovery, especially for multi-channel color image. The traditional iterative engine, such as the Relaxed Averaged Alternating Reflections (RAAR), can be applied to reconstruct the image channel-wisely. Due to the \emph{relative uniqueness} of the solution, the restoration is not automatically aligned with the accurate orientation for each channel. Hence the reconstructed image is far away from the underlying manifold of the solution. To address this issue, by penalizing the mismatch of the image channels, a diffusion model as the strong prior of the color image is leveraged and incorporated into the iterative engine. The combination of the traditional iterative engine and the diffusion model provides an effective solution to the oversampled Fourier phase retrieval. The formed algorithm, \emph{DiffFPR}, is validated by experiments.