We introduce MoNet, a novel method that merges end-to-end learning with modular network designs for self-supervised and interpretable sensorimotor learning. MoNet consists of three functionally distinct neural modules: Perception, Planning, and Control. By leveraging its modularity with a cognition-guided contrastive loss function, MoNet efficiently learns task-specific decision-making processes in latent space without requiring task-level supervision. Moreover, our method integrates an online, post-hoc explainability approach, enhancing the interpretability of end-to-end inferences without compromising sensorimotor performance. In real-world indoor environments, MoNet demonstrates effective visual autonomous navigation, outperforming baseline models by 7% to 28% in task specificity analysis. We also explore the interpretability of our network through a post-hoc analysis of perceptual saliency maps and latent decision vectors. This provides valuable insights into the incorporation of explainable artificial intelligence within the realm of robotic learning, encompassing both perceptual and behavioral perspectives.