Recently, in visual Reinforcement Learning (RL), various pre-training methods have significantly enhanced agent performance. However, their ability to generalize in diverse environments is not fully understood, mainly due to evaluations being limited to in-distribution (ID) environments and non-unified experimental setups. To investigate this, we introduce the Atari Pre-training Benchmark (Atari-PB), which assesses the generalizability of pre-trained models on identical (ID), visually shifted (Near-OOD), and task-shifted (Far-OOD) environments. Employing a ResNet-50 model, pre-trained on 10 million transitions from 50 Atari games, we demonstrate that understanding spatial and temporal dynamics enhances generalization across various evaluation distributions. However, while task-specific knowledge proves beneficial in ID environments, it does not excel in out-of-distribution (OOD) generalization. Our findings provide key insights for developing more broadly applicable pre-training objectives in visual RL.