Local steps are crucial for Federated Learning (FL) algorithms and have witnessed great empirical success in reducing communication costs and improving the generalization performance of deep neural networks. However, there is limited study on the effect of local steps in the heterogeneous FL. A few works study this problem from the optimization perspective. Woodworth et al. (2020) showed that the iteration complexity of Local SGD, the most popular algorithm in FL, is dominated by the baseline mini-batch SGD in the heterogeneous data regime, which does not show the benefits of performing local steps. In addition, Levy (2023) proposed a new local update method that provably benefits over mini-batch SGD under the conventional heterogeneous assumption in FL. However, in the same setting, there is still no work analyzing the effects of local steps in generalization. Motivated by our experimental findings where Local SGD learns more distinguishing features than parallel SGD, this paper studies the generalization benefits of local steps from a feature learning perspective in a heterogeneous FL setting. We propose a novel federated data model that exhibits new forms of heterogeneity, under which we formally show that a convolutional neural network trained by GD with global updates will miss some pattern-related features, while the network trained by GD with local updates can learn all features in polynomial time. Consequently, local steps help the neural network generalize better under our data model. Our experimental results also confirm the benefits of local steps in improving test accuracy on real-world data.