We study generalizable policy learning from demonstrations for complex low-level control tasks (e.g., contact-rich object manipulations). We propose a novel hierarchical imitation learning method that utilizes scalable, albeit sub-optimal, demonstrations.Firstly, we propose an observation space-agnostic approach that efficiently discovers the multi-step subgoal decomposition (sequences of key observations) of the demos in an unsupervised manner.By grouping temporarily close and functionally similar actions into subskill-level segments, the discovered breakpoints (the segment boundaries) constitute a chain of planning steps to complete the task. Next, we propose a Transformer-based design that effectively learns to predict the chain of subskills as the high-level guidance for low-level action. We couple action and subskill predictions via prompt tokens and a hybrid masking strategy, which enable dynamically updated subskill guidance at test time and improve feature representation of the trajectory for generalizable policy learning.Our method, named Subskill Predictive Control (SPC), consistently surpasses existing strong baselines on a wide range of challenging low-level manipulation tasks with scalable yet sub-optimal demos.