During human brain development through infancy and childhood, microstructural and macrostructural changes take place to reshape the brain's structural networks and better adapt them to sophisticated functional and cognitive requirements. However, structural topological configuration of the human brain during this specific development period is not well understood. In this study, diffusion magnetic resonance image (dMRI) of 25 neonates, 13 toddlers, and 25 preadolescents were acquired to characterize network dynamics at these 3 landmark cross-sectional ages during early childhood. dMRI tractography was used to construct human brain structural networks, and the underlying topological properties were quantified by graph-theory approaches. Modular organization and small-world attributes are evident at birth with several important topological metrics increasing monotonically during development. Most significant increases of regional nodes occur in the posterior cingulate cortex, which plays a pivotal role in the functional default mode network. Positive correlations exist between nodal efficiencies and fractional anisotropy of the white matter traced from these nodes, while correlation slopes vary among the brain regions. These results reveal substantial topological reorganization of human brain structural networks through infancy and childhood, which is likely to be the outcome of both heterogeneous strengthening of the major white matter tracts and pruning of other axonal fibers.