Polymorphisms of the apolipoprotein E (APOE) promoter rs405509 are related to Alzheimer's disease (AD). The T/T allele of rs405509 decreases the transcription of the APOE gene and leads to impairments in a specific brain structural network in aged individuals; thus, it is an important risk factor for AD. However, it remains unknown whether rs405509 affects white matter networks during aging. Here, we investigated the effect of the rs405509 genotype (T/T versus G-allele) on age-related brain white matter structural networks via construction of the graph theory-based structural connectome using diffusion MRI data in a large cohort. Network communication efficiency was quantified, along with the network's betweenness centrality (Bc), global efficiency, local efficiency, and shortest path length. Regarding cognition, TT carriers had significant negative correlations between age and memory performance and between age and executive functions. A network analysis showed that TT carriers had an accelerated age-related loss of Bc and that regional Bc decreased in the left inferior frontal gyrus pars opercularis, the left posterior cingulate cortex, the right inferior occipital gyrus (IOG.R), and the left angular gyrus (ANG.L). Additional brain-behavior relationship analyses showed that polymorphism of rs405509 and age have strong interaction effects on the association of nodal Bc and cognition, mainly in the IOG.R and ANG.L. These results demonstrate that the rs405509 T/T allele of APOE causes an age-related cognitive decline in non-demented elderly people, possibly by modulating brain network communication efficiency, which may be beneficial for understanding the neural mechanisms of rs405509-related cognitive aging and AD pathogenesis.