Background Alzheimer's disease (AD) is characterized by progressive neurodegeneration and cognitive decline with age. The genetic architecture of AD involves multiple loci, including the apolipoprotein E gene (APOE). The polygenic risk scores for AD (AD-PRS) provide a comprehensive genome-wide assessment of AD risk, yet their age-related effects on brain structures and cognitive function in cognitively unimpaired individuals remain largely undefined. Methods We analyzed cognitively unimpaired, genetically unrelated Caucasians from the UK Biobank (N = 21,236, 64.5 +/- 7.6 years). AD-PRS was derived using a Bayesian approach incorporating approximately 5 million genetic variants (UK Biobank's standard PRS). Brain structures were measured with regional gray matter (GM) volumes and tract-wise microstructural white matter (WM) integrity. Cognitive performance was evaluated with executive function, visuospatial function, reasoning, and memory. Sliding window analyses were performed to investigate age-related polygenic effects on brain structures, and mediation analyses tested whether structural changes mediated the gene-cognition relationship across different age groups. Analyses were replicated using two custom PRSs-one including APOE and the other excluding APOE regions-calculated with the clumping-and-thresholding approach. Results High AD-PRS was associated with accelerated GM atrophy (particularly in the hippocampus, thalamus, and parahippocampus), increased cerebral ventricular volume, and reduced WM integrity (especially in the fornix, cingulum, and superior fronto-occipital fasciculus). These polygenic effects demonstrated significant age-related amplification (p(Bonf) < 0.05), with the strongest effects in individuals aged >= 75. Elevated AD-PRS was linked to lower cognitive performance across aging, especially in executive function, reasoning, and memory, which were significantly mediated by structural brain changes in subcortical and posterior limbic regions and their WM connections, predominantly in late aging (p < 0.05). Sensitivity analyses confirmed the robustness of these findings, emphasizing the dominant contribution of APOE, while also identifying age-specific effects from non-APOE variants. Conclusions High polygenic risk for AD may be associated with accelerated cognitive decline in healthy aging, mediated by structural changes within hippocampal-thalamic regions and their connecting WM tracts. We provide insights into the early pathogenesis of AD and support the potential for age-targeted screening and early intervention for individuals at high genetic risk.