The relationship between atherosclerosis and cognitive function is less well studied in Chinese populations. In addition, the results among middle-aged adults have been mixed. We aimed to investigate the association of atherosclerosis measured by carotid intima-media thickness (CIMT) and cognitive function in middle-aged and older adults from a Chinese community. Participants in the Asymptomatic Polyvascular Abnormalities in Community study (APAC) who had completed the CIMT detection and cognitive function measurements in 2012/2013 were included. Cognitive function was measured using the Mini-Mental State Examination (MMSE). Multivariate linear regression analysis was used to analyze the association between CIMT and MMSE. Then, a stratified analysis was performed separately in middle-aged and older adults. A total of 3227 participants were included in this study (mean age 57.9 years, range 43-93 years); 56.6 % of them were men, 66.0 % were middle-aged adults. After adjusting for potential confounders, larger CIMT was associated with lower MMSE scores, with a 0.75-point decrease in MMSE score for every 1-mm increase in CIMT (beta = - 0.75, P = 0.0020). The association remained statistically significant in middle-aged adults (beta = - 0.57, P = 0.0390), and was stronger in older adults and adults with low education levels. There is a significant association between CIMT and cognitive function among middle-aged and older adults sampled from a Chinese population. This association was stronger in older adults and adults with low education levels.