ApoE gene polymorphism has been reportedly associated with serum lipids and cognition. However, very few studies have explored the combined effects of ApoE gene polymorphism and gender on serum lipid profile with subsequent impacts on cognition in Chinese population. A total of 1,000 Chinese community dwellers aged 55 years and above were recruited in this cross-sectional study. Demographic information of the participants was collected using well designed self-administered questionnaires. The Montreal Cognitive Assessment (MoCA) test was employed to evaluate the cognitive status of the participants. Semi-quantitative food frequency questionnaire (FFQ) was used to obtain the dietary intake information. Fasting venous blood samples were taken for ApoE genotyping and serum lipid measurements. Significant gender differences in cognition, serum lipid profile and dietary fat-rich foods consumption were observed (p < 0.05). Cognition of the subjects was found to be associated with ApoE genotypes (p < 0.05). ApoE rs429358 and rs7412 variants demonstrated a significant effect on cognitive performance in the male subjects; especially within the attention and language cognitive domains as well as the total MoCA score (p < 0.05), respectively. Serum lipid profile and cognition of Chinese adults are significantly linked with gender and ApoE genetic polymorphism. The ApoE variant rs429358 is found to be notably associated with cognition in aging male Chinese population.