Background: We aimed to investigate the correlation of factors involved in the change of Mini-Mental State Examination (MMSE) and type 2 diabetes in the elderly. Methods: This study was a secondary analysis of a prospective cohort study. Type 2 diabetes patients aged > 55 years were recruited and assigned into three groups based on their glycated hemoglobin (HbA1c) levels: HbA1c < 7, 7% ≤ HbA1c < 8% and HbA1c ≥8%. MMSE decline was considered the endpoint. Factors related to MMSE decline were identified by univariate and multivariate regression analyses. Results: Altogether, 1519 subjects were included, 883 in the Low group, 333 in the Median group, and 303 in the High group. Age ≥ 75 years, education below elementary school level, not participating in seminars or consultation on healthcare, physical activity less than 30 min/day, cerebrovascular disease history, MMSE score at baseline, and HBA1c ≥8% were associated with cognitive decline by univariate and multivariate analysis. When the other factors were adjusted for, HBA1c ≥8% was independently associated with the severity of cognitive decline (β = 0.58, 95%CI:0.06-1.11, P = 0.029) and the occurrence of cognitive decline (odds ratio (OR) = 1.55, 95%CI:1.13-2.12, P = 0.007). Conclusions: In elderly patients with type 2 diabetes, HbA1c ≥8% is an independent factor for cognitive decline and is also associated with the severity of the cognitive decline. © 2020 The Author(s).