BACKGROUND The relationship between low physical activity and cognitive impairment in type 2 diabetes mellitus (T2DM) patients remains unclear. AIM To explore this association and identify risk factors for cognitive impairment in elderly T2DM patients. METHODS A retrospective analysis was conducted on 245 elderly T2DM patients treated at Xuanwu Hospital, Beijing, in 2023. Patients were categorized into low physical activity (n = 126) and non-low physical activity (n = 119) groups. After propensity score matching (PSM) of 100 pairs, univariate and binary logistic regression analyses identified risk factors for cognitive impairment. A predictive model was constructed and evaluated using receiver operating characteristic curve analysis. RESULTS Before PSM, the percentage of cognitive impairment was higher in the low physical activity group (P < 0.05), but after PSM, this difference was not significant (P > 0.05). Additionally, on regression analyses after PSM, age, occupation type, history of stroke, malnutrition, and frailty remained independent factors associated with cognitive impairment, while low physical activity did not. The constructed risk prediction model for cognitive impairment in elderly T2DM patients exhibited an area under the curve of 0.77. CONCLUSION Low physical activity was not associated with cognitive impairment in our study population. Some results differed before and after PSM analysis, indicating that PSM supports objective assessment of risk factors by controlling for selection bias and confounding factors related to population characteristics. The constructed cognitive risk model provides insight for the development of a clinical tool for early prevention of cognitive impairment in elderly T2DM patients.