Alzheimer's disease (AD) is a progressive neurodegenerative disorder. Mild cognitive impairment (MCI) is the prodromal stage of AD. Accurate identification of these conditions is crucial for the early diagnosis of these diseases. Brain tissue atrophies, observable in regions such as the hippocampus and cortices, serve as an essential biomarker for MCI and AD. However, similar atrophies are also present in elderly individuals with normal cognitive function, albeit to a lesser extent, and can be found in other non-biomarker brain tissues. To address this challenge, we introduce an atrophy disentanglement network (AD-Net), designed to decouple age-related normal atrophies and disease-specific pathological atrophies in structural magnetic resonance imaging images. Specifically, we first design a dual-task prediction module, guiding the model to differentiate normal and pathological atrophies. Subsequently, we devise a feature orthogonality module for enhanced separation of the two types of atrophies. Our extensive experiments demonstrate that AD-Net outperforms existing methods, highlighting the efficiency of the devised dual-task prediction and feature orthogonality modules in disentangling normal and pathological image features and further improving the diagnosis for AD and MCI. The source code is publicly avaliable at https://github.com/CVAPPS24/ADNet.git.