Although extensive research on neural plasticity resulting from hearing deprivation has been conducted, the direct influence of compromised audition on the auditory cortex and the potential impact of long durations of incomplete sensory stimulation on the adult cortex are still not fully understood. In this study, using voxel-based morphometry, we evaluated gray matter (GM) volume changes that may be associated with reduced hearing ability and the duration of hearing impairment in 42 unilateral hearing loss (UHL) patients with acoustic neuromas compared to 24 normal controls. We found significant GM volume increases in the somatosensory and motor systems and GM volume decreases in the auditory (i.e., Heschl's gyrus) and visual systems (i.e., the calcarine cortex) in UHL patients. The GM volume decreases in the primary auditory cortex (i.e., superior temporal gyrus and Heschl's gyrus) correlated with reduced hearing ability. Meanwhile, the GM volume decreases in structures involving high-level cognitive control functions (i.e., dorsolateral prefrontal cortex and anterior cingulate cortex) correlated positively with hearing loss duration. Our findings demonstrated that the severity and duration of UHL may contribute to the dissociated morphology of auditory and high-level neural structures, providing insight into the brain's plasticity related to chronic, persistent partial sensory loss.